Planning FreeNAS Config on a BackBlaze Pod

[Chewie71]

I have one of the BackBlaze pods, default hardware as sold by Protocase.
http://www.protocase.com/products/index.php?e=Backblaze

Hardware Specs are on this page.
http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/

But basically...

8GB RAM
2 x Syba PCI Express SATA II 4-Port RAID Controller Card SY-PEX40008
9 x CFI-B53PM 5 Port Backplane (SiI3726)
45 x 3TB WD Red NAS hard drives

If I remember correctly, 8 backplanes connect off the PCIe SATA cards and one off the MB.

I've install FreeNAS 8.2.0-p1 onto a thumb drive. I've read the forums quite a bit about the most performant way to assemble the ZFS pool. It seems the conventional wisdom says to create a pool, and populate it with a bunch of 6-disk vdevs. I've partly done this and my setup is below...

ZFS_POOL
- raidz2
- - ada0p2
- - ada1p2
- - ada2p2
- - ada3p2
- - ada4p2
- - ada5p2
- raidz2
- - ada6p2.nop
- - ada7p2.nop
- - ada8p2.nop
- - ada9p2.nop
- - ada10p2.nop
- - ada11p2.nop
- cache
- - ada45p1
- spares
- - ada44p2
- - ada43p2
- logs
- - ada42p2
- - ada41p2

Every time I add another 6 disk raidz2 vdev to the pool, it's going to eat up two hard drives. So out of the 135TB it looks like I may only get a little over half of that as usable space....which seems like an awful waste to me. Maybe I'm designing this wrong? I'm open to suggestions.


The other question I have which I've seen asked about but not really answered...is the siisch timeout problem. I switched to FreeNAS from the default Debian install because the port multiplier issue was giving me fits in Linux. I also wanted a better way to manage the storage and the ZFS support was another reason to switch. But in some early testing when I mistakenly had created a single vdev in a pool using ALL the drives...I was starting to see the siisch timeout problems in FreeNAS too. Maybe with my ZFS redesign those will go away, but I'm not too hopeful.

I'm waiting to do anything further though until I get a better understanding of the ZFS design and whether I should make any changes.

Thanks,
Matt

 

[toddos]

Every time I add another 6 disk raidz2 vdev to the pool, it's going to eat up two hard drives. So out of the 135TB it looks like I may only get a little over half of that as usable space....which seems like an awful waste to me. Maybe I'm designing this wrong? I'm open to suggestions.

It looks like you're including your cache, spares, and logs in that calculation. I would assume those are fixed (for every 6 new drives, you don't add one more to cache, logs, and spares). So right now, with only two vdevs, you're getting less than half of your total storage space as usable (9 cache/spares/logs/parity drives vs 8 data drives). But if you add a third 6-drive Z2 vdev, you're only adding 2 parity drives (total of 11 "unusable") and 4 data drives (total of 12 "usable"). And if you add a fourth, it's another 2 parity, 4 data.

If that ratio is not good enough for you, you can rethink what you're doing with cache, spares, and logs. Do you really need them? Can you survive with 1 spare? Do you need "hot" spares at all with 2-drive parity per vdev, so that you can lose up to two drives per vdev and still run while waiting on replacements? Do you really need cache (ARC vs. ZIL)? There are calculations you can do based on your expected workload to determine if you need them. Etc.

 

[Chewie71]

I don't know if I really need the cache or not. This will be almost all writes as this is supposed to mostly be storage for offsite backups. But...the cache is actually running on the internal drive connected to the mobo and is not one of the drives from the big disk bay. So that is not eating up any extra.

I can probably drop the spares. I've got a box of new drives on hand to swap out when existing drives go bad....and I'd have to lose three drives from any one vdev. And from what I've read the current FreeBSD implementation does not do auto failover of hot spares anyway. Just makes me nervous. :)

From some more reading I've done it also looks like maybe 10 drive vdevs are a good number for raidz2. That would increase the number of usable disks vs. the unusable parity and log drives.

 

[paleoN]

8GB RAM
2 x Syba PCI Express SATA II 4-Port RAID Controller Card SY-PEX40008
9 x CFI-B53PM 5 Port Backplane (SiI3726)
45 x 3TB WD Red NAS hard drives

You need more RAM.

I've read the forums quite a bit about the most performant way to assemble the ZFS pool. It seems the conventional wisdom says to create a pool, and populate it with a bunch of 6-disk vdevs.

The most performant way, with double-parity, would be striped 3-way mirrors.

- cache
- - ada45p1
- spares
- - ada44p2
- - ada43p2
- logs
- - ada42p2
- - ada41p2

Unless these, cache & log, are something besides the WD Reds, remove them. Designating a few disks as spares on particular controllers ahead of time may be rather useful. Spares, cache & logs are per pool if you didn't know.

Every time I add another 6 disk raidz2 vdev to the pool, it's going to eat up two hard drives. So out of the 135TB it looks like I may only get a little over half of that as usable space....which seems like an awful waste to me. Maybe I'm designing this wrong?

Perhaps you are thinking wrong. What are your IOPs/Latency/Throughput/Storage requirements?

The other question I have which I've seen asked about but not really answered...is the siisch timeout problem. I switched to FreeNAS from the default Debian install because the port multiplier issue was giving me fits in Linux.

Port multipliers = poor performance & problems. I've read the siisch chipset on SY-PEX40008 is well supported on FreeBSD. I don't know about the backplanes though. The disk firmware also needs to play well with the backplanes/controller/driver.

I can probably drop the spares. I've got a box of new drives on hand to swap out when existing drives go bad....and I'd have to lose three drives from any one vdev. And from what I've read the current FreeBSD implementation does not do auto failover of hot spares anyway.

The more I've read, the more inclined I am to not use hot spares. Warm spares are your friend even if they are inconvenient. Cold spares can work as well.

From some more reading I've done it also looks like maybe 10 drive vdevs are a good number for raidz2. That would increase the number of usable disks vs. the unusable parity and log drives.

It would be better to say a 10 drive vdev is an aligned number for raidz2. It's a wider vdev than I would want to try to resilver with slow 3TB drives or any number of other drives. Optimal raidz2 vdev does seem to be six. Keep in mind the more vdevs you have the greater performance & redundancy the pool will have.

 

[Chewie71]

You need more RAM.

OK...I'll look into that.

The most performant way, with double-parity, would be striped 3-way mirrors.

Unless these, cache & log, are something besides the WD Reds, remove them. Designating a few disks as spares on particular controllers ahead of time may be rather useful. Spares, cache & logs are per pool if you didn't know.

Perhaps you are thinking wrong. What are your IOPs/Latency/Throughput/Storage requirements?

This is for backup storage only. I'm actually more concerned about getting lots of usable space out of it than performance. It just has to be fast enough to receive VEEAM backup jobs. The bottleneck on the jobs is most likely going to NOT be the storage.

Are you saying I don't need cache or log drives? Or that I shouldn't use the WD Red drives from the storage bay for cache/logs?

Port multipliers = poor performance & problems. I've read the siisch chipset on SY-PEX40008 is well supported on FreeBSD. I don't know about the backplanes though. The disk firmware also needs to play well with the backplanes/controller/driver.

Through experimentation I've found that if I'm sending data out to every backplane at once...that caused major communication problems. I initially thought I could protect myself better by building arrays using only one drive per backplane. But the more backplanes I have to read/write at once....the worse the problem was. So now my thinking is to just build arrays using contiguous drives from one (or as few as possible) backplanes to reduce or maybe eliminate that issue.

I'm not sure if having multiple contiguous raidz2 vdevs in one pool will affect that. If I have 7x6disk vdevs in a pool (multiple vdevs from multiple backblanes in a pool)....when I write to a volume in the pool will it only write to one vdev or will it be writing all at once across all vdevs in the pool? I'm not familiar enough with ZFS yet to understand how that works.

The more I've read, the more inclined I am to not use hot spares. Warm spares are your friend even if they are inconvenient. Cold spares can work as well.

Agreed...I think I'll eliminate the hot spares.

It would be better to say a 10 drive vdev is an aligned number for raidz2. It's a wider vdev than I would want to try to resilver with slow 3TB drives or any number of other drives. Optimal raidz2 vdev does seem to be six. Keep in mind the more vdevs you have the greater performance & redundancy the pool will have.

How does having multiple vdevs in a pool help redundancy? I thought if one vdev went down it would affect the entire pool of vdevs.

If I create 6 disk vdevs in one pool, and leave aside 2 drives for logging, no spares....my config would be something like this.

ada0 - ada5: vdev1
ada6 - ada11: vdev2
ada12 - ada17: vdev3
ada18 - ada23: vdev4
ada24 - ada29: vdev5
ada30 - ada35: vdev6
ada36 - ada41: vdev7
ada42 - ada43: logs
ada44: warm spare

A config like that is going to give me around 75TB usable. If I went to 10disk vdevs I'd get somewhere near 100TB, but as you say (and I've read more online) the resilvering will take way too long to complete, so probably not a great plan.

Thanks,
Matt

 

[paleoN]

OK...I'll look into that.

What does the motherboard max out at?

This is for backup storage only. I'm actually more concerned about getting lots of usable space out of it than performance. It just has to be fast enough to receive VEEAM backup jobs. The bottleneck on the jobs is most likely going to NOT be the storage.

How many VEEAM jobs from how many servers? How much space do you actually require?

Are you saying I don't need cache or log drives? Or that I shouldn't use the WD Red drives from the storage bay for cache/logs?

Certainly the second, but possibly the first as well. As to the second, you want to use devices that have a few orders of magnitude less latency than the main pool.

I'm not sure if having multiple contiguous raidz2 vdevs in one pool will affect that. If I have 7x6disk vdevs in a pool (multiple vdevs from multiple backblanes in a pool)....when I write to a volume in the pool will it only write to one vdev or will it be writing all at once across all vdevs in the pool? I'm not familiar enough with ZFS yet to understand how that works.

It depends on the size of the file. ZFS distributes the writes across the top level vdevs. If the file is small enough it won't make it across all of the vdevs. Of course during a normal txg commit you will likely have enough writes to write to all the vdevs concurrently.

How does having multiple vdevs in a pool help redundancy? I thought if one vdev went down it would affect the entire pool of vdevs.

How doesn't it? You are correct in that a multiple raidz2 vdev pool is still a double-parity pool. If you lose the wrong 3 disks you lose the entire pool. However with multiple vdevs you can lose more than 3 disks as long as they are in different vdevs. You are also more likely to not run in a 'critically' degraded state, e.g. more than 2 disks down in a single vdev.

A config like that is going to give me around 75TB usable. If I went to 10disk vdevs I'd get somewhere near 100TB, but as you say (and I've read more online) the resilvering will take way too long to complete, so probably not a great plan.

Keep in mind, barring natural disasters, storage is cheap and only getting cheaper. How full to you plan on filling the pool?

 

[solidvxv]

I realize this post is dead as a doornail but I have the same setup and was wondering how it went im getting terrible writes.