Looking for vdev layout advice - 24x HGST 3TB SAS6

[sunshine931]

I'm re-building my pool, and am looking for suggestions around vdev design.

I've got 24x HGST 3TB SAS6 drives to work with for my data vdevs, installed in NetApp DS4246 attached to a SAS6 HBA (not RAID controller).

Does anyone have a suggestion for other vdev layouts?

My use-case is a blend of:

• VMs via iSCSI, mostly lab workloads (lab as in playing with stuff, not scientific laboratory data)
• Home-users using SMB/CIFS shares via gigabit ethernet
• Plex media server VMs accessing the data via NFS, multiple Plex users in the home via gigabit ethernet, and external users at no more than 20mbps

I'm considering:

• 12x mirrored pair vdevs (2 drives each vdev, 1/2 "wasted")
• 8x raidz1 (3 drives each vdev, 1/3 "wasted")
• 2x raidz2 (12 drives each vdev, 2/12 "wasted")

I believe (but have not yet tested to confirm) that:

• the 12x mirrored pairs will perform best, but it's a wasteful layout
• the 8x raidz1 would be a good compromise between performance and waste
• the 2x raidz2 would be very resilient to outage and the least wasteful, but less performant than the other options

I should mention that:

• System has 64GB ECC RAM
• I have SSD available to use as cache, but need to assess the read performance vs. the data vdevs first
• I will very likely be adding slog to address sync-writes from the VMs, but again, will want to do some testing

 

[SweetAndLow]

If you want to run vm storage you should use mirrors. If you don't care about performance as much raidz 2 with 3 vdevs

 

[Heracles]

Hi Sunshine,

I will start with what NOT to do.

The 3x 8 drives RaidZ1 is a clear NO-GO for me. RaidZ1 does not protect well enough and should not be used, even less on a vDev with as many as 8 drives, and even less 3 times instead of 1 in the same pool. Big NO-NO here.

The 2x 12 drives RaidZ2 is a little better, but not much. Yes each vDev can now sustain the loss of 2 drives, but being larger, they are at a higher risk by default.

As for the 12 vDev of 2 drives, the equivalent of Raid-10, this is a very good choice for many aspects, including performance. As you observed, the cost is about the capacity of the pool. Such a config will give you a pool of 12x 3 TB = 36 TB. Is that enough for your need ? If it is, there should be no hesitation.

Other options could be :
2x 12 drives with RaidZ3 ; 18 drives of usable space / 6 drives for redundancy. The ratio of redundant drives per vDev is good.
3x 8 drives with RaidZ2 ; similar to the previous case. To have smaller vDevs usually helps with re-silvering process.
4x 6 drives with RaidZ2 ; an increase in redundancy / reduction of pool size and again, smaller vDevs rebuild faster.

There are options like keeping some drives as hot spare. That is another way to improve redundancy and when you have many drives, it can be an interesting solution as well.

Hope this will give you better ideas of what to do and what to avoid,

 

[Mlovelace]

I'm re-building my pool, and am looking for suggestions around vdev design.

I've got 24x HGST 3TB SAS6 drives to work with for my data vdevs, installed in NetApp DS4246 attached to a SAS6 HBA (not RAID controller).

Does anyone have a suggestion for other vdev layouts?

My use-case is a blend of:

• VMs via iSCSI, mostly lab workloads (lab as in playing with stuff, not scientific laboratory data)
• Home-users using SMB/CIFS shares via gigabit ethernet
• Plex media server VMs accessing the data via NFS, multiple Plex users in the home via gigabit ethernet, and external users at no more than 20mbps

I'm considering:

• 12x mirrored pair vdevs (2 drives each vdev, 1/2 "wasted")
• 8x raidz1 (3 drives each vdev, 1/3 "wasted")
• 2x raidz2 (12 drives each vdev, 2/12 "wasted")

I believe (but have not yet tested to confirm) that:

• the 12x mirrored pairs will perform best, but it's a wasteful layout
• the 8x raidz1 would be a good compromise between performance and waste
• the 2x raidz2 would be very resilient to outage and the least wasteful, but less performant than the other options

I should mention that:

• System has 64GB ECC RAM
• I have SSD available to use as cache, but need to assess the read performance vs. the data vdevs first
• I will very likely be adding slog to address sync-writes from the VMs, but again, will want to do some testing

Since you mentioned running VMs over iSCSI I would go with mirrors. Here is some info that might help you balance what is most important to you.

Pool Geometry
If small random IOPS are of primary importance, mirrored vdevs will outperform raidz vdevs. Read IOPS on mirrors will scale with the number of drives in each mirror while raidz vdevs will each be limited to the IOPS of the slowest drive.

If sequential writes are of primary importance, raidz will outperform mirrored vdevs. Sequential write throughput increases linearly with the number of data disks in raidz while writes are limited to the slowest drive in mirrored vdevs. Sequential read performance should be roughly the same on each.

Both IOPS and throughput will increase by the respective sums of the IOPS and throughput of each top level vdev, regardless of whether they are raidz or mirrors.

 

[sunshine931]

Thanks for the feedback everyone.

I realize now I forgot that ZFS isn't "perfect" in raidz1 parity waste. By that I mean 3 disks in raidz1 does not give the capacity of 2 disks, with 1 disk waste. It's actually closer to 50% waste based on the tables I've been reviewing. This makes an 8 vdev of 3 raidz1 unattractive. I might as well go mirror.

So, between that and all the great points made above in your replies, I am going 12 x 2 mirror vdev.

Thanks all.

 

[SweetAndLow]

Thanks for the feedback everyone.

I realize now I forgot that ZFS isn't "perfect" in raidz1 parity waste. By that I mean 3 disks in raidz1 does not give the capacity of 2 disks, with 1 disk waste. It's actually closer to 50% waste based on the tables I've been reviewing. This makes an 8 vdev of 3 raidz1 unattractive. I might as well go mirror.

So, between that and all the great points made above in your replies, I am going 12 x 2 mirror vdev.

Thanks all.

There isn't any more waste than any other filesystem. Not sure what you mean by waste