5-Wide Z2 vdevs or Mirrored? Help!

[firewallwhisperer]

Hey all,

Setting up a new FreeNAS box. I'm running a 36 bay super micro. I've got 16 8tb drives right now. Expandability is key. No VM's stored in FreeNAS but read/write performance is still important to me. Mirroring seems pretty much superior in every way except for storage efficiency. The 50% efficiency really sucks.

I've basically narrowed it down to these two options for pool configuration:
5-wide Z2 vdevs (Once the bays are maxed, I'll have 7 devs of 5 drives with one hot spare. Total usable storage space would be 168TB)
OR mirrored vdevs (Once full, I'll have 144tb usable storage.)

I figure the smallish size of 5-wide vdevs might be a good compromise of quick resilver times, storage efficiency, and easy scalability (only need 5 drives to increase capacity.)

My questions are:
1) If I do mirrored vdevs, does the size of the vdev really matter? I was thinking 4-wide mirrored vdevs.
2) The performance of z2 vdevs will be bad at first will it not? Simply because I don't have that many vdevs? How bad are we talking here? Once I expand and have more vdevs, how will the performance compare to mirroring?
3) If I'm storing all my VM's in local vSAN storage on my esxi hosts, am I overstating the importance of a performant pool?
4) What would YOU do in my shoes?

 

[Samuel Tai]

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.ixsystems.com

 

[MikeyG]

I think that a 5 wide zd2 vdev will be measurable slower for random IO than a mirror vdev. However, if you are not storing VMs on the pool, I think the question of what you are actually storing on it and how much performance you really need is relevant. What is your intended workload?

When you say "4-wide mirrored vdev" are you meaning a vdev where there are 4 disks mirroring each other? As in 25% storage efficiency? This seems like pretty severe overkill, but your IO and reads would be very good, but 4X the IO of writes.

 

[sretalla]

Expandability is key.

Then certainly mirrors as RAIDZ2 will mean 5 disks at a time, whereas 2-4 disks for a mirror (I would recommend 3 if you're uncomfortable with 2 for your level of comfort).

 

[firewallwhisperer]

I think that a 5 wide zd2 vdev will be measurable slower for random IO than a mirror vdev. However, if you are not storing VMs on the pool, I think the question of what you are actually storing on it and how much performance you really need is relevant. What is your intended workload?

When you say "4-wide mirrored vdev" are you meaning a vdev where there are 4 disks mirroring each other? As in 25% storage efficiency? This seems like pretty severe overkill, but your IO and reads would be very good, but 4X the IO of writes.

Sorry, let me clarify - 4 Disks, two may mirror - 50% storage efficiency. The most demanding workload (that the box sees on a regular basis) I anticipate would be multiple 4k Plex streams. When we get up to 7 RAIDz2 vdevs do we begin to creep up on comparable performance of a mirrored pool?

 

[firewallwhisperer]

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.ixsystems.com

Hey Samuel,

Sorry if this post is redundant! Really appreciate the link you sent over. Great read. A heavy read, still digesting and deciphering bits of it, but very informative. Thanks a lot

 

[MikeyG]

I don't think Plex streams would be that demanding from an IO perspective. 4k Blurays cap out around 80mb. I would think even 10 of those should be easy for RAIDZ2 vdevs. It's all sequential data, so the extra IO speed of mirrors gets you nothing in that scenario.

Someone might correct me on this, but I think that 7 RAIDZ2 vdevs of any size will be slower for random IO than 7 mirror vdevs. There's overhead to RAIDZ2 that just isn't present in mirrors. Each RADZ2 vdev still only gives the IOPS of a single disk for reads and writes, whereas a mirror vdev has double IOPS for reads.

For your 36 bay server, and for storing 4k media, I would create RAIDZ2 vdevs of 6 drives. Easily divisible into 36, and adding capacity 6 drives at a time isn't unreasonable.

 

[Yorick]

I’d do 6-wide raidz2, if 4k streams is all you’re going to run off that. I have a single 5-wide and it can handle multiple 4k streams easily. Streaming, it can saturate the Gig link I have, same for large file write.

It’s not until you get into 10Gbit that you’ll scratch the performance of a single vdev (for large file streaming), and with multiple vdevs, you’ll be set.

Plex and general file storage are not demanding use cases.

VM storage and databases are a different story, but you won’t be doing any of that.