Storage setup advice to start my first FreeNAS setup

[Phil1295]

Hi,
I am reading and preparing to step to FreeNAS since a few months now.

Storage specs:
- 7 spare bays. I plan to use 6 for the pools and keep one spare for maintenance and easy increase of pool disk size if needed.
- Disk sizes will range from 4 TB to 16 TB

Basically, I am hesitating between these 2 approaches:
1- One pool with 6 drives in z2
vs
2- Three distinct pools in Mirror mode

I don't care loosing space with option 2 and I priorise security + fast rebuilds when recovering from a degraded pool.

I read in one of the pinned pdf/powepoint docs that Mirror is not recommended with large capacity disks, but no explanation.
Option 2 is really the way I thought to go because of 2 main aspects:
- less risk to loose all the pool data on a failure/error, that is the whole 4 (+2 recovery) disks data
- faster recovery from a degraded pool

Is this really a bad choice because of the disks capacity or something I am missing ?

Thank you and best regards

 

[Jasse Jansson]

This might be from the Solaris days, but I vaguely remember that you can set up a raidZn pool of 6 disks and add the 7:th as a spare.
I believe that you can make the spare to replace a failed disk automatically.
At least there was discussions about it in the Solaris community release forums.
Probably only works with disks of the same size.

 

[Phil1295]

I think I did not be enough clear.
By spare disk, I mean I will keep one of my available SATA slots free to be able to connect a disk for faster resilvering when a disk fails in a Z2 pool

So, my main question is if there are really any issues running 3 pools of mirror z1 pairs with HDD disks of up to 16 TB vs having one pool of 4+2 disks in Z2 (all of the same size obviously)

Again, my priority is:
- less risk to loose all the pool data on a failure/error, that is the whole 4 (+2 recovery) disks data
- faster recovery from a degraded pool

Best regards

 

[Phil1295]

Any help please on my question ?

 

[Ericloewe]

nning 3 pools of mirror z1 pairs with HDD disks of up to 16 TB vs having one pool of 4+2 disks in Z2 (all of the same size obviously)

I'm not sure I understand your question. Mirrors and RAIDZ1 are very different things. If you mean three mirrors vs 6-wide RAIDZ2, the former has three times the IOPS, the latter is more reliable.

 

[ornias]

Also @Ericloewe I've no idea what this guy is trying to tell us here:
"By spare disk, I mean I will keep one of my available SATA slots free to be able to connect a disk for faster resilvering when a disk fails in a Z2 pool"

Because there is no way how an empty slot is going to speedup resilver on a raidz vdev.
This ain't draid(yet)

 

[Ericloewe]

I assume it is to minize the delay between disk failure and start of resilver.

 

[Phil1295]

I assume it is to minize the delay between disk failure and start of resilver.

About the "spare disk slot": I exactly meant what you said, minimize the delay to change a disk before failure or to increase the vdev size when changing disks with bigger ones. If I well understood, It will avoid running in degraded state which would take too long + more risks during the resilvering

I'm not sure I understand your question. Mirrors and RAIDZ1 are very different things. If you mean three mirrors vs 6-wide RAIDZ2, the former has three times the IOPS, the latter is more reliable.

Sorry, I fixed my second post removing the z1 typo.

My question is: I read in the Powerpoint or another PDF guide in the forum that running a Mirror on very large disks is a very bad idea. I am planning to have 2x 18 TB mirrors + 2x 8TB Mirrors. The 3rd pair is smaller. Is it really a problem because of the disks size ? On that pdf/ppt nothing was mentioned about the reasons, so I am not sure if I am missing a zfs limitation with such disk sizes in Mirror

Hope I am more clear now

PS: disks are of server grade quality and I don't want to invest in bigger equal disks to run 6 disks in z2, but rather use the pairs I have. In the future, I will try to have disks of same size progressively

 

[HoneyBadger]

Well, if you're going to keep the disk spinning anyways, you could just run a RAIDZ3 and have "zero delay" in a sense.

Edit: Ah. Different sized disks cannot be (efficiently) combined into a single vdev - you'll only have the capacity of the smallest disk. You can combine different sized vdevs in a single pool, or use three separate pools.

 

[Phil1295]

Well, 3 mirror vdevs in a single pool would be more risk + no benefit than 3 separate pools (less risk to loose whole pool on a vdev failure...)
I have backups in any case

So, if I understand, there is no issue related to disk sizes at all if I run them in Mirror: 1 pool with 2x 18TB, one pool with 2x 8TB and one pool with the other pair (6 TB) ?
Once I upgrade disks, I could in run a Z2 with 6 disks to spare space of one disk compared to the mirror setup where I loose 50% capacity

That was really my only concern, 18TB disks in Mirror. They work perfectly on my ReadyNAS in BTRFS. But that mention in the ppt/pdf guide made me doubt

 

[Ericloewe]

So, if I understand, there is no issue related to disk sizes at all if I run them in Mirror: 1 pool with 2x 18TB, one pool with 2x 8TB and one pool with the other pair (6 TB) ?

You can have them all in one pool, that's not a problem. It's mixing within vdevs that gets inefficient.

 

[HoneyBadger]

Once I upgrade disks, I could in run a Z2 with 6 disks to spare space of one disk compared to the mirror setup where I loose 50% capacity

To be clear, you will lose the space of 2 disks in a Z2 setup (hence the "2" in "RAIDZ2") - I wouldn't recommend RAIDZ1 with large drives.

That was really my only concern, 18TB disks in Mirror. They work perfectly on my ReadyNAS in BTRFS. But that mention in the ppt/pdf guide made me doubt

You're technically in the same situation using large disks in a mirror of BTRFS, hardware/software RAID, any of it. This isn't unique to ZFS - by design, a mirror will leave a large amount of data unprotected in case of a single device failure. ZFS you can at least do N-way mirrors for really important data that is also performance sensitive (ruling out Z2) - and since you have backups, this mitigates the risk as well.

 

[Phil1295]

Ok, thank you all, I understand better
So the concern about big disks in mirror mentioned in that pdf was probably related to the failure of one disk, like in a z1. It was not a compatibility/performance issue with zfs/FreeNAS !

I understand it and have backups. But for me, restoring one pair of failed mirrors from a backup is not the same as restoring 3 pairs when one pair fails if they are on the same pool.

But sure, once I can get more disks of same size and replace the old smaller ones, I would run a single Z2 pool

Thank you again as I think I got well answered