how many mirrored vdev would make you uncomfortable?

[ShimadaRiku]

1x 2-way vdev is a no brainer, but what about 48x 2-way vdevs?

Both drives failing in the same vdev would destroy the whole pool. Recovering from backups would be a pain.

How many mirrored vdevs would make you uncomfortable?

 

[Heracles]

Hey @ShimadaRiku,

Indeed, a failed drives in the same vDev will toast your pool. But when you have 48 mirrors (96 drives), you can loose any one as a first. Then, you can loose any of the 94 drives in an untouched mirror out of the 95 remaining drives total. That is, you have 98.9% of chances to survive. For the third one, you can loose any of the 92 drives still in the remaining mirrors, so 92 out of 94. That means you have 97.9% of chances to be OK. Putting these 2 together, you have 96.8% of chances to survive the loss of 3 drives.

So as you see, the more drives you have and more failures you will have but statistically, you are still very strong. This also illustrates why backups are a hard requirement and that the restore is part of the backup process. Should your restore process not be up to the task of putting you back online in a timely reaction, then it is not up to the task.

For that, you may use a second NAS and replicate to that one. In case of a failure, you clone the proper snapshot on that second NAS and go back online from that second one right away. No delay, no pain. You then take the time you need to restore the main server.

 

[c77dk]

With that many drives and the statement of restoring being a pain, you could also (still with a solid backup strategy as proposed by @Heracles ) make two of the disks hot-spares, so a failed disk can be resilvered ASAP. It still leaves the option of pool crash, but minimizes the timeframe.

 

[Arwen]

Agree with @c77dk, when you have a great many disks in use, it is wise to have some spares. And that would be regardless of Mirrored vDevs or RAID-Zx vDevs. Either:

• Cold, (aka on the shelf, but bad block tested)
• Warm, (not assigned as hot spare to a any pool, but spinning and awaiting replacement into a specific pool)
• Hot

Note that you can share hot spare disks between pools, but some people prefer to manage spares manually. Thus, they may have a disk preinstalled, awaiting a failure.

PS: However, you cannot export a pool with an in-use shared spare unless you use the zpool export -f (force) option.

 

[ShimadaRiku]

Hey @ShimadaRiku,

Indeed, a failed drives in the same vDev will toast your pool. But when you have 48 mirrors (96 drives), you can loose any one as a first. Then, you can loose any of the 94 drives in an untouched mirror out of the 95 remaining drives total. That is, you have 98.9% of chances to survive. For the third one, you can loose any of the 92 drives still in the remaining mirrors, so 92 out of 94. That means you have 97.9% of chances to be OK. Putting these 2 together, you have 96.8% of chances to survive the loss of 3 drives.

So that means if you had 1024 drives you'll feel even more comfortable than 96 drives? Even if drives of the same batch/mfg might fail together?

 

[Heracles]

So that means if you had 1024 drives you'll feel even more comfortable than 96 drives?

Not more ; just as much. Because more drives means that more will fail. As such, out of 1024 drives, you must be ready to loose more drives than when you have only 8 or 96. Still, despite the higher number of failed drives you need to sustain, the statistic says that your probabibility is not worst at 8, 48, 96 or 1024 drives.

And mostly, in all cases, you always need a complete backup that include a restore procedure up to what you need as recovery time.

 

[AlexGG]

Even if drives of the same batch/mfg might fail together?

It is actually better with mirrors than with any other layout. With mirrors, you only need two different sets (batches) of drives, and then make sure each mirror uses one drive from each batch. With RAIDZ1, you need to ensure that no vdev contains two drives of the same batch, so you need at least three batches.

 

[Arwen]

I went a bit over-board on my last NAS for avoiding same batches. I purchased a WD Red, (CMR), in retail packaging. Later bought a WD Red, (CMR), in non-retail packaging. Then bought 2 WD Red Pros months apart. After some testing, I created a RAID-Z2 pool of those 4 disks in my FreeNAS Mini. (Instead of the other option, 2, 2 disk Mirrored vDevs...)

My next NAS will have a simple 3 way Mirror vDev. My storage needs are modest, so using larger disks makes sense today. Yet, I will still plan on using disks from different manufacturers.

Some people call me paranoid. But, I like to think of it as long term planning for the worst.


Back to the original subject. If the desire is to have better recovery than 2 disk Mirrored vDevs, yet not have more than 50% of the space used by redundancy, you can always create 24, 4 disk RAID-Z2 vDevs. Those RAID-Z2 vDevs can have 2 disks lost PER vDev without data loss. So, out of say 96 disks, you can loose 48, with better odds than simple 2 disk Mirrored vDevs. (At least I think it's better odds... don't quote me, elves don't do math

The speed of RAID-Z2 is less than a simple 2 way Mirror. But, striping across 24 RAID-Z2 does have it's advantages.


None of that means backups are not suggested, (or required for more important data).

 

[Chris Moore]

1x 2-way vdev is a no brainer, but what about 48x 2-way vdevs?

Both drives failing in the same vdev would destroy the whole pool. Recovering from backups would be a pain.

How many mirrored vdevs would make you uncomfortable?

Mirror vdevs are generally not a problem. We have a system at work that has a little over300 drives. Mind you, it was installed by Sun/Oracle, but it is using their version of ZFS in mirrors. I don't think there is really a limit as long as you keep a carful eye on the system or have hot spares in the system.

 

[Sawtaytoes]

Note that you can share hot spare disks between pools, but some people prefer to manage spares manually.

How do you share hot spares? Mine are always assigned to a single pool via FreeNAS.

 

[Arwen]

I don't know how the GUI would support adding an already assigned, (but not in use), hot spare. But, command line, (discouraged), states it is possible;

Code:

Spares can be shared across multiple pools, and can be added with the zpool add command and removed with
the zpool remove command. Once a spare replacement is initiated, a new spare vdev is created within the
configuration that will remain there until the original device is replaced.  At this point, the hot spare
becomes available again if another device fails.

 

[blanchet]

I have no worries whith the number of vdevs but I am unconfortable when the pool needs several JBODs.

Therefore I prefer to have one pool per JBOD.

 

[Whattteva]

1x 2-way vdev is a no brainer, but what about 48x 2-way vdevs?

Both drives failing in the same vdev would destroy the whole pool. Recovering from backups would be a pain.

How many mirrored vdevs would make you uncomfortable?

You know... you could calculate such things right? The numbers are actually better the more drives you have.

For example, the probability of 2-drive failures taking out your entire pool for:
1 vdev: 1/1 = 100%
2 vdev: 1/3 = 33%
3 vdev: 1/5 = 20%
4 vdev: 1/8 = 14.3%
etc...

The above assumes all other things being equal for simplicity. Of course, that is only 1 factor in the equation. Different drives have different failure rates depending on age and production batch.

There is also a confounding factor between RAIDZ vs mirrors where a failure in a vdev in a RAIDZ puts a ton of extra load on the rest of the surviving drives (instead of just 1) and the resilvering also takes far longer in that vdev adding a lot more failure rate to the other drives that doesn't really happen in a striped mirrors. For example, a 6-drive RAIDZ1 has 5 times extra I/O load while resilvering compared to a striped mirror and the calculations get worse for RAIDZ2 and higher. This is because for every block written to resilver a new drive, a block has to be read from each of the surviving drives.

I run striped mirrors for the reasons stated above. Way faster resilvering (especially with drives pushing closer and closer to 20TB and beyond), minimal degraded performance and just better performance overall and way more flexibility for upgrade path and even redundancy (3+ way mirroring). And of course, I have backups. So, the choice is a no-brainer for me.

 

[Sawtaytoes]

I have 3 zpools with 12 or more mirrors. One is the main zpool, one is a backup, and the third is an offsite backup.

As long as I have backups, I don't mind having a bunch of vdevs. It feels very safe.

I also prefer mirrors since you can add and remove them at will including the metadata (special) devices. This is super convenient.

It's also super fast to resilver (as others have stated) which lets hot-spares bring your pool back to 100% resiliency that much quicker.

While I like the fact that RAID-Z2 and RAID-Z3 have better integrity checking than 2-drive mirrors, you can scale mirrors up to 3-drive or more for better data integrity at any time. Changing from RAID-Z1 to RAID-Z2 isn't so simple. Any RAID-Z vdevs are pretty-much stuck in your pool as-is. Maybe good for enterprise, but horrible for home use where you might buy a set of larger drives for your limited storage device, but don't want to buy as many.

---

Take for example you have a RAID-Z2 of 8 drives. If you wanna buy larger drives, you need to buy 8 of them, not 2, not 4, all 8 need to be replaced and _then_ you can expand the vdev's capacity in that zpool.

If you made those drives into 4 mirrors, you lose a ton of space, but you can upgrade in pairs. It's cheaper and more-efficient. Also, the resilvering process is faster, so upgrading takes less time and is less likely to kill a troublesome disk.

Lastly, you can change a 2-drive mirror into 3 when upgrading and bring it back down to 2 when you're done. This is honestly the recommended path as well. If you can, try to avoid going down to 1-drive in your mirror, but even if you do, the resilver is much faster.

 

[Arwen]

Another way to look at it, is a bunch of 4 disk RAID-Z2. Same storage as a 2, 2 disk Mirrors. But, any 2 disks can fail. And yes, you can "replace in place" when either growing a vDev or replacing a failed disk.

The caveat are:
- To get good IOPS, you need more than 1 vDev
- Growing a vDev does require 4 new disks, instead of 2
- More complex data redundancy, parity verses plain copy / mirror
- Can't get the read performance improvement of mirrors

That said, with lots of disks, say 12, it can make sense if you don't need more than 50% available space. Yet want better redundancy than simple 2 way Mirrors.

Or you could do 6 disk RAID-Z3... for 3 way redundancy, which would be suitable for 4 way Mirror replacement.

 

[Sawtaytoes]

1x 2-way vdev is a no brainer, but what about 48x 2-way vdevs?

Both drives failing in the same vdev would destroy the whole pool. Recovering from backups would be a pain.

How many mirrored vdevs would make you uncomfortable?

Dunno why I never posted this screenshot, but I guess I did post in January :p. This does make me uncomfortable, even with SSDs:

And I could and would add more vdevs, but at this point in time, I'm actually wanting to move to dRAID for more capacity (I have 120 SSDs in this system), and I'm still working that out right now or else this would be more like 48 mirror vdevs.