ZFS Configuration

[RichR]

Question about this... http://www.solarisinternals.com/wik...Configuration_Example_.28x4500_with_raidz2.29

If a scrub or resilver is being performed, is the entire pool degraded, or just the raidz2 that is being scrubbed/resilvered?

I have 45 drives, 7 x 6-drive raidz2s, plus 3 spares... but they are all separate "Volumes" in FreeNAS, maybe it would be better to make one volume such as this? Then, the volume size would not be limited other than the entire size of one single volume.....? Currently, 6x3TB raidz2 yields 10.7TB for each Volume (so I have 7 of these volumes). If I would create the volume from the linked example at the top, would I get a single volume, but with 74.9TB? (10.7TB x 7)

Rich

 

[jgreco]

A scrub does not degrade the pool, though it may reduce performance because it is causing components to be very busy.

A resilver does degrade the pool.

The difference is that the data on a scrubbing vdev is assumed to be in optimal condition and should be completely usable - you're just verifying that that's the case. However, on a component that is being resilvered from the rest of the vdev, you have suffered a loss of redundancy, because the component being resilvered does not have a complete or correct version of the data it is supposed to hold. That's pretty much the canonical definition of degraded redundancy.

For your situation, the answer is basically as follows:

As it stands, you have a highly resilient setup. You have 7 6-drive RAIDZ2's. The loss of any two drives is tolerable without data loss. The loss of any three drives is very likely to be tolerable, but if it were to happen within a single vdev, you lose that "volume" (i.e. that one ZFS pool) ... and therefore 1/7th of all your data. The other six ZFS pools go on their happy way.

You can instead set up a single volume made up of seven 6-drive RAIDZ2's. Again, the loss of any two drives is tolerable without data loss. The loss of any three drives is very likely to be tolerable, but if it were to happen within a single vdev, you lose the entire ZFS pool, and all of your data.

You can certainly make a single 75TB volume, and in doing so, you may make it substantially easier to store data without worrying where to put your bits. However, do be aware that your existing setup is more resilient to disk failures, and that your exposure to total loss scenarios is much smaller.

 

[bollar]

The entire zpool is degraded if any vdev in that zpool is resilvered. In a raidz2 scenario, if you lose more than two drives in any single vdev, the zpool will be lost.

Aside from the benefit of having a single large volume, striped vdevs are also somewhat faster.

 

[RichR]

Yeah, your last paragraph points to the real question at hand.... administration (where to put the bits). I currently have to juggle what is going where, which customers use how much space, and at what point do I need to make an additional symlink on their server for their storage to go somewhere else (a different pool).

I certainly wouldn't make a single pool with a single vdev with 42 drives, that's just insane and too risky - but the single pool with 7 vdevs of 6 drives each was what I was really curious about. I get the scrubbing part, although with separate pools, I could stagger scrubbing hence not affect performance everywhere. The worrisome part was resilvering. The entire pool is degraded even if one of the 7 vdevs is resilvering -- that's the balancing act -- worrying about bit placement vs. worrying about degraded state time.......

Thanks for the reply.

 

[jgreco]

ZFS doesn't work well with large vdev's, and generally wants to see a power-of-two for the number of data drives (not incl parity drives) in a RAIDZ{1,2,3}. I don't remember exactly what the current guidelines are, but you would be fine with a pool made out of however many 6-drive RAIDZ2 vdev's.

 

[Z300M]

A scrub does not degrade the pool, though it may reduce performance because it is causing components to be very busy.

A resilver does degrade the pool.

The difference is that the data on a scrubbing vdev is assumed to be in optimal condition and should be completely usable - you're just verifying that that's the case. However, on a component that is being resilvered from the rest of the vdev, you have suffered a loss of redundancy, because the component being resilvered does not have a complete or correct version of the data it is supposed to hold. That's pretty much the canonical definition of degraded redundancy.

Do you mean that the pool is degraded even after resilvering is complete, or do you mean only while resilvering is still taking place?

 

[jgreco]

It should be degraded until the specified level of redundancy is restored and verified.