SOLVED why loss of data when RAIDZ1 vdev loses 1 disk?

[skimon]

I'm ramping up on zfs and related topics and fiddling around with my vmware sandbox version before i create a new system and migrate . I was reading cyberjock's FreeNAS guide for nubs (me). I saw this:

"If your VDev has only 1 disk of redundancy(RAIDZ1 or RAID1) and you have to replace
a failed disk and any other disk has even 1 bad sector you are literally running a RAID0(striped pool) until the rebuild is complete. Statistically you can also expect some ZFS corruption and data loss. The data loss can vary from a single file to complete file system corruption resulting in loss of multiple files or metadata. This is not a limitation of ZFS but is a limitation of having only 1 disk of redundancy."

By extrapolation doesn't this mean the with RAIDZN if you lose N disks *and* any other disk has 1 bad sector you are guaranteed loss? This would seem like big deal, to the point where i would just say RAIDZN is trashed in freenas (and not degraded) as soon as you have lost N disks. Maybe i am missing something here, unless a bad sectors on those remaining disks is particularly rare, but that would make RAIDZ2 only good for 1 disk loss, and RAIDZ3 good for 2 disk loss.

Is there any way to increase the parity storage by default so that you still have some wiggle room on the remaining disks in the vdev when you have lost N disks?

Thanks !

 

[Jailer]

What he is expalining is that if you have data corruption one disk beyond what your vdev has for redundancy you will lose your data. RAIDZ1= lose 1 disks + data corruption on another and your data is lost, RAIDZ2 = lose 2 disks + data corruption on another and your data is lost and so on.

 

[SirMaster]

"why loss of data when RAIDZ1 vdev loses 1 disk?"

Because RAIDZ1 only has 1 disk worth of parity data. So when you take a disk away then every sector that's remaining is completely necessary and any error in any sector will result in corruption of that data.

It's really the same concept as just running a single disk. On a single disk, every sector is necessary, if any sector goes bad, then you will have data corruption.

The main difference for ZFS or RAID is that it's worse in an array because generally your are dealing with even more sectors than a single disk. Sector error rates increase linearly with the number of sectors, so if you have 5 4TB drives striped and no parity because you lost your 6th 4TB disk, then there is statistically a 5x greater likelihood of a corrupt sector than if you just had a single 4TB disk, because there are 5x more sectors in your array than on a single disk.

You also have to remember than RAID isn't a backup or a substitute for a backup. It's purpose is to increase performance (through striping), and to increase uptime (through mirroring and parity), and in the case of ZFS, to increase data integrity (through a combination of checksums, mirroring, and parity).

There is always the chance to get corrupt files and lose pools which is where the backup comes into play.

If you had a RAIDZ1 and lose a drive, you will start to rebuild it. In the best case, the other disks will operate as they are supposed to and not have any errors, and your array will rebuild without problem.

A worse case is that during the rebuild another disk will have an error in a sector. Fortunately for ZFS it knows what sectors are a part of which files so it will tell you which file was corrupted and unrecoverable after the resilver finishes. In this case you delete the corrupt file and restore it from your backup.

Then in the worst case this corruption occurs in the zpool metadata. This can take out your whole pool. Fortunately again for ZFS however is that ZFS always stores at least 2 copies of the metadata, so a single corrupt sector (even with no parity) cannot take down your whole zpool. You would need multiple corrupt sectors in specific places to lose both copies of your metadata and lose the pool.

RAIDZ2 and RAIDZ3 simply come with better guarantees of greater uptime and greater data integrity, but you should still be keeping backups for the same reasons as the RAIDZ1 example.


Just an extra bit on the contrast of ZFS RAIDZ1 compared to normal RAID5. In a normal RAID5 if you lose a disk and then encounter a corrupt sector during a rebuild, the rebuild will generally completely fail simply because the RAID5 array doesn't know which file includes the unrecoverable sector. Some RAID systems allow you to ignore the error and continue on, but you still won't know what file the corruption has taken place in and thus won't know what file(s) you need to restore from your backup, so you basically have to just restore everything. Also other RAID5 systems may not keep multiple copies of the metadata and can suffer complete array loss with less corruption than it would take to take out a zpool.

This is why ZFS is so awesome and above standard RAID.

 

[skimon]

Ah ok so basically if you lose N disks in RAIDZN you would not throw everything out , rather let it rebuild once you have replaced the disks and let freenas let you know which files get corrupted and then restore those from a good backup?

I think that makes sense , thanks. What i didnt understand was that it would tell you what was corrupted which is good. Without that i thought it would be better to just restore everything from backup.

 

[SirMaster]

Yes, depending on how big your array is and how often you scrub and how well you stay on top of any SMART errors and pre-emptively replace disks before they get bad your chances of successfully rebuilding even a RAIDZ1 can still be pretty high (again assuming it's not too large).

 

[skimon]

Yes, depending on how big your array is and how often you scrub and how well you stay on top of any SMART errors and pre-emptively replace disks before they get bad your chances of successfully rebuilding even a RAIDZ1 can still be pretty high (again assuming it's not too large).

Ok sorry just so i understand what do you mean by chances of successfully rebuilding - do you mean chance of rebuilding without having any corrupted files and thereby needing to restore anything from backup - or do you mean chances of rebuilding at all?

 

[SirMaster]

Ok sorry just so i understand what do you mean by chances of successfully rebuilding - do you mean chance of rebuilding without having any corrupted files and thereby needing to restore anything from backup - or do you mean chances of rebuilding at all?

Chance of rebuilding without any corruption.