5x 4TB RAIDZ1 Array Rebuilding (with NRE/URE issue)

[Sonic]

I am planning to have a 5x 4TB RAIDZ1 array which is 16TB usable space. What I expect is data wouldn't loss when lossing one disk. However, issue regarding to Non-recoverable Read Errors (NRE)/ Unrecoverable Read Error(URE) come out when i do some search on RAID5 rebuild.

For non-enterprise hard drives, NRE/URE rate is normally 1 per 10^14 bits read, which means that NRE/URE occurs every 11.37TB. For my case, when array is full and one disk fails, system have to read 16TB data to rebuild the array. There is a very high probability that NRE/URE occurs during rebuild process.

My questions are:-
(1) What will Freenas do when NRE/URE occurs?

(2) Can Freenas still finish rebuilding if there is NRE/URE occurrence in the process?

(3) If I use ECC ram & cpu, can I ignore this issue?

(4) If answers of question 2 & 3 are "No", it means raid 5 rebuild process fails most of the time for array over 11.37TB usable space, right?

(5) Without using expensive enterprise hard drives, what other measures can be taken to protect an array from one disk failure under NRE/URE rate of 1 per 10^14 bits read?

Many thanks

 

[cyberjock]

I am planning to have a 5x 4TB RAIDZ1 array which is 16TB usable space. What I expect is data wouldn't loss when lossing one disk. However, issue regarding to Non-recoverable Read Errors (NRE)/ Unrecoverable Read Error(URE) come out when i do some search on RAID5 rebuild.

For non-enterprise hard drives, NRE/URE rate is normally 1 per 10^14 bits read, which means that NRE/URE occurs every 11.37TB. For my case, when array is full and one disk fails, system have to read 16TB data to rebuild the array. There is a very high probability that NRE/URE occurs during rebuild process.

My questions are:-
(1) What will Freenas do when NRE/URE occurs?

(2) Can Freenas still finish rebuilding if there is NRE/URE occurrence in the process?

(3) If I use ECC ram & cpu, can I ignore this issue?

(4) If answers of question 2 & 3 are "No", it means raid 5 rebuild process fails most of the time for array over 11.37TB usable space, right?

(5) Without using expensive enterprise hard drives, what other measures can be taken to protect an array from one disk failure under NRE/URE rate of 1 per 10^14 bits read?

Many thanks

I'll assume you mean a ZFS RAIDZ1.

If you are using RAID5 hardware, then it depends completely on what your hardware does and all of my answers won't provide any insight.

If you are using UFS RAID5 then I'm not sure because I'm pretty sure RAID5 isn't supported with FreeNAS 8, which means you are using a different product that had the same name(FreeNAS 7 was renamed to NAS4Free and you should go there for help in that case).

1. ZFS will continue the rebuild, but you will have some amount of corruption. It could range from a single file being corrupted to metadata corruption that causes a lot of files to be lost or the zpool to be unmountable(a loss of all data).
2. ZFS will continue to rebuild, but again, that's based on what is corrupted. As mentioned in #1 the consequences can range from minor to major.
3. No. ECC has nothing to do with NRE/URE.
4. It means you will have "some" amount of corruption. It depends on what data is corrupted. As mentioned in #1, the consequences can range from minor to major.
5. Make a RAIDZ2/RAID6.

The stuff you are discussing is precisely why RAID5 was deemed "dead" back in 2009 or so. If you google "why RAID5 is dead" you'll find an article from 2009 explaining pretty much everything you've discussed here. On a harware RAID you are typically screwed. With ZFS its like gambling. You might be okay or you might lose everything.

 

[Sonic]

1. ZFS will continue the rebuild, but you will have some amount of corruption. It could range from a single file being corrupted to metadata corruption that causes a lot of files to be lost or the zpool to be unmountable(a loss of all data).
2. ZFS will continue to rebuild, but again, that's based on what is corrupted. As mentioned in #1 the consequences can range from minor to major.
3. No. ECC has nothing to do with NRE/URE.
4. It means you will have "some" amount of corruption. It depends on what data is corrupted. As mentioned in #1, the consequences can range from minor to major.
5. Make a RAIDZ2/RAID6.

The stuff you are discussing is precisely why RAID5 was deemed "dead" back in 2009 or so. If you google "why RAID5 is dead" you'll find an article from 2009 explaining pretty much everything you've discussed here. On a harware RAID you are typically screwed. With ZFS its like gambling. You might be okay or you might lose everything.

So, for one disk failure, what will Freenas do when NRE/URE occurs during ZFS RAIDZ2 rebuilding? Can I have a full recovery?

 

[cyberjock]

As long as you don't have a third disk in that "stripe" of data that has an error, then the recovery should proceed normally.

 

[Sonic]

As long as you don't have a third disk in that "stripe" of data that has an error, then the recovery should proceed normally.

If choosing RAIDZ2, according to users guide, it is better to have 6x 4TB array for optimal performance.

Do you mean Freenas can fully recover a 6x 4TB array (5 existing & 1 new) with NRE/URE occurrence during rebuilding process?

How Freenas deal with the disk having NRE/URE? Will it considers the disk as fail disk, or just markes down the bad sectors and aviod to use that section?

 

[cyberjock]

Do you mean Freenas can fully recover a 6x 4TB array (5 existing & 1 new) with NRE/URE occurrence during rebuilding process?

It's just like I said above...

As long as you don't have a third disk in that "stripe" of data that has an error, then the recovery should proceed normally.

How Freenas deal with the disk having NRE/URE? Will it considers the disk as fail disk, or just markes down the bad sectors and aviod to use that section?

It doesn't mark disks as "good" or "bad". They are either attached to the system and part of the zpool, or they are not. ZFS doesn't mark down "bad" sectors, it simply rewrites the good data when corruption is found and trusts that the hard drive is capable of self maintenance. The hard drive should be remapping bad sectors as necessary. If the hard drive isn't, you shouldn't be using them anyway.

 

[Sonic]

It's just like I said above...

I just want to make sure I do not misunderstand your meaning.

Besides, while the theoretical maximum protected usable capacity of RAID 5 is 11.37TB for disks with 1 per 10^14 NRE/URE rate, how about RAID 6?

 

[thelambentonion]

Besides, while the theoretical maximum protected usable capacity of RAID 5 is 11.37TB for disks with 1 per 10^14 NRE/URE rate, how about RAID 6?

RAIDZ2/RAID6 sacrifices an additional drive for parity (N+2), which means that in the event of a URE during rebuild, there is still a parity drive to protect you. The author of the article cyberjock mentions seems to believe that RAID6 will stop making sense by 2019 for similar reasons, so keep that in mind.

EDIT: I feel that I must also add the following: RAID isn't really a backup solution, and the steps that everyone on this forum advocates are simply to minimize downtime. If your array fails, it should be inconvenient, not life-shattering.

 

[cyberjock]

I agree with the math, but disagree with the logic... kinda.

For hardware RAID, you may be in trouble if your controller drops the drive on a NRE/URE.

But, ZFS, as long as it can repair that "stripe" of data, will continue on. So as long as you don't end up in a situation where the same stripe will have 3 bad pieces of data/parity, then all is well. I believe that the typical stripe is 128kbytes, so it would be very hard for me to be convinced that in 2019 that reading 128Kbytes of data is likely to return a NRE/URE regularly. Also keep in mind that an additional failed disk and whatnot causes a big poop stain on the carpet when you start considering probabilities.