Hi guys, I manage our entreprise freenas server. I just want to share my recent experience. We have only +-900gb of data and we are from 4 to 6 to work on documents, drawings, etc... So this is not a really demanding environment. My only preocupation is data integrity.
Our setup is 4 drives, in a raid 1 + raid 0 with a hot spare. In my head, I was bullet proof with this setup. I tought that I was able to loose 3 drives. After only 4500 hours, one of my hard drive started to have problems. 2 times, freenas crashed. So after the second reboot, the da4 was considered faulty. So I had to manually tell freenas to replace the drive by the hot spare.
Guess what, during resilver, the number of checksum errors on the da3 began to rise to over 1000. For those that might interest, I have not lost anything and I had a backup anyways.
Conclusion ;
-Raid 0 + raid 1 with 4 drives is not as good as it seems to be. If 2 mirror drives fail, it's useless to have 2 copies of only one half of your data. Your are screwed.
-In your setup, plan for a multiple hard drive failure at the same time. It's probable, if drives are mirror. They are doing the exact same thing, they have been produced the same day, put in service at the same time, then they can have the same defect and they can fail at the same time.
Other consideration ; I was suprised that the failing drive was participating to the resilver of the hot spare. 40% of the data (80mb/sec) was comming from the failing one and 60% ( 120mb/sec ) from the remaining good drive. This bring me this point ; if you can, use machine with more bays than you need. That way, you will be able to use hot spare to replace drives. That way, you reduce pressure you put on your remaining good drives. As my experience shows, your remaining drive could be near the end also, so reducing pressure on it might be a good idea. Finally I will replace the da3, so my both mirrors were about to fail.
Next, I plan to replace my raid 0 + 1 by a Raid Z-2 five drives + 1 hot spare. Any better idea ?
Francis
Our setup is 4 drives, in a raid 1 + raid 0 with a hot spare. In my head, I was bullet proof with this setup. I tought that I was able to loose 3 drives. After only 4500 hours, one of my hard drive started to have problems. 2 times, freenas crashed. So after the second reboot, the da4 was considered faulty. So I had to manually tell freenas to replace the drive by the hot spare.
Guess what, during resilver, the number of checksum errors on the da3 began to rise to over 1000. For those that might interest, I have not lost anything and I had a backup anyways.
Conclusion ;
-Raid 0 + raid 1 with 4 drives is not as good as it seems to be. If 2 mirror drives fail, it's useless to have 2 copies of only one half of your data. Your are screwed.
-In your setup, plan for a multiple hard drive failure at the same time. It's probable, if drives are mirror. They are doing the exact same thing, they have been produced the same day, put in service at the same time, then they can have the same defect and they can fail at the same time.
Other consideration ; I was suprised that the failing drive was participating to the resilver of the hot spare. 40% of the data (80mb/sec) was comming from the failing one and 60% ( 120mb/sec ) from the remaining good drive. This bring me this point ; if you can, use machine with more bays than you need. That way, you will be able to use hot spare to replace drives. That way, you reduce pressure you put on your remaining good drives. As my experience shows, your remaining drive could be near the end also, so reducing pressure on it might be a good idea. Finally I will replace the da3, so my both mirrors were about to fail.
Next, I plan to replace my raid 0 + 1 by a Raid Z-2 five drives + 1 hot spare. Any better idea ?
Francis