ZFS file system and Hardware RAID

[ivkeee]

HI all!
I have one simple question that I could not find answer on the internet.
Can I use FreeNas zfs file system on hardware RAID? (NOT RAID-Z, only zfs file system)
In my case hardware RAID is mandatory because of the company policy.
I understand that combination of hardware RAID and zfs software RAID (RAID-Z) is not recommended and I don't plan to use zfs software RAID.
When I install FreeNas I see only one hdd (hardware RAID 10 is presented as one hdd).
In my case I cannot use RAID-Z because FreeNas see only one hdd.
FreeNas is installed on the flash disk.
All I could find on the internet is one comment from one of the FreeNas release:
"If you have to use hardware RAID, the ZFS is recommended instead of UFS."

Please do not send references to zfs wiki pages because they do not have the answer to my question.

 

[JaimieV]

Yes, you can certainly do this. You still gain the benefits of ZFS' filesystem features like data scrubbing, replicas, compression and so on.

 

[ivkeee]

Thank you JaimieV!
A lot of people are mixing terms ZFS and RAID-Z.
At a lot of forum posts I found something like "Do not use ZFS and hardware RAID!" which is not true.
This should be renamed to "Do not use software RAID-Z and hardware RAID".

 

[JaimieV]

Right - mixing RAIDZ's on top of hardware RAID will make for Interesting Times when something goes south.

It is generally easier to solve problems when the whole stack of underlying support is known, which is why it's generally heavily recommended that you use the ZFS RAIDZ functions rather than builtin hardware ones - it means you can get help more easily because people can repeat your setup. But you're stuck with the environment you have, and ZFS's data scrubbing is worth the price of admission alone.

 

[Joshua Parker Ruehlig]

Yes, you can certainly do this. You still gain the benefits of ZFS' filesystem features like data scrubbing, replicas, compression and so on.

If FreeNAS is presented with only one drive he won't get the main benefit from scrubbing. All it will do is read the disks and tell you that some of your data is corrupted. It won't be able to fix the bitrot unless you have software redundancy, but i guess everything else would work as normal.

 

[cyberjock]

Quoted from the FreeNAS manual:

NOTE: instead of mixing ZFS RAID with hardware RAID, it is recommended that you place your hardware RAID controller in JBOD mode and let ZFS handle the RAID. According to Wikipedia: “ZFS can not fully protect the user's data when using a hardware RAID controller, as it is not able to perform the automatic self-healing unless it controls the redundancy of the disks and data. ZFS prefers direct, exclusive access to the disks, with nothing in between that interferes. If the user insists on using hardware-level RAID, the controller should be configured as JBOD mode (i.e. turn off RAID-functionality) for ZFS to be able to guarantee data integrity. Note that hardware RAID configured as JBOD may still detach disks that do not respond in time; and as such may require TLER/CCTL/ERC-enabled disks to prevent drive dropouts. These limitations do not apply when using a non-RAID controller, which is the preferred method of supplying disks to ZFS.”

It's really not recommended to mix hardware RAID with ZFS RAID. I'm sure the OPs searching found me saying this too. Personally, if I had to use a hardware RAID I think I'd consider going with UFS. There is no fsck for ZFS, and I'm not sure how ugly it would be if the bits in the file system got messed up. ZFS may or may not be able to fix the error. It would definitely suck to have a file system that got trashed because of bitrot and you couldn't run any tool to fix it. You'd be left with the only option being to destroy and recreate the zpool. YUK!

 

[JaimieV]

One would like to think that any data scrubbing read problems would only trigger a "oh hey, problem" at the hardware RAID level, which should allow you to replace a dodgy disk before any data loss occurs... with the hardware RAID delivering it's own CRC'd data from the other disks. Am I too optimistic? I can't see how a scrub error would actually appear at the ZFS layer without being caught first at the hw RAID layer.

 

[Joshua Parker Ruehlig]

One would like to think that any data scrubbing read problems would only trigger a "oh hey, problem" at the hardware RAID level, which should allow you to replace a dodgy disk before any data loss occurs... with the hardware RAID delivering it's own CRC'd data from the other disks. Am I too optimistic? I can't see how a scrub error would actually appear at the ZFS layer without being caught first at the hw RAID layer.

scrub has nothing to do with the error. the error is bitrot, and no hardware raid takes it into account. Nothing fixes bitrot except zfs with redundancy.
Now you could scrub and see what files are corrupted, grab them from a backup and overwrite the bad version, but this is what zfs is supposed to do automatically.

 

[JaimieV]

I probably should have said "on-read" error, but in a ZFS system that's probably more likely to happen at scrub time than any other!

Replication sets on the same volume wouldn't help here, I expect.

Still, at least you would be notified of the bitrot, which is a big step up from finding out about it after the rot has cycled through all your backup history.

 

[paleoN]

I would suggest presenting two LUNs to FreeNAS and using a ZFS mirror or alternatively using copies = 2 on the main dataset. Otherwise ZFS can only report errors and not fix them. IMO, the multi-LUN setup would have slightly better redundancy.

 

[JaimieV]

Top tips, paleoN - thank you. I'd forgotten all about copies = 2.

 

[cyberjock]

Personally, I'm not convinced that the RAID controllers I have used in the last few years actually use the parity data to verify that data read was correct.

Example:

With my Windows Server (hardware RAID6) if I stream a movie I get small reads from 1 disk at a time. If the RAID was actually checking for bitrot then it would be reading 1 "stripe" of the RAID at a time so it could do parity compare. So I tend to think that most RAID controllers don't do parity compare for reads.

I have noticed this behavior with Areca, Adaptec, Highpoint, and 3ware.

 

[jgreco]

No, in normal operation RAID controllers absolutely DO NOT use the parity data to verify that the data read was correct. To do so would require all spindles to seek to read the blocks in question in order to obtain all the needed parity reconstruction information; this is what a RAID does in degraded mode to "recover" your data and allow things to keep working when disks are failed. You don't need to read a stripe at a time or anything like that, but you do need to read the sectors involved, which would have approximately the same IOPS hit (because IOPS are usually more about seeks than transfers). Most hardware RAID controllers are too stupid to act with more finesse and their stripe sizes are usually far too small to worry about such finesse anyways, but when you're doing software RAID with a many-MB stripe size, it matters.

 

[JaimieV]

I feel educated. Thanks to all.