ZFS vs hardware raid question

[Milkwerm]

Hi
With hardware raid, in order to maximise IO, "WriteBack" (with a BBU) is often used (and prefered for hosting Virtual machines)
My question is, Does a ZFS Raidz set get the same advantage from any particular setting/configuration in order to maximise IO throughput?

 

[cyberjock]

From my experimenting with ZFS on my raid controller(with 4GB of on-card RAM) your best bet is to disable the write cache and leave the read cache enabled with "normal" or "conservative". This provides the best performance and reliability.

Using the write cache on the RAID controller can cause corruption because when ZFS commits writes to the hard drive they are silently written to the raid cache and not to the hard drive. This can really mess up a zpool and your data on a loss of power or system crash. Not to mention using the write cache can cause conflicts between reads and other writes because the hard drive will be busy when ZFS thinks the drives are idle.

 

[Milkwerm]

Thanks Cyberjock.
By read cache "on" and write cache "off" your talking about a "writethrough" setup which is usually the fall back option for Writeback if/when a BBU battery fails?. Very interesting information you have posted that I hadn't thought about, but not why I was asking . My fault, I'll try to re-explain the question.
I disable the raid card altogether when setting up RaidZ disks (raidz1), taking hardware caching out of the equation. This is on an old S5000PAL intel board and I remove the Raid key from the daughter board. So in a completely ZFS software controlled raid are there options like writethrough/writeback that can speed up the I/O or do we have to look to SSD drives for arc cache etc as eqivilents?

 

[cyberjock]

No. On my card (Areca 1280ML) I have settings for the read cache and write cache.

Write cache options: Disabled, Write-Through, Write-Back
Read ahead-cache options: Disabled, Conservative, Normal, Aggressive.

So all write cache options should be off/disabled. This is as recommended by ZFS for reliability and happens to provide the best performance anyway. So turning it on is pretty much a bad idea for both reasons. I have done one-on-ones with people and it seems that regardless of the controller used, write cache off seems to always work best even if you ignore the ZFS reasons. Even if it did provide a performance boost, unless performance was terrible and the absolute only way to fix it was enabling it I would strongly recommend against using it. You can basically unplug your BBU because it will serve no purpose at all.

For read ahead I set mine to off at first, then changed it to normal. Normal seems to result in a significant(more than 150% increase in performance!) of scrubs. Also the system is much more responsive and generally seems to just run smoother.

If you have one of those $1000 RAID controllers with 2GB+ of RAM on card and a BBU and you were looking to save some money I'd ebay the card(or return it to the manufacturer) and get an IBM 1015 RAID card. The 1015s are typically less than $100 so you can pocket quite a bit of money by changing cards. The only reason I kept mine is because I'm too lazy to sell mine. But I know the day is coming when I'll finally get smart and spend some time to ebay my card.

Edit: Note that if you make these changes using a controller CLI utility you will most likely have to reboot for the changes to take effect.

 

[ioudas]

I signed up to say... this advice is about the worst advice ive ever seen on the internet anywhere. Ever. Disabling will just give you more problems.

Food for thought.

 

[warri]

ioudas, do you have any backing on why it is a "crappy" advice? Some explanation would be helpful at least.

As far as I now, cyberjock is somehow right here. The general recommendation for ZFS seems to be to disable write-back mode if you have no battery backed cache on the RAID controller. In this case I guess you could still use "write-through" - since that is supposed to write the changes to both the disk and cache.
On the other hand, if you have a controller with a BBU it should be fine to use "write-back" - if you get your system up and running in a reasonable time frame after a power failure.

Disclaimer: Solely based only on google and some reading, have never used such a setup myself ;)

Some sources:
http://forums.freebsd.org/showthread.php?t=39040
http://forums.freebsd.org/showthread.php?t=29233
http://serverfault.com/questions/114547/what-is-a-raid-controller-bbu-for

 

[ioudas]

Failure to understand the concepts of hw raid mostly. From cyber jocks post he mentions nothing about BBU raid controllers. Your comments here actually reflect reality.

Edit also: you will notice performance differences with disabled vs write back.. Disabled not being the best.

 

[cyberjock]

If you read around here at all you'd know I had a BBU, and deliberately removed it because I had to disable the write cache to get reasonable performance. All of this is discussed above as well as plenty of other posts explaining in detail why having a write cache enabled doesn't garner you extra performance.