Resource icon

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter.

This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort, and even in "JBOD" or "HBA mode" generally hide the physical device. If you cannot see the type of device (such as "ST6000DX000-1H217Z" in "camcontrol devlist", you DO NOT HAVE A TRUE HBA. If you cannot get the output of "smartctl" for a device, you DO NOT HAVE A TRUE HBA. A true HBA passes communications through itself directly to a drive without further processing. No amount of marketing department wishful thinking can change that technical reality.

Note that having device names in "camcontrol devlist" and getting "smartctl" results is not any sort of proof that you do actually have an HBA instead of a RAID card. It's just an easy test that weeds out a wide range of RAID cards.

2) FreeBSD/Linux has incredibly robust support for the LSI HBA's.

FreeBSD's LSI HBA (mps/mpr) drivers are authored by LSI and carefully designed to work with their HBA firmware. The TrueNAS userbase has installed many thousands of these cards which have, in aggregate, BILLIONS of problem-free run-hours. Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error. SMART is properly supported. Forum members are incredibly familiar with all the variations on these and can provide useful assistance. Cards such as the LSI 9240-8i, IBM ServeRAID M1015, Dell PERC H200 and H310, and others are readily available on the used market and can be converted to LSI 9211-8i equivalents.

3) You must crossflash to IT/IR firmware

If you don't crossflash, then a lot of the remainder of this ALSO applies to LSI non-IT-20.00.07.00 HBA's!! The IR firmware is also fine but is a few percent slower. It is not clear there is any value to doing this as you would never want to use an IR virtual device with FreeNAS. We used to do this in the old days for boot devices, but with ZFS boot this may no longer relevant.

The LSI 9211-8i (PCIe 2.0 based on LSI 6Gbps SAS2008) and LSI 9207-8i (PCIe 3.0 based on LSI 6Gbps SAS2308) both require firmware 20.00.07.00.

The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware 16.00.10.00 or the special 16.00.12.00 available via iXsystems.

4) FreeBSD/Linux may or may not have good support for other HBA's/RAID controllers.

FreeBSD and Linux do support a wide range of HBA's and RAID cards, but in many cases the drivers have been reverse-engineered, or may cause various problems when there is an underlying issue with a component disk. ZFS has the capability to place immense I/O workloads on your HBA, and you don't really want the controller chip crashing and taking a bunch of your disks offline with it (known to happen). ZFS expects that its ability to talk with correctly functioning HDD's is 100.000% and bad things start to happen when controllers or drivers freak out.

It might seem very tempting to use some alternative brand of HBA or RAID controller that you happen to "have handy", but the problem is that you will be on your own private little adventure. Things might seem to work fine, until one day something bad happens. And then you might be outta luck. Some RAID cards do things like encapsulating their JBOD VD's in a partition, making it effectively impossible to plug the drives into a SATA port or LSI HBA.

If you have come to TrueNAS with the intention of making your NAS into a guinea pig for testing of an unknown and untested controller, then, by all means, go ahead. Just please be aware that the measure of success isn't "I got it to make a pool." It is possible for things to work for weeks, months, even years before something adverse happens. We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours. You will not be able to duplicate that sort of rigorous testing.

4a) But FreeBSD or Linux has a driver or command line tool for my FooBlar HBA or WooGreat RAID! Surely it is fully supported!

FreeBSD and Linux have a lot of drivers for a lot of stuff, but not all of it was written with ZFS in mind. The TrueNAS developers have included drivers and support for a lot of stuff, some known to be catastrophically bad. This is very helpful when you are importing data from an old hardware RAID system into your TrueNAS, but it is not meant to imply that your janky old Adaptec or 3Ware RAID card is an acceptable controller for use with ZFS in TrueNAS. Drivers and/or command line tools for 3Ware, Adaptec, Areca, DPT, Highpoint, Intel RAID, LSI MFI, LSI MegaRAID, Microsemi, Mylex, and possibly others may be included in the FreeBSD base system, but this does not imply suitability for use. If, on the other hand, you merely need to use it to help you import data off an old disk or array onto a new ZFS pool, by all means, go for it.

5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.

In these devices, you are relying on the RAID card driver to communicate from the host to the controller. As previously noted, the LSI HBA drivers have billions of proven run-hours, but in many cases, RAID drivers aren't as solid. Some of FreeBSD's RAID drivers have been tweaked to cope better with device error handling on the theory that you have redundancy (JBOD isn't), many do not allow you to poll the drive's SMART status, and in many cases you can inadvertently set up bad situations with write caching, etc. ZFS has the ability to generate immense amounts of I/O traffic that can be a crushing workload for the weedy little CPU's on a RAID controller, can totally flood the cache on a RAID card, etc. As mentioned in the previous section, many RAID cards also do things such as encapsulation of JBOD within a partition, which effectively locks you into having to use that RAID card. This is super-bad for error recovery. With SATA ports or LSI HBA ports, SATA drives are completely interchangeable.

6) A RAID controller with write cache is particularly bad.

A RAID controller with a write cache is likely to get swamped by the massive I/O ZFS is pushing. These devices are typically sized to cope with the sorts of I/O a standard server would push around, update a file here, read an executable from there, do some database updates... but ZFS will perform operations such as scrubs and resilvers that will maintain massive I/O pressure for hours or days. Even the normal I/O is demanding, as a ZFS transaction group can easily be a gigabyte, every five seconds. Hiding the actual performance of devices behind a tiny RAID card cache is not a good idea as it leads to less-predictable performance.

7) Do not fear the HBA.

For many SOHO, power users, home users, hobbyists, etc., the HBA may be the first exposure to unfamiliar server hardware and different kinds of cabling. It is simply a server-grade version of a SATA add-on card. The cabling is designed for high density inside servers, but you can buy break-out cables to transform SFF8087 to SATA connectors. It can support SAS and SAS expanders. The forums have a nice resource to bring you up to speed on SAS, and it's really quite nice to work with.

8) Fear the HBA.

There's one caveat. Keep your HBA cool. It is an embedded computer and throws off about 10 watts. Failure to have airflow directed over your HBA can cause overheat, and in extreme cases LSI HBA's have been found to vomit random bits all over, which isn't good for ZFS.

9) The only exception.

Because LSI uses a similar MIPS CPU and software design on all its cards, people long ago learned to crossflash LSI 9240 low end RAID cards and OEM low end RAID cards into SAS2008 or SAS2308 IT-mode HBA cards. It is the software running on the card and the driver in use in FreeNAS that matters here.

It is theoretically possible to turn the LSI high end RAID cards into IT-mode HBA's as well, but this is veering off into "that might not be a good idea"-land, because the high end RAID cards have additional components such as cache, flash, and battery circuits that the IT firmware doesn't expect to be there. If you can figure out the black magic to crossflash one of these to IT-mode and it actually works (be sure to test extensively!), then from the perspective of this article, congratulations, you have an HBA, and it's actually kinda likely to work correctly with FreeNAS. However, there might have been software components in the RAID firmware that controlled things like battery charging, so I'd suggest removing batteries or supercaps to reduce the chance of some unexpected issue that might result in a battery fire. Changing the card out for one that's known to work correctly is still a better option.

2021/12/29: Update for LSI high end RAID cards:
@bartuc dug up this link for the ServeRAID M5110/H1110 cards. I have not tried it. If it works for you, be sure to go on over to this thread and say thanks.

10) The Age of SATA is Ending, and SAS is Probably Dying Too

For many years, SATA and SAS have played a major role in the storage world. However, the SATA Cabal never generated a useful standard for anything past 6Gbps, partially because HDD's can only manage around 3Gbps peak. For SSD, the advent of NVMe allowed those manufacturers to bypass the logjam entirely and also the need for dedicated HBA/RAID controllers. The result is that there isn't a lot of demand for development of higher speed SAS. It is likely to remain a technology with a long tail for attaching shelves of hard disks, but most of the storage controller vendors have collapsed and been acquired. Microchip owns Microsemi owns Adaptec. Broadcom owns Avago owns LSI owns 3Ware. Etc. I doubt any new players are developing new HBA's, so my point is this: the options described in this document are effectively the end of the road. We are likely to see some new weird stuff like NVMe-for-HDD, but HDD itself is in a race to the end, as flash memory prices continue to drop.
Author
jgreco
Views
121,346
First release
Last update
Rating
5.00 star(s) 2 ratings

More resources from jgreco

Latest reviews

There should be a link to this article or the content should be included in the storage section of the TrueNAS doc site.

This was exactly what I was looking for. I am migrating from SATA based workstations to a storage bridge bus (SBB) based Supermicro 6036ST-6LR two node system. The SAS2008 based card is in IR mode but running older (11.0) firmware. I didn't want to migrate to the IT firmware unless required because the RAID HBA is packaged onto the motherboard and has proprietary connections to the midplane. Looks like I just need to upgrade the firmware to 20.00.07.

What brought me here? I see all of the disks in `camcontrol devlist`, I get unique serial numbers when enumerating them with `camcontrol inquiry <disk> -S`. I can add disks to ZFS vdev's in FreeBSD 13.0, but TrueNAS 13-U3 doesn't show any available disks. I was nearly confident I was going to have to go to IT firmware to get FreeNAS to show them as available, and I am unsure if I will be able to use the (Supermicro noted as required RAID functionality) without IR firmware when sharing access between the two nodes to the same storage with RSF-1-HA. It now looks like I just need FW 20.0 on the

I will follow up with edits for any issues I run into... just wanted to share my appreciation for this article.
jgreco
jgreco
The TrueNAS developers have always been resistant to draw lines in the sand for what hardware can be used with TrueNAS. They aren't tasked with the job of day-to-day support, which is unfortunate. It leaves us here in the forum holding the bag, so to speak. I'm more interested in providing pragmatic documentation on how things work and what's expected to work, because I'm pointing users at my resources multiple times per day.

While I cannot promise any particular result, I would note that historically, it's been fine to use true LSI IR firmware -- not vendor hacked-up firmware -- in place of the IT code. I have machines as recent as TrueNAS 12 running IR cards and they're fine. I would just try running IR fw 20.00.07.00 and see if that works out.
Very descriptive and helpful.
Top