Confused about that LSI card? Join the crowd ...

Status
Not open for further replies.

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
A recent post prompted me to think, there's a lot of confusion regarding LSI HBA/RAID cards. If you're confused, it is probably understandable... it can be a bit complicated!

ZFS users on FreeNAS should avoid using hardware RAID cards. ZFS and FreeNAS work best when the drive is managed directly by FreeNAS, including having SMART data available from the drive. Any use of "virtual drives" or one-drive RAID0 JBOD modes likely involves the controller writing its own proprietary configuration to the drive and then that means that if you ever need to switch controllers, you're going to have extra special trouble because you need to copy the data off the old drive, not just move the old drive to a new SATA controller port. See this linked article: What's all the noise about HBAs, and why can't I use a RAID controller?

LSI makes a lot of hardware. They make HBA ("Host Bus Adapter") and RAID cards. In general, a HBA is probably a better choice than a RAID card. The HBA cards I've seen are most often driven by the LSI Logic Fusion-MPT2 SAS driver in FreeBSD ("mps") but older ones may be driven by the LSI Logic Fusion-MPT SCSI driver ("mpt"). Most of the current RAID cards seem to be driven by the LSI Logic MegaRAID SAS driver ("mfi") which comes with some significant caveats, including being unable to see your drives via the FreeBSD CAM subsystem or access diagnostics via SMART, and all maintenance and management must be done via the "mfiutil" tool or via the BIOS. If you have an older controller of some sort, one that doesn't do SATA-III, you may need some other drivers and you will also be limited to no more than 2TB drives. If you are considering buying one, don't, unless you're getting it for $10 or free or something like that... such controllers include the IBM ServeRAID BR10i and Intel RAID Controller SASUC8I (LSI SAS3082E-R/LSI SAS1068E chipset) and are driven in FreeBSD by the LSI Logic Fusion-MPT SCSI driver ("mpt"). It is possible to crossflash these cards to be a generic LSI SAS1068E card, but the silicon is still capped at 2TB drives.


A favored card in the FreeNAS community is the IBM ServeRAID M1015, a budget RAID card that can be found inexpensively (~$75) on eBay. This card comes by default with IBM's version of the LSI RAID firmware on it. It's basically an LSI 9240-8i. You don't want to run that, you instead want to crossflash it to be an LSI 9211-8i (also known as LSI SAS2008) in IT mode, making it a basic SAS/SATA HBA card. Two SFF8087 connectors provide up to 8 SAS/SATA channels directly via breakout cables, or, if a compatible SAS expander is used, possibly many more SAS/SATA channels. The card consumes around 10 watts and should have at least some airflow in order to maintain proper cooling. It is important to crossflash this card! If you do, it works under the "mps" driver, and it becomes a plain HBA. If you don't, then it is under the "mfi" driver and it is a RAID card. If you get an M1015 and crossflash it to IT mode, you end up with one of the best HBA controllers available for FreeNAS, in my opinion. These cards often come without brackets or with a low-profile bracket to fit into 2U rackmount servers. Getting a proper bracket is recommended.


Configuration of LSI 6Gb/s HBA products

This is the BIOS probe of an M1015 in IT mode with 8 drives attached via a single SFF8087 and an LSI SAS expander (Supermicro 24-drive SAS backplane). Directly attached drives look just about the same, just fewer of them:

screen6.PNG

You can hit control-C during that to get to the card configuration utility.

screen7.PNG

Shows you the adapter type and firmware revision. Note firmware version 15 and IT mode. You can look under SAS Topology at your attached disks, but really there's not a whole lot to be seen here in IT mode.

screen8.PNG

Within FreeBSD, that'll probe as a controller serviced by the mps driver, and the drives will appear as normal "daX" devices and appear in "camcontrol devlist", so pretty straightforward pleasure there. A "dmesg | grep mps" would show something like:

screen9.PNG

And so far, things like hot adding a drive seem to work fine. It has been a very pleasant and flexible controller to work with, as we use them with ESXi in -IR mode as well. I expect that other LSI SAS 6Gb/s HBA's would closely resemble the above; if you run into something different, please let me know.


On to LSI 6Gb/s RAID:

The higher end LSI RAID products are a bit different. Aimed at high performance and high availability applications, they tend to have more ports, cache, perhaps battery for the cache, and options for managing and configuring the array. These devices are intended to abstract out the physical storage and present the system with virtual devices, which means that they've got an incredibly flexible and rich featureset and the ability to attach lots of drives both directly and indirectly, in many RAID levels, etc., which is great for Windows but not so great for FreeNAS and ZFS, where the operating system is quite capable and competent at dealing with the raw disk devices. Using a hardware RAID card introduces numerous additional failure points such as write caching complications, device drivers that aren't up to snuff, encapsulation inside a controller-provided partition, etc. It's usually easy to make a RAID card work, but that's "work" in the sense that you got it to run, not that it runs correctly or under adverse conditions -- things you definitely should demand from your HBA. Therefore you should avoid high end RAID cards and use a true HBA. A hardware RAID card that claims to offer JBOD or HBA isn't sufficient.

In some cases, users have come to the forum inquiring about a many-port controller where some combination would do. It is worth noting: With ZFS, you do want a decent SATA connection for your drives. Motherboard ports are generally fine. The $1 PCIe add-on card in the clearance bin at the local computer store is probably dodgy. A quality HBA from LSI will be reliable. If you need 10 SATA-III ports, you do NOT NEED A 16-PORT LSI RAID CONTROLLER FOR $800. You don't even need a 16-port LSI HBA for $500. You can get away with an M1015 and two motherboard SATA ports. Really! If you need 20 ports, consider two M1015's and four motherboard SATA ports. Even a SATA-II port can go much faster than a contemporary spinny hard drive can. Yes! It works just fine. If you need 24 ports, you definitely DO NOT NEED A 24-PORT LSI MegaRAID 9280-24i4e LSI RAID CONTROLLER FOR $1500. Get the point?

If you haven't figured it out, the point is to steer away from port-dense and extremely pricey RAID controllers. Don't let me stop you from spending $1500, but if you're going to spend $1500, buy a nice Supermicro chassis with a built-in SAS expander backplane like the SC846BE26-R920B along with an M1015 and cables, it is a better choice for FreeNAS.

But if you're stuck with an LSI RAID controller, here's some hopefully helpful information because you're probably sitting there wondering "what now."

NOTE: In the many years since this was written, there's been lots of evidence that you do NOT want to do this. Please see the linked article: What's all the noise about HBAs, and why can't I use a RAID controller?

We have a Supermicro LSI 2208 built into the mainboard on this server. Planning to use it for ESXi so yes we needed a RAID controller, but right now it is running FreeNAS for fun testing. The "Driver for new LSI card" thread came across so I went looking to see what the deal was.

This is how you can identify the LSI BIOS that uses MFI-based drivers. When you're booting, it'll say "SAS-MFI BIOS" and talk about "WebBIOS". Our Supermicro LSI MegaRAID 2208:

screen1.PNG

If you hit control-H to go into the WebBIOS (yay for consistency), it'll give you tools to configure your controller and the attached drives. They try to make it look Windows-y and while it's possible to control from the keyboard (use ALT plus the highlighted characters, and ENTER to select), it helps to have a mouse. The controller selection screen there will tell you the make and firmware of your controller.

screen2.PNG

The LSI intention is for individual drives to be aggregated into virtual drives, such as that virtual drive 0 which is a RAID1 of two SSD's, which is a common config here for our ESXi hosts. But that's not really what you want for FreeNAS. If you just plug drives into the LSI, they'll show up as "Unconfigured Good" drives:

screen3.PNG

so at this point you know the controller and the drives are talking. So the question becomes, what then. Apparently it is possible to configure the LSI controllers to pass through unconfigured-good drives to the underlying OS but they don't do it by default, and I don't see an immediately obvious way to set that. And it is more difficult because the disks don't show up in "camcontrol devlist". A "dmesg | grep mfi" from the FreeNAS CLI yields

screen5.PNG

which shows the RAID1 virtual device happily showing up as mfid0, but no mfid1. You can do a "mfiutil show drives" to see attached drives and "mfiutil show volumes" to show the available volumes. So what you probably want to do is do "mfiutil show drives" and then take note of the number (first column) of each "UNCONFIGURED GOOD" drive, then run "mfiutil create jbod NUMBER" for each of those numbers. Be warned that creating JBOD is almost certainly guaranteed to be overwriting what is on the disk. I don't think the MegaRAID Firmware Interface system has a way to cope with acting as a dumb SATA/SAS controller, so you will not be able to migrate disks back and forth from an LSI MFI controller to other random types of controller. So you're locked in to LSI if you have to use MFI, at least as far as I can tell. You also lose the ability to talk to the drive via SMART to look at its diagnostics directly.

Once you have used mfiutil (or the BIOS) to create your virtual devices on the RAID controller, they will be visible to FreeNAS and should be available in the volume manager. FreeNAS will not see MFI-attached devices that are "UNCONFIGURED GOOD", you must configure them and make them into virtual devices for the controller, at which point they'll be presented to FreeNAS. However, even once that happens, they will not show up as FreeBSD CAM devices, so if you're used to being able to do "camcontrol devlist" or other camcontrol ops, they won't be there for management in that manner.

The takeaway from all of this? Avoid MFI/MRSAS based (or other high end RAID) controllers if you plan to use ZFS. They're just about hopeless, and the inexpensive HBA models are much more appropriate and better suited to the task. The LSI HBA drivers have literally billions of aggregate problem-free runtime hours under FreeNAS and are trusted to the task.
 

9C1 Newbee

Patron
Joined
Oct 9, 2012
Messages
485
Thank You for the education. M1015 on the way.
 

mjt5282

Contributor
Joined
Mar 19, 2013
Messages
139
this tutorial was very helpful, thank you.
 

9C1 Newbee

Patron
Joined
Oct 9, 2012
Messages
485
PROBLEM: While I was flashing the M1015 to IT mode, during the process I got an error message "Failed to initialize PAL".

SOLUTION: I used a different computer to perform the flash without issue. Once the M1015 was flashed to IT mode, I was able to place the card back into the intended machine and use it with out issue.

A quick search of the web turned up the same results for others during the flash process. The error seems to manifest on certain Intel chipsets. That made sense due to the fact I was attempting to flash the controller with Supermicro X9SCM-F with an Intel chipset, which resulted in the error. I ended up successfully flashing the M1015 in a Dell inspiron. I am surprised it worked because getting %#ck^d over by Dell machines seems to be the status quo for me lately.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Good point, I had forgotten about that. There are so many vendors with so many stupid flash update requirements that we normally don't bother trying to upgrade card firmware in-place unless it is something like the HP gear which just gives you a CD image to update everything in the system automatically.

It's totally unrealistic to require Windows in order to update firmware on a network or storage controller, because you're not going to install Windows on your massive UNIX server just to update some firmware. It is a great reason to have a spare PC laying around running something like XP or Vista, with a floppy and a CDROM drive.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
Actually for LSI cards you can make a DOS bootable using some old USB thumb drive with the firmware files on it and slip that into your uber Unix box and boot it up and do the firmware work without messing up your nice unix box.

As per the readme make sure you use firmware version 13. There is a very tight relationship between the FreeBSD driver and the firmware of the card. While using a newer version of the firmware can be tempting I've found that LSI tends not to test newer firmware with FreeBSD unless they also update the FreeBSD driver also. And once they do update the FreeBSD driver you need to either compile it yourself or wait for FreeNAS to include it(at which time I'm sure they'll update the readme to list the required firmware version).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Actually for LSI cards you can make a DOS bootable using some old USB thumb drive with the firmware files on it and slip that into your uber Unix box and boot it up and do the firmware work without messing up your nice unix box.

You would think this to be the case, but some of LSI's firmware update tools don't work on some mainboards, annoyingly including some of Supermicro's. I can't remember the exact message or I'd Google it and show you the entertaining discussions of the issue, which basically have Supermicro support blaming LSI, and LSI doing the same. It was *more* entertaining because the mitigation suggested was to use the EFI based tools, which were a complete and total disaster. *My* mitigation of sticking it in an HP desktop and flashing it... once I discovered that the problem was the combination of mainboard and LSI... five minutes and worked perfectly. I long ago forgot the specifics and so when we get a batch of M1015 in, I just go dredge out the bench PC and flash away. One of the things that led to my frustrations in this thread.

It does mean I've got to actually get off my butt and lay hands on hardware, which is kind of offensive in this age of IP-KVM and IPMI/BMC.

Oops. I'm an idiot. Look right above at 9C1 Newbee's post. There's the error and a synopsis of the cause.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
I know what you mean, you need LSI's EFI tools to flash the card and some kind of EFI boot tool. In other words save yourself a ton of trouble and pull the card and put it in a junk box and flash it.

Now I've been able to do standard firmware upgrades from inside FreeNAS once, but the switch from IR to IT mode requires a DOS or EFI environment for some reason :(
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I had the "Failed to initialize PAL" error with my M1015 card. It is caused by UEFI and there is a work around. You need to boot into UEFI shell and use the sas2flash.efi utility that is included with the download pack from servethehome.com (the link is above). This article explains the process: http://brycv.com/blog/2012/flashing-it-firmware-to-lsi-sas9211-8i/

I performed the flash on my Supermicro board and it worked fine after I figured out what was going on.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
It's may be supported. If you have one already I'd try it and post back. If you don't have one I wouldn't spend money in it. Sure, it's "newer" but it won't affect the performance of ZFS....

Edit: Remember, the LSI 2008 (IBM 1015) is just under $100 on ebay, and I'm betting you won't pay that little for the 2308.
 

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
A recent post prompted me to think, there's a lot of confusion regarding LSI HBA/RAID cards. If you're confused, it is probably understandable... it can be a bit complicated!

<snip>

A favored card in the FreeNAS community is the IBM ServeRAID M1015, a budget RAID card that can be found inexpensively (~$75) on eBay. This card comes by default with IBM's version of the LSI RAID firmware on it. You don't want to run that, you instead want to crossflash it to be an LSI 9211-8i (also known as LSI SAS2008) in IT mode, making it a basic SAS/SATA HBA card. Two SFF8087 connectors provide up to 8 SAS/SATA channels directly via breakout cables, or, if a compatible SAS expander is used, possibly many more SAS/SATA channels. The card consumes around 10 watts and should have at least some airflow in order to maintain proper cooling. It is important to crossflash this card! If you do, it works under the "mps" driver, and it becomes a plain HBA. If you don't, then it is under the "mfi" driver and it is a RAID card. If you get an M1015 and crossflash it to IT mode, you end up with one of the best HBA controllers available for FreeNAS, in my opinion. These cards often come without brackets or with a low-profile bracket to fit into 2U rackmount servers. Getting a proper bracket is recommended.


Configuration of LSI 6Gb/s HBA products

This is the BIOS probe of an M1015 in IT mode with 8 drives attached via a single SFF8087 and an LSI SAS expander (Supermicro 24-drive SAS backplane). Directly attached drives look just about the same, just fewer of them:

View attachment 1987

You can hit control-C during that to get to the card configuration utility.

View attachment 1988

Shows you the adapter type and firmware revision. Note firmware version 15 and IT mode. You can look under SAS Topology at your attached disks, but really there's not a whole lot to be seen here in IT mode.

View attachment 1989

Within FreeBSD, that'll probe as a controller serviced by the mps driver, and the drives will appear as normal "daX" devices and appear in "camcontrol devlist", so pretty straightforward pleasure there. A "dmesg | grep mps" would show something like:

View attachment 1990

And so far, things like hot adding a drive seem to work fine. It has been a very pleasant and flexible controller to work with, as we use them with ESXi in -IR mode as well. I expect that other LSI SAS 6Gb/s HBA's would closely resemble the above; if you run into something different, please let me know.

<snip>

The takeaway from all of this? Try to avoid MFI based controllers if you plan to use ZFS. They're not hopeless, but the inexpensive HBA models are more appropriate and better suited to the task.

Any reason not to just buy an LSI9211-8i in the first place (at the right price) rather than cross-flashing an M1015?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
because the M1015 is usually about 1/4th the price on ebay ...?


Sent from my iPhone using Tapatalk 2
 

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
because the M1015 is usually about 1/4th the price on ebay ...?


Sent from my iPhone using Tapatalk 2
M1015 cards on eBay these days are considerably more expensive than I see mentioned in older messages in these forums, and I see LSI 9211-8i cards having sold recently for little more, and in some cases even less, than the M1015.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
So yes the going rate for the M1015 is about $90-$100, which is about $15 more than it used to go for. The eBay price on the 9211-8i appears to have dropped substantially, I don't really know what the reason for that might be. The retail price is still around $250, so the idea that you should acquire your SAS2008 HBA of whatever branding off of eBay is still valid.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I was gonna say I was just involved with the purchase of 2 M1015s and they were about 1/2 the price of the 9211-8i.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Why is it that whenever I go to buy something on eBay, I can see that some of them sold two months ago for half what people are asking for it today. Heh.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I dunno. Paid $95 each for my cards. :P
 

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
So yes the going rate for the M1015 is about $90-$100, which is about $15 more than it used to go for. The eBay price on the 9211-8i appears to have dropped substantially, I don't really know what the reason for that might be. The retail price is still around $250, so the idea that you should acquire your SAS2008 HBA of whatever branding off of eBay is still valid.
On further consideration and after further reading, I see claims that some of the "LSI"-branded boards from China are in fact fakes and are perhaps even using sub-par chips. Even some of the "Made in China" boards being sold by US-based vendors on eBay may be of that kind. So it may be better to buy an IBM-branded board and flash it to IT mode after all.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Fakes are becoming a bigger and bigger problem. I always stick with buyers that have good return policies in case they are fakes. Generally with ebay I stick to purchases from inside the USA if its an expensive purchase. You're more likely to get a refund from a scammer if they are in the USA than overseas from my experience.
 
Status
Not open for further replies.
Top