CAM hate for MRSAS?

[jenksdrummer]

I have a Supermicro box with a 3108 controller; which, granted, *right now* I am using in full RAID mode (60, to be exact) with my storage pool consisting of da0 (plus a pair of mirrored meta vdev, mirrored nvme dedup vdev, and a nvme cache vdev...

But seems when I put it in JBOD mode, boot up still includes a solid minute of "Root mount waiting for: CAM" messages; until it times out and then flies through the rest of booting up. Again, same if running JBOD or RAID mode with a 3108. Seems Odd. I realize CAM wants more than what the RAID controller will allow; but in JBOD mode, it presents the disks and all the smart data much the same way as my 3008-IR controller does with no configuration (though, the 3108 seems to be faster in JBOD mode than the 3108-IR; and it's really fast in RAID mode...)

 

[sretalla]

You need to read this: https://www.ixsystems.com/community...s-and-why-cant-i-use-a-raid-controller.81931/

Then flash the card to IT mode assuming it is capable if you want good results with FreeNAS moving forward.

 

[jgreco]

I have a Supermicro box with a 3108 controller; which, granted, *right now* I am using in full RAID mode

Aside from getting a hacksaw and cutting it in half, which might give you a "half RAID mode", the 3108 is a RAID controller and is not expected to work well for all the reasons listed in @sretalla 's linked post.

"The disk shows up" is not the measure of compatibility with FreeNAS. The MRSAS and its predecessor MFI driver are known to be problematic under significant load and under adverse conditions. It's not a good thing to try to use.

 

[jenksdrummer]

You need to read this: https://www.ixsystems.com/community...s-and-why-cant-i-use-a-raid-controller.81931/

Then flash the card to IT mode assuming it is capable if you want good results with FreeNAS moving forward.

OK, so let me clarify my use case. LAB. Also, the 3108 is a RAID card, there is no "IT mode"...it has dedicated hardware for parity calculations and built in 2GB cache; unlike the 3008 which can be cross-flashed IR or IT; with IR being able to do non-parity RAID levels and limits with the number of disks in the array.

I have a 3008-IR onboard with this system, but flashing it to IT-mode, per Supermicro, voids my warranty which is still for another 2 years. Not happening. I also bought the box with a 3108 controller as at the time I was debating between dropping this as an all-in-one virtualization box with local storage (VMWare loves RAID Controllers) or going to FreeNAS as a SAN solution w/ separate compute nodes. As it turned out, I went with separate compute nodes. Again - LAB environment. Hate the idea of the 3108 sitting there doing nothing because it does RAID better than FreeNAS with the 3008-IR or the 3108 in JBOD. As in, significantly better.

To that, I've ran 11 just fine w/ 3108 in JBOD mode. Did some testing comparing JBOD vs RAID back then, and no real difference noted; so went JBOD. Later, moved over to the 3008-IR as the disks are presented raw to the OS and pulled the 3108 thinking to sell it, but it's a darn good card! Again, seemed fine. But, I really want to use that 3108...I paid for it! 3rd time...LAB!

Post was more or less aimed at why is MRSAS getting this kind of hatred from FreeNAS/CAM and delays the boot by about 2 minutes.

 

[jgreco]

OK, so let me clarify my use case. LAB. Also, the 3108 is a RAID card, there is no "IT mode"

So let me clarify the answer. This is an irrelevancy, as is the remainder of your message. I would love to be able to run FreeNAS on a Raspberry Pi, yet I cannot. It is not suitable. No amount of grinching about it here is going to make that work.

The solution is to buy a compatible card, or to crossflash the card that exists on your mainboard.

If you do not like the way the MRSAS driver is working, the author's e-mail address is at the bottom of the manual page. To save you a few moments, I will make note that the driver was authored by LSI by an LSI employee, who might very well have moved on, but I don't really know. LSI got sold and then sold again, so I am not expecting to see large improvements in old product lines anymore.

I have a 3008-IR onboard with this system, but flashing it to IT-mode, per Supermicro, voids my warranty which is still for another 2 years.

That seems like a clear violation of Magnuson-Moss, but I understand your reluctance.

 

[jenksdrummer]

That seems like a clear violation of Magnuson-Moss, but I understand your reluctance.

OK, let's fix this one....Magnuson-Moss act gets thrown around a LOT by people who do not understand it...

Magnuson-Moss does nothing FOR you without a lawyer. You can't call up Supermicro and say "I flashed this" and they say "warranty denied"; and you say "but...but...but...MM Act!" and they bend over and hand you a new motherboard; no, they'll laugh you off the phone, file away the phone recording for the potential lawsuit.

You have to file a lawsuit ($$$). You have to prove that your actions did not cause the failure vs their certified engineers saying what you did in fact did cause the issue. Oh, and you'll have a counter-suit pending that outcome that covers their lawyers (likely $$$$$) and their engineer's time ($$$$) vs spending the money ($$$) for a replacement board.

Lets use a car as an example. If you flash the ECU to change the fuel mix and timing to be more aggressive, where you need to use higher octane fuel or even have to add octane to your fuel, and your SO drives the car and fills it up with normal unleaded gas and forgets, and and the engine is toast from detonation. Magnuson-Moss won't do squat. Change it up; instead of the engine, the clutch burns up. Nope. Dropped a valve spring. Nope. Differential failed; they could claim it was abuse as you clearly modified the engine. MM nope...basically unless it's not drivetrain or suspension related, you'll have to fight it out in court and prove they were wrong. Lets also forget for a moment that a warranty is not a guarantee...it's an insurance policy these days. What do insurance company excel at? Not paying out. That's literally the way they survive and are profitable.

What Magnuson-Moss does allow you to do, is to not void the warranty by using oil for the engine supplied by a brand that is not from the manufacturer, as long as it meets all the certified requirements. This parallels being able to install any brand of RAM that meets the specification, or attach any hard drive; or any RAID controller...but flashing the firmware? That's a modification. MMA does not protect you from that.

 

[jenksdrummer]

If you do not like the way the MRSAS driver is working, the author's e-mail address is at the bottom of the manual page. To save you a few moments, I will make note that the driver was authored by LSI by an LSI employee, who might very well have moved on, but I don't really know. LSI got sold and then sold again, so I am not expecting to see large improvements in old product lines anymore.

So, it's interesting that the MRSAS gets thrown under the bus with FreeNAS (curious if this is FN specific, or FBSD) as it seems it works fine under other operating systems? 3108 is still a current production controller with current drivers being produced, so I'm still at a bit of a loss as to how/why FreeNAS (or FreeBSD) either hasn't or won't bother tackling it. Use cases can and likely do exist, performance numbers I can show as well, where using a hardware RAID card with ZFS as a file system and volume management can make sense; combine that with FreeNAS's replication/UI...not sure I understand the logic.

As for failures under load, I have slapped the crap out of my system and nothing. Not saying it's not possible, but, perhaps since all that came about, those articles written, is it remotely possible that the MRSAS driver got updated and is now stable? Just the FreeNAS community beats the drum over and over about how you MUST USE AN HBA...and ONLY THESE HBAs...and ONLY THIS FIRMWARE....

Not that long ago, there's a thread about iX including a firmware update to HBAs as part of the 12 package. I'm not keen on allowing anyone to update firmware on something they don't own, manage, maintain, or lease...without notification and an opt-out default...anyhow.

 

[jgreco]

What Magnuson-Moss does allow you to do, is to not void the warranty by using oil for the engine supplied by a brand that is not from the manufacturer, as long as it meets all the certified requirements. This parallels being able to install any brand of RAM that meets the specification, or attach any hard drive; or any RAID controller...but flashing the firmware? That's a modification. MMA does not protect you from that.

It's the same thing. It's a computer. You can use a different version of the software on it. It's a reversible change. People have sent in their boards with crossflashed firmware and Supermicro fixes them. They just don't want the headache of supporting random firmware.

 

[jenksdrummer]

It's the same thing. It's a computer. You can use a different version of the software on it. It's a reversible change. People have sent in their boards with crossflashed firmware and Supermicro fixes them. They just don't want the headache of supporting random firmware.

I don't agree; a modification is not protected under MMA. Flashing firmware is a modification.

SM doing it under a warranty claim; what are the odds that the submitting party disclosed that information beforehand, and SM resolved it without considering it as a charity; vs it wasn't disclosed and SM did it without knowing?

 

[jgreco]

So, it's interesting that the MRSAS gets thrown under the bus with FreeNAS (curious if this is FN specific, or FBSD) as it seems it works fine under other operating systems? 3108 is still a current production controller with current drivers being produced, so I'm still at a bit of a loss as to how/why FreeNAS (or FreeBSD) either hasn't or won't bother tackling it.

It's fantastic that you expect someone else to do significant and complicated technical work that probably involves needing LSI internal documentation about their proprietary firmware, for free.

Use cases can and likely do exist, performance numbers I can show as well, where using a hardware RAID card with ZFS as a file system and volume management can make sense; combine that with FreeNAS's replication/UI...not sure I understand the logic.

As for failures under load, I have slapped the crap out of my system and nothing. Not saying it's not possible, but, perhaps since all that came about, those articles written, is it remotely possible that the MRSAS driver got updated and is now stable? Just the FreeNAS community beats the drum over and over about how you MUST USE AN HBA...and ONLY THESE HBAs...and ONLY THIS FIRMWARE....

Because every time someone thinks they know better, it eventually turns into a ****show. If you choose to use hardware that isn't recommended, the least you can do is to own that choice and take responsibility for your actions.

But what has generally happened is that something goes awry one day, a drive drops off the controller or performance sucks during a scrub or other "strange" things, and people are unwilling to own their decisions. So there are things we know work swimmingly well, and we definitely do encourage you to use these things, because when you don't, and you have problems, you are basically wasting the time of a bunch of well-intentioned community members who would like to help you.

Not that long ago, there's a thread about iX including a firmware update to HBAs as part of the 12 package. I'm not keen on allowing anyone to update firmware on something they don't own, manage, maintain, or lease...without notification and an opt-out default...anyhow.

I agree with that.

 

[jgreco]

I don't agree; a modification is not protected under MMA. Flashing firmware is a modification.

No, getting my SMD soldering station out and removing the ROM is a modification. Software is something that is not a physical change to the system.

 

[jenksdrummer]

I agree with that.

At least there's that. :)



Never once did I say I wasn't willing to own being outside of the recommended topology (mentioned lab 3 times); just surprised that, specifically, MRSAS seems to be getting ignored and or the hate level is increased between 11 and 12.

Relating to software is something...

One would A) have to prove that to jurors and a judge vs a slick legal team who would dismiss your credibility vs their own engineers. MMA won't pay for legal fees.

To also place a point, flashing the ECU in your car will, and does often, void warranties. Check out any "tuner" or performance car forum. Many vehicles these days have a flash-counter to them; if you take it to a dealer for any kind of service, the dealer checks against the VIN, if it shows the vehicle has been updated X by them, and your vehicle has been updated X+N(some arbitrary threshold) they figure you flashed it then re-flashed it before bringing it in for the issue (or flashed it and left it...); if it's in any way related to the issue...warranty denied; pay the service fee to find out and here's your keys.

 

[jgreco]

Never once did I say I wasn't willing to own being outside of the recommended topology (mentioned lab 3 times); just surprised that, specifically, MRSAS seems to be getting ignored and or the hate level is increased between 11 and 12.

Well you can mention lab 3 times or click your heels 3 times. It probably won't help. Sorry.

iXsystems has no reason to go paying a developer to fix a driver they don't use. iXsystems is in this to be able to fund the development of their enterprise NAS product. FreeNAS is essentially the same codebase with some goodies stripped out. They're a small company offering a premium product in a tight marketplace.

LSI is the author of their own driver, so if you want to lay "getting ignored," then by all means, they're the ones who sold the hardware and wrote the driver. I *guarantee* you that we're not aware of anyone from LSI ever having shown up here in the forums.

If you want someone *else* to fix the driver, for free, bear in mind that this involves them buying a piece of hardware that is relatively expensive, and most of the driver hackers already have plenty of totally unsupported hardware they would like to support. Getting into NDA issues to get ahold of LSI's internal firmware documentation is going to be a pain too.

The hate level is increased between 11 and 12 because I have been on the front lines here for nearly a decade, explaining to people why it is that their PC-grade hardware that only barely works with Windows isn't going to make a good FreeNAS. We've had people come in with Adaptecs and LSIs and PERCs and HighPoints and CISS-based controllers are are all "works with FreeBSD!" but after various misadventures it generally becomes clear that the drivers are not up to snuff. I can't (or, rather, won't, because I don't hack on device drivers anymore) make your random stuff work. There is a golden path to success with FreeNAS and it's easier to tell you what that is and ask you to follow it.

You don't have to like that, and if you think you can do a better job, you are welcome to contribute driver patches or hop on the high-volume contributor list or remediate whatever it is you think is coming up short.

Relating to software is something...

One would A) have to prove that to jurors and a judge vs a slick legal team who would dismiss your credibility vs their own engineers. MMA won't pay for legal fees.

To also place a point, flashing the ECU in your car will, and does often, void warranties. Check out any "tuner" or performance car forum. Many vehicles these days have a flash-counter to them; if you take it to a dealer for any kind of service, the dealer checks against the VIN, if it shows the vehicle has been updated X by them, and your vehicle has been updated X+N(some arbitrary threshold) they figure you flashed it then re-flashed it before bringing it in for the issue (or flashed it and left it...); if it's in any way related to the issue...warranty denied; pay the service fee to find out and here's your keys.

Well, speaking as the owner of a company that has professionally designed and sold servers on a commercial basis, and that includes warranties, I suspect I've spent more money on lawyers to provide me with professional opinions on these topics. Sorry, I'm working off of professional opinions and not anecdotal evidence here. I fully understand the difference between claiming something isn't covered under warranty and being able to void a warranty on the basis of something a customer did.

 

[jenksdrummer]

So is this driver a FreeBSD issue or is it FreeNAS? (guess I could load up FreeBSD and find out...)

To that, I've ran Linux distributions on it using the 3108, and the only thing that keeps me coming back to FreeNAS is the ease of the UI to do otherwise CLI items that I'd rather not spend a lot of time learning (since it's my LAB...rather do LABby stuff...lol). Plus my critical data is on another Supermicro box (gasp, using Intel SATA-mode); that, and I've not yet figured out ways to optimize it as an iSCSI host to get the same performance levels. Yet.

I hope you never need to pull the MMA card or get it pulled on you. From what I've seen it likely never will happen because the cost of either side is easier to displace without going to court; but odds are in the corporate fence; most always is.

 

[jgreco]

So is this driver a FreeBSD issue or is it FreeNAS? (guess I could load up FreeBSD and find out...)

It's hard to know. You're coming into a community of enthusiasts, most of whom are hobbyists. They aren't likely to have a $500 RAID card around just for random FreeNAS compatibility testing when it's already been established that it's a generally bad card for the purpose. Even those of us who do this professionally, what we have in inventory here is mostly 6Gbps stuff. If you're not doing SAS 12Gbps storage, the only thing the 3108 has going for it is the extra cache.

To that, I've ran Linux distributions on it using the 3108, and the only thing that keeps me coming back to FreeNAS is the ease of the UI to do otherwise CLI items that I'd rather not spend a lot of time learning (since it's my LAB...rather do LABby stuff...lol). Plus my critical data is on another Supermicro box (gasp, using Intel SATA-mode); that, and I've not yet figured out ways to optimize it as an iSCSI host to get the same performance levels. Yet.

Intel AHCI SATA is probably the *best* way to attach disks for FreeNAS, simply because you avoid the overhead of having the RAID-on-Chip processor proxying commands. The IT firmware doesn't cut the HBA CPU out of the loop, it just makes it into a minor speed bump.

Optimizing FreeNAS for iSCSI isn't that hard. https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/

I hope you never need to pull the MMA card or get it pulled on you. From what I've seen it likely never will happen because the cost of either side is easier to displace without going to court; but odds are in the corporate fence; most always is.

Well the corporate lawyers say that "warranty void if broken" stickers don't relieve one of warranty obligations if the sticker is broken, but it's not illegal to apply them. Applying them is still a "thing" at some computer stores because so many people obediently believe stuff like that without question, and trying to "win" at that is a lose for all involved.