Firmware version P21?

[billgreenwood]

Support at Supermicro this evening replied to my email and confirmed that P20 is the latest firmware version.

 

[QuillOmega0]

Like here?

https://forums.freenas.org/index.php?threads/9-10-nightlies-p21-lsi-firmware-warning.44775/

Forum post != Bug report

Why on earth did they create a P21 driver when there is no P21 firmware?

Can we just backlevel the driver to the P20 version instead of either (a) having mismatched driver versions (but without the warning message) or (b) reverting to the previous version of FreeNAS?

Why?

They're compatible with each other, just different versions. Makes no sense to backlevel something that's compatible when it's easier and more proper to just set an exception in the warning.

 

[BigDave]

This might be a stupid question, but I'll ask it anyway as this always seems to pop up as an issue :D

FreeNAS works with many different motherboards, processors, disk drives, etc. so why can't it accommodate a number of different SATA/SAS cards?

What's so special here?

A number of manufacturers don't produce drivers that work well (or not at all) with FreeBSD, it's the
companies that make the products/drivers that are not accommodating, not the other way around.

 

[Apollo]

If I recall, on one of a the updates, Supermicro didn't provide the driver only days or weeks after Freenas updates went through.
It might be the same here.

 

[cyberjock]

Thanks CJ!

Edit: Just as an aside Avago bought Broadcom ;) http://fortune.com/2015/05/28/avago-buys-broadcom-for-37b/

Thanks. I thought it was the other way around. I know my contact for this issue has a broadcom email, so I thought it was the other way around. :p

This might be a stupid question, but I'll ask it anyway as this always seems to pop up as an issue :D

FreeNAS works with many different motherboards, processors, disk drives, etc. so why can't it accommodate a number of different SATA/SAS cards?

What's so special here?

It does. What's special is that LSI controllers that are HBAs have a boatload of firmware versions and drivers. LSI tests only one driver and one firmware version and promises those will work, everything else is gambling. When doing HBA only stuff (which is what FreeNAS users should be doing) you can expect that if you use 2 random driver and firmware versions that don't match, you may not have problems. But you are still gambling, so we've added the warning since:

1. The vast majority of FreeNAS users use this one LSI chipset.
2. The vast majority of FreeNAS users aren't aware of this nuance to LSI.
3. Don't realize that you do run the risk of total destruction of your zpool, without warning, by doing mismatched drivers and firmwares (this really happened to some people with P16 firmware and p20 drivers).

It's not the first time though, as we've been from P14 to 16 to 20 and now 21.

What's so important to match the driver and firmware on these cards? Can't it deal with a mismatch?

Just trying to understand and gain some knowledge (as I have one on my motherboard, although all I know is I can plug some drives into the ports!)

Read above. LSI only promises that a given driver and firmware will work. If you upgrade the driver and don't upgrade the firmware, you are gambling (which is bad).

Thanks for the update. Forgive me if I'm reading that wrong, but don't you mean to say that FreeNAS is going to be changed to not throw a warning for P20 firmware with the new P21 driver?

Yes, FreeNAS will be coded to allow this one specific case.

Why on earth did they create a P21 driver when there is no P21 firmware?

Can we just backlevel the driver to the P20 version instead of either (a) having mismatched driver versions (but without the warning message) or (b) reverting to the previous version of FreeNAS?

The guy that did the P21 changes was trying to be helpful by fixing a few things. He decided that the appropriate thing would be to increase the version since he made changes (reasonable to be honest). This creates problems though. ;)

Like here?

https://forums.freenas.org/index.php?threads/9-10-nightlies-p21-lsi-firmware-warning.44775/

Yeah, nobody noticed that at iXsystems (I'm the only employee that reads and understands stuff here regularly, and I've been so busy lately I have gotten behind). A bug report should have been filed on that, but apparently wasn't. The good news is that there is no harm or foul with this. It's an anomaly that only FreeNAS users using the latest version are likely to even notice right now.

 

[c32767a]

It's not the first time though, as we've been from P14 to 16 to 20 and now 21.

What's so important to match the driver and firmware on these cards? Can't it deal with a mismatch?

Just trying to understand and gain some knowledge (as I have one on my motherboard, although all I know is I can plug some drives into the ports!)

As others have said, Avago(LSI) provides the firmware and driver to FreeBSD.

It requires a lot of extra code (read: adds complexity, reduces performance, increases cost to maintain driver) to ensure that random driver versions can talk to random firmware versions. Some manufacturers choose to omit all the guardrails and safety nets in the name of simplicity and performance. When they do that, they have to test specific pairings of driver and firmware versions and only matched sets are supported in production.

In Avago's case, that means the version numbers must match.

If you run an untested pairing, it may lead to data corruption, which ZFS may not tolerate well. Better to match the versions than roll the dice that there are no bugs or problems between disparate versions.

 

[George777]

Like here?

https://forums.freenas.org/index.php?threads/9-10-nightlies-p21-lsi-firmware-warning.44775/

This is problematic. How was this release considered stable and something this significant not detected? I think we all dodged a bullet on this one that the driver mismatch didn't create any data corruption errors, but this lack of quality control is concerning. Also that someone reported the issue and was basically questioned on why he was using the nightlies, instead of logging it as an issue to look in to.

Thank you cyberjock for the full background. I'll be waiting even longer before accepting updates from FreeNAS in the future.

 

[mrichardson03]

Additionally, there was no mention of a driver version change in the release notes.

 

[TXAG26]

Additionally, there was no mention of a driver version change in the release notes.

Hit the nail right on the head! As much as its harped on people to make sure their firmware and driver match, it's unacceptable that the driver was updated without that even being mentioned in the release notes! That should have been the first item listed in the release notes, and it should have been bold, underlined, and includes blinking lights.

 

[wblock]

Yeah, nobody noticed that at iXsystems

Actually, Dru reported this very early. But it apparently took a while to figure out what had happened with the driver.

 

[Z300M]

A number of manufacturers don't produce drivers that work well (or not at all) with FreeBSD, it's the
companies that make the products/drivers that are not accommodating, not the other way around.

But in this case there is no such thing as P21 firmware for which Broadcom/Avago/LSI could have provided an updated driver.

 

[dlavigne]

See the comments and commit in https://bugs.freenas.org/issues/16649.

 

[Casiquey2k]

I would suggest testing the performance of your FreeNAS. I am not sure if it is just me but I noticed performance issues when writing/reading files to my shares. As soon as I reverted back to the boot environment before the update, my issues seemed to disappear. Anything simple from renaming files, moving files between folders in my shares, etc. would take a few seconds, and the system would even freeze momentarily. I am no storage expert, but maybe someone who knows of more comprehensive tests can confirm if there are issues caused by the drivers mismatch?

Thanks.

 

[dlavigne]

It would be worth having a dev look at it. To do so, create a bug report at bugs.freenas.org which includes a debug file (created from System -> Advanced -> Save Debug), and post the issue number here.

 

[Casiquey2k]

It took me a while, but here it is Bug #16669. I am sure they will let me know if more info is needed.

 

[monovitae]

I've read all the way through this thread and am unsure of what to do. So whats the tl;dr here? Is it ok to stay on the latest update or should the prudent man revert to the previous boot environment for a while?

 

[bestboy]

I'm going back. If the developer of the driver just made some minor fixes, then he should have made a point release and leave the major version alone. Considering he did a major version update I suspect profound changes in the driver. And I'm a bit reluctant to trust that this new version is compatible. I mean, by whom were these changes even tested, when the Avago test team is only testing - and thus guaranteeing - for matching firmware/driver combinations?

 

[a-freenas-user]

The guy that did the P21 changes was trying to be helpful by fixing a few things. He decided that the appropriate thing would be to increase the version since he made changes (reasonable to be honest).

Perhaps the new owners will take notice of the fact that not even their own Dev's can understand their driver/firmware numbering system nevermind us mere end-users...

 

[TXAG26]

Would it be that difficult to maintain the last couple of driver versions in FreeNAS so that FreeNAS can pick and use the correct driver version based upon the HBA firmware it sees? It seems like this would be a reasonable item to implement if a firmware/driver mismatch really can bork up a ZFS pool beyond repair.

 

[jpaetzel]

The issue here is complex, but I'll try to distill it down.

Broadcom now owns Avago, who bought LSI. This has resulted in some changes on their end, one of which being they now have a developer who is also a FreeBSD committer. He was able to commit some changes to the mps driver directly to FreeBSD. He bumped the mps driver version during one of his commits. I saw this change, and assumed that there would be a firmware to match this driver getting released from Broadcom (just like there has been every time there's been a driver bump for the last 6 years). Well, it turns out, the process has changed, and now Broadcom is saying that the version 21 driver is compatible with the phase 20 firmware, which is a break from the way things have been for the last several years. Furthermore they have no plans to release a phase 21 firmware. Of course I didn't find this out until *after* the release but that's more of a timing thing. My contact at Broadcom (who I've been working with since she was at 3Ware in 2009) was on vacation, and I had no reason to suspect anything was changing, and why pull the firehandle over this when it would've delayed several real fixes from getting out to people....

As far as firmware mismatches causing data corruption on ZFS pools, we've never had anything but apocryphal evidence that that can even happen. The bugs I have seen that I have hard data on have all been things like "drive 0 doesn't show up" or "when a drive emits this specific error code XXXX, then the controller infinitely retries an error that was reported as a fatal error that shouldn't be retried and it hangs the controller at boot"

The official word from Broadcom/Avagao/LSI is "They test the firmware and driver in matched sets, so by using a matched firmware and driver you are using a tested combination" It's possible the current situation of v21 driver and p20 firmware has not gone through their internal regression test suite. I'm working on verifying that at this point.

It's acceptable to silence the alert for now, the next FreeNAS update won't consider the p20/v21 pair alert worthy.