One of My Two SAS9220-8i Adapters Has Died

[BobCochran]

Greetings!

I have a FreeNAS system as described in my signature block below. I built the system at a time when I did not know very much about LSI products, so I installed two SAS9220-8i adapters purchased on Ebay. I believe they are rebranded as an IBM HBA.

I have not kept the overall system up to date in terms of FreeNAS version upgrades. It worked very well, and the user was very happy and did not want the system taken down. Over many months we both effectively forgot it.

This evening, one of the two adapters has died, and the system is unusable. When I boot the system, this text shows up first:

Code:

LSI CORPORATION MPT SAS2 BIOS
MPT2BIOS-7.25.00.00 (2012.02.17)
Copyright 2000-2012 LSI Corporation

Initializing.../

Then I see this message:

Code:

Adapter configuration may have changed, reconfiguration is suggested!
Searching for devices at HBA 0...

..then I'm presented with a list of devices that appear to be associated with HBA 0. The first line identifies the firmware revision I think:

Code:

LSI SAS2008-IT 13.00.01.00

Then after FreeNAS appears to boot, I get many error messages that reference 'SATA passthru failure' and other messages that read like this:

Code:

mpssas_fw_work: failed to add device with handle 0xe
mpssas_prepare_remove 476 : invalid handle 0xe
mps_config_get_sas_device_pg0: page read with error; iocstatus = 0x22
...

Question: is this sequence of messages and the error messages themselves consistent with an LSI adapter failing?

How do I fix the issue?

I do have a spare LSI9220-8i adapter. However, I must flash this into IT mode with the same firmware revision as the still-living adapter. I assume that I can do this, and just swap the new adapter (properly flashed, of course) and boot the system up and it will work as before. Does my assumption seem correct?

Another option is, I could buy a new LSI adapter entirely set entirely. I'm really out of date now on how to build a FreeNAS system in 2016. If it makes sense to buy a new LSI adapter, what should I get? Would such an adapter work with the latest release of FreeNAS?

Thanks a ton for your help!

 

[Robert Trevellyan]

I must flash this into IT mode with the same firmware revision as the still-living adapter. I assume that I can do this, and just swap the new adapter (properly flashed, of course) and boot the system up and it will work as before. Does my assumption seem correct?

The important thing is to match the firmware version you flash with the driver version in whatever version of FreeNAS is installed. Then replacing the HBA should take care of the problem.

EDIT: typo

 

[BobCochran]

The important thing is to match the firmware version you flash with the driver version in whatever version of FreeNAS is installed. Then replacing the HA should take care of the problem.

Thank you! I will work on this and report back.

 

[jgreco]

Yeah, IT firmware version 13 is probably a bad idea unless you're running a REEEEEALLLLLY old version of FreeNAS.

 

[BobCochran]

Hi!

Thank you for your help.

I'm using FreeNAS 9.2.0, should I be using firmware version P16 instead?

I think I have found what is wrong with one of my two M1015/LSI 9220-8i adapters. One cable plugged into adapter cable is frozen in place, it looks like it is fused to the port from heat.

I'm also having a problem with using the megarec utility on my spare HBA. It gets a page fault under FreeDOS 1.1 when I try to run it with 'megarec -writesbr 0 sbrempty.bin'. I'm following this post from "GeekGoneOld" which is way down near the bottom:

https://forums.freenas.org/index.ph...id-m1015-and-no-lsi-sas-adapters-found.27445/

I'm not sure what the page fault means. I haven't tried moving the adapter to the other PCIe slot on the motherboard.

Thanks

Bob

 

[jgreco]

One cable plugged into adapter cable is frozen in place, it looks like it is fused to the port from heat.

Holy mackerel.

 

[Ericloewe]

I think I have found what is wrong with one of my two M1015/LSI 9220-8i adapters. One cable plugged into adapter cable is frozen in place, it looks like it is fused to the port from heat.

Phew, that's a rather nasty failure for something that should have neither high voltages nor high currents.

 

[BobCochran]

Thanks, Eric! I don't know what to make of it, but I can't get the SFF-8087 end of the cable out of the LSI adapter. The other cable plug came off just fine. I just received new cables from an Ebay supplier, and I'll put together the system soon. I plan to order an Intel expander card (the RES2V360 model) and install it since I need to expand my drive capacity anyhow. I'll make a low-profile wooden shelf for the expander card large enough so that I can also add strain reliefs for the cables.

Bob

 

[jgreco]

Wait. You know that you need to push the connector in (towards the socket), while depressing the metal release tab on the connector, prior to pulling it out, right?

I've also occasionally had SFF8087's that needed a little coercing by prying the metal top of the socket up a little. The latch mechanism on these things can be finicky.

 

[BobCochran]

Wait. You know that you need to push the connector in (towards the socket), while depressing the metal release tab on the connector, prior to pulling it out, right?

I've also occasionally had SFF8087's that needed a little coercing by prying the metal top of the socket up a little. The latch mechanism on these things can be finicky.

I see. After twiddling with this more, and following your suggestions with the coercion, I did get the cable free from the HBA socket. I thought it had heat-fused in there somehow. Thank you!

The frustrating thing for me is that I can't seem to get the second HBA adapter recognized by the system at boot time any more. Let me relate what I imagine should happen when a FreeNAS 9.2.0 system is booted with two LSI / INM M1015 adapters plugged into it. The reality might be different, and perhaps someone will give me a reality check.

First, I imagine that the boot sequence is like this:

1. The motherboard's BIOS boots.
2. The BIOS for IBM M1015 #1 will boot, and search for devices connected to it. I indeed see this happen with HBA 0.
3. I imagine that since a second IBM M1015 adapter is plugged in, that should boot next, and likewise display the devices connected to it.
4. Then FreeNAS 9.2.0 boots up and a nice happy storage system starts.

But in my case -- all of a sudden -- step 3 does not happen any more. If it ever did happen, I mean. I never took photos of the boot process when it was healthy, and I wish I had.

So I am under the impression that the adapter is dead, and doesn't present any of its connected devices to FreeNAS. I get the error messages shown in my first quote in this thread. I have swapped 3 different M1015 adapters and installed one new SFF-8087 cable. I still get the impression that HBA 0 is able to "start" and find its devices, but the next HBA (HBA 1?) reports nothing.

Does anyone have ideas about what to do?

Thanks a ton

Bob

 

[Robert Trevellyan]

Try a different slot?

 

[Mirfster]

Just curios if they all had the SAS Address wiped or set to all 0s (zeros)?

 

[BobCochran]

Just curios if they all had the SAS Address wiped or set to all 0s (zeros)?

Thank you! I will check on that. I might have wiped out the SAS addresses when I updated the cards to the P16 firmware. I was using these incantations:

Code:

sas2flsh -o -e 6

Perhaps the above also wiped out the SAS address of the adapter? Then I would run

Code:

sas2flsh -o -f 2118it.bin -b mpt2sas.rom

Which seemed successful.

I will check on this since I am having great difficulty getting the system to boot into FreeNAS.

Thanks

Bob

 

[BobCochran]

Here is another update. Earlier, I had thought I should see one list of devices that are connected to each of my two M1015 adapters. It was as if the hard drives from one adapter were getting passed through to FreeNAS, but not the devices connected to the second adapter. A list of devices for the second adapter never even showed up. So when the first message appeared to indicate M1015 adapters are initializing, I pressed Control-C to get into the LSI Configuration Utility. I noticed that both the adapters appeared, but the top adapter is listed as a "9211-8i" device and the bottom adapter is listed as a "9210-8i" device. Also, the top adapter had a "Boot Order" coded into it, but the bottom adapter did not. Perhaps this was why I was not seeing the second HBA present its device list? So I changed the boot order so that one device was boot order 0 and the second was boot order 1.

This gave me a list of all the devices attached to both adapters, as shown below. So this means that both adapters are booting up, right?

My problem now is, the system isn't booting into FreeNAS. Once the system gets past the above screen, it tries to PXE boot, instead of boot from the USB device that contains FreeNAS 9.2.0. I don't understand why this is happening. The Supermicro motherboard BIOS does indeed have the FreeNAS USB drive set as the first boot device. But now the system does not boot from that. I thought it might be that the USB flash drive itself has gone bad, so I made up a spare flash drive with a FreeNAS 9.2.1.8 system on it. However, the system does not boot from the new flash drive, either. It is as if the USB drive is being ignored and the system tries to PXE boot. Can anyone tell me why this is happening? Perhaps my playing with the LSI Configuration Utility options was a mistake?

Thanks for any advice!

Bob

 

[Mirfster]

Thank you! I will check on that. I might have wiped out the SAS addresses when I updated the cards to the P16 firmware. I was using these incantations:

Nah, the command that wipes the SAS Address is: sas2flsh -o -e 7
You should be fine with what you ran.

But, what is the output of: sas2flsh -list

That command should show you the card's SAS Address

 

[BobCochran]

Nah, the command that wipes the SAS Address is: sas2flsh -o -e 7
You should be fine with what you ran.

But, what is the output of: sas2flsh -list

That command should show you the card's SAS Address

Thank you! I just did that, and both boards show SAS addresses that match what is printed on the label affixed to each board. Is that the answer you wanted? Did you want me to post photos showing the full output of the command for each board? I ran it on each board individually (that is, only one board was plugged into a PCIe slot and the board was not connected to any hard drives each time I ran sas2flsh.) I received a message "Adapter removed from boot order!" when I did this.

I am interested in the "NVDATA Product ID" and ""Board Name" fields shown my this command. On one of my boards, these fields both have the value "SAS9211-8i". In the second board, the "NVDATA Product ID" is "SAS9211-8i" and the "Board Name" is "SAS9210-8i". Do you think this might be part of my problem? Should both boards have exactly the same Product ID and Board Name values?

Thanks

Bob

 

[Mirfster]

Should both boards have exactly the same Product ID and Board Name values?

I would tend to think so if they are the same make/model and are flashed the same. But if they come from different manufacturers, then I can see where that would be different.

 

[Robert Trevellyan]

It is as if the USB drive is being ignored and the system tries to PXE boot. Can anyone tell me why this is happening?

On some boards you have to explicitly set the USB emulation to Hard Drive instead of Automatic. Do you have that setting anywhere in your BIOS?