HBA Hardware issue with LSI 9400-16i

[rivimey]

I recently bought an LSI 9400-16i for my server, installed it and ran up truenas (which had been using motherboard and basic pcie sata cards) but all ok.

To my consternation it essentially failed to boot into the kernel: lots of bus resets, failure to connect to disks and general mayhem. Removing the card and reinstating the previous sata controller connections and order was restored.

The system is a P8Z68 MB with i7 4790K (not OCd), 12GB DDR3 mem, and the LSI card was in what would normally be the graphics card pcie slot. The other pcie slots had a TPLink 10/5/2.5G ethernet card and a radeon 54-series graphcs card. While the LSI card is PCIe Gen3 capable but the MB is only PCIe Gen2, I would expect it to derate gracefully to Gen2 x8... yes?

The card was bought as being 'in IT mode' and it does detect all the drives in its own bios screen, and the UEFI bios on the MB also sees them. 'lspci -vv' reports the card as I would expect and without complaints.

Is there something I've forgotten or is this a faulty card that I should talk to the seller about?

 

[PhilD13]

It should drop down to pcie2 and ithe card should be compatible. Make sure you are using a SAS to SATA Forward Breakout Cable and not a SAS to SATA Reverse Breakout Cable to connect the drives to the card. The forward and reverse cables, are not the same although they look the same. If the wrong cable is used then I would not expect things to work as they are wired differently though look the same.

If the host-controller side (HBA) has a SAS connector (SFF-8470) and the target side are just SATA drives then you must always use a SAS to SATA Forward Breakout Cable from the HBA card. If The Motherboard/host-controller side are regular SATA connectors and you have a backplane with a SAS connector that your drives are plugged into then you must always use a SAS to SATA Reverse Breakout Cable. For SATA to SATA you just use a "SATA" red cables between the motherboard and drives.

Using a GPU boost slot might not be the best idea either the motherboard may do something funky with the slot, but if the card is seen correctly then it should work.

 

[Ericloewe]

Make sure you are using a SAS to SATA Forward Breakout Cable and not a SAS to SATA Reverse Breakout Cable to connect the drives to the card

We're past that because the disks are recognized.

While the LSI card is PCIe Gen3 capable but the MB is only PCIe Gen2, I would expect it to derate gracefully to Gen2 x8... yes?

Theoretically yes, though it's not unheard of for such things to be problematic. It is rare, though.

The system is a P8Z68 MB with i7 4790K

You might want to double-check that, as the CPU does not fit in the motherboard.

To my consternation it essentially failed to boot into the kernel: lots of bus resets, failure to connect to disks and general mayhem. Removing the card and reinstating the previous sata controller connections and order was restored.

Are you trying to boot from the card? If so, avoid that if possible. In any case, you probably want to update to the latest firmware and get rid of the BIOS option ROM (keep the UEFI option ROM if you're booting from the card).

 

[Davvo]

Do I wrongly remember the need to set something in the BIOS in order to boot from HBA?

 

[rivimey]

Do I wrongly remember the need to set something in the BIOS in order to boot from HBA?

My boot drive is an SATA SSD on the motherboard controller (at first, an NVME SSD drive).
The HBA was intended to be only the 'pool' drives.

 

[rivimey]

It should drop down to pcie2 and ithe card should be compatible. Make sure you are using a SAS to SATA Forward Breakout Cable and not a SAS to SATA Reverse Breakout Cable to connect the drives to the card. The forward and reverse cables, are not the same although they look the same. If the wrong cable is used then I would not expect things to work as they are wired differently though look the same.

If the host-controller side (HBA) has a SAS connector (SFF-8470) and the target side are just SATA drives then you must always use a SAS to SATA Forward Breakout Cable from the HBA card. If The Motherboard/host-controller side are regular SATA connectors and you have a backplane with a SAS connector that your drives are plugged into then you must always use a SAS to SATA Reverse Breakout Cable. For SATA to SATA you just use a "SATA" red cables between the motherboard and drives.

Using a GPU boost slot might not be the best idea either the motherboard may do something funky with the slot, but if the card is seen correctly then it should work.

I'm sure I'm using the right cabling. Been round *that* loop before

 

[rivimey]

You might want to double-check that, as the CPU does not fit in the motherboard.

Oops... sorry that was another board I used at first. /memory!

It's an Asus Z97-AR MB.

Are you trying to boot from the card? If so, avoid that if possible. In any case, you probably want to update to the latest firmware and get rid of the BIOS option ROM (keep the UEFI option ROM if you're booting from the card).

No, I'm not.

 

[rivimey]

Using a GPU boost slot might not be the best idea either the motherboard may do something funky with the slot, but if the card is seen correctly then it should work.

Slot 1 is the only one the UEFI BIOS will give more than 4 lanes to...

(yes, am wondering about new MB but already overspent the budget on this!)

 

[Ericloewe]

Mostly sidenote, since it was never super likely: every component you listed should be PCIe 3.0, so difficulties in negotiating down shouldn't apply.

 

[Davvo]

My boot drive is an SATA SSD on the motherboard controller (at first, an NVME SSD drive).
The HBA was intended to be only the 'pool' drives.

Mmmmh, I would try reflashing it.

 

[rivimey]

What's the best site to download a new flash image?

 

[Davvo]

I guess from Broadcom's website (query). Version 24.00.00.00.
File attached for posterity.

 

[rivimey]

Thank you

 

[rivimey]

I have the image but am struggling to use it. I thought I could use the sas[23]flash programs to do so within truenas, but whatever options I use I get a message about no adapters found. I can see the adapter in an lspci listing:

$ lspci -s 1:0.0 -v 01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) (rev 01) Subsystem: Broadcom / LSI HBA 9400-16i Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 1 Memory at d0100000 (64-bit, prefetchable) [size=1M] Memory at d0000000 (64-bit, prefetchable) [size=1M] Memory at df900000 (32-bit, non-prefetchable) [size=1M] I/O ports at e000 [size=256] Expansion ROM at dfa00000 [disabled] [size=64K] Capabilities: <access denied> Kernel driver in use: mpt3sas Kernel modules: mpt3sas

but even using the '-cpci' option of sas3flash it says no avago controller found:

$ sas3flash -cpci 1:0.0 Avago Technologies SAS3 Flash Utility Version 16.00.00.00 (2017.05.02) Copyright 2008-2017 Avago Technologies. All rights reserved. No Avago SAS adapters found! Limited Command Set Available! ERROR: Command Not allowed without an adapter! ERROR: Couldn't Create Command -cpci Exiting Program. $ lspci -s 1:0.0 01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) (rev 01)

similarly the -listall command fails. For completeness I tried using sas2flash but it fails in the same way?

{edit: this is with nothing but the boot drive in use and no drives connected to the LSI controller.}

Is there something I'm missing here?

 

[Davvo]

Is there something I'm missing here?

Yup, wrong command apparently.

Reddit - Dive into anything

www.reddit.com

Broadcom Inc. | Connecting Everything

docs.broadcom.com

If you look into the readme.txt in the archive there should be instructions.

 

[Ericloewe]

Also, you don't want to update the firmware with the pool mounted. It's best to boot into the EFI shell and use the EFI version of storcli.

 

[rivimey]

I thought I could get to the EFI shell from the UEFI BIOS screen, but I can't see any way to do so... how is it done?

 

[Ericloewe]

It's not unheard of for consumer boards to not ship an EFI shell in their firmware image. You can get around this by acquiring an EFI shell binary and having the firmware boot to it on the same USB flash drive you'll use for storcli and the SAS firmware.