Don't see drive status LED after connecting it HP H220 HBA

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
Hi

I recently added an HP H220 HBA card to my DL380 G8 server to replace the i420p I had on the motherboard.

I updated my firmware to version 20 using the 2308T207.ROM file (IT mode).

I noticed that the status LED indicators on my drives are not working anymore.

I tried using the sas2ircu command but while the command was successful, the LED light didn't turn on

Code:
sas2ircu 0 locate 1:30 on wait 10
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

SAS2IRCU: LOCATE command completed successfully.
SAS2IRCU: Command LOCATE Completed Successfully.
SAS2IRCU: Utility Completed Successfully.


Here is the output of the display parameter:

Code:
root@freenas[~]# sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS2308_1
  BIOS version                            : 7.39.02.00
  Firmware version                        : 20.00.07.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 255
  Concurrent commands supported           : 8192
  Slot                                    : 2
  Segment                                 : 0
  Bus                                     : 7
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 28
  SAS Address                             : 5000c50-0-5760-fbed
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0023
  Firmware Revision                       : 0006
  Serial No                               : Z1X10PG3
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 28
  SAS Address                             : 5000c50-0-573e-7881
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0001
  Firmware Revision                       : 0002
  Serial No                               : Z1P66CMH
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 29
  SAS Address                             : 5000c50-0-412a-3f41
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0001
  Firmware Revision                       : 0001
  Serial No                               : Z1P1L05R
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 29
  SAS Address                             : 5000c50-0-558a-70dd
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0001
  Firmware Revision                       : 0002
  Serial No                               : Z1P40ZRT
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 30
  SAS Address                             : 5000c50-0-4124-7891
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0001
  Firmware Revision                       : 0001
  Serial No                               : Z1P1GZL0
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 30
  SAS Address                             : 5000c50-0-5760-a8c1
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0023
  Firmware Revision                       : 0006
  Serial No                               : Z1X12ZBX
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 31
  SAS Address                             : 5000c50-0-5760-ce9d
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0023
  Firmware Revision                       : 0006
  Serial No                               : Z1X129ZF
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 31
  SAS Address                             : 5000c50-0-412a-2b65
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : SEAGATE
  Model Number                            : ST2000NM0001
  Firmware Revision                       : 0001
  Serial No                               : Z1P1KZD8
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 50030480:00000000
  Numslots                                : 8
  StartSlot                               : 28
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.


I would love to have an idea which drive is which if something goes wrong.

Here are more details on the system:
  • HP Proliant 665553-B21 DL380p Gen8
  • 2 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
  • 128GB RAM
  • 8 x Segate ST2000NM0001 2TB SAS 3.5inc drivers
  • HP H220 w/firmware v/20 IT mode
  • 1 x Samsung SSD 850 EVO 500GB for log drive
  • 1 x PC401 NVMe SK hynix 512GB for cache
Thanks in advance

Itamar
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
It seems to think there are two drives per slot. Wouldn't you expect each drive to show up in its own enclosure slot?

Is there firmware for that back plane? Maybe that'll make it "play nice" with the P20 firmware of the HBA.
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
It seems to think there are two drives per slot. Wouldn't you expect each drive to show up in its own enclosure slot?

Is there firmware for that back plane? Maybe that'll make it "play nice" with the P20 firmware of the HBA.

Hi

Yes, I would have expected it to show the 8 drivers.

I was looking online and there several available F/W to download, none of them describes this issue.

The drive status was working when i was connecting them to my on-board p420i controller

I will keep on playing with it, it might be a BIOS settings on the controller

thanks

Itamar
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
I expect it's how the SAS extender represents the drives, not the HBA itself. And, I don't have one of those to play with.
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
I expect it's how the SAS extender represents the drives, not the HBA itself. And, I don't have one of those to play with.

Ok, I will need to re-firmware my solution, the BIOS is messed up.

1588953753597.png


Do you recommend using version 19 instead of 20?

On version 19, during POST, the controller init shows all drivers while version 20, shows only one

1588953967386.png


Thanks

Itamar
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
I have no experience with that, I'm using a SAS2008 variant cross-flashed to 9211-8i. There's a mention over there that there are different steppings of the H220, some are actually 9207-8i. Maybe start with identifying the stepping and making sure you are matching board to firmware. See https://www.ixsystems.com/community/threads/cross-flash-hp-h220-to-lsi.67775/

So, after some research and talking with #theartofserver. We figured out that the LED display is related to HP propriety code.

I have reverted the firmware back to version 15 (HP Stock IT one) and it is working now.

From performance perspective, I actually see an improvement in performance when I run the the stock F/W one.

Code:
root@FR4G[~]# dd if=/dev/zero of=/mnt/FR4G/testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 16.634269 secs (2521483847 bytes/sec)

root@FR4G[~]# dd of=/dev/zero if=/mnt/FR4G/testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 6.157737 secs (6811437314 bytes/sec)


Before that, the read performance was around 500MBs and the write was less than 200MBs

Now, the real question is this, what is the impact of running the stock F/W vs the v20 one?

Thanks in advance
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Now, the real question is this, what is the impact of running the stock F/W vs the v20 one?

Ranging from nothing up to total failure. The driver and firmware are designed to work together, and running older or newer firmware than what the driver is expecting can be problematic, because they work as a team, passing messages back and forth between the driver and the controller. The firmware is the controller half.

It is not supported to run firmware other than IT 20.00.07.00. Doing so may not necessarily kill you, but there is definitely risk, things may not work right, especially unusual conditions such as disk errors or other exceptional conditions that are virtually impossible to test. It is absolutely known that even 20.00.04.00 has some issues. It would be a better idea to see if you can update the firmware on the expander and see if it works better with IT fw 20.00.07.00. It doesn't have to "mention this issue" because the issue you're seeing may only be a symptom rather than the underlying problem.
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
Ranging from nothing up to total failure. The driver and firmware are designed to work together, and running older or newer firmware than what the driver is expecting can be problematic, because they work as a team, passing messages back and forth between the driver and the controller. The firmware is the controller half.

It is not supported to run firmware other than IT 20.00.07.00. Doing so may not necessarily kill you, but there is definitely risk, things may not work right, especially unusual conditions such as disk errors or other exceptional conditions that are virtually impossible to test. It is absolutely known that even 20.00.04.00 has some issues. It would be a better idea to see if you can update the firmware on the expander and see if it works better with IT fw 20.00.07.00. It doesn't have to "mention this issue" because the issue you're seeing may only be a symptom rather than the underlying problem.

Appreciate the input. I have upgraded the F/W back 20.00.07.00 and the performance is looking good.

I think reverting back to the HP stock F/W and then just updating the 2308 F/W may have also "fixed" whatever mismatched F/W and BIOS combination I had. Performance looks good now.

I will keep on looking for a way to have the disk LED light up, at least on command.

Thanks

Itamar
 

NickF

Guru
Joined
Jun 12, 2014
Messages
760
I have the same issue with a number of HP/Lefthand shelves which we repurposed with FreeNAS and HP H220/IT Mode, and I was never able to make the drive indicator lights work either. I also believe it has to do with the SAS Expander and how it communicated with the "stock" P410's.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
How are those SAS expanders powered, screwed in physically, and connected to the cages?

You know where I am going: It might be so proprietary there’s no hope, and: Maybe a standard SAS expander would fit, to replace the custom HP one?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
How are those SAS expanders powered, screwed in physically, and connected to the cages?

You know where I am going: It might be so proprietary there’s no hope, and: Maybe a standard SAS expander would fit, to replace the custom HP one?

Because an SAS expander is just a chip, typically they are powered by and soldered onto the backplane. It is the easiest and most reliable method.
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
Because an SAS expander is just a chip, typically they are powered by and soldered onto the backplane. It is the easiest and most reliable method.

So I tried to see if there is a F/W update for my backplane.

Found this in HP support site:

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX-cf50ab7b3cfe47888831b05c06

Tried to run it but it claims that there is no supported device.

So I am trying to thing about this a bit holistically.

Do we know if HP has any other 9211-8i based controllers out there that may have F/W updates to version 20?

Thanks

Itamar
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
How are those SAS expanders powered, screwed in physically, and connected to the cages?

You know where I am going: It might be so proprietary there’s no hope, and: Maybe a standard SAS expander would fit, to replace the custom HP one?

Do you have a SAS extender in mind for me to test?

Thanks

Itamar
 

bawalker

Cadet
Joined
Sep 11, 2022
Messages
3
Do you have a SAS extender in mind for me to test?

Thanks

Itamar
Was there ever any resolution to the hard drive indicator lights you were experimenting with the P20 firmware? If not, what was your process for downgrading back to the HPE H220 firmware? I have the exact same setup, but I won't be using TrueNas on it, instead I'll be using ProxMox, but wanted to fix the hard drive indicator issue regardless.

Thanks!
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
Was there ever any resolution to the hard drive indicator lights you were experimenting with the P20 firmware? If not, what was your process for downgrading back to the HPE H220 firmware? I have the exact same setup, but I won't be using TrueNas on it, instead I'll be using ProxMox, but wanted to fix the hard drive indicator issue regardless.

Thanks!
Nope, It never worked for me. It was an HP thing, on my other Dell server, no problem at all.
 

bawalker

Cadet
Joined
Sep 11, 2022
Messages
3
Nope, It never worked for me. It was an HP thing, on my other Dell server, no problem at all.
So even reverting back to OEM HP firmware on the H220 still prevented the LED drive indicator lights from working? Thanks for sharing your input as I was starting to pull my hair out on this ordeal.
 

hysel

Explorer
Joined
Apr 11, 2020
Messages
69
So even reverting back to OEM HP firmware on the H220 still prevented the LED drive indicator lights from working? Thanks for sharing your input as I was starting to pull my hair out on this ordeal.
That is weird. Did you check your backplane because that is what ultimately controls the LED indicators?
Also, when you reverted to the HP OEM Firmware, did you revert the H220 BIOS to the HP one?
 

bawalker

Cadet
Joined
Sep 11, 2022
Messages
3
That is weird. Did you check your backplane because that is what ultimately controls the LED indicators?
Also, when you reverted to the HP OEM Firmware, did you revert the H220 BIOS to the HP one?
I haven't reverted to back to the OEM firmware yet. I haven't been able to find the correct firmware/BIOS and flashing instructions for downgrading. I wasn't sure what your process was for when you did that previously.
 
Top