TrueNAS only seeing one disk per LSI slot

[CJRoss]

I've had TrueNAS Core machine running for many years now. I've upgraded it to the current version. I've been having some data issues whenever the pool is under heavy load but it's been localized to the LSI HBA drives so I think my 2008 is starting to go south.

I picked up a 2308 and flashed it to the latest IT firmware (20.00.07.00-IT) on a different machine. Then I swapped out the 2008 for the 2308.

When log into TrueNAS Core, my pool doesn't show up and looking at the disk screen shows that I'm missing 4 of the 8 drives on the LSI. When I go into the LSI config utility, I see all 8 drives. Comparing the serials with the SAS Topology screen shows me that the controller has two drives on each slot and TureNAS is only seeing one. It lists Slots 28-31, each with two drives. camcontrol devlist shows the same drives as the disks menu.

The only thing that has changed is replacing the 2008 with the 2308 so I'm assuming I have a setting set incorrectly in the config, but I can't seem to find it. I've reset all of the LSI configs to default.

Any suggestions?

 

[jgreco]

What do you mean by "each slot"? What is your setup? Are these SAS drives (which would have a primary and secondary path)? Is there an SAS expander involved?

 

[CJRoss]

What do you mean by "each slot"?

When I go into the LSI config utility and look at the SAS Topology, to the left of each drive it lists a Slot #. There is no header above the Slot field. It looks like this.
Slot 28 Drive 1 SATA Slot 28 Drive 2 SATA Slot 29 Drive 3 SATA Slot 29 Drive 4 SATA Slot 30 Drive 5 SATA Slot 30 Drive 6 SATA Slot 31 Drive 7 SATA Slot 31 Drive 8 SATA

TrueNAS is seeing drives 2, 4, 6, and 8.

What is your setup? Are these SAS drives (which would have a primary and secondary path)? Is there an SAS expander involved?

I have the one LSI controller breaking out each SAS port to 4 SATA ports. No expander and all SATA drives.

 

[jgreco]

When I go into the LSI config utility and look at the SAS Topology, to the left of each drive it lists a Slot #. There is no header above the Slot field. It looks like this.

This stuff always happens when I don't have any test gear handy. That topology description feels wrong to me; when using the directly connected SAS lanes I'm used to seeing them numbered as "Slot" 0 thru 3 and then 4 thru 7. The middle column should have a heading of "Device Identifier"; your devices really identify themselves as "Drive 1" etc?

I have the one LSI controller breaking out each SAS port to 4 SATA ports. No expander and all SATA drives.

Just a nitpick, what you're referring as a "SAS port" isn't. A SAS port looks just like a SATA port. The SFF-8087 connector is a four lane connector that carries four SAS lanes. That breakout cable you have breaks out a SFF-8087 to four individual SAS connectors, which can be used to connect to various kinds of backplanes such as Supermicro TQ or the two-drive 2.5" gizmos. Important to understand what's actually going on; SAS includes features such as dualporting that allow a primary and secondary channel to each drive, for example.

 

[CJRoss]

This stuff always happens when I don't have any test gear handy. That topology description feels wrong to me; when using the directly connected SAS lanes I'm used to seeing them numbered as "Slot" 0 thru 3 and then 4 thru 7.

I don't recall how they were listed previously on the 2008 as I haven't touched it in years.

The middle column should have a heading of "Device Identifier"; your devices really identify themselves as "Drive 1" etc?

No, they show the appropriate drive models and if I go into the device info I can see all of the serial numbers. That's how I was able to verify that it's only seeing one drive per slot.

Just a nitpick, what you're referring as a "SAS port" isn't. A SAS port looks just like a SATA port. The SFF-8087 connector is a four lane connector that carries four SAS lanes. That breakout cable you have breaks out a SFF-8087 to four individual SAS connectors, which can be used to connect to various kinds of backplanes such as Supermicro TQ or the two-drive 2.5" gizmos. Important to understand what's actually going on; SAS includes features such as dualporting that allow a primary and secondary channel to each drive, for example.

You're correct. I forgot what the SFF-8087 was called and didn't bother to look it up. I am connecting to the drives through a SuperChassis 743T-665B backplane but that's just a straight passthrough AFAIK.

The only times I've had drives not show up previously are when I managed to not fully insert the caddy, but in those cases the drive wouldn't show up in the BIOS or HBA.

 

[jgreco]

Ahh, so that'll be either a CSE-SATA-743 or SAS743TQ backplane in there, but yeah that looks like mostly a bog standard TQ board with SES2 enclosure management by the classic MG9072 stuff Supermicro loves. I was hopeful for a moment that the introduction of a backplane would provide a eureka moment... but really this may only explain why it is assigning higher slot numbers to the drives.

The manual for that backplane is at

https://www.supermicro.com/manuals/other/BPN-SAS-743TQ.pdf

but as of right now I'm short on ideas. It feels like some configuration issue, and you should probably make sure that the HBA is reset to factory.

After that, it might be curious to see if you could directly cable one of the drives and see what "Slot" number it comes up with. This feels a bit like something (SES2?) has assigned something weird and while I can imagine several different possibilities, I am having trouble coming up with reasonable explanations for how such states would have come to be.

 

[CJRoss]

Ahh, so that'll be either a CSE-SATA-743 or SAS743TQ backplane in there, but yeah that looks like mostly a bog standard TQ board with SES2 enclosure management by the classic MG9072 stuff Supermicro loves. I was hopeful for a moment that the introduction of a backplane would provide a eureka moment... but really this may only explain why it is assigning higher slot numbers to the drives.

The manual for that backplane is at

https://www.supermicro.com/manuals/other/BPN-SAS-743TQ.pdf

but as of right now I'm short on ideas. It feels like some configuration issue, and you should probably make sure that the HBA is reset to factory.

I think it's a config issue as well, since all I've done is replace the 2008 with a 2308. I did a reset defaults in all of the menus but that didn't help. What do you mean by reset to factory?

After that, it might be curious to see if you could directly cable one of the drives and see what "Slot" number it comes up with. This feels a bit like something (SES2?) has assigned something weird and while I can imagine several different possibilities, I am having trouble coming up with reasonable explanations for how such states would have come to be.

I'll take a look, but I'll probably try swapping in the other 2308 I have first and seeing what that does. If it doesn't fix the issue I'll pop the original 2008 in and see what it says for slot numbers.

 

[CJRoss]

I swapped in the other 2308 I had and now TrueNAS is seeing all 8 of the drives. Looking through the LSI config menus everything appears to be the same as the previous 2308. It lists the same slot numbers and config.

The only two things I tell are that the LSI config version is different and it gave me a warning about needing to reconfigure the boot. I think this card is the one that didn't come with a boot rom.

I'm back up and running, but I'd really like to figure out what's going on with the other 2308. Any troubleshooting suggestions?

 

[jgreco]

Two.

One, and don't laugh, it's happened to many folks, is to make sure the contacts on the SFF-8087 cable and socket are both clean. I recommend electronic contact cleaner. Also, make sure that your plug is fully inserted into the socket and clicks. You should not be able to remove the cable once correctly seated.

Two, and it's tragic, try swapping the "A" and "B" SFF-8087 cables (some HBA's do not mark them). See if the problem follows the cable or seems to indicate that the HBA port is malfunctioning. HBA's can and do go bad.

 

[CJRoss]

Two.

One, and don't laugh, it's happened to many folks, is to make sure the contacts on the SFF-8087 cable and socket are both clean. I recommend electronic contact cleaner. Also, make sure that your plug is fully inserted into the socket and clicks. You should not be able to remove the cable once correctly seated.

Two, and it's tragic, try swapping the "A" and "B" SFF-8087 cables (some HBA's do not mark them). See if the problem follows the cable or seems to indicate that the HBA port is malfunctioning. HBA's can and do go bad.

While I didn't clean the cable before inserting it, I did make sure it was properly inserted and clicked in.

Wouldn't any issues with the cables, connection, etc cause the drives not to show up in the LSI config? That's the part that gets me. The card appears to be seeing the drives just fine but somehow isn't presenting them to the OS?