ahci driver problems with Marvell 88SE9230 (Dell BOSS-S1)

[Fab Sidoli]

Hi All,

I'm copying the title of a bug report submitted to FreeBSD, which hasn't got much traction there, in the hope that someone here can help.

I have three Dell PE R740xd2 servers with these BOSS cards in, all running FreeNAS 11.3, and can verify that the latest firmware for the BOSS breaks the ability for FreeNAS to boot, or for the FreeNAS installer to see the disks if a fresh install is attempted. A Windows installer can see the disks, however, so I feel like this is a FreeNAS issue.

Please see the bug report below (link and content paste provided) which describes the issue I'm having.

Apologies if the format of this post is poor form.

Many thanks,
Fab

FreeBSD bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243401

Contents:

"This feels more like a firmware problem than a driver problem but since it apparently works in Windows and Linux, but not in FreeBSD I figured I'd report it here anyway...

(Probably not meaningful to try to report it to Dell since FreeBSD isn't officially supported by them)


Dell BOSS-S1 (Marvell 88SE9230 based) M.2 "RAID" cards running Dells latest firmware (2.5.13.3022 A06 or 2.5.13.3022 A05) does something strange - when the kernel has loaded (from the drives on this card) it fails to detect the disks ("unconfigured" disks, non-RAID setup) and then root fs mounting fails...

(We have two M.2 SSDs connected to that controller)


With firmware 2.5.13.3016 A04 it gives a couple of errors at kernel boot time, but does detect the disks and the system boots.

With firmware 2.5.13.3011 A03, 2.5.13.2009 A02 or 2.5.13.2008 A01 no errors are printed and the disks are found just fine.

(But there are bugs fixed in the later releases that probably would be nice to have.. I have had M.2 drive go "offline" for me at 2008/A01-firmware so that's why I tried the later versions...


A summary of the (Dell) firmware fixes and my test results:

2.5.13.3022 A06
Fixes: None
Enhancement: Added support for 15G platforms

2.5.13.3020 A05
Status:
- Does not work, gives errors:
- 'ahcich16: stopping AHCI engine failed'
- Detects a 'pass23', but no disk devices:
pass23 at ahcich16 bus 0 scbus19 target 0 lun 0
pass23: <Marvell Console 1.01> Removable Processor SCSI device
pass23: Serial Number HKDP221516WL
pass23: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)
Fixes:
- Fixed an issue where system will hang during
Boot when PERC is in HBA mode with BOSS-S1
- When CLI is running, default temporary file
directory & permission in Linux and ESXi Operating
systems are changed as appropriate
Enhancement: N/A

2.5.13.3016 A04
Status:
- Works and detects all disks, but gives errors about:
- 'ahcich14: stopping AHCI engine failed'
- 'ahcich15: stopping AHCI engine failed'
- 'ahcich16: stopping AHCI engine failed'
Fixes:
- Fixed a behavior of BOSS-S1 firmware incorrectly marking M.2 drive offline/failed
- Fixed a behavior where ESXi Host goes unresponsive
- Fixed a behavior where BOSS-S1 Management path will not respond to Management commands
- Fixed a behavior where BOSS-S1 boot partition becomes inaccessible
- Fixed a behavior where ESXi host results in PSOD due to unexpected I/O timeout
- Fixed a behavior where rebuild will not be proceed during error handling condition
Enhancement:
- Enhanced/ Added MVCLI events for command timeout
- Added SLES15 Support

2.5.13.3011 A03
Status:
- Works
Fixes:
- Fixed M.2 disk failure when medium error is present
Enhancement:
- Enhanced medium error handling

2.5.13.2009 A02
Status:
- Works
Fixes:
- Fixed Sideband functionality issue
Enhancement:
- Added support for Rollback of Controller Firmware through iDRAC/LC

2.5.13.2008 A01
Status:
- Works
Initial release


Kernel boot output (the relevant parts) from a firmware 3016 A04 boot:

ahci2: <Marvell 88SE9230 AHCI SATA controller> port 0x8028-0x802f,0x8034-0x8037,0x8020-0x8027,0x8030-0x8033,0x8\
000-0x801f mem 0xb8800000-0xb88007ff at device 0.0 numa-domain 0 on pci9
ahci2: AHCI v1.20 with 3 6Gbps ports, Port Multiplier not supported
ahci2: quirks=0x900<NOBSYRES,ALTSIG>
ahcich14: <AHCI channel> at channel 0 on ahci2
ahcich15: <AHCI channel> at channel 1 on ahci2
ahcich16: <AHCI channel> at channel 2 on ahci2
...
ahcich16: stopping AHCI engine failed
ahcich16: stopping AHCI engine failed
...
ahcich16: stopping AHCI engine failed
ahcich15: stopping AHCI engine failed
ada0 at ahcich14 bus 0 scbus17 target 0 lun 0
ada0: <SSDSCKJB120G7R N201DL43> ACS-3 ATA SATA 3.x device
ada0: Serial Number PHDW817002Z4150A
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors)
ada1 at ahcich15 bus 0 scbus18 target 0 lun 0
ada1: <SSDSCKJB120G7R N201DL43> ACS-3 ATA SATA 3.x device
ada1: Serial Number PHDW817002WC150A
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors)
pass25 at ahcich16 bus 0 scbus19 target 0 lun 0
pass25: <Marvell Console 1.01> Removable Processor SCSI device
pass25: Serial Number HKDP221516WL
pass25: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)


On 3022 the ada0 and ada1 devices never get detected, and it only complains about not being able to stop ahcich16, nothing about 14 & 15."

 

[Fab Sidoli]

Hi All,

If you need more information from me please let me know.

Otherwise, does anyone have any helpful suggestions?

Thanks,
Fab

 

[StevenPowell]

Did you get any where with this i have exactly the same issue

 

[Fab Sidoli]

Hi Steven,

Not quite*, although I have the Dell techs looking into their firmware which I think is a little broken. They seem to be sticking to the line that FreeNAS isn't a supported OS so I'm not sure if they'll attempt a fix in A07. They are at least looking into it though. Is this is a Dell system? If so, have you reported it? The more instances that are reported the more likely they'll take this seriously.

*I have reverted the BOSS to the A03 firmware and this does work for me. There are some critical fixes in A04, 05 and 06 that I'm a little worried about so ideally I'd like not to be running A03. The other option I have been told to try is to RAID the BOSS with the A06 firmware as this has apparently been test by Dell on a near identical system. For the boot disk this would probably be OK as no strict need for ZFS.

 

[StevenPowell]

Im running the same as you really Dell 740xd2

2x 12 core
256gb ram
2x 256gb m2 sticks
and 6 x 7.6tb ssd nvme

 

[Fab Sidoli]

In that case, if you need to get this working I'd suggest either trying to RAID the m2 sticks on the BOSS, running the latest firmware. Or, downgrade the BOSS firmware to A03 and leave the BOSS in JBOD.

What controller do you have for the 6 storage disks? Do you have a PERC or an HBA?

 

[StevenPowell]

Im using a dell S140 but have basically switched off any of the raid stuff and let freenas do it.

Im very new to this but slowly getting there

Steven

 

[Fab Sidoli]

You really should consider getting an HBA instead.

We initially went with a RAID because at the time we thought the HBA mode would be fine if we went down the ZFS route. Turns out, it wasn't.

I encrypted the drives in FreeNAS and on the first power-cycle I got locked out of the disk and the RAID card refused to pass them through to the OS. I ended up down a rabbit hole of pain to "fix" this.

As it turns out, I'm still having encryption issues with the HBA but at least the disks are being passed through to the OS so I can do something with them instead of having to boot off media to fix.

 

[dirtyfreebooter]

Ran into the same thing today with 2.5.13.3024, A07 ... TrueNAS Core 12.0-RC1 installer does not recognize any m.2s attached to BOSS controller. Downgraded to A03 and then TrueNAS installer was able to see the 2 m.2 SATA drives in JBOD mode.

 

[Fab Sidoli]

There seems to be little appetite to look into this issue seriously so as it stands the only work around I have is to RAID the boss via the BIOS and then it happily sees the disks with the latest Dell Firmware versions.

 

[Diff]

As of Sept, 2021 running into same problem on TrueNAS Core 12.0-U5.1 & ROM (2.5.13.3024) and firmware (2.5.13.1215).

I see there is more recent version of ROM 2.6.13.3024, A07 released, but somehow iDRAC do not see that update. Anybody got any luck resolving this recently?

If not, that's probably good reason to move to TrueNAS Scale BETA.1, since that one seems to detect it correctly...

 

[abongiorno_bbanda.it]

To avoid this problem you may apply the configuration steps below, if them fits yours needs.

1. Activate UEFI boot mode in BIOS setup
2. Use Dell BOSS-S1 BIOS configuration utility, to configure RAID 1 using both drives (drives must be M.2 SATA no PCIe NVMe)
3. Install TrueNas on the BOSS RAID 1 drive obtained

Following these steps I've successfully installed "TrueNAS Scale 22.02.4" , on Dell R640 with BOSS-S1 [ROM (2.5.13.3024) and firmware (2.5.13.1215)] using two Micron 5100 M.2 SATA drives.

The installation fail to reboot if you omit step 1, (leaving BIOS legacy boot mode enabled). You will get a similar faulted behaviour, if you don't configure the RAID 1, trying to use both drives to install TrueNAS.

No other scenario has been tested by me.

I don't have much time to partecipate in conversations, so I posted this contribution only to share my experience, hoping this can help someone else to address the problem.

 

[KarlHenselin]

These steps worked for me also.

1. Activate UEFI boot mode in BIOS setup
2. Use Rufus to make a UEFI USB stick for Truenas.
3. Use idrac configuration utility, to configure RAID 1 on the boss card using both drives (drives must be M.2 SATA no PCIe NVMe)
4. Install TrueNas on the BOSS RAID 1 virtual drive.

 

[richardvrusso]

Hey everyone,
Old IT infrastructure guy here. New to the forum and Core/Scale too.

I'm attempting an install to a pair of M.2's on the same Boss-S1 card. I have a decomm Dell XC640 with the M.2s, two SAS SSD and two SAS HDD. The other day I did a round of updates to the bios/idrac/nic/etc. So now i'm running ROM (2.5.13.3024) and firmware (2.5.13.1215). Seems simple enough to RAID1 the M.2s and install to the one virtual drive during OS install. Although I feel at a lose because then ALL the drives are not entirely managed from within TrueNAS Scale.

Has anyone heard if a newer or even beta rom/firmware has been released?

Regards, Rich