HP DL380e G8, LSI 9300-8i, HGST 4Tb NAS, FreeNAS 9.10 (only sees 1 HDD of 12)

[SamM]

Long story short: FreeNAS only sees 1 of 12 data drives while POST & some other OS'es see all 12.

Per forum rules, here's the hardware I have available to work with:
Server: HP DL380e G8 (12 LFF) w/ SAS Expander Backplane,
CPU: Intel Xeon E5-2440 2.4GHZ 6 core 15MB x2
RAM: 32GB DDR3 ECC Registered (4GBx8)
HDD: HGST 4TB (7200RPM, 128MB, SATA3) NAS edition x12 (Connected to HBA via SAS Expander)
- Kingston RBU-SNS4151S3/16GD 16GB M.2 SSD x2 (Connected to onboard SATA)
- Samsung 960 EVO MZ-V6E250BW M.2 250GB NVMe PCI-Express 3.0 x4 SSD x2 (Connected via PCI-E to M.2 card)
HBA: LSI 9300-8i
NIC: HP NC552sfp dual port 10GbE

Some of this hardware was picked out primarily out of a tiny budget but also due to misunderstood and/or outdated 'best practice' incase you're wondering about the odd selection. But this is what I have, so I'll try to make the best of it. The original (outdated) idea was to boot off the redundant 16GB SSD's, use the 250GB SSD's as mirrored Read/Write Cache drives, with the primary storage pool being a pair of RAID-Z1 (6 HDD's) mirrored with each other much like how the HP Lefthand SAN at work has. Updates to the original plan include:

1. Not sure which is better option for boot drives: Just select both drives during FreeNAS install and let it do it's thing (No idea how that works or impacts of a failed SSD) OR RAID1 (onboard controller) the two drives and install FreeNAS on that logical drive.
2. FreeNAS can't do a mirrored RAID5 equivalent the way Lefthand does. So now I'm thinking of striping 4 sets of 3 mirrored drives. I was hoping for 20TB of usable space and this method only yields an estimated 16TB (4TB x4 if my math is right), but redundancy is a major concern and at least (if I understand it correctly) I get significant IOPS gains for multipurpose VM's & iSCSI LUNs.
3. Correct me if I'm wrong, it seems that 'write cache' equivalents, or ZIL's/SLOG's, no longer need to be mirrored and can only use 16GB anyways making the 250GB SSD's a silly choice. So how best to repurpose them given that the intention for this FreeNAS Server will be to host VM's & other iSCSI shares? I'm currently considering using one as a ZIL device (and try to look the other way about the wasted 234GB on given SSD) and use the other as a ARC2 device.
4. I saved the "best" for last. Even with (minor) issues 1-3 aside, I'm having a problem with FreeNAS seeing only one of the twelve LFF drives attached to the LSI HBA via the backplane's SAS Expander.

In regards to the missing drives problem, here's (A LOT) of background info:

1. When I first build this server and installed FreeNAS 9.10, the HBA presumably had firmware P12 (IT). When I upgraded to FreeNAS 9.10-U3, the GUI issued a warning about the HBA firmware being version P12 and the driver wanting P14. I went through the surprisingly difficult process updating the HBA. Up until now, I've never encountered a device that treated the BIOS & Firmware as two separate files/updates, which was really confusing. I figured it out and got the card to version P14, which made the warning in the GUID disappear. This is when the missing drive problem started...
2. My Google-Fu was weak and I couldn't find any solid information on the problem I was having with the missing drives. I came across something that suggested I try version P13 and ignore the resulting GUI warning. After a nightmarish process of trying to downgrade the HBA to version P13, I temporarily ended up bricking the card and hastily ordered another one hoping it would come with P12 installed again where I could just leave it at that.
3. While 'sleeping on it' while waiting for the replacement card to arrive, I decided to take another stab at flashing the bricked card. After all, what do I have to lose? Well, I was able to resurrect the HBA and flash it to P13. It was all for not though as FreeNAS still would only see one drive.
4. Not that I figured out how to properly flash these LSI SAS 9300-8i HBA's, I flashed it back to P12 thinking that should put me back at square-one and that I could live with the warning after that experience. No dice, 1 drive problem still remained.
5. At this point, I'm just 'throwing Science at the wall and seeing what sticks'. I've tried reinstalling. Wiping drives then reinstalling. Tried FreeNAS 9.3, which generated similar GUI warning about Firmware vs driver mismatches AND still only saw one drive. I've tried booting to G-Parted & Windows Installer. G-Parted could see and manipulate (the file systems anyways) all drives (12 HGST +2 Kingston's, +2 Samsung's, +1 Boot USB Flash drive). Windows 2012 installer could not see any of the LFF (did see all 4 SSD's) even after loading the LSI F6 driver. But to be fair, the HBA was the second one running Firmware (IT) 05.00.00.00, BIOS 08.11.00.00, and UEFI 06.00.00.00 when the driver's download archive filename specifically referenced P14. I've tried disconnecting one of the SAS (4 lane) cables to minimize the complexity of the SAS fabric. Nada.
6. One really odd thing is that I took the 1st HBA (P12 at the time) home, put it in a basic SOHO PC and attached (jerry-rigged) 4 similar 80GB SATA drives via a SAS to SATA x4 Fan-out cable, then installed FreeNAS (9.10-U3) on a 5th drive. FreeNAS worked exactly as it should leading me to think the problem is somewhere between the HBA (Hardware/Firmware/BIOS) to SAS Expander vs Fan-out cable to the drives.
7. I'm guessing that this is a problem with the driver FreeNAS is using. For what it's worth, the one drive that does show up is listed as "da0" which I believe at least hints as to what driver is being used. I'll keep playing with the combination of BIOS/Firmware/UEFI code vs FreeNAS and hopefully something will pan out. I'll try out same server, same card (and card related software), but Windows 2016 installer. I'm also going to try taking this card out and putting it into a spare HP DL585 G7 server that also has a SAS Expander backplane, throw a few test drives in it, and compare results.

Thoughts?

Thanks in advance...

*** Update ***
The HP DL585 G7 does not have a SAS Expander, just a pass-thru backplane. Connecting 7 drives with one cable to the HBA resulted in both POST and FreeNAS Installer seeing 4 usable HDD's. So while that technically worked, the test results didn't yield anything insightful.

The Windows Server 2016 Installer test yielded the same results as the Windows 2012 Installer tests. It could see all 4 SSD's (though it won't install to PCI-E NVMe modules because the system BIOS is unable to boot from these devices, nor will it install to the other SSD's for obviously being too small). I'm reluctant to flash this 2nd HBA given all the trouble I had with flashing the 1st HBA. I'd rather (re)flash the 1st HBA and play with that, but I forgot it at home. Still, since the 2nd HBA has already been proven to not work, I suppose there's no reason to not flash it... So after updating the HBA to P12, and using the P12 driver on Windows 2016 Installer, no drives on the HBA still show up. Needless to say, I tried FreeNAS after the P12 update and it didn't work there either.

During the POST cycle, the HBA's BIOS mentions PCI Slot 3, Encl Slot 56, LUN Num 0. Not sure if that 56 is a problem though I don't see why it would be.

 

[SamM]

Long story short, switching from the two LSI 9300-8i's to a LSI 9207-4i4e (LSI00303) solved the one of twelve drives problem.

The LSI 9300-8i would see four SATA drives in BIOS and OS if attached via a fan-out cable (ie: NOT using a SAS Expander); but would see 12 drives in BIOS and only 1 in OS if connected the the SAS Expander of a HP DL380e G8 (12 LFF) w/ SAS Expander Backplane. OS'es included Windows 10, Windows Server 2012, Windows Server 2016, FreeNAS 9.2 (Stable), FreeNAS 9.10 (Stable), and FreeNAS 10 (Stable). Oddly, G-Parted worked under all conditions. Maybe I have a faulty SAS Expander, though it works fine with the 9207. Maybe the 9300 isn't actually SAS Expander compatible. I didn't have any other SAS Expanders to test against, so who knows...

 

[recurry]

Did you eve resolve this with the 9300? Sound like a driver issue to me.