No luck installing on Super Micro Opteron server (SC487/H8DGU-F)

[LockeCJ]

After months in analysis paralysis on a from-scratch build, I happened upon the following on eBay:

Supermicro 4U 36 BAY FREENAS ZFS UNRAID Server AMD 16 Cores 64GB 36x Trays RAIL | eBay

Find many great new & used options and get the best deals for Supermicro 4U 36 BAY FREENAS ZFS UNRAID Server AMD 16 Cores 64GB 36x Trays RAIL at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Here are the relevant specs for posterity:
Processor: 2x AMD Opteron AMD 6212 Octo (8) Core 2.6Ghz (Total 16 Cores)
Memory: 64GB DDR3 (8 x 8GB - DDR3 - REG PC3-10600R (1333MHZ)
Controller: 1x LSI 9211-8i HBA JBOD FREENAS UNRAID (Flashed with IT firmware)
Server Chassis/ Case: CSE-847E16-R1400UB
Motherboard: H8DGU-F (IPMI)
Backplanes:
- BPN-SAS2-846EL1 24-port 4U SAS2 6Gbps single-expander backplane
- BPN-SAS2-826EL1 12-port 2U SAS2 6Gbps single-expander backplane
Boot Drive: SK hynix Gold S31 500GB
Data Drives: 8xSeagate IronWolf 6TB NAS (Not installed yet)
I did a bit of reconnaissance and didn't find any obvious red flags. It's an older model, but it seemed like it would suit my needs to start. In the future, I figured I could keep the case, HBA, and backplanes and replace the CPUs, motherboard, and RAM.

Unfortunately, my experience thus far has been...sub-optimal.

The first problem I encountered

is that I cannot get any recent version of FreeNAS/TrueNAS to boot from USB on this hardware.
I have tested the following:
- FreeNAS 11.3-U4 (Current Stable)
- TrueNAS 12.0-BETA
- TrueNAS 11.2-Release
In all cases, flashing to a USB stick and attempting to boot would get to the BTX Loader displaying the version, and hang.

I did some digging and found that there was a change to FreeNAS to switch from grub to BTX after BTX added support for booting from ZFS, so I went back to the latest version of FreeNAS that used grub (11.1-U7) and tested it. This time, I was able to get to the installer, and completed the initial setup without any errors.

I had placed the SSD in a 3.5" bracket adapter and put it into one of the bays on the front of the chassis. Unfortunately, on reboot the system hung on a blank screen (with a blinking cursor in the upper-left corner) right after the grub prompt. I did a bit more digging and discovered that the HBA config can mark a drive as bootable otherwise it will try to boot from the first device. I went ahead and set that even though I had only installed the SSD in the chassis up to that point, but it didn't make a difference. I even did a fresh install afterwards just in case.

Finally, I connected the SSD directly to the motherboard via SATA based on advice I've seen on this forum not to boot from the HBA. Instead of failing after grub, BIOS reports that the disk has failed, prompts for F1, and moves on to the next boot method. I did another fresh install just in case, but no change in behavior.

I'd really appreciate any help in getting this to work.​

 

[pschatz100]

I don't have experience with your motherboard, but on my X9SCM system, the motherboard is fussy about using BIOS or UEFI boot options on certain devices. Some devices boot OK from UEFI boot - others don't. I suspect it has something to do with the firmware on the device you are attempting to boot from, but I never bothered to figure this out for certain.

When I migrated my boot device from USB flash drive to an SSD, I had to re-install the FreeNAS image for BIOS boot and also set for LEGACY boot in the motherboard bios.

 

[LockeCJ]

I've managed to get past the issue with booting by buying a used SSD off of eBay based on the list from Super Micro's website, an Intel 520 series. I was able to boot 11.2-Release from USB, successfully install, and reboot into FreeNAS. I now have the storage vdev set up, and I went ahead and set up the other SSD as an L2ARC. I've got a smaller SSD that I'm going to set up as an SLOG.

However, my original issue still exists. I don't currently see any way to install a post-2017 version of FreeNAS/TrueNAS on this hardware since the BTX Loader will not boot. Is this an issue where it's just not compatible with this hardware, or is there some array of settings that I just need to get correct first? Please keep in mind that this motherboard does not have UEFI as an option.

 

[codepoet_atx]

I have the exact same build from the same vendor. I've had one hurdle after another with that beast, but it does show promise.

For me, I imaged the 11.3 install onto a USB drive and shoved it into the USB port on the MB near the SATA ports. I put two ADATA 2.5" SSDs into conversion sleds and loaded them into the SAS backplane. After I set the BIOS to boot to USB media first, it just came up. I installed and then it rebooted and nothing happened. That's when I learned someone disabled booting from the SAS card. So I rebooted, held control-C to get to that BIOS and enabled booting from SAS. Another reboot just brought up FreeNAS from the SSDs.

Reset your IMPI and BIOS, then ensure they're on the latest versions. I know my BIOS was several revisions behind, despite the seller's claim they updated it. (I found this out when it didn't POST, contrary again to their claims; it didn't support the CPUs they put in there, but a later BIOS did.)