How to boot Supermicro X9SCM-F with all SATA ports used for pool?

[coldplug]

It seems that Supermicro X9SCM-F does not support booting from PCIe NVMe drive. I went through TrueNAS installation, installer correctly detected NVMe drive in PCIe slot and installed OS there. However, after reboot, I don't have BIOS option to boot from that drive.

I searched over forums and it seems that I'm out of luck. There are mentions that BIOS modding is required but I'm unable to tell for sure if that will allow the board to boot TrueNAS from this drive as in those threads Windows are used a lot.

Maybe I'm missing something though. I'm strugling for second day with this now, still without progress, so I decieded to search for help.

So, is it definite that there is no way to boot this board from PCIe NVMe drive? If that's true, can it be achieved with BIOS modding? What are my other options? I have all 6 SATA ports populated with spinners to create RAID-Z2 pool and reducing that to 5 drives to be able to boot is last thing I want to do. Installing OS on USB stick does not look very reliable option so I dismissed that in general.

Would PCIe SATA drive make any difference compared to PCI2 NVMe?

Any other idea?

Thank you,
Ivan

 

[Redcoat]

Welcome to the forums!

I have 2 X9SCM-F's - I have not tried to use a PCIe device boot, but I would attempt this procedure if I did as PCI-E NVMe SSD s are only supported in UEFI mode Bios:

Make sure you have the latest bios.

1. Go into the bios to the section PCIe/PCI/PnP Configuration and change the Device Option Rom Configure “UEFI mode” from Legacy on Riser card slot or PCIe slot.

2. Save and exit the bios.

3. During the post press F11 and select “UEFI: DVD-ROMxxxxxxxx” to start the installation (Hopefully Truenas includes the drivers?).

4. Reboot after OS installation, press F11 again and select the PCIe device to load the OS. Change the boot sequence in the BIOS so that the PCIe is the first boot device in the BIOS.

Good Luck with this. Let us know how it goes if you try it.

EDIT Your alternative is to get an HBA. No, because it's not a SATA device

 

[Etorix]

Would PCIe SATA drive make any difference compared to PCI2 NVMe?

There's no such thing as "PCIe SATA". Being on PCIe protocol means it's NVMe. You may have confused with M.2, which is a connector, and may be used to carry PCIe (NVMe) or SATA signals.

If @Redcoat 's manoeuvre does not work, using a SATA SSD on a SATA-to-USB adapter could be an option. The bad part about booting from USB sticks is that sticks generally wear out quickly under load.

 

[coldplug]

Welcome to the forums!

I have 2 X9SCM-F's - I have not tried to use a PCIe device boot, but I would attempt this procedure if I did as PCI-E NVMe SSD s are only supported in UEFI mode Bios:

Make sure you have the latest bios.

1. Go into the bios to the section PCIe/PCI/PnP Configuration and change the Device Option Rom Configure “UEFI mode” from Legacy on Riser card slot or PCIe slot.

2. Save and exit the bios.

3. During the post press F11 and select “UEFI: DVD-ROMxxxxxxxx” to start the installation (Hopefully Truenas includes the drivers?).

4. Reboot after OS installation, press F11 again and select the PCIe device to load the OS. Change the boot sequence in the BIOS so that the PCIe is the first boot device in the BIOS.

Good Luck with this. Let us know how it goes if you try it.

EDIT Your alternative is to get an HBA. No, because it's not a SATA device

Thank you for reply!

I already discovered tips like this one but I'm not sure that I have such entries under PCIe/PCi/PnP Configuration. Please, take a look at attached image how it is here.

Anyway for now I installed TrueNAS onto external USB HDD drive and that works. However, I would still like to avoid having external case attached and use all components inside the case.

I have another question, as I'm new to TrueNAS. Now, if I chose to stick to booting from external USB HDD, can I then use this NVMe drive for something good? Will it be useful for caching? I have only one, so no mirroring there; what would happen if I set it for caching and then it dies?

Thank you,
Ivan

 

[Etorix]

A cache (L2ARC) is a duplicate of what's in the pool. If the cache drive fails, ZFS falls back to reading the pool: No data is lost, only performance.
Using a L2ARC takes up RAM. It is advised to add RAM, preferably at least 64 GB, before considering a L2ARC, and then not to use L2ARC larger than 5x to 10x RAM.

 

[Ericloewe]

There's no such thing as "PCIe SATA". Being on PCIe protocol means it's NVMe.

Not technically true, though not off by much.
Before NVMe was a thing, there were multiple solutions, from proprietary drivers to just AHCI - yes, the SATA controller protocol. Of course, all that stuff is obsolete and almost irrelevant.

Sidenote: It should be possible to boot from NVMe, by either modifying the system firmware image to include an NVMe driver and whatever else the stock firmware might need, or - more reasonably - by using a custom EFI loader (e.g. rEFInd) to do the whole "load driver, load OS bootloader" thing. Of course, you then need to store the custom loader somewhere (e.g. USB).
Needless to say, this is not very practical.

 

[Redcoat]

I already discovered tips like this one but I'm not sure that I have such entries under PCIe/PCi/PnP Configuration. Please, take a look at attached image how it is here.

Well I just poked around in the BIios on one of mine - we have the same information displayed, and, if anything I would say that the bios seems to be set in the closest set of options that appears to match the method we are discussing.

What happens(ed) when you try step 3 and hit F11 during the post after you exit the bios?

 

[trueJack]

How about something like this?

Set up GRUB 2 to boot FreeBSD using either Ubuntu tools or LiveUSB to find what partitions?

I currently have Windows 7 and Ubuntu 11.04 (Natty) installed on my computer. I also made a FreeBSD liveUSB (or installer, not sure; just dded their .img to the USB stick) and installed it without

unix.stackexchange.com

You could boot from a small USB stick with GRUB and have GRUB load your NVME disk installation.

You would permanently need the USB stick for booting, but you could buy two sticks so you always have a backup. If you save the config, you would always be able to recreate one as well. Performance-wise, this should be pretty fast as almost all files will be loaded from the SSD.

 

[coldplug]

Well I just poked around in the BIios on one of mine - we have the same information displayed, and, if anything I would say that the bios seems to be set in the closest set of options that appears to match the method we are discussing.

What happens(ed) when you try step 3 and hit F11 during the post after you exit the bios?

Hi,

Well, with F11, I get device list to boot from, which is basically the same thing as if I enter BIOS and set boot priority device. Not much to do here, I can select Sharkoon USB stick with TrueNAS installer and install it again to PCIe device, but I don't see a point of doing that, as there is no PCIe device on the list to boot from. See attached image. There is WD Elements that is USB-HDD drive in external enclosure where I installed TrueNAS and running from now. HGSTs are obviously spinners on MB SATA ports and IBA device is network as far as I can tell.

 

[Redcoat]

Thanks for the additional feedback. Sorry that you didn't get it to where you wanted it to be. But it sound like you have a workable solution.

 

[trueJack]

How about this?

You could create a mini partition on the front of one of your HDDs. You could install GRUB to that. GRUB could boot your SSD. You would lose that partitioned space on that HDD though, so you could not entirely use that HDD for RAID. You could still use all your HDDs for RaidZ2, or so I suspect. I think ZFS can use partitions.

Example:
HDD1 [ [grub partition (very small) ] [ partition for raid ] ]
HDD2 [ [empty partition (very small) ] [ partition for raid ] ]
HDD3 [ [empty partition (very small) ] [ partition for raid ] ]
HDD4 [ [empty partition (very small) ] [ partition for raid ] ]
HDD5 [ [empty partition (very small) ] [ partition for raid ] ]
SSD1 [ [ OS installation ] ]

You could have a RaidZ2 across the 5 raid partitions. You would hardly lose any space.

Disclaimer: I am new to True NAS. I have not tested this.

 

[pschatz100]

@coldplug The easiset and simplest solution will be to use a small sata SSD with usb adapter as your boot device. The NVMe device can be used as a data device because the necessary drivers will be loaded after the system boots. Unfortunately, you can't boot from the NVMe device.

 

[trueJack]

By the way, to follow up on my suggestion to use partitions on HDD to boot-load the NVME, that seems to be not recommended, if not impossible.

 

[jgreco]

By the way, to follow up on my suggestion to use partitions on HDD to boot-load the NVME, that seems to be not recommended, if not impossible.

It's not impossible, but it is a terrible idea. It messes with the system's ability to be able to operate in the manner in which it was designed. Doing hacky things like that is fine for your FreeBSD or Linux desktop machine if you must, but for a NAS appliance, you want to follow the paradigms that the NAS was designed to work with.

 

[trueJack]

Sorry to revive this thread, but I have been playing with my Supermicro board that I just got a week or two ago.

Try booting:
IBA GE Slot

I know that is a network card, but I found that when I chose that option, it would reliably jump to a boot drive not listed under F11 or in the BIOS boot order list, after trying the network.

If that works, add IBA GE Slot as the only boot order option.

Also, next to your SATA ports it looks like there is an embedded USB port. You could leave a permanent small-footprint USB thumb drive in there for boot loading. I daresay it is just for these types of use cases.

 

[Ericloewe]

Try booting:
IBA GE Slot

I know that is a network card, but I found that when I chose that option, it would reliably jump to a boot drive not listed under F11 or in the BIOS boot order list, after trying the network.

AMI's UEFI implementation is extremely buggy when it comes to choosing where to boot from, to the point of features being removed by motherboard manufacturers because they're so broken (e.g. the boot device override option).
It is definitely a good idea to reduce the number of variables to the bare minimum: only UEFI or CSM and remove from the boot order anything that is not meant to be bootable. Also check all the stupidly specific options, like "Hard Drive Priorities" or "Removable Storage Priorities" or similar - it's anyone's guess where a specific device will end up being listed.

 

[artlessknave]

from what i was able to find, X9SCM-F simply doesnt boot from pcie, there was no bios updates that enable it.

 

[Ericloewe]

That's not strictly true. The X9 series never received an update which added an NVMe UEFI driver, which would allow booting from NVMe. This can be worked around, as reported up-thread, and has no bearing on any non-NVMe PCIe devices, which would typically be expected to supply their own extension ROMs.

 

[artlessknave]

oops, I meant PCIe NVME.
I didn't catch that there was a workaround. might have to try that.