HBA Upgrade on SuperMicro board | HBA guidance needed

[involut]

Hi all,

I am currently happily running a TrueNAS CORE on a SuperMicro X11SCL-IF with 32 GB ECC ram and 4 Seagate IronWolf ST4000VN008 hdds. Currently, I am looking to upgrade a few things on the hardware side.
Especially, whether I should add an HBA. I came across the LSI SAS 9207-8i. Is it suitable? I am not sure if I need 4 SATA ports on the card as well? *confused*
Which HBA is recommended? Is there a reference?

 

[demon]

Yes, a 9207-8i would be quite suitable for use with TrueNAS. I've previously used 9211-8i HBAs (same controller chip, just a slightly different layout), and am now using a Broadcom/LSI 9400-16i. Make sure to watch for CRC errors with your MiniSAS fanout cables though - when I was building my current TrueNAS box, I ended up with occasional CRC errors that required different ones to correct. Make sure you're using IT firmware on the board though, so it just provides raw drive access - the IR firmware has RAID functionality, which you don't want. There are crossflashing guides available, but it's pretty easy to do either from DOS or an EFI shell.

 

[involut]

Hi,

Thank you @demon for providing the information. I came across a Fujitsu CP400i card which apparently uses a LSI 3008 chipset. I will install the card into the server and confirm the IT firmware flag.
One question about moving the storage pool onto this HBA is not entirely clear to me. Once I know that the card is ready to go, do I just need to unplug the drives from the onboard SATA ports to the HBA and boot up? Will TrueNAS recognise this move automatically and not have any issues whatsoever?

Thank you very much for your feedback.

 

[danb35]

Will TrueNAS recognise this move automatically and not have any issues whatsoever?

Yes, there wouldn't be any problem there at all. But why add a HBA? Are you planning on adding more drives to the system? Because using a HBA doesn't really benefit you compared to the onboard SATA ports otherwise.

 

[involut]

Yes, there wouldn't be any problem there at all. But why add a HBA? Are you planning on adding more drives to the system? Because using a HBA doesn't really benefit you compared to the onboard SATA ports otherwise.

Thank you for the information. That is very nice to know it's more or less plug and play.

I have added the HBA to have some headspace for the future. Right now I am using 4 SATA drives but the controller allows for 8 drives. In addition, I wanted to move away from using TrueNAS on a USB stick. Instead I would like to use a single SSD on one of the SATA onboard ports.

 

[Ericloewe]

A much more sane option, that should be cheaper and use less power, is to use an M.2 SSD as the boot disk and avoid the HBA until such a time when it's needed. Also, fewer cables flying around. Your board supports SATA and NVMe, so get whatever's cheapest and still reputable.

 

[involut]

A much more sane option, that should be cheaper and use less power, is to use an M.2 SSD as the boot disk and avoid the HBA until such a time when it's needed. Also, fewer cables flying around. Your board supports SATA and NVMe, so get whatever's cheapest and still reputable.

Hi @Ericloewe,

Thank you for your input. I actually like the minimum energy consumption approach and would like to follow that route now.

As of now, I am running my TrueNAS installation from a 16 GB USB stick. I do have a 450 GB M2.SSD installed that currently only serves as a pool for jails. My idea is to wipe the SSD, partition into a smaller TrueNAS base system and the remaining storage for jail pool storage.

I came across this link here where I believe the necessary steps to be done are outlined. https://www.truenas.com/community/t...itioned-single-ssd-with-boot-and-jails.85867/

Is this the right way to do this or is there an easier way? I would love to mirror my USB drive install onto the smaller SSD partition. Is this possible?

Any feedback is highly appreciated.

 

[Ericloewe]

You can also have a look at this one: https://www.truenas.com/community/t...sds-for-boot-pool-and-data.81409/#post-590563

As stated, it's not supported and you're on your own for:

• Disk replacements
• The actual setup
• Not nuking the other pool on the same disk with the TrueNAS installer or something like that

I would love to mirror my USB drive install onto the smaller SSD partition. Is this possible?

Possible, yes, but it's of limited usefulness. I can imagine it making the setup process a bit easier (just mirror the existing install to the prepared partition on the new disk), but it would be basically pointless in production after that. Of course, since it's a mirror, you can easily remove the USB drive once the process is complete.

 

[involut]

You can also have a look at this one: https://www.truenas.com/community/t...sds-for-boot-pool-and-data.81409/#post-590563

As stated, it's not supported and you're on your own for:

• Disk replacements
• The actual setup
• Not nuking the other pool on the same disk with the TrueNAS installer or something like that

Possible, yes, but it's of limited usefulness. I can imagine it making the setup process a bit easier (just mirror the existing install to the prepared partition on the new disk), but it would be basically pointless in production after that. Of course, since it's a mirror, you can easily remove the USB drive once the process is complete.

Hi @Ericloewe

I guess the fact that I am on my own when it comes to my setup basically applies for my entire setup. haha
Seriously, I understand the risks involved but actually running it from the SSD gives me more confidence instead of running it from the USB stick - in terms of reliability.
In preparation I already deleted the pool that was on the SSD. So I am basically looking at an empty SSD now.

I found the thread you have mentioned as well. But reading through the command lines I find the one I have linked very understandable.

There is just one question I have regarding the thread I have linked when it comes to mirroring. It says:

#setup boot as mirror from our usb drive (usb drive B is da0, da0p2 is the freenas-boot pool)
zpool attach freenas-boot /dev/da0p2 /dev/ada0p3

#wait for resilver! run 'zpool status' to check status
zpool offline freenas-boot /dev/da0p2
zpool detach freenas-boot /dev/da0p2

Here - from what I understand - you are actually taking the exact same partition from the USB stick and mirror it to the newly created partition on the SSD, correct? This would mean I 'copy' all of the files and configurations from the USB stick as well.
I will back up in any case and don't have a problem with a fresh install. Hmm

I guess one thing to consider are the partition sizes I am creating on the SSD. Because if they are not identical as the ones on the USB stick, I will waste space if I just blindly mirror from USB-->SSD. Hmm

 

[NugentS]

You won't waste space - you just won't use it.
For example - boot USB = 16GB, M.2=500GB
If you mirror the USB to M.2 - you end up with a 16GB M.2 boot disk - great. Modify BIOS, remove USB and boot from M.2 - all good.

You COULD then manually add partitions to the M.2 and use it as a pool - beware, here lie dragons

 

[Ericloewe]

Here - from what I understand - you are actually taking the exact same partition from the USB stick and mirror it to the newly created partition on the SSD, correct? This would mean I 'copy' all of the files and configurations from the USB stick as well.
I will back up in any case and don't have a problem with a fresh install. Hmm

Right.

I guess one thing to consider are the partition sizes I am creating on the SSD. Because if they are not identical as the ones on the USB stick, I will waste space if I just blindly mirror from USB-->SSD. Hmm

You would, but you're not copying the partition - you're adding the partition as a leaf vdev to the single vdev in your boot pool, to make it a two-wide mirror. After removing the old disk, the pool should automagically grow.

 

[involut]

Hi,

so... I followed above mentioned steps. Was able to do all steps without any errors. But upon reboot, there is not a way to select a bootable device in BIOS for UEFI. As well as booting from the USB stick does not work, it just leaves me with with:

I am a bit clueless now :)

 

[Ericloewe]

Try outright disabling all CSM options and related stuff. The setup menus can be rather buggy around boot devices.

 

[involut]

Hi @Ericloewe ,

CSM options and related stuff? I am confused what you mean by that... :)

I just flashed a different USB stick and booted into the shell of the installer.

Found this which is looking okay I guess:

 

[Ericloewe]

CSM options and related stuff?

In the system firmware setup menu. Also known as legacy boot, BIOS boot, etc.

 

[involut]

In the system firmware setup menu. Also known as legacy boot, BIOS boot, etc.

Hi,

So I disabled all of the secure boot and CSM option ins BIOS.

But within my boot priorities, the M2 drive does not show up as a 'bootable' device. What you see above happened while still trying to boot from the previously working installation where I executed all of the commands from.

Thus, when I boot the BIOS tells me to select a bootable device.

Does that make sense?

 

[Ericloewe]

But within my boot priorities, the M2 drive does not show up as a 'bootable' device.

The menus can be rather unintuitive, and there may be multiple lists. It's hard to be specific, AMI seems to keep changing these settings around.

 

[involut]

Hm so I believe I looked at all the options available. But there is no option to select the NVMe drive.

But, I booted into the UEFI shell. Doing this, I am presented with a list of BLK devices, which perfectly match the table I presented in the screenshot above. BLK0-BLK4 all refer to the NVMe drive.

If I do a
bcfg boot dump -v

I only have one option available wich ist the built in shell.

My idea was to mount a blk device (most likely blk1 as I assume it refers to the freebsd-boot nvd0p1 partition form above. And manually create an boot option pointing to a .efi file on blk1.
But I somehow fail to do mount the blk1 device and can't access what is on that device.

Something is really off here....

 

[involut]

Thank you all for your contributions. It was a bit rough but I was successful.

I have written a complete how to about the necessary steps here:

HOWTO: Setup TrueNAS Core (v. 12+) on partitioned NVMe and UEFI

Hi, While I was in the process of refreshing my hardware, @Ericloewe provided me with some great feedback in terms of a lean and energy efficient setup when it comes to boot devices. Up to then, I used to run a SuperMicro X11SCL-IF, 4 Seagate IronWolf ST4000VN008 connected through the onboard...

www.truenas.com