Dell R720XD does not recognize H310 (full size) as a boot option

[MR5Fhd]

Hello

This is my first time setting up a FreeNAS/TrueNAS. I run into a weird issue that I cannot find answers anywhere.

My hardware:
Dell PowerEdge R720XD chassis
2x Intel Xeon E5 2667 v2
256GB of RAM
12x 4TB SAS at front, 1x Samsung SSD at rear flex.
1x H310 mini mono
1x H310 Full size

Everything else is just basic R720XD hardware (I believe it's 12th gen).

My problem is that the the system does not allow me to boot from H310 Full size.


Background:

I purchased this server not only for FreeNAS/TrueNAS, but also for virtualization on Proxmox. The plan is to install Proxmox on the one SSD at rear flex and run a FreeNAS VM. And then pass the front 12 SAS drives to the FreeNAS VM. I specifically picked this CPU because it has the highest frequency among the same generation. So it is very useful for a lot of other virtualization tasks besides FreeNAS.

You might wonder why I have two HBA controllers. At first I did not know H310 mini and H310 are different. I proceeded to flash H310 mini with the H310 instruction and ended up bricking it. After I learned my mistake, I quickly purchased a H310, along with a USB eeprom programmer to fix H310 mini. Eventually I have both HBA flashed to IT mode. Both of them are tested working.

I proceeded with H310 mini to setup Proxmox and TrueNAS core. But then I realized that I need to pass through the HBA to TrueNAS for best performance. Being a noob, I did just that. And Voilà! My proxmox crashed because it lost access to the one and only HBA.

Well I still have a full size H310. So I think maybe I can connect both HBA to the server, and connect the SSD to only the H310 full size, and leave the rest to H310 mini.

However, no matter what I do, the system won't recognize H310 as a bootable device. It can only boot from H310 mini. I checked BIOS settings. All PCI-E slots are enabled to be bootable. Both H310 and H310 mini are recognized by BIOS. If I boot to a iso (through USB or virtual disk), I can use lspci | grep LSI to see both HBAs and use lsscsi to see all drives attached to them. To make it less confusing. I also removed H310 mini leaving only H310 full size. The system simply does not recognize H310 full size as a bootable device.

I tried to connect the whole back plane to H310 full size. Does not boot.
I tried to connect only the SSD to H310 full size through cable. Does not boot. System shows a error saying a cable connected to wrong port.
I also taped B5 and B6 on H310. Nothing changed.

I booted to a live OS to check whether H310 is working properly for all of the above configurations. The OS can recognize both the H310 and the hard drive attached.
./sas2flash -c 0 -list shows this result. It seems some information is N/A. Is that normal? I lost my original SAS address, so I just made up one.

I believe my H310 full size is working properly, but the system refuse to recognize it as a bootable device. I even successfully installed proxmox on the SSD that was connected to H310 while the back plane with the SAS drives connected to H310 mini. It means that both HBA can operate together.

Please provide me some thoughts. I have exhausted all my research.

P.S. I was able to have a TrueNAS core VM working by passing through hard drives instead of HBA. According to my research, this is not the best practice. I achieved this set up with H310 mini.

 

[MR5Fhd]

Welp....

Problem solved. Many H310 IT mode flash instructions omit the last part that flashes BIOS rom and EFI rom. If you run your FreeNAS/TrueNAS bare metal on a USB drive, you don't need the roms. But in my case, I need them to boot from my drive.

It's driving me crazy

 

[Touffy-Ra]

I’m trying to replicate this exact setup (albeit with different cards - H710 mini and LSI 9211).

1. Any advice on the cabling required for this to work? Especially regarding the A1 and BP cables that go from rear backplane to front backplane.
2. Any special requirements on BIOS / iDRAC versions?

Would really appreciate your help with my new 720XD

 

[MR5Fhd]

I’m trying to replicate this exact setup (albeit with different cards - H710 mini and LSI 9211).

1. Any advice on the cabling required for this to work? Especially regarding the A1 and BP cables that go from rear backplane to front backplane.
2. Any special requirements on BIOS / iDRAC versions?

Would really appreciate your help with my new 720XD

I am quite new to this myself. I don't know how reliable my answer could be. I'll do my best.

1, Cabling is a pain in the butt. In my case, my original server came with everything installed. So by default, every drive is already connected to the H310 mini. Everything fit exactly where it is. There is little room to customize the cables. Even the length of the cables are exact fit. They wouldn't fit in if they are one inch longer. That is why I had to connect H310 full size to the SSD at rear flex. There is simply no other way. H310 mini is in a fixed place. All the cables fit exactly where they are. Luckily, there seems to be some pre-defined space left for some sort of cables at the rear part of the server. I'm not really sure whether are they really for cables, I used them to hold cable for H310 full size anyway. It was not a nice fit, but a fit nonetheless.

2, I don't think iDRAC matters. iDRAC allows to remotely access your server. It basically is a super powerful SSH. I had the licensed iDRAC simply because I want the functionalities. I wouldn't worry too much about BIOS either unless your server is super old. To my knowledge BIOS are super compatible. It will always work. UEFI might cause issue though. I tried UEFI once. It didn't work.

 

[Touffy-Ra]

Thanks so much for the reply. I ended up getting it to work doing exactly what you suggested.
One issue I encountered though was not being able to access the new LSI card’s BIOS to set the SSDs as bootable discs. I ended up fixing it my temporarily disabling the mini card from the BIOS, and that allowed me to access the LSI BIOS.

 

[MR5Fhd]

Thanks so much for the reply. I ended up getting it to work doing exactly what you suggested.
One issue I encountered though was not being able to access the new LSI card’s BIOS to set the SSDs as bootable discs. I ended up fixing it my temporarily disabling the mini card from the BIOS, and that allowed me to access the LSI BIOS.

I'm glad it worked for you. I'm very surprised that no one talked about this kind of set up before us. I wonder if everyone was rich enough to afford a dedicated server for one task only.

 

[dir_d]

I think this is why people buy the 720XD, I have one that has the front 12 drives passed through to a TrueNas VM. ESXi is running on a bootable USB and all VMs are sitting on a Raid 1 1TB PM863 drives in the flex slot. 1TB is actually a lot of space for VMs if all data resides on the TrueNas pass through drives.

I found it the easiest and fastest way to setup TrueNas after coming from bare metal. I think there are alot more people out there running the same setups we have than we think. You just dont hear much about it because its an easy setup and just works.

 

[MR5Fhd]

I think this is why people buy the 720XD, I have one that has the front 12 drives passed through to a TrueNas VM. ESXi is running on a bootable USB and all VMs are sitting on a Raid 1 1TB PM863 drives in the flex slot. 1TB is actually a lot of space for VMs if all data resides on the TrueNas pass through drives.

I found it the easiest and fastest way to setup TrueNas after coming from bare metal. I think there are alot more people out there running the same setups we have than we think. You just dont hear much about it because its an easy setup and just works.

Oh!!!! I see. That's a smart way to set up too! Do you also use another controller to manage the two drives at rear flex? Or do you just pass everything to TrueNAS and set them to a different pool?

 

[dir_d]

The 2 drives in the rear Flex Slot are on Raid 1 on a PERC H710 and NOT pass through. I have a LSi 2008 passed through to the TrueNas VM, that has the front 12 drives JBOD connected.

 

[MR5Fhd]

The 2 drives in the rear Flex Slot are on Raid 1 on a PERC H710 and NOT pass through. I have a LSi 2008 passed through to the TrueNas VM, that has the front 12 drives JBOD connected.

I see. Similar to mine.