Raid Card Guidance for R720xd 12 Disk w/ Flex Bay

[kjparenteau]

Greetings! I have been a long time user of FreeNAS, and my current system (Frankenstein build) has been cobbled together over the years, but recently I was able to get some new enterprise grade server hardware finally and I am now in need of some advice. Heres what I have...

R720xd, 12x 3.5" bays in the front with 2x 2.5" flex bays in the rear
2x CPU E5-2630 v2 @ 2.60GHz
384GB RAM
PERC H710 Mini
12x 6TB SATA
2x 256GB Samsung 860 EVO

From what I am seeing and now understanding, the H710 doesn't support JBOD natively. My Frankenstein build has an IT Mode LSI 9211-8i SAS SATA 8-port PCI-E 6Gb/s Controller Card, but it only supports 8 SATA ports with the cabling I have. It looks like the card itself can support more disks than that, but I am not sure of the cabling that I would need to make it connect to the backplane of the 720xd...

Ultimately, I don't want a solution that has a performance impact, I want native JBOD passthrough so FreeNAS and ZFS can work the magic. I would prefer to use the LSI 9211-8i card I have (I would purchase a 2nd one for this new server) since it's been rock solid, but not sure how many disks it handles, what size disks it supports, and even more importantly, what cables I can use to connect to the R720xd to make this a clean build... If I need a different card, please point me in the right direction for something comparable. If I need some kind of cable or expansion card to connect all these drives, please let me know what I need. This is a new venture for me trying to bypass the native Dell raid controllers.

Ideally, I want the FreeNAS OS on the 2x Samsung 860 SSD's mirrored, and then the rest just pass through so FreeNAS can use them as ZFS RAIDZ2 or RAIDZ3 storage.

Thank you in advance for any advice, please don't burn me too bad for ignorance. :) Hopefully it is just another LSI 9211-8i card and a special cable to make this all work!

 

[JDCynical]

A quick google turned up this:
Guide: Flashing H310/H710 Mini & full size to IT Mode

This might be the better option?

 

[kjparenteau]

A quick google turned up this:
Guide: Flashing H310/H710 Mini & full size to IT Mode

This might be the better option?

First of all, THANK YOU for the reference guide. I didn't see that in my searching and it was very helpful to make progress. It indeed flashed properly to IT mode and recognizes the drives etc., but I am having an issue for booting I cannot seem to resolve...

As mentioned in the original post, I was wanting to install FreeNAS to the 2x Samsung 860 SSD's so the configuration is mirrored. I would prefer to not use flash drives if I don't need to, but maybe I have to?? Here's what I have done so far...

When I mount the FreeNAS ISO through the iDRAC, I run the installer just fine, select the 2 drives as the install destination, and proceed like normal, however once done, the system will not recognize the installation as a bootable option. I did also install the flashboot /root/Bootloaders/mptsas2.rom optional boot image and it does list the card as a bootable device in the R720's BIOS. I also changed the boot order to make it the first boot device, but it continually refuses to boot as shown below. Am I missing something?

Just for kicks... I took the 9211-8i card from my original FreeNAS setup (which is already flashed to IT mode for the other system) and connected directly with THESE SAS CABLES and it too has the same boot issue. Keep in mind on the original system I am booting from a single USB flash drive, which to me is a single point of failure, thus wanting to run from the 2x SSD's mounted in the flex bay of the R720xd.

I am going to try returning to the original factory default on the H710 Mini and running a RAID1 on that, then install FreeNAS on the RAID1 virtual disk which I know will boot and it works. Then leave the 9211-8i installed for the remaining drives so FreeNAS will pick them up.

Ideally I would love to just leave the H710 mini in IT mode and use just the single controller for all of this, but please advise if I am missing something to make those drives boot. Thank you in advance!

 

[Samuel Tai]

It’s possible you flashed a BIOS boot ROM, and you’re trying to boot FreeNAS using UEFI, or vice versa.

 

[kjparenteau]

It’s possible you flashed a BIOS boot ROM, and you’re trying to boot FreeNAS using UEFI, or vice versa.

Good point, but that is not the case here. UEFI is not enabled, and the boot options are set in the BIOS appropriately. FreeNAS also installed with BIOS option during the install, and the BIOS boot rom does indeed show up in the boot options as a selectable device to boot from in the BIOS settings of the R720xd. But for whatever reason, the system will not boot to those devices. Thank you for adding additional info for this thread.

 

[Apollo]

Good point, but that is not the case here. UEFI is not enabled, and the boot options are set in the BIOS appropriately. FreeNAS also installed with BIOS option during the install, and the BIOS boot rom does indeed show up in the boot options as a selectable device to boot from in the BIOS settings of the R720xd. But for whatever reason, the system will not boot to those devices. Thank you for adding additional info for this thread.

I would try installing Freenas on only 1 SSD, leaving the other one unattached. I don't know how well will the BIOS handle mirrored disk when it is managed by ZFS in the event of a failure of one of the SSD at reboot.

 

[kjparenteau]

I would try installing Freenas on only 1 SSD, leaving the other one unattached. I don't know how well will the BIOS handle mirrored disk when it is managed by ZFS in the event of a failure of one of the SSD at reboot.

I did try that, but the same issue is present. I think the issue is in how the BIOS is interpreting the HBA's boot options/drives since the install does complete successfully whether it is using either 2 drives or 1.

 

[kjparenteau]

Here's the current summary of what is going on with this... I apologize for the book, but I figured an up to date summary might be beneficial for information / logging purposes...

I was pointed to the article H310/H710/H710P Mini & Full Size IT Crossflashing which worked perfectly for getting the card into IT mode, (thank you for that!) however I am having troubles with booting while in IT mode... And yes, I installed the BIOS Boot image, and it shows as a bootable device in the BIOS settings. I changed the boot order to make sure it was top on the list, but it won't boot to FreeNAS on the R720xd, even when manually choosing the card as the boot device from the BIOS boot menu. So far all of my attempts to make this work the way I have envisioned have failed.

SETUP
R720xd, 12x 3.5" bays in the front with 2x 2.5" flex bays in the rear
2x CPU E5-2630 v2 @ 2.60GHz
384GB RAM
PERC H710 Mini D1
12x 6TB SATA
2x 256GB Samsung 860 EVO

GOAL
Use all 12 bays in the front for storage
Use 2x flex bays with Samsung SSD's in the rear for the FreeNAS OS (mirrored)

ISSUE
FreeNAS does see both SSD's just fine during installation with the IT mode card, and I can select both drives for installation. Install succeeds just fine, but when I reboot, it refuses to boot FreeNAS OS and says it is unable to find a bootable device. Manually selecting the device from the BIOS Boot Menu, or changing the BIOS Boot Order yields the same result.

WORKAROUND
If I install FreeNAS to a USB drive, the system works just fine for adding drives to storage pools, etc., but then the OS is running from USB flash media and not mirrored. I would much prefer to run FreeNAS from mirrored SSD's for better performance and reliability.

NOTE
I do have an LSI 9211-8i PCI-E card flashed to IT mode that I put in to try bypassing the H710 to see if I would have different results, but unfortunately it has the EXACT same issue... It seems like the card itself may not be the issue, but rather how the R710 is interpreting the boot options of the cards.

Another side note... I can revert the H710 Mini to the factory default boot image, and connect the H710 cable's just to the rear flex slot which works for setting RAID1 on the 2x SSD's, and also is bootable, and then using the LSI 9211-8i in IT mode for the front bay storage with different SAS cables. This works, but there are 3 issues I've seen with this route so far... First, the increased fan noise... Secondly, using RAID1 vs ZFS for the FreeNAS OS, and Third, the HDD status lights do not seem to work at all when connected to the LSI card...


Anyway... That is all of the information I know of to date... Any suggestions that I might be missing? Maybe a newer boot image or something for this card that might have improved the ability to boot? I am at a loss and would love some guidance so I can put this beast of a machine into good use.

Thank you for any advice in advance,
Kevin

 

[kjparenteau]

Latest update... So I went ahead and bought a H310 Mini to try and see if it solves the issue of booting to a pair of SSD's for FreeNAS OS. Did a factory reset on the card which set all of the drives to non-raid, and then I selected the boot device (one of the 2 drives I will install FreeNAS to). Sure enough... It worked! And the cool part is the drives show in the iDRAC interface. That being said... There is a new issue present, as well as a question...

First of all, the issue is with SMART... It will not start, even when clearing the disks and recreating the pools, it refuses to start. From what I understand, this will be a deal breaker in determining the setup because I want FreeNAS to report errors and such accurately. So my guess is the H310 is somehow preventing SMART from starting since it's using the standard PERC firmware from Dell maybe? Here's the error I see on the console...

freenas smartd[4414]: Configuration file /usr/local/etc/smartd.conf parsed but has no entries

So functionally, it is now doing what I originally set out to do in terms of booting, mirroring the OS drives, and even reporting to the iDRAC is a neat bonus, but seems like I might need to flash this card to IT mode as well? But then I assume I will end up in the same situation as with the H710 where it will not allow me to boot...

The second question I have is in terms of performance... If the H310 is simply passing the drives through, will the performance be pretty much the same as the H710, or even the LSI 9211-8i card? Essentially they're doing the same thing and all support 6Gbps rates, but maybe I am missing something...

I am so close I can taste it! Can anyone help me by answering the performance question, and if the H310 will work for performance, how can I enable SMART??

Thank you!

 

[kjparenteau]

A quick google turned up this:
Guide: Flashing H310/H710 Mini & full size to IT Mode

This might be the better option?

Thank you for this comment. This link was the most helpful. There was a hidden command set I didn't see previously as it was not clearly defined in the instructions to set a specific drive to boot. It is operational and been holding steady for a few days now. Thank you again for this recommendation, it was exactly what was needed.

 

[sophware]

Hi. Would you mind sharing the hidden command set for setting a specific drive? I have the same setup and same issue.

 

[sophware]

Found it. For anyone else, here is the information: https://www.youtube.com/watch?v=dJyQVLPjWbo&list=UUKHE9DEep52XlmwLbZUKvyw&index=28

 

[JaimieV]

Latest update... So I went ahead and bought a H310 Mini to try and see if it solves the issue of booting to a pair of SSD's for FreeNAS OS.

Rather belatedly, but with my R520 there is a spare mobo SAS socket ("software raid") that you can break out to 4xSATA, or use the two CD/Tape (!) SATA ports to feed the rear 2.5" drive bays. Looks like it may be the same with the R720.