SOLVED Installing FreeNAS-9.10.1 on Dell servers with PERC H730 controller

[chownp]

After some newbie pain installing FreeNAS on a Dell server with PERC H730 controller https://forums.freenas.org/index.ph...-freenas-9-10-1-running-on-dell-r730xd.46594/ please find herewith some tips.

The PERC H730 is in the HCL for FreeBSD https://www.freebsd.org/relnotes/9-STABLE/hardware/support.html#disk so should work.

Item no 1 on your installation check list should be to enter the BIOS and ensure the controller is set to HBA mode.

From the BIOS select:
Device Settings - Integrated RAID Controller1 - Controller Management
Scroll to bottom of pane and select: "Advanced Controller Management"
Click "Switch to HBA Mode"
Re-boot the server for the mode change to take affect


Unfortunately that's the easy bit.

The HCL states that the PERC H730 controller is supported by the mrsas(4) driver. My experience has been that FreeNAS loads that driver but then defaults to using mfi.

If you are truly using the H730 in HBA mode then the devices listed by camcontrol will be of the form da<n> e.g.

Code:

# camcontrol devlist
<HGST HUS726060AL5214 KK06>  at scbus1 target 0 lun 0 (pass0,da0)
<HGST HUS726060AL5214 KK06>  at scbus1 target 1 lun 0 (pass1,da1)
<HGST HUS726060AL5214 KK06>  at scbus1 target 2 lun 0 (pass2,da2)

NOT

Code:

# camcontrol devlist
<HGST HUS726060AL5214 KK06>  at scbus0 target 0 lun 0 (pass0)
<HGST HUS726060AL5214 KK06>  at scbus0 target 1 lun 0 (pass1)
<HGST HUS726060AL5214 KK06>  at scbus0 target 2 lun 0 (pass2)

The mrsas(4) manpage https://www.freebsd.org/cgi/man.cgi?query=mrsas&sektion=4&manpath=freebsd-release-ports gives some useful tips on setting the priority of the device drivers.

To ensure the mrsas driver is being used edit /boot/device.hints, add the following line and reboot

Code:

hw.mfi.mrsas_enable="1"

 

[Ericloewe]

Hmm, so the LSI SAS2208 and SAS2308 (the latter when in HW RAID mode) also support mrsas? Fascinating.

 

[kossuth]

This is a super helpful piece of info. Additionally want to add this. If you have a NAS built using a PERC H730 disk controller and continually either have SMARTD fail or it alarms in the panel because it can't check the status of the disks your issue may reside with the mrsas driver. Same fix as above in my case.

 

[ovizii]

So basically with these modifications FreeNAS runs fine on a PERC H730 in HBA mode?
I'm asking because I am looking for a new server and I only have the choice between Dell PERC H330 or Dell PERC H730 and the threads I found so far are confusing.

 

[chownp]

So basically with these modifications FreeNAS runs fine on a PERC H730 in HBA mode?
I'm asking because I am looking for a new server and I only have the choice between Dell PERC H330 or Dell PERC H730 and the threads I found so far are confusing.

I've been running FreeNAS 9.10 on a PERC H730 for 6 months without issue once I'd nudged FreeNAS in to using the mrsas driver.

The PERC H730 is built round the LSI SAS 3108 controller which is supported by the mrsas driver as noted on the man page I linked in my previous post above.

The PERC H330 is based on the LSI SAS 3008 which is also supported by the mrsas driver. So I think either should be viable (but I stress have no experience of the H330).

I'm a relative FreeNAS newbie and don't have the experience of a range of hardware to be making recommendations but if you do go for an H330 then I would assume that a better bet would be the HBA330 which is a non raid (which you were only going to turn off anyway) card built around the LSI SAS 3008.

 

[ovizii]

Thanks for the feedback. I am looking to rent a server online for a specific project and those 2 controllers are the only options right now. I'm not even going to run FreeNAS on there, its just that the FreeNAS forum had the best info on controller compatibility with ZFS

 

[Rhino81]

I know this is old. Did you have all other options on this card disabled, write cache for example? Or does that matter?
I've got the MrSAS driver setup as above and really appreciate your guide!

 

[Chris Dill]

I just installed 11.2 on PERC H730 after setting HBA, and it came up as da<n> successfully without further configuration.

 

[MRBIQ]

I just installed 11.2 on PERC H730 after setting HBA, and it came up as da<n> successfully without further configuration.

I have H730 min so need any settings or only make it HBA and everything will work perfect , one more thing if I have H730 can use raid + HBA modes together in sometime or will not work some of them say make your raid and other drives make it no-raid so you will see them in vm

 

[Chris Dill]

I have H730 min so need any settings or only make it HBA and everything will work perfect , one more thing if I have H730 can use raid + HBA modes together in sometime or will not work some of them say make your raid and other drives make it no-raid so you will see them in vm

Sorry man, it's been 4 years. I don't even remember the server this was done on. I do know that I am currently running Freenas 11.3 on a Dell R6 or 700 series. with RAID controller, but I use FreeNAS for RAIDZ.

 

[MRBIQ]

Sorry man, it's been 4 years. I don't even remember the server this was done on. I do know that I am currently running Freenas 11.3 on a Dell R6 or 700 series. with RAID controller, but I use FreeNAS for RAIDZ.

I said also this before my commnet maybe will never remember any way thnaks for your commnet ,

 

[ERM-Consulting]

Advices from CHOWNP are very helpful!
Using Dell R730 with Raid H730 Mini we testet with recent freeNAS release, build on FreeBSD 11.3-RELEASE-p11.
=> direct installation on USB-stick immediately accesses RAID (running in HBA-mode).
BUT:

1. speed is VERY BAD (150.000MB/s transfers) [We have testet RAID and SEAGATE disks in 12GBit/s mode before].
2. using VMWare 7.0 for booting the server and passthrough the RAID: controller is recognized on freeNAS-boot, but no access to controller nor disks after boot (e.g. camcontrol devlist does not show the discs)
3. using VMWare 7.0 and using controller in RAID-mode: 12 Gbit/s in VMWare esxi host, but only 300.000MB/s in freeNAS-VM. Its the same result if we use the disks in a RAID and attach the VMWare storage to the VMWare or if we configure JBOD-disks and attach single disks to the VMWare.

I think we are missing a bit. The FreeBSD mrsas driver is supporting 12Gb/s (see mrsas(4)) but we get 0,3Gb/s as a maximum speed

 

[Ericloewe]

Why not just replace it with an HBA330? It's a drop-in replacement. And cheap, last time I checked.

This honestly sounds like a hypervisor problem, since you mention that using hardware RAID on the hypervisor and giving FreeNAS a virtual disk is equally bad. Either that or you have failing hardware in the chain that went bad after you got the acceptable performance numbers.

 

[ERM-Consulting]

I reply backwards:

Either that or you have failing hardware in the chain that went bad after you got the acceptable performance numbers.

Rebooted from VMWare-stick - full transfer speed given:
Drive /c0/e32/s2 Device attributes :
==================================
SN = ZA19GCMG
Manufacturer Id = SEAGATE
Model Number = STMPSKD1CLAR8000
NAND Vendor = NA
WWN = 5000C50094D74014
Firmware Revision = UN0B
Firmware Release Number = N/A
Raw size = 7.277 TB [0x74702556 Sectors]
Coerced size = 7.276 TB [0x746e0000 Sectors]
Non Coerced size = 7.276 TB [0x746e2556 Sectors]
Device Speed = 12.0Gb/s
Link Speed = 12.0Gb/s
Write Cache = Disabled
Logical Sector Size = 4 KB
Physical Sector Size = 4 KB
Connector Name = 00

Why not just replace it with an HBA330? It's a drop-in replacement. And cheap, last time I checked.

It does not seems to make sence to buy a HBA330 as the H730 can be used in HBA-Mode too. freeNAS finds the controller and FreeBSD says:
Invader/Fury 12Gb/s MR controller: DELL PERC H730/P is supported

This honestly sounds like a hypervisor problem, since you mention that using hardware RAID on the hypervisor and giving FreeNAS a virtual disk is equally bad.

Yes, there is any issue that we want to solve.
BUT:
To exclude a hypervisor problem we booted freeNAS directly from USB-stick: that should give us full access to the controller. I mean: it is a FreeBSD that has a 12Gb/s device driver for the PERC H730. That should be solvable issue

Glad to get any further advices!

 

[jgreco]

It does not seems to make sence to buy a HBA330 as the H730 can be used in HBA-Mode too. freeNAS finds the controller and FreeBSD says:

The following was written for people in your situation:

https://www.ixsystems.com/community...bas-and-why-cant-i-use-a-raid-controller.139/

There is no such thing as an "HBA mode". Either it is running LSI IT firmware or it is not. If it is not, it will not work properly with FreeNAS. This isn't a topic that is open to debate. It doesn't matter that FreeBSD "works" with it. It doesn't **WORK** with it in the manner ZFS needs.

 

[Ericloewe]

Fundamentally, even in the best-case scenario, with mrsas talking to stock LSI firmware on an SAS3108 or similar chip, an SAS3008 in IT still has way more hours behind it doing ZFS than anything else in the SAS world. And your situation is not ideal. Dell has a... let's call it comprehensive management solution. They're likely to have customized their RAID firmware to support this solution, and the last thing they care about is making sure that direct-attach disks work correctly in every situation. They'll gladly sell you an HBA330 for that (but do yourself a favor and get one from a reputable reseller and save hundreds of bucks).

 

[ERM-Consulting]

... we have ordered a DELL HBA330 ...
So first thing to do when it arrives is to flash the firmware?
Do you have a recommended link?

 

[ERM-Consulting]

OK, Dell R730 with HBA330 is up and running. Starting freeNAS from USB-stick: HBA330 Controller is found and drives are usable.

BUT:
The idea was: booting R730 with VMWare 7.0, passthrough HBA330 to freeNAS. So we can use RAM and CPU for ESXI and Controller with drives for ZFS. So we started the server from USB-stick with esxi 7.0 in a second step.
1. If we do not attach any drive to the controller, we can passthrough HBA330 to VM (freeNAS)
2. If we attach ANY disk to the controller, we cannot passthrough HBA330 anymore. (we tried simple data-disks, boot-disks etc. - as soon as we attach a disk, VMWare 7.0 does not passthrough the HBA330 anymore. (We still boot from USB-Stick!)

Many people described similar installations with ESXI (6.5/6.7) and passthrough to freeNAS. Any idea how to solve this issue?

 

[ERM-Consulting]

Done - so far:

This is the first time I can see the disks in XigmaNAS-VMWare 7.0 - and 12GB/s transfer rate.
Steps:
1. didn't used esxi-Web-Client but vSphere-Client to assign passthrough PCI-Device to VM
2. didn't attached disk to the controller during start of VMWare
3. started VM with XigmaNAS
4. attached disk to HDA330 controller (insert disks in hotswap-bay)

Main issue still has to be tested and solved: After reboot (with disks attached) the PCI device is not passthrough anymore:
VM-Settings:

Host -> PCI Devices:

No Passthrough possible with attached disks:

Any advices or tipps? Be aware: this is for VMWare 7.0

VMWare has changed the passthrough technologies and there seems to be an issue or it is just missing documentation.
Trying to configure the passthrough in /etc/vmware/esx.conf is removed after restart of the host.
???... may be I have an issue with saving configuration settings in VMWare 7.0 ... ???

 

[ERM-Consulting]

... sorry for pasting with XigmaNAS: we have configured both: freeNAS and XigmaNAS: same behavior on both systems. I did not wanted to generate the screenshots doubled for both systems. ...