SOLVED Installing FreeNAS-9.10.1 on Dell servers with PERC H730 controller

[ERM-Consulting]

Question to everybody using VMWare:

Is there anybody who can send a BUG-MESSAGE to VMWare? We are using the Essential Kit without separate support contract. We would have to pay 300 Euro for single case support to be able to report the bug.

I can provide further information if needed. But that is definitely a bug in VMWare 7.0

 

[ERM-Consulting]

OK, explanation and conclusion for all of you who read this threat searching for solution:

Three main issues:
1. BE AWARE: VMWare 7.0 does NOT install a scratch Disk if you install on USB-STICK! That means: you will never be able to store all of your configuration and after reboot you will have the old configuration again! Configure your scratch disk: https://blogs.vmware.com/vsphere/2020/07/vsphere-7-system-storage-when-upgrading.html

2. DO NOT USE Dell customized

VMware-VMvisor-Installer-7.0.0-16324942.x86_64-DellEMC_Customized-A02.iso

if you want to passthrough HBA330!
If you attach disks to your controller the passthrough is not possible anymore! Use the original VMWare iso:

VMware-VMvisor-Installer-7.0b-16324942.x86_64.iso

3. If you try to passthrough HBA330 on Dell R730 you will tap into a serious Dell-ISSUE:
During SOME boots (e.g. 1 of 5) your system will hang at
mpr0: Out of chain frames, consider increasing hw.mpr.max_chains.
[...]
mpr0: Calling Reinit from mpr_wait_command, timeout=60, elapsed=60
mpr0: Reinitializing controller

This issue could not be solved from Dell until now (you will find several threats about this issue).
- If you are using HBA330 as PCI-Card: using another port can help.
- If you are using HBA330 MINI: you cannot workaround the issue.

Further information:
As we have two systems with the same configuration, we have tested the speed of disk usage:
1. System: Dell R730, 128 GB RAM, RAID H730, 5 SAS Seagate 8 GB 12 GBit/s, VMware 7.0b,
booted on mirrored ATA HDs (slot 1+2 of Raid),
Xigmanas 12.1.0.4 - Ingva /or/ FreeNAS-11.3-U4.1.iso , 4 CPU, 32 GB RAM, 24 TB Storage on VMWare Datastore.

2. System: Dell R730, 128 GB RAM, HBA330 (passthrough), 5 SAS Seagate 8 GB 12 GBits/s, VMWare 7.0b,
booted on USB-HD,
Xigmanas 12.1.0.4 - Ingva /or/ FreeNAS-11.3-U4.1.iso, 4 CPU, 32 GB RAM, 39(!) TB Storage on Raid-Z2 direct disk management (passthrough controller)

We have tested the speed with:
A. dd if=/dev/zero of=<zfs-pool>/<dataset>/<directory>/output.txt bs=1G count=1
=> Raid slightly quicker

B. dd if=/dev/zero of=<zfs-pool>/<dataset>/<directory>/output.txt bs=512 count=1000
=> Raid 33 times quicker

C. cp <zfs-pool>/<dataset>/<directory>/input.txt <zfs-pool>/<dataset>/<directory>/output.txt
=> 100 GB File: Raid 3,25 times quicker

In our special case it seems to make sense NOT to use passthrough HBA330 but to use Raid H730 in VMWare with Datastore for freenas/xigmanas.

 

[Cptnemo]

1. System: Dell R730, 128 GB RAM, RAID H730, 5 SAS Seagate 8 GB 12 GBit/s, VMware 7.0b,
booted on mirrored ATA HDs (slot 1+2 of Raid),
Xigmanas 12.1.0.4 - Ingva /or/ FreeNAS-11.3-U4.1.iso , 4 CPU, 32 GB RAM, 24 TB Storage on VMWare Datastore.

Can you give some more information about this configuration, would be interesting! Raid amd VMWare settings would be great!

 

[ERM-Consulting]

Best configuration is pretty default:
We are using 3,5 Slots.

Raid via PERC H730:
Slot 1+2: Raid 1 configured via BIOS with any two disks > 80 GB
Slot 3-7: Raid 6 with SAS Seagate 8 GB 12 GBit/s configured later via perc commands (or you do it with the bios-configurator in advance.

VMWare:

1. use Dell-VMWare-Image "VMware-VMvisor-Installer-7.0.0-16324942.x86_64-DellEMC_Customized-A02.iso". Put it onto a stick and boot server from stick.
2. Install VMWare on Raid 1. (takes about 8 GB on Raid 1)
3. Boot VMWare and create datastore on Raid 1 (call it "BootDatastore") > e.g. 40-60 GB
4. Create VM for TrueNas or Xigmanas. Attach Disk (e.g. 40 GB) from "BootDatastore"
5. Install TrueNas or Xigmanas into VM
6. Create Raid 6 => login to VMWare via ssh and use percli command: ./perccli /c0 add vd type=raid6 drives=32:2-6
7. Create Datastore on Raid 6 and assign it to Truenas/Xigmanas VM
8. Boot Truenas/Xigmanas VM and configure

Attention: The idea of ZFS is direct disk management (Passthrough HBA330 controller to Truenas/Xigmanas VM would be correct.)
BUT: For this you have to use HBA330 (JBOD-controller) instead of PERC H730 (Raid-controller). Dell has a runtime issue with HBA330 using VMware with passthrough HBA330 controller - at least if you are using HBA330 MINI. Using PERC H730 just assign the Raid and use ZFS on "single-disk".

Attention2: If you are using MS domain controller on VM instead of integrated SAMBA domain controller, you should install the VM with the MS Domain Controller on "BootDatastore" and start that VM first. For backup reasons you can sync the Domain-Controller to a second DC stored on ZFS after full system boot.

 

[Cptnemo]

Thanks for the inside!
I was a bit wondered because normally everybody is warning to run trueNAS on Hardware RAID... I'm a bit uncertain because standing in front of the same problem like you described. Additionally I do not want to put an extra controller for the OS Storage in ESXi.....

 

[jgreco]

You are absolutely fine running ESXi's datastores on any RAID controller ESXi supports. Using a RAID controller for this, along with RAID 1 or better, is best practice. It has nothing to do with FreeNAS, really, even if FreeNAS boots from a vmdk on the RAID datastore.

It is not absolutely necessary to run ESXi on a RAID controller, as long as you understand that ESXi is funny about writing to datastores that do not include some sort of write acceleration, and of course your VM's will vanish if the directly-attached disk fails.

Most of the FreeNAS hosts here are virtualized on top of an ESXi platform; an LSI HBA is passed through to FreeNAS using PCI passthru, while a RAID controller like a LSI 9270CV-8i handles datastores for ESXi.

The problem with RAID controllers is when you try to use them with FreeNAS. Just please don't.

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

 

[Cptnemo]

The problem with RAID controllers is when you try to use them with FreeNAS. Just please don't.

But this is exactly what is described above, 5 HDDs with external RAID -> ESXi Datastore -> freeNAS as Single Disk ZFS

 

[jgreco]

But this is exactly what is described above, 5 HDDs with external RAID -> ESXi Datastore -> freeNAS as Single Disk ZFS

I don't understand your confusion.

Hardware RAID controllers are generally a terrible choice of way to attach disks directly to FreeNAS.

What is being described above is using VMware virtual disks with FreeNAS. The fact that they're being supplied to ESXi by a RAID controller is irrelevant (well, at least, much less relevant).

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

explains why, and it's largely to do with drivers and SMART monitoring and stuff like that. If ESXi is taking care of the drivers and SMART monitoring, then this just breaks down into my recommended solution for those who absolutely must do that:

https://www.truenas.com/community/t...ative-for-those-seeking-virtualization.26095/

which I described about six years ago, and appears to be exactly what we're talking about.

 

[ERM-Consulting]

But this is exactly what is described above, 5 HDDs with external RAID -> ESXi Datastore -> freeNAS as Single Disk ZFS

Going LIVE we have changed this detail: now ESXI is providing 9 vmdks (2.4 TB) to the NAS VM. We are using 3 vmdks per virtual disk (RaidZ1) and combine 3 virtual disks to one Pool. This allows zfs to maintain the disks.
Please see attached PDF with graphic representation of host/storage.
Transfer rate is only limited by physical disk transfer rate as we are using traditional harddisks.