TrueNAS on ESXi7u2 PE R7525 with AMD EPYC 7352 and Intel/Dell Express Flash NVMe P4610 3.2TB SFF

[TDA IT Dept]

No RAID card. Well there is a BOSS-S1 card for ESXi and TrueNAS OS storage. Idea is to use the ZFS RAIDz for storage for the ESXi datastore via iSCSI on 40Gb NICs + QSFP+ connectors. w/ 2 identical R7525 servers, double everything, NICs, switches all of it.

Disclaimer1: I have read every one of the posts/threads about VM and Free/TrueNAS. And searched for threads on AMD, EPYC, NVMe, ESXi. Happy to read anything else too.
Disclaimer2: This is hypothetical idea from my boss about how to best use this R7525 with light speeds as an ESXi host for ALL our production server VMs probably in HA. He has NOT read any of the forum posts. We are open to other ideas if this isn't going to work. Just seems like it'd be cool and fast af.

So far, installed ESXi7.0u2 on a BOSS-S1 card with two Micron NVMe cards in a RAID1 for OS. All good. Created a VM, gave it a bunch of resources, reserved, etc. Installed TrueNAS 12.0u8. All good so far.

Went into Hardware > Manage PCI Devices > and found the 5 Intel/Dell Express Flash NVMe P4610 3.2TB SFF devices and enabled Pass-Through.
Things started getting a little hairy about here.
The console started posting repeated "Missing Interrupt" messages over the top of the normal console choose options screen output. And alternating "SET FEATURES" and "Invalid Field" stuff with sqid's and cid's (see attached screenshot).
Restarted the TrueNAS server at this point.

Same things after reboot.
In the TrueNAS UI disks see the 5 NVMe drives nvd0-4. See the serial numbers, S.M.A.R.T. (see attached screenshot).

But will not create Pool. just sits there creating pool for long time. Hours.

Did I miss something?
Is this a fool's errand?
Happy to buy tech support from iXsystems. Probably need to do that anyway for our file share and replicants.

Happy St. Patrick's Day good people of the forum!!

Jeff

 

[Nick2253]

Did you try rebooting the ESXi server after doing the passthrough? Are you sure that you don't have any of those drives in use in the ESXi system somewhere?

Some ideas from this thread may help: https://www.truenas.com/community/threads/nvme0-missing-interrupt-after-esxi-and-bios-update.78745/

 

[TDA IT Dept]

Thanks Nick I'll check that out.
No there is nothing on this Host yet. Just ESXi and now a TrueNAS core VM.
Appreciate the info link!

 

[TDA IT Dept]

So, after failing to uncover any reason why TrueNAS Core/FreeBSD did not like my wicked fast NVMe drives I decided to try Scale/Debian as a VM and that appears to have worked great. I was able to turn off the Core VM and remove the drives. Then passthru those drives to the SCALE instance (VM) and that worked fine. Then able to add the drives into a Scale RaidZ-2 and a data pool.
Will make virtual iSCSI between the ESXi host and the TrueNAS-SCALE data pool next. Then try to host some VMs on it.