Mellanox passthrough to FreeNAS guest on ESXi

wormified

Cadet
Joined
Aug 19, 2020
Messages
3
Hi, I have a FreeNAS guest running on ESXi with a single 50 GbE port on a ConnectX-5 card passed through to it. FreeNAS doesn't show the interface and dmesg -a | grep mlx returns:

mlx5_core0: <mlx5_core> mem 0xe4000000-0xe5ffffff irq 19 at device 0.0 on pci6
mlx5_core0: INFO: firmware version: 12.24.1000
mlx5_core0: ERR: enable msix failed
mlx5_core0: ERR: mlx5_load_one failed -6

I've attached a debug dump from the install as well. Any help would be greatly appreciated!
 

Attachments

  • debug-FreeNAS2-20200819091540.tgz
    404.1 KB · Views: 551

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Moderator note: Moved from Networking Forum to Hypervisors subforum because this is not a Networking question, as it is clearly an issue between FreeNAS, ESXi, and a bit of hardware. The fact that the bit of hardware is a network card does not make it a Networking question (sorry!).

Virtualization guru comment: Try disabling msi. See for ex. https://nvidia.custhelp.com/app/answers/detail/a_id/4135/~/vmware-vdga-/-gpu-passthrough-requires-that-msi-is-disabled-on-vms but your specifics may be different.
 

wormified

Cadet
Joined
Aug 19, 2020
Messages
3
I'm sorry! Will try to be more mindful of the categorization in the future.

I tried disabling msi for all passthru devices (for simplicity) and it didn't affect this behavior. One thought is that perhaps the Mellanox drivers expect msix? That seems to be the case for some devices with Windows guests:


There also seems to be some discussion of whether or not MSI-X and its implementation in BSD vs ESXi is broken in some way? Although this discussion is a bit beyond me tbh:

The MLX4 driver documentation suggests it's possible to disable MSI-X:
But I'm not sure how this interacts with passthru, especially since I'm only passing through one port on a two port adapter.

This card was working under ESXi and with traffic routed through a vswitch, but FreeNAS saw that portgroup only as 10 Gb and I get line rates at that speed (~1.1-1.3 GB/s). I'm hoping to get more out of the 50 Gb pipe that's available hence why I'm trying to pass the 50 Gb port in directly, but I wonder if I need to either pass the whole card through or enable SR-IOV (which I can't without upgrading beyond my essentials license) to do this properly. At that point it would be cheaper just to add an additional 50 Gb card dedicated for the FreeNAS guest.
 

wormified

Cadet
Joined
Aug 19, 2020
Messages
3
Confirmed that this error persists even when passing through both ports of the same card.
 

lipei

Cadet
Joined
Jun 9, 2021
Messages
1
I have the same problem like you. After doing a lots of research I tried put following variable to Tunables and it just fix the MSI-X problem.
variable hw.pci.honor_msi_blacklist
value 0

It is a surprise to me that an old bug causing FreeBSD all those years to stop using MSI-X in esxi.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I have the same problem like you. After doing a lots of research I tried put following variable to Tunables and it just fix the MSI-X problem.
variable hw.pci.honor_msi_blacklist
value 0

It is a surprise to me that an old bug causing FreeBSD all those years to stop using MSI-X in esxi.

That's sort of mischaracterizing the problem. MSI/MSI-X handling is just generally messy, and some combinations of some things sometimes don't work, because PC hardware sucks (in my opinion), but lots of stuff also DOES work. The FreeBSD bug thread above is an enlightening case of not-entirely-undeserved finger pointing while also giving some idea as to just how complicated it is to do these things correctly.
 
Top