VSphere 7 SR-IOV Intel 10GB problems.

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
Okay, I did a search for sr-iov and did not come up with any recent helpful threads.

I have a second instance of FreeNAS running as a VM on Vsphere 7.

The configuration for the VM was initially 2 VMX3 interfaces, and a LSI card passed through. For fun, I switched from the regular pci passthrough, to dynamic pci passthrough. I disabled the virtual interfaces, and passed through 2 of the virtualized interfaces (sr-iov versions), and did the same for the LSI controller.

The LSI controller works just fine (and I have been able to assign it to multiple machines, but obviously not both on at the same time.. well I have not tried that oddly enough.. another exp for another day).

However, the intel interfaces are not working.
"IXV0 MSI-X config error"
is what gets thrown.

Any ideas?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
I'm trying to cross compile the latest intel driver source (dated May this year). Another solution may be the VMWare native driver sucks, so may need to install Intels vmware driver.
 

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
Whelp... that did not work. Neither did vanilla FBSD12. Centos worked fine, but Ubuntu did not... This is going to be fun, and it's not even a random crappy nic, well known popular intel 82599 based chipset (HP 560 FLR cards)
 

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
I got it to work.

I'll post a writeup later, but it was a pain. The intuitive way of pci passthrough that we all use for HBA cards.. does not work for network cards when using SRIOV. It turned out I was fighting the wrong fight, it had less to do with FreeNAS (and FreeBSD), and more to do with the way to configure the devices in VMWare.. if you don't have the documentation up for Vsphere and the bug reports/posts for FreeBSD as it relates to SRIOV
 

warriorcookie

Explorer
Joined
Apr 17, 2017
Messages
67
I got it to work.

I'll post a writeup later, but it was a pain. The intuitive way of pci passthrough that we all use for HBA cards.. does not work for network cards when using SRIOV. It turned out I was fighting the wrong fight, it had less to do with FreeNAS (and FreeBSD), and more to do with the way to configure the devices in VMWare.. if you don't have the documentation up for Vsphere and the bug reports/posts for FreeBSD as it relates to SRIOV

Would be great to know how you did this. I'm fighting the good fight on this right now...
 

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
I'm assuming you have enabled sriov on the pci devices you want to pass through?
Originally I was trying to add a pci device, and selected the virtual device (I picked 5 per 10G card).. and it would never work. After much googling.. and failures on getting it to work on multi OS's, I came across the fact that the devices are blocked, and that I was doing it completely wrong.

For FreeBSD (FreeNAS or generic), you need to add the sysctl (FreeBSD) or under System/Tunables (FreeNAS) hw.pci.honor_msi_blacklist=0 (Apparently the Intel card I have is blacklisted, there are some reasons behind it, not sure, but the above makes them work)
Then under the VM, add a new Network card, not a PCI device, expand the network device section in VMWare where you usually select VMX or E1000 network device, and there is an option for SRIOV in the drop down, pick that. Save your settings and it should work.
 

warriorcookie

Explorer
Joined
Apr 17, 2017
Messages
67
For FreeBSD (FreeNAS or generic), you need to add the sysctl (FreeBSD) or under System/Tunables (FreeNAS) hw.pci.honor_msi_blacklist=0 (Apparently the Intel card I have is blacklisted, there are some reasons behind it, not sure, but the above makes them work)
Then under the VM, add a new Network card, not a PCI device, expand the network device section in VMWare where you usually select VMX or E1000 network device, and there is an option for SRIOV in the drop down, pick that. Save your settings and it should work.

Sorry, not meaning to hijak your thread, If the conversation goes much deeper than this I can start my own:

Card is Intel x520-DA1

SR-IOV is enabled on host and I was able to successfully configure it for a Windows 10 VM as a SR-IOV network device.

Doing the same for FreeNAS, it does not show up in my list of adapters in the gui.
demsg shows the following:
Code:
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.5.13-k> mem 0xfd1f8000-0xfd1fbfff,0xfd1fc000-0xfd1fffff at device 0.0 on pci6
ixv0: MSI-X config error
device_attach: ixv0 attach returned 6


The outcome is identical wether the hw.pci.honor_msi_blacklist=0 sysctl tunable is set or not.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Last edited:

warriorcookie

Explorer
Joined
Apr 17, 2017
Messages
67

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
:)

It took me a while to finally find that, there are not too many references to it. It needs to be set on boot, so loader.conf in /boot or in FreeNAS in the tunable section.
 

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
My bad on the sysctl.. it can be set after boot, then unload and reload if_ix kernel module... but that makes no sense when you can just load it at boot. It was late/early in the morning.
 

warriorcookie

Explorer
Joined
Apr 17, 2017
Messages
67
My bad on the sysctl.. it can be set after boot, then unload and reload if_ix kernel module... but that makes no sense when you can just load it at boot. It was late/early in the morning.
Haha, no problem. Biggest thing was I couldn't figure out why I could see it in demsg but it wasn't loading the driver. Why would it be blacklisted?

It's working, but iperf performance is disappointing. Using vmxnet3 I was seeing 6Gbps, now with sr-iov it's just over 3Gbps. I was expecting a substantial increase in performance, not decrease....

I'll have to play with it this weekend and if I can't figure it out I'll start a new thread. Thanks for your help.
 

ljvb

Dabbler
Joined
Jul 14, 2014
Messages
30
Haha, no problem. Biggest thing was I couldn't figure out why I could see it in demsg but it wasn't loading the driver. Why would it be blacklisted?

It's working, but iperf performance is disappointing. Using vmxnet3 I was seeing 6Gbps, now with sr-iov it's just over 3Gbps. I was expecting a substantial increase in performance, not decrease....

I'll have to play with it this weekend and if I can't figure it out I'll start a new thread. Thanks for your help.
I am actually seeing the opposite. using replication between 2 FreeNAS servers, one bare metal, one VM with SRIOV network cards, I am seeing around 6 to 8G transfer rates, whe using VMX, I was seeing 2 to 3G. What switch are you using. I have 2 Brocade ICX6610s
 
Top