FreeNAS and x552 SFP+ via ESXi Pass-though

[zey]

So I've done some research and haven't exactly found my issue. However I have found some starting about a year ago with Intel's x552 SFP+ on FreeNAS as a VM on ESXi. I'm looking for some input.

I've set the interface to pass-through to the FreeNAS VM but I don't see it as an option after it boots. I do see it while booting but I'm fairly new to FreeNAS.

 

[bigphil]

Are you trying to use SR-IOV or passthrough of the entire network controller? If attempting to use SR-IOV, you'll likely need to build the newest driver from Intel, here, to get it to work correctly...or you've configured something wrong (like you enabled the device for PCI passthrough, but never added it as a PCI device to the VM). SR-IOV would be the way to go though...assuming all of your hardware supports it.

 

[zey]

Are you trying to use SR-IOV or passthrough of the entire network controller? If attempting to use SR-IOV, you'll likely need to build the newest driver from Intel, here, to get it to work correctly...or you've configured something wrong (like you enabled the device for PCI passthrough, but never added it as a PCI device to the VM). SR-IOV would be the way to go though...assuming all of your hardware supports it.

I'm new to pass-through. All I did was enable it from the hardware tab. How would I go about using SR-IOV?

BTW, I'm using a Supermicro X10SDV-7TP4F

 

[bigphil]

It looks like all of your components support sr-iov. You'll need to make sure the features are enabled in your mo-bo BIOS (vt-x and vt-d enabled in processor settings, sr-iov enabled in pci settings). After that, you'll need to logon to the vSphere web client and enable sr-iov for the physical network adapter. The method depends on if you use vCenter or not. If using vCenter, you can enable sr-iov by going to "physical adapters" on the ESXi host and editing the settings for the physical adapter you want to enable. There is an sr-iov column that will show disabled/enabled for supported adapters. If NOT using vCenter, you may logon to the ESXi host ui (https://host/ui) and go to Host > Manage > Hardware > PCI devices > configure sr-iov. After all that is done, you can edit the hardware settings for a VM and add a "shared pci device." The vf device should then appear upon bootup of the vm. If you do all of that and the vf nic doesnt appear in the FreeNAS vm, you'll need the latest vf driver from Intel. Also...don't configure the entire physical network card for passthrough in ESXi...sr-iov is different and uses its own settings.

 

[zey]

Ok, so I went ahead and checked the BIOS settings and verified that I have VT-d enabled but I couldn't find VT-x. ASPM and SR-IOV are enabled. Now I'm getting an error when I attempt to configure SR-IOV on any of the two X552 10 GbE SFP+. I tried enabling SR-IOV with it in active passthrough and disabled passthrough. I still receive the error.

Code:

Update Passthru Config
Key
haTask-ha-host-vim.host.PciPassthruSystem.updatePassthruConfig-198981845

Description
Update PCI passthrough device configuration

State
Failed - An error occurred during host configuration.

Errors

Which means what? I hate that about ESXi.

 

[bigphil]

I haven't seen that error. Couple of things to check. Is ESXi up to date (latest build #6765664), is the ixgbe driver updated (either 4.5.1 or 4.5.2)? If you are at those build numbers and it still doesn't work, you can try the ixgben native driver. The VMware kb says that sr-iov isn't available in the native driver, but it's been updated quite a few times so it's worth a shot to try if the previous option doesn't work out. Also not sure if you can update the Intel NVM/firmware image on the card, but worth a check.

 

[zey]

Thanks a lot for the assistance, I'll look into the drivers. As for ESXi I just downloaded its newest image (6.5.0 Update 1 (Build 6765664). This is a brand new build.

Just to be sure do I set the NIC to passthrough then configure SR-IOV? I'm pretty sure that a yes but I don't want to assume.

 

[bigphil]

No, don't configure for pass through as that passes the entire controller through to one vm. By using sr-iov instead enables you to use it for multiple vm's. Only enable sr-iov.

 

[zey]

Looks like the drivers ESXi was using, did not have SR-IOV enabled/support. I installed the latest drivers (net-ixgbe_4.5.3-1OEM.600.0.0.2494585) via ssh/esxcli. I was able to enable SR-IOV.

I added it to the VM (PCI Device). However I am not seeing the interface when I'm FreeNAS' web client nor console. Will I need to install the FreeBSD drivers? I feel like that should already be part of FreeBSD 11.