vmxnet3.ko for FreeNAS 9.X in ESXi 5.5

Status
Not open for further replies.

David E

Contributor
Joined
Nov 1, 2013
Messages
119
Updated the first post with a 9.3 module, and integrated ajohnson's great build suggestions into the instructions.
 
Joined
Dec 24, 2013
Messages
3
Thanks for all your work David; I've been using your compiled modules for close to a year now!

In case anyone else experiences this - I updated to 9.3-STABLE-201501151844 last night and lost the vmx3f0 interface. I have not tried to troubleshoot the issue; this is the first time it hasn't worked after a 9.3 update. Rebooting into the previous version (9.3-STABLE-201501090144) worked flawlessly and immediately fixed the issue.
 

David E

Contributor
Joined
Nov 1, 2013
Messages
119
Thanks for all your work David; I've been using your compiled modules for close to a year now!

In case anyone else experiences this - I updated to 9.3-STABLE-201501151844 last night and lost the vmx3f0 interface. I have not tried to troubleshoot the issue; this is the first time it hasn't worked after a 9.3 update. Rebooting into the previous version (9.3-STABLE-201501090144) worked flawlessly and immediately fixed the issue.

You are very welcome! Glad it is useful.

Per your specific problem, if the kernel interface changes at all it can break the bindings that the VMware driver was compiled against causing the problems you are seeing. Likely if you are using these weekly releases you may have to compile against each specific version you are using.
 

fisherwei

Dabbler
Joined
Oct 28, 2013
Messages
47
Greeting David,

Today, I upgrade my nas from 9.2.1.5 to FreeNAS-9.3-STABLE-201501212031, and use 9.3 vmxnet3.ko which is downloaded from this thread.
But no luckly, it cannot work.

So, is that mean, I should install a freeBSD 9.3 VM and checkout freenas from git, and complie it by myself?
 

sifebr

Cadet
Joined
Aug 28, 2014
Messages
2
The vmxnet3.ko in VMware's ESXi 5.5 tools does not work with FreeNAS 9.2.X, I have recompiled the (unmodified) source code for this driver, and am making available a ready to go kernel module for the FreeNAS versions listed below. You will want to temporarily use the E1000 NIC type so you can copy the driver into FreeNAS, then change it to VMXNET3 on the next boot.

For instructions on how to install the driver, see the latter part of the following post:
http://itanalyses.blogspot.com/2013/02/jumbo-frames-in-virtual-freenas-831-on.html

Instructions for compiling yourself
Rough guide of the necessary steps, substitute 9.2.1-RELEASE below with the FreeNAS version you are interested in. Updated to include ajohnson's faster build below:
  • Set up a FreeBSD 9.2 VM in VirtualBox
  • Create and attach a second hard drive to the VM (20G or so)
    • Find the disk, ls /dev look for ada*
    • zpool create tank ada1
  • Install Dependencies
    • cd /usr/ports/devel/git
    • make install clean
    • cd /usr/ports/sysutils/cdrtools
    • make install clean
    • cd /usr/ports/ports-mgmt/pbi-manager
    • make install clean
  • Check out FreeNAS
    • cd /tank
    • git clone git://github.com/trueos/trueos.git
    • cd trueos
  • Find version to build against
    • git tag -l
    • git checkout 9.2.1-RELEASE
  • Unpack vmware tools
    • scp the tools to the vm in /root (vmware-freebsd-tools.tar.gz)
    • cd /root
    • gunzip vmware-freebsd-tools.tar.gz
    • tar xf vmware-freebsd-tools.tar
    • tar xf /root/vmware-tools-distrib/lib/modules/source/vmxnet3.tar
  • Build
    • cd /tank/trueos
    • make buildenv
    • cd /root/vmxnet3-only
    • SYSDIR=/tank/trueos/sys make
Resulting kernel module will be in /root/vmxnet3.ko

Available Kernel Modules
  • ESXi 5.5 FreeNAS 9.3 x64 shasum: 4ea6e8dc8069fbcbb735599bf76286bb8a58b721
  • ESXi 5.5 FreeNAS 9.2.0 x64 shasum: 831f99c967713c88a02fa240c2542cf3b41cbd7d
  • ESXi 5.5 FreeNAS 9.2.1 x64 shasum: 69baa7d0621ee604277cb6768a504f79cb72b872
  • ESXi 5.5 FreeNAS 9.2.1.1 x64 shasum: 378cf7b6d4267991b31e412cc90ddc4b1a6afb59
  • ESXi 5.5 FreeNAS 9.2.1.3 x64 - use the module from 9.2.1.1
  • ESXi 5.5 FreeNAS 9.2.1.6 x64 shasum: ff156321029d71f353f8752878a2e6d157f93f22

Thank U so much ... Finally I could use the vmxnet3 NIC , The performance of E1000 sucks

BTW, I'm a guy from china ,Thanks again!!!!
 

David E

Contributor
Joined
Nov 1, 2013
Messages
119
Greeting David,

Today, I upgrade my nas from 9.2.1.5 to FreeNAS-9.3-STABLE-201501212031, and use 9.3 vmxnet3.ko which is downloaded from this thread.
But no luckly, it cannot work.

So, is that mean, I should install a freeBSD 9.3 VM and checkout freenas from git, and complie it by myself?

Did you get an error printed to the console or logs when starting it?

Has anyone else had a problem?
 

fisherwei

Dabbler
Joined
Oct 28, 2013
Messages
47
Did you get an error printed to the console or logs when starting it?

Has anyone else had a problem?


it is OK, after upgrading freenas to the latest version, and remove/re-add network interface by vsphere client.
 
Last edited:

deasmi

Dabbler
Joined
Mar 21, 2013
Messages
14
Has anyone else had a problem?

I've seen the same.

A clean install today of FreeNAS-9.3-STABLE-201502060108 will not work with the driver.
Running a full update and the module loads and see the interfaces.
I didn't need to remove and re-add interfaces.
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
I just realised that my VMXNET3 seems to crash under heavy load. I'm using the original VMWare vmxnet3.ko from the latest tools and I tried the one I can download here. E1000 works flawless (but slow) at any time.
Upgrading to the latest version FreeNAS-9.3-STABLE-201503200528 did not fix the problem.
Booting with FreeNAS-9.3-STABLE-201502210408 solved the problem.


What has changed after FreeNAS-9.3-STABLE-201502210408 or the following version?

Regards,
Nik
 

David E

Contributor
Joined
Nov 1, 2013
Messages
119
So it looks like they have moved to a new mechanism for releases, where they are much more frequent, and aren't receiving new revision numbers, just dates. I updated the original post, but you have to figure out the date of the release you are upgrading to, open up its MANIFEST file and find the Git revision number, then build the kernel for precisely that revision. I just upgraded to 9.3 myself, so I haven't seen what the in-GUI update mechanism looks like, does anyone know if it lists a date there? I'm a little bit frustrated by this release process since it means everything will be far more frequent, with less end user communication about each one, and I have to assume less testing. So far the module I built for the latest rev seems to be working, but it will need more testing. If anyone has been having problems, feel free to update to the rev posted in the OP and let me know how it goes. And keep in mind, you can't take arbitrary updates anymore without possibly breaking your network :(
 

Nanderson87

Cadet
Joined
Jun 11, 2015
Messages
1
Just wanted to offer my thanks here and bump this thread. My ESXi host was getting crashes from my FreeNas VM due to, what seemed like, the E1000 NIC adapter. Had to switch to vmxnet3 adapter to try and clear it up. OF course, even in the nice new 9.3, the vmxnet3 isn't recognized. This guide along with your link fixed me right up, and I'm about the farthest thing from a Linux guy you can get.

I downloaded your stable release, unzipped and put into an .iso. Mounted the .iso to the virtual machine, mounted it to a cd drive folder in the FreeNAS shell, copied the vmxnet3 file from the cd folder to /boot/modules, added the tunable per your guide, rebooted and it sees them just fine. Had to reconfigure the adapter since it replaced an E1000, then reboot once more, and we're back in business.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Thank you for this post!

I have been testing FreeNAS-9.3-STABLE-201506042008 running on VMware ESXi 6.0.0 build 2715440, and the VMware-supplied VMXNET3 driver does indeed fail under load when used for either NFS or iSCSI datastores.

My test is simple: I spin up two Windows VMs and simultaneously run the ATTO disk benchmark on them. Things seem great at first, with incredibly fast transfer rates. But inevitably the underlying datastore goes South and the benchmarks stall.

With iSCSI-based datastores I see this sequence of FreeNAS console messages:

Code:
WARNING: 10.0.58.1 (iqn.1998-01.com.vmware:554fa508-b47d-0416-fffc-0cc47a3419a2-2889c35f): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 10.0.58.1 (iqn.1998-01.com.vmware:554fa508-b47d-0416-fffc-0cc47a3419a2-2889c35f): connection error; dropping connection


With NFS-based datastores there are no console or error messages or any other indication of a problem, except that the vSphere client shows the NFS datastore in italicized typeface when it goes off-line. Oddly, despite their underlying datastore being off-line, the Windows VMs don't halt or crash.

Eventually the datastores 'wake up' and the Windows VMs respond to input. The benchmarks then run to completion, albeit very slowly...

Today I will try to find time to compile the driver using these instructions and see if these problems persist.

EDIT: Let me add that when I move the datastores over to an E1000-based configuration, the test case works fine, both for NFS and iSCSI. But both have slower and/or less consistent read rates coupled with abysmally slow write rates.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Anyone have any news on this?
I was able to successfully compile and then test the vmxnet3.ko module, but to no avail; I still got the same errors I described above.

It turns out that FreeNAS 9.3 does not support VMware's VMXNET3 adapter, though support is promised for version 10. So I have dropped back to using the E1000 adapter, which works fine.
 

hpnas

Dabbler
Joined
May 13, 2012
Messages
29
What an odd claim. It is faster than the VMware drivers in some cases.

If you want 10 Gbps then you require the VMXNET3. E1000 is only 1 Gb. Typically there are not a lot of reasons to use E1000. ..

Looking forward to a stable relate.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
E1000 is not "only 1Gb." It'll happily plow through as fast as it can. I've posted numbers here in the past showing it beating VMXNET3 in various scenarios.

We had horrid problems with VMXNET3 misfeatures/stability at one point and were surprised to find that E1000 was rock stable.
 

Xelas

Explorer
Joined
Sep 10, 2013
Messages
97
With many SuperMicro boards sprouting 2 NICs, why not passthrough (VT-d) one of them to freenas so that it gets it's own dedicated hardware, real, NIC? If you need to, you can then create a virtual "bridge" vswitch that isn't attached to any physical NICs and use it only for passing traffic between other VMs and freenas (only to keep that traffic off the physical interface), and use the other physical NIC with another vswitch for the ESXi host and the other VMs to use for network/internet access.

I think the additional benefit is that you know that your freenas config works with the physical NIC in your box - that way, if/when ESXi goes south or Yet Another Update for either ESXi or Freenas breaks vmxnet3 or something, you just boot from a USB with the identical config file that you keep updated and backed up (right? of course you do!), and you can just truck along.

You obviously need to know what you're doing to avoid network loops, and you'll probably need to create the "bridge" virtual switch and set client VMs up with with static IPs (on a different subnet, I think) since it cannot easily be connected to a DHCP server without some hassle. It also entails some additional configuration.

Thoughts?

UPDATE - I went ahead and added the 2nd NIC to passthrough, and added it to Freenas. 1 reboot later, it's running just fine after the usual configuration on the console (set IP, etc). I also disconnected the vmxnet3 NIC from any active physical interfaces to avoid network loops, and set it up with a static IP on a separate vswitch (that currently doesn't have anything else on it).

Now my server has 3 network cables - 1 IPMI, 2 for the VMs.
 
Last edited:
Status
Not open for further replies.
Top