NFS ESX datastore disconnects with LACP on FreeNAS

[cpfarhood]

I setup a 4x1GB LAG/LACP link to a Dell PowerConnect 5324 switch and all of my XBMC/Openelec front ends and FreeBSD/Linux virtual machines are able to attach to NFS perfectly but for some reason my ESXi server no longer wants to stay connected to the NFS datastore. It will connect when the host boots and dies immediately after I power on a VM with the VMs then showing inaccessible. I ran a crossover direct from ESXi to FreeNAS and can attach to the same NFS datastore flawlessly. After nearly a whole day of tinkering with settings I'm at a loss. Ideas?

FreeNAS 9.2.1.3 - HW:
SuperMicro X8DTL-iF
2x E5645 CPUs
24GB Ram
Intel i340-T4

ESXi 5.5 - HW:
ASRock H87M Pro4
Core i7 4770k
32GB Ram
Intel i340-T4

 

[Serverbaboon]

The fact connecting directly solves the issue obviously points the LACP setup. I am using LACP on my Freenas with a DLINK switch with no obvious problems and with Vmware (5.1) I am running an unnecessarily complex network setup with ISCSI MPIO and NFS on the ESXI host connecting to a Freenas box with a dual port LACP interface dishing out data over 4 vlans so it does work (at least on 5.1)

You don't say what the network setup on your ESXI box is, I know there are some updates to LACP handling in ESXI 5.5 that could be making a difference, I only have have my NFS on 1 NIC configured with 2 VMkernals (ISCSI and NFS) so not using LACP on the ESXI host.

What type of LACP are you using on the switch?

 

[cpfarhood]

It's a Dell PowerConnect 5324 and I've tried both L2/L3 and L2/L3/L4 hashing, both work fine with pfSense and not with FreeNAS. I ran wireshark and found LOTS of out of order packets and the throughput seems to be perfectly balanced across all 4 links even with a single TCP session, which I know isn't right. I'm not running LACP on ESXi at all, just multiple trunks with port virtual id balancing.

 

[cpfarhood]

It seems LACP+VLAN+Jumbo Frames is breaking something either in my switch or on FreeNAS itself, dropping the MTU back down to 1500 everything works as expected.

 

[dlavigne]

From ifconfig, what is the driver name for the freenas interfaces? Not all drivers support jumbo frames.

 

[cpfarhood]

It's an Intel i340-t4 adapter as stated above, the driver is igb, it works perfect with jumbo frames without LACP as I ran it in failover lag mode with vlan tags and jumbo frames successfully for quite some time before moving to LACP. I'm thinking it's the switch that's choking on it but I need to acquire a different switch to test and verify, thinking of just picking up a Cisco 3750G as I want something with L3 support anyway.

 

[zambanini]

http://www.vmguru.nl/wordpress/2014/04/vmware-vsphere-5-5-update-1-nfs-disconnects/ just found today in my feedly pool