10g Network - SMB Copy TO FreeNAS issues (FROM works)

[bldink]

Overview:
My FreeNAS SMB shares can't seem to be written to normally from Windows 10 desktop via the 10Gb network (1Gb is fine). Reading FROM those share works great on both connections.

Detailed Explanation:
I have FreeNAS (v11.2 U7) installed as a VM on a ESXi host (6.7 U3 in case it matters). The main "VM Network" is run off one of the 2 1Gb motherboard Intel NICs. My "10G Network" is via an installed Aquantia AQN-100-104-SFA PCIe card. The FreeNAS VM has 2 vNICs (VMXNET 3), one for each network. I've statically assigned the IP v4s for both interfaces within FreeNAS and set the MTU to 9000 for the 10G connection. Anyways, here's the output from ifconfig:

root@freenas[~]# ifconfig
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:7e:1f:03
hwaddr 00:0c:29:7e:1f:03
inet 192.168.0.38 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::20c:29ff:fe7e:1f03%vmx0 prefixlen 64 scopeid 0x1
inet6 2600:8802:2301:7c00:20c:29ff:fe7e:1f03 prefixlen 64 autoconf
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:7e:1f:0d
hwaddr 00:0c:29:7e:1f:0d
inet 192.168.10.3 netmask 0xffffff00 broadcast 192.168.10.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo

On my Windows 10 desktop, I have a similar networking setup; 1Gb NIC (Killer E2200 unfortunately) and the same Aquantia card as mentioned above. My 10G NIC is also statically assigned; IPv4 192.168.10.2 with 255.255.255.0 subnet mask. I also disabled IPv6; because I'm not savvy enough with IPv6 to statically assign it correctly on both ends... I have "Jumbo Packets" advanced property set to the 9014 bytes option (closest to 9000); the rest I have left at their defaults.

Also note the 10G Aquantia cards are directly attached via a SFP-DAC cable; no switch.

I have been able to use FreeNAS normally (as far as I can tell at least), via the 1G/main/web GUI connection (192.168.0.38 in this case). SMB shares seem to perform as expected from my main Windows 10 desktop; copying to and from. However, when accessing those same shares via the 10G connection (192.168.10.3 in this case), I have different experiences. Reading and copying FROM the shares are great; regularly maxing out my drive speeds, i.e., over 350 MB/sec or well over 3 Gbit/s receive. Writing and copying TO the shares however I get issues.

When I try to start a copy TO one of the shares, e.g., \\192.168.10.3\Downloads\, Windows brings up the file copying/transfer progress dialog (or whatever you want to call it), but it seems to stall immediately at "Calculating...". It will eventually fail. If I cancel the file transfer the first file will show up in the share but it is corrupted (seemingly full sized, but not readable/functional).

Testing/Troubleshooting I've done:
Focusing on my Windows 10 desktop machine, googling for a while, I wasn't able to find any forum and/or blog posts with similar situations. However, suggestions about disabling "Large Send Offload" options did sound somewhat worthwhile, but didn't seem to do anything. I also tried a few different Aquantia drivers, nothing seemed to change, so I went back to the original/suggested driver.

I then thought to test the ESXi side. I created a Windows 10 VM on the same host with the same networking config/setup as the FreeNAS VM (except for 192.168.10.10 for the IPv4 static IP on the vNIC attached to the ESXi "10G Network"). After disabling the windows firewall on the Windows VM I was able to copy to and from a shared folder on the VM as well as the other way around. ["Jumbo Packets" set to "Jumbo 9000"; FYI]

This leads me to believe there is some configuration I'm missing on the FreeNAS VM. Just a hunch though given how the Windows VM worked...

Any and all help is appreciated!

 

[dlavigne]

Were you able to resolve this?

 

[Akai]

Sorry to revive an old thread but i had similar issues as you did until i enabled Large Receive Offload on the network adapter. From the shell try "ifconfig vmx0 lro" or try Network > Interfaces > Edit, then add the lro option and reboot.

The issue i've got now is that gets disabled when i start a jail.

 

[bldink]

Tanks so much for the post. I was still dealing with that issue, so after giving it a shot my initial reaction and reply was simply: "OMG, that was it. Thanks!!!".

Who would have thought that would have to be set for a virtual interface, and that it would be differently locally (between VMs) vs remotely.

Thanks again.

 

[Akai]

No problem, I was stumped by that one for a few hours until i tried disabling and re-enabling offloading via the GUI and noticed it started working. Digging deeper it was when offloading was re-enabled in FreeNAS GUI on the interface that it turned on the LRO setting for the NIC but it wasn't turning it on by default.

I was reading that FreeBSD enables the feature automatically if certain 10Gbe network adapters and chips are detected (broadcom) but since we are using virtual ones it doesn't have physical access to the card to see what chipset is in use so it gets left off. Adding the option manually to the NIC seems to resolve the problem as LRO is then enabled. The only exception to this is when i go to start/restart a Jail it restarts the network stack and doesn't seem to bother applying the manually entered options for the NIC and it breaks again -> i suspect this is a bug with FreeNAS though.