Slow 10GbE performance and a weird issue/issues.

[Daniel Claesson]

Hi all,

I have some issues getting my newly added 10GbE equipment to work as intended. Any advice and pointers are much welcome.

Network performance in my homelab is "wonky" and slow on the 10GbE storage network.
Performance goes from about 300MB/s (both directions, read/write) over SMB between my workstation and the server to 8kb/s-800kb/s.
On a freshly rebooted server the performance is good but just suddenly after some time performance goes down the drain.

The 1GbE network don't have any problems at all. It will stay at 100-110MB/s.

I have tried to figure this out, but at the moment i am clueless.


Workstation:
A Windows 10 system with the following specs:
Intel Core i7-6700K/16GB RAM/2.5TB of SSD's/Intel X540-T1

FreeNAS Server:
OS: 9.10.2-U6
Logicboard: Supermicro X9SRL-F
CPU: Intel Xeon E5-2620 v1
RAM: 32GB ECC
NIC: 2x1GbE Onboard + Intel X540-T1 PCIe
Storage: 6x2TB in RAIDz2 (Seagate Constellation ES.3)
Boot: 2x150GB Intel SSD DC S3520 in mirror.

Network:
Cat6 cables, all under 10m.

- Meraki MX65 FW -> Netgear XS708E (10GbE network, 192.168.10.xxx) -> Workstation, FreeNAS, ESXi (iSCSI).
- Meraki MX65 FW-> Unifi Switch 24 POE (1GbE network, 192.168.5.xxx) -> ESXi (LAN), FreeNAS, Lab servers, Mac mini server, Unifi AP.
- Meraki MX65 FW-> Meraki MS220-8P Switch (1GbE network, 192.168.5.xxx) -> Workstation, Meraki AP's, Laptops, IOT, Printer, etc.

FreeNAS settings:
I have the "autotune" setting enabled (might have been a bad idea?) and under tunables i have manualy added the "hw.ix.enable_aim=0" variable.
The problem is present with and without this variable.

Other settings added by autotune is:


Other then that it's pretty standard.

Testing done so far:

- iperf: PC -> FreeNAS
------------------------------------------------------------
Client connecting to 192.168.10.100, TCP port 5001
TCP window size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.10.205 port 64704 connected with 192.168.10.100 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.41 GBytes 2.93 Gbits/sec

- iperf: FreeNAS -> PC
Didn't get it to work?!? (Don't know this tool all that well)

Tested some settings on the Windows driver for the Intel 10GbE nic, didn't do anything.
I have thought about Jumbo frames, but every second thread (and person i asked) says to stay away from it.

Where should i begin investigating? Any more test's that i shall run? The iperf test didn't get that great of a result.

Best Regards
Daniel Claesson

 

[dlavigne]

Were you able to resolve this?

 

[chris crude]

Many people with Intel 10GB NIC need to set a few tunables. This is what i did to unlock the full potential of mine.

FreeNAS=I didn't set jumbo frames, but used the tunables here https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/page-7#post-272868
WIN10= again didn't set jumbo frames but used this as guide http://www.cinevate.com/blog/confessions-of-a-10-gbe-network-newbie-part-5-smb3/

 

[bigphil]

Have you tried taking the 10Gb switch out of the equation and just direct connect FreeNAS and your workstation to see if the same issue persists?