SOLVED Very slow network performance

[ondjultomte]

[Solved]

I moved from ESXi AiO to baremetal .

on the server Im using Chelsio T420-CR and on the win workstation mellanox connect x2.
Before when I was using beta nic drivers for ESXi I got around 7Gbps in iperf, now I get 0.4 Gbps. I first thought that I got to have massive CRC/errors but not according to netstat, so layer 1 seems ok , cables etc .

netstat -e on wks

Code:

:\Program Files\iperf3>netstat -e
Interface Statistics

						   Received			Sent

Bytes					 839846399	  1323635568
Unicast packets		  1296624341	  2655336968
Non-unicast packets		 1113225		  820848
Discards						  0			  10
Errors							0			  20
Unknown protocols				 0

netstat on fn 11

Code:

netstat -I cxgbe0	
Name	Mtu Network	   Address			  Ipkts Ierrs Idrop	Opkts Oerrs  Coll
cxgbe  1500 <Link#1>	  00:07:43:11:41:50   402529	 0	 0	  145	 0	 0
cxgbe	 - 192.168.1.0/2 192.168.1.96		 81690	 -	 -	   16	 -	 -

iperf

Code:

in/iperf.exe -c 192.168.1.96 -P 1 -i 1 -p 5001 -f m -t 10
------------------------------------------------------------
Client connecting to 192.168.1.96, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[300] local 192.168.1.21 port 51579 connected with 192.168.1.96 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[300]  0.0- 1.0 sec  4.98 MBytes  41.7 Mbits/sec
[300]  1.0- 2.0 sec   127 MBytes  1067 Mbits/sec
[300]  2.0- 3.0 sec  92.2 MBytes   774 Mbits/sec
[300]  3.0- 4.0 sec  34.0 MBytes   285 Mbits/sec
[300]  4.0- 5.0 sec  35.6 MBytes   299 Mbits/sec
[300]  5.0- 6.0 sec  65.3 MBytes   548 Mbits/sec
[300]  6.0- 7.0 sec  64.2 MBytes   539 Mbits/sec
[300]  7.0- 8.0 sec  84.5 MBytes   709 Mbits/sec
[300]  8.0- 9.0 sec  65.7 MBytes   551 Mbits/sec
[300]  9.0-10.0 sec  65.8 MBytes   552 Mbits/sec
[300]  0.0-10.0 sec   639 MBytes   535 Mbits/sec
Done.

Im really surprised to see such low performance when moving to bare metal. Something is really wrong here.

ANy tips on what I should look at ?

 

[SweetAndLow]

Any tunable set? After you using a switch? Try direct attaching everything. Full hardware specs? Freenas version?

Sent from my Nexus 5X using Tapatalk

 

[ondjultomte]

Im currently in the middle of a restore from backup job so I can disconnect the cable for another few days.
No special tunings a pure fn11 install.

Build FreeNAS-11.0-RELEASE (a2dc21583)
Platform Intel(R) Xeon(R) CPU E5-2630L v3 @ 1.80GHz
Memory 65389MB

I got a spare sfp+ here that I can try later but since I dont get any L2 errors there is no physical damage, if I got some contamination I would see l2 errors.

 

[SweetAndLow]

Im currently in the middle of a restore from backup job so I can disconnect the cable for another few days.
No special tunings a pure fn11 install.

Build FreeNAS-11.0-RELEASE (a2dc21583)
Platform Intel(R) Xeon(R) CPU E5-2630L v3 @ 1.80GHz
Memory 65389MB

I got a spare sfp+ here that I can try later but since I dont get any L2 errors there is no physical damage, if I got some contamination I would see l2 errors.

Full hardware specs means everything. Motherboard, CPU, memory, power supply, chassie, hba, HDD, switch, client specs and os. You need to provide information otherwise no one can help.

Sent from my Nexus 5X using Tapatalk

 

[ondjultomte]

Finally after almost 2 weeks of backing up and restoring I can do changes again. Applied the latest updates and rebooted FreeNAS and everything works great. ~7Gbit from my mellanox wks now.