Cannot get full 10Gbe on Tx

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
I finally got fiber runs from my basement homelab to my desktop, however I found that the Tx bandwidth of my FN box was limited to ~5Gbit/s, while Rx side is OK.


Code:
c:\iperf>iperf -c 192.168.20.101
------------------------------------------------------------
Client connecting to 192.168.20.101, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.30.101 port 51823 connected with 192.168.20.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.7 GBytes  9.15 Gbits/sec

c:\iperf>iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ ID] Interval       Transfer     Bandwidth
[  4] local 192.168.30.101 port 5001 connected with 192.168.20.101 port 15309
[  4]  0.0-10.0 sec  5.18 GBytes  4.43 Gbits/sec

192.168.20.101 is the FN box and 192.168.30.101 is my desktop.

To determine where the problem lies, I run another set of test to a FN VM (192.168.20.122)
Code:
c:\iperf>iperf -c 192.168.20.122
------------------------------------------------------------
Client connecting to 192.168.20.122, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.30.101 port 51809 connected with 192.168.20.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  9.24 GBytes  7.94 Gbits/sec

c:\iperf>iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ ID] Interval       Transfer     Bandwidth
[  4] local 192.168.30.101 port 5001 connected with 192.168.20.122 port 25230
[  4]  0.0-10.0 sec  10.3 GBytes  8.86 Gbits/sec

It appears to be some headroom due to virtualization but still decent numbers. This also suggest that something is happening on the FN box, so I reboot the box into Ubuntu live CD (192.168.20.110)
Code:
c:\iperf>iperf -c 192.168.20.110
------------------------------------------------------------
Client connecting to 192.168.20.110, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.30.101 port 52428 connected with 192.168.20.110 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  9.75 GBytes  8.38 Gbits/sec

c:\iperf>iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ ID] Interval       Transfer     Bandwidth
[  4] local 192.168.30.101 port 5001 connected with 192.168.20.110 port 57248
[  4]  0.0-10.0 sec  11.0 GBytes  9.42 Gbits/sec



The problem goes away. Was there any issue on FreeNAS with X540? As far as I know there used to be some problems with ixg drivers but that should be resolved AFAIK.
 
Joined
Dec 29, 2014
Messages
1,135
There is another thread (don't have it handy) relating to system tunables for 10G performance. I suspect it a limit on the amount of unacknowledged data on the FreeNAS side. FWIW, here are the tunables from my primary FreeNAS (256GB RAM, Chelsio T520 10G NIC). I can max out iperf between my two FreeNAS systems (see sig for a complete description). Be careful with the tunables so you don't jack up something else. :smile:

1547653223943.png
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
I suspect it a limit on the amount of unacknowledged data on the FreeNAS side.
I am not sure if that is true. The FreeNAS VM has much more limited resource (8G RAM, 8vCPU, 8G vdisk, etc.) comparing to the physical box, yet it can do 8Gbit/s both ways. Both the box and VM has not been tuned.

I will try those tuneables, thanks.
 
Joined
Dec 29, 2014
Messages
1,135
I will try those tuneables, thanks.
Remember that my bare metal FreeNAS has a whole lot more memory than your VM, so I wouldn't use those values verbatim.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
Remember that my bare metal FreeNAS has a whole lot more memory than your VM, so I wouldn't use those values verbatim.
Oh, my FN VM (on a ESXi box) can do 8Gbit/s out of box. It's my bare metal FreeNAS box (with 128GB RAM) having the problem stated in OP.
I will take a snapshot of the boot drive though.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
There is another thread (don't have it handy) relating to system tunables for 10G performance. I suspect it a limit on the amount of unacknowledged data on the FreeNAS side. FWIW, here are the tunables from my primary FreeNAS (256GB RAM, Chelsio T520 10G NIC). I can max out iperf between my two FreeNAS systems (see sig for a complete description). Be careful with the tunables so you don't jack up something else. :)

View attachment 27836
I let FreeNAS run auto-tune, and it set the tunables very similar to yours, except that some are halved because I only got 128G RAM
freenas   FreeNAS 11 1 U6  caffd76fa .png

Unfortunately the Tx speed was still capped at ~5Gbits.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
Interestingly if I turn off the offloading features manually ifconfig ix0 -rxcsum -txcsum -tso -lro , the Tx speed can go to ~8Gbit/s briefly, but soon dropped to ~5Gbit/s.
FN_Tx.png

And in this state it's the Rx side being capped at 5Gbit/s
FN_Rx.png
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
After playing around I found out disabling TSO makes the Tx throughput hover just above 7Gbps, while Rx stays above 9Gbps.

Code:
root@freenas:~ # ifconfig ix0 -tso
root@freenas:~ # iperf -c 192.168.30.101 -t 600
------------------------------------------------------------
Client connecting to 192.168.30.101, TCP port 5001
TCP window size:  513 KByte (default)
------------------------------------------------------------
[  3] local 192.168.20.101 port 12831 connected with 192.168.30.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-600.0 sec   499 GBytes  7.14 Gbits/sec


Not the really fully utilize 10Gbe, but I fill it's acceptable.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
Does anyone know the tunable for turning off TSO? I guess I can write a post-init command but a tunable sounds more elegant.
 

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
Ender117, I'm having the exact same problem as you are, except my 10gb interface drops to low 100Mbps! And it's only on Tx side as well. I've tried so many different tunable settings, but nothing fixes it except a reboot. But within 24 hours, the problem happens again.

To turn off TSO, set it in your interface options. Just add -tso and anything else you want to turn off.

Let me know if you find out the cause. I'm forced to go back to my gigabit NIC since I still get full speed both directions
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
Ender117, I'm having the exact same problem as you are, except my 10gb interface drops to low 100Mbps! And it's only on Tx side as well. I've tried so many different tunable settings, but nothing fixes it except a reboot. But within 24 hours, the problem happens again.

To turn off TSO, set it in your interface options. Just add -tso and anything else you want to turn off.

Let me know if you find out the cause. I'm forced to go back to my gigabit NIC since I still get full speed both directions
I didn't look into this further but settled with 7Gbit sending. Thanks for your tip though

Have you tried if you can reproduce the problem with different OS? You may also want to look into the logs because it only happen some time after boot
 

VolumeTank

Dabbler
Joined
Dec 23, 2018
Messages
38
Ender117, I'm having the exact same problem as you are, except my 10gb interface drops to low 100Mbps! And it's only on Tx side as well. I've tried so many different tunable settings, but nothing fixes it except a reboot. But within 24 hours, the problem happens again.

To turn off TSO, set it in your interface options. Just add -tso and anything else you want to turn off.

Let me know if you find out the cause. I'm forced to go back to my gigabit NIC since I still get full speed both directions

wow!! This post is more than year old but sure it helped me, I was trying to figure out my problem and BOOM a simple -tso

Before -tso
Code:
------------------------------------------------------------
Client connecting to 10.10.10.1, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.2 port 49482 connected with 10.10.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.64 GBytes  2.27 Gbits/sec
imac-2015:Downloads gioriv$ ./iperf -c 10.10.10.1
------------------------------------------------------------


After -tso
Code:
Client connecting to 10.10.10.1, TCP port 5001
TCP window size:  131 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.2 port 49740 connected with 10.10.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  9.32 GBytes  8.00 Gbits/sec
imac-2015:Downloads gioriv$ ./iperf -c 10.10.10.1
 
Top