Network issue 2,5/10Gbe receiving data very slow

[Darksand]

Ok, after many, many hours of troubleshooting, I now come here for your help

My issue in short: iperf3 receiving is very slow while sending is fine.

Hardware:

Truenas is installed on:

Supermicro SSG-5018A-AR12
8 core Intel(R) Atom(TM) CPU C2750 @ 2.40GHz
64 GB Memory.
Intel Dual 10Gbe 82599

Issue was originally on Truenas 12, but while troubleshooting, I upgraded to 13.0-U5.3. Issue persists.

Tests are performed from:
a ESXi Windows VM
Direct from a Proxmox host

The ESXi host hardware is identical to the procmox host.
Gigabyte Gigabyte GB-BRR7H-4800
Ryzen 7 4800u
64GB DDR4 memory
Multiple 2.5Gbit USB interfaces.

everything is connected directly via:
a QNap switch QSW-M2108-2S 8x2,5 + 2x10Gbe

Here is my issue:

iperf3
ESXi Windows VM > Proxmox host: iperf3 -c 192.168.4.51 -V
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 262 MBytes 2.20 Gbits/sec

fine for 2.5gbit to 2.5gbit

ESXi Windows VM < Proxmox host: iperf3 -c 192.168.4.51 -V -R
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 251 MBytes 2.10 Gbits/sec

again fine

Truenas > Proxmox Host: iperf3 -c 192.168.4.51 -V
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 247 MBytes 2.07 Gbits/sec 1 793 KBytes

again fine

Now here look at this!!

Truenas < Proxmox Host: iperf3 -c 192.168.4.51 -V -R
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 92.2 MBytes 774 Mbits/sec

Which is way to slow. (as an extra, yes i did try this also to the esxi vm, same result)

What did I do to try to resolve?

Updated Qnap switch
Upgraded truenas from 12 to 13
exchanged the Intel 10gbe NIC for a spare one.
exchanged the fibre cable/spf
tickled, as far as I know. all receive buffer settings. (on Truenas aswell on the proxmox host, because at first i thought it was a proxmox issue)
Switched off hardware offload, however then the (Truenas > Proxmox) score will go down as well.
changed congestion algorithm to dctcp aswell as cubic.

does anyone of you have any more ideas? do I maybe need another driver for this Intel 10Gbe Nic?

All ideas are welcome!

 

[Darksand]

Did try the tso/lro setting, didn't help

found here, SOLVED 10GbE ESXi 6.5 VMXNET 3 performance is poor with iperf3 tests.

 

[NickF]

Have you read through this resource:

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

 

[Darksand]

Have you read through this resource:

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

Thanks for your suggestion, however, I actually used during my troubleshooting among others others

 

[NickF]

It sounds like you may be hitting a CPU bottleneck during this testing. I am seeing little evidence to suggest otherwise. What does htop look like while running the iPerf3 test on the TrueNAS?

Your other two servers are much faster, and for the testing you are doing, the relatively low single threaded performance is likelt the culprit of your problems.

 

[Darksand]

Yes, the CPU was my first suspect, it is not the fastest. But also on multiple treads it was very slow. My instinct was that it atleast should be better than 1gbit. But....

I think I found my issue and solution.

Like I said I tried everything, so now I just updated my mainboard Bios. The server had to reboot. And after that i see that send and receive went up, but still receive half the speed....

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 145 MBytes 1.21 Gbits/sec 0 1.33 MBytes
[ 5] 1.00-2.00 sec 142 MBytes 1.20 Gbits/sec 0 1.33 MBytes
[ 5] 2.00-3.00 sec 141 MBytes 1.18 Gbits/sec 0 1.33 MBytes
[ 5] 3.00-4.00 sec 141 MBytes 1.18 Gbits/sec 0 1.33 MBytes
[ 5] 4.00-5.00 sec 142 MBytes 1.20 Gbits/sec 0 1.33 MBytes
[ 5] 5.00-6.00 sec 141 MBytes 1.19 Gbits/sec 0 1.33 MBytes
[ 5] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 1.33 MBytes
[ 5] 7.00-8.00 sec 141 MBytes 1.18 Gbits/sec 0 1.33 MBytes
[ 5] 8.00-9.00 sec 142 MBytes 1.20 Gbits/sec 0 1.33 MBytes
[ 5] 9.00-10.00 sec 141 MBytes 1.18 Gbits/sec 0 1.33 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.39 GBytes 1.19 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
CPU Utilization: local/sender 11.3% (0.0%u/11.3%s), remote/receiver 93.0% (14.8%u/78.2%s)

It is better, like from 70-90MB to 140 all over the board. And then I see the CPU load, 93%! And take it from me, i did try it with multiple threads -P option Before. And I saw with 4 threads like 13-18MB per, and after some optimizations between 20-25. now however I see 280mb/P. So on P4 I see.

[ 5] 9.00-10.00 sec 79.0 MBytes 663 Mbits/sec 24 146 KBytes
[ 7] 9.00-10.00 sec 71.4 MBytes 599 Mbits/sec 31 114 KBytes
[ 9] 9.00-10.00 sec 62.3 MBytes 523 Mbits/sec 36 12.7 KBytes
[ 11] 9.00-10.00 sec 67.8 MBytes 569 Mbits/sec 38 65.2 KBytes
[SUM] 9.00-10.00 sec 281 MBytes 2.35 Gbits/sec 129

I am not sure if the Bios update was the solution, because before the reboot I did many buffer changes. not sure these became active only after reboot.

Thanks for thinking with me.

 

[Darksand]

aarch, and it is back :/

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 41.6 MBytes 349 Mbits/sec
[ 7] 0.00-1.00 sec 41.8 MBytes 351 Mbits/sec
[ 9] 0.00-1.00 sec 40.2 MBytes 337 Mbits/sec
[ 11] 0.00-1.00 sec 39.6 MBytes 332 Mbits/sec
[SUM] 0.00-1.00 sec 163 MBytes 1.37 Gbits/sec

What makes this happen?

 

[Darksand]

Hmm, removed the netdata plugin and the speeds are normal again