Replication task running at 60MB/s on 100Gbit network

R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Elliot Dierksen said:
Code:
83:00.0 Ethernet controller: Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
                 LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
                LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Click to expand...
So Link is capable of 8GT/s x8,
Link current status is 8GT/s x8 as well.

1577619833689.png


According to Wikipedia 8 GT/s is PCIe 3.0


I think the v2 is not related to PCIE version but to the reporting segment, you got more v2 segments and also some v1 ones a bit below...
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
radambe said:
Once lspci -vv began reporting the expected result of 16 lanes / 8GT/s, that’s when I was able to get iperf3 up to ~ 37Gbps, but “real world” tests like a drag’n’drop file transfer, command line FIO tests, command line Blackmagic TestIO test, GUI Blackmagic Disk Test, AJA System Test, and Atto disk test all continued to show very volatile, unstable and relatively slow (never more than 3 to 4 Gbps average) results that were completely unacceptable for our use case, and on average still fell very short of the Intel X540 10GbE interface’s performance.
Click to expand...

So I just ran a couple of tests with iperf2 (iperf3 is single threaded and such harder to scale up with since you need multiple processes).

This is with the newest driver on Windows 2012R2, FreeNas 11.2 U7 on the other side and T62100-LP-CRs on both boxes.

On Windows side initially there was no optimization, on FreeNas I have jumbo & TOE on 192.168.10.27 and nothing on 192.168.11.27

1577639233027.png


This scaled me up to 54GBit/s (4/8/16 threads).
I then doubled all buffers on 10.27 and also added Jumbo there, but that did net only another 2-3 Gbit/s, not really worth it.

Code:
C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.26 port 49226 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  20.9 GBytes  18.0 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.11.27
------------------------------------------------------------
Client connecting to 192.168.11.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.11.26 port 49239 connected with 192.168.11.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.3 GBytes  8.88 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -w 1M
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[  3] local 192.168.10.26 port 49248 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  21.6 GBytes  18.6 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 4
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.26 port 49255 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49257 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49254 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49256 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  4.28 GBytes  3.68 Gbits/sec
[  6]  0.0-10.0 sec  17.2 GBytes  14.8 Gbits/sec
[  3]  0.0-10.0 sec  18.7 GBytes  16.1 Gbits/sec
[  5]  0.0-10.0 sec  13.2 GBytes  11.4 Gbits/sec
[SUM]  0.0-10.0 sec  53.5 GBytes  46.0 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 8
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.26 port 49267 connected with 192.168.10.27 port 5001
[ 10] local 192.168.10.26 port 49273 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49272 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49266 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49270 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49268 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49269 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49271 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  8.78 GBytes  7.54 Gbits/sec
[ 10]  0.0-10.0 sec  7.81 GBytes  6.70 Gbits/sec
[  9]  0.0-10.0 sec  7.81 GBytes  6.71 Gbits/sec
[  3]  0.0-10.0 sec  7.96 GBytes  6.83 Gbits/sec
[  7]  0.0-10.0 sec  7.75 GBytes  6.65 Gbits/sec
[  5]  0.0-10.0 sec  7.72 GBytes  6.63 Gbits/sec
[  6]  0.0-10.0 sec  7.72 GBytes  6.62 Gbits/sec
[  8]  0.0-10.0 sec  7.75 GBytes  6.66 Gbits/sec
[SUM]  0.0-10.0 sec  63.3 GBytes  54.3 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 8 -w 1M
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[ 10] local 192.168.10.26 port 49287 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49286 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49285 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49284 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49283 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49281 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49280 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49282 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 10]  0.0-10.0 sec  7.79 GBytes  6.69 Gbits/sec
[  9]  0.0-10.0 sec  7.78 GBytes  6.69 Gbits/sec
[  8]  0.0-10.0 sec  7.84 GBytes  6.74 Gbits/sec
[  7]  0.0-10.0 sec  8.25 GBytes  7.09 Gbits/sec
[  6]  0.0-10.0 sec  7.92 GBytes  6.81 Gbits/sec
[  4]  0.0-10.0 sec  8.14 GBytes  6.99 Gbits/sec
[  3]  0.0-10.0 sec  8.14 GBytes  6.99 Gbits/sec
[  5]  0.0-10.0 sec  7.90 GBytes  6.79 Gbits/sec
[SUM]  0.0-10.0 sec  63.8 GBytes  54.8 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 8 -w 4M
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size: 4.00 MByte
------------------------------------------------------------
[ 10] local 192.168.10.26 port 49300 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49298 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49299 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49297 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49294 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49296 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49293 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49295 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 10]  0.0-10.0 sec  8.21 GBytes  7.06 Gbits/sec
[  8]  0.0-10.0 sec  7.89 GBytes  6.78 Gbits/sec
[  9]  0.0-10.0 sec  7.90 GBytes  6.79 Gbits/sec
[  7]  0.0-10.0 sec  8.08 GBytes  6.95 Gbits/sec
[  4]  0.0-10.0 sec  7.77 GBytes  6.68 Gbits/sec
[  6]  0.0-10.0 sec  8.08 GBytes  6.94 Gbits/sec
[  3]  0.0-10.0 sec  7.79 GBytes  6.69 Gbits/sec
[  5]  0.0-10.0 sec  7.77 GBytes  6.67 Gbits/sec
[SUM]  0.0-10.0 sec  63.5 GBytes  54.6 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 8
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  5] local 192.168.10.26 port 49354 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49355 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49353 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49358 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49352 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49356 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49357 connected with 192.168.10.27 port 5001
[ 10] local 192.168.10.26 port 49359 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  7.62 GBytes  6.54 Gbits/sec
[  6]  0.0-10.0 sec  8.50 GBytes  7.30 Gbits/sec
[  4]  0.0-10.0 sec  7.66 GBytes  6.57 Gbits/sec
[  9]  0.0-10.0 sec  8.27 GBytes  7.10 Gbits/sec
[  3]  0.0-10.0 sec  8.40 GBytes  7.21 Gbits/sec
[  7]  0.0-10.0 sec  8.50 GBytes  7.30 Gbits/sec
[  8]  0.0-10.0 sec  8.39 GBytes  7.20 Gbits/sec
[ 10]  0.0-10.0 sec  8.68 GBytes  7.45 Gbits/sec
[SUM]  0.0-10.0 sec  66.0 GBytes  56.7 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 16
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ 18] local 192.168.10.26 port 49382 connected with 192.168.10.27 port 5001
[ 17] local 192.168.10.26 port 49381 connected with 192.168.10.27 port 5001
[ 16] local 192.168.10.26 port 49380 connected with 192.168.10.27 port 5001
[ 15] local 192.168.10.26 port 49379 connected with 192.168.10.27 port 5001
[ 14] local 192.168.10.26 port 49378 connected with 192.168.10.27 port 5001
[ 12] local 192.168.10.26 port 49376 connected with 192.168.10.27 port 5001
[ 13] local 192.168.10.26 port 49377 connected with 192.168.10.27 port 5001
[ 10] local 192.168.10.26 port 49374 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49372 connected with 192.168.10.27 port 5001
[ 11] local 192.168.10.26 port 49375 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49371 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49373 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49370 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49367 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49369 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49368 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 18]  0.0-10.0 sec  4.01 GBytes  3.45 Gbits/sec
[ 17]  0.0-10.0 sec  4.09 GBytes  3.51 Gbits/sec
[ 16]  0.0-10.0 sec  4.26 GBytes  3.66 Gbits/sec
[ 15]  0.0-10.0 sec  3.89 GBytes  3.34 Gbits/sec
[ 14]  0.0-10.0 sec  3.98 GBytes  3.42 Gbits/sec
[ 12]  0.0-10.0 sec  4.27 GBytes  3.67 Gbits/sec
[ 13]  0.0-10.0 sec  3.98 GBytes  3.42 Gbits/sec
[ 10]  0.0-10.0 sec  4.41 GBytes  3.79 Gbits/sec
[  8]  0.0-10.0 sec  4.24 GBytes  3.64 Gbits/sec
[ 11]  0.0-10.0 sec  4.61 GBytes  3.96 Gbits/sec
[  7]  0.0-10.0 sec  3.97 GBytes  3.41 Gbits/sec
[  9]  0.0-10.0 sec  3.91 GBytes  3.36 Gbits/sec
[  6]  0.0-10.0 sec  3.94 GBytes  3.38 Gbits/sec
[  3]  0.0-10.0 sec  4.00 GBytes  3.44 Gbits/sec
[  5]  0.0-10.0 sec  4.12 GBytes  3.54 Gbits/sec
[  4]  0.0-10.0 sec  3.97 GBytes  3.41 Gbits/sec
[SUM]  0.0-10.0 sec  65.7 GBytes  56.4 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 16
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ 18] local 192.168.10.26 port 49407 connected with 192.168.10.27 port 5001
[ 17] local 192.168.10.26 port 49406 connected with 192.168.10.27 port 5001
[ 16] local 192.168.10.26 port 49405 connected with 192.168.10.27 port 5001
[ 15] local 192.168.10.26 port 49404 connected with 192.168.10.27 port 5001
[ 14] local 192.168.10.26 port 49403 connected with 192.168.10.27 port 5001
[ 13] local 192.168.10.26 port 49402 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49398 connected with 192.168.10.27 port 5001
[ 12] local 192.168.10.26 port 49401 connected with 192.168.10.27 port 5001
[ 11] local 192.168.10.26 port 49400 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49395 connected with 192.168.10.27 port 5001
[ 10] local 192.168.10.26 port 49399 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49397 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49396 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49393 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49392 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49394 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 18]  0.0-10.0 sec  4.16 GBytes  3.58 Gbits/sec
[ 17]  0.0-10.0 sec  3.95 GBytes  3.39 Gbits/sec
[ 16]  0.0-10.0 sec  4.14 GBytes  3.56 Gbits/sec
[ 15]  0.0-10.0 sec  4.01 GBytes  3.45 Gbits/sec
[ 14]  0.0-10.0 sec  4.50 GBytes  3.87 Gbits/sec
[ 13]  0.0-10.0 sec  4.18 GBytes  3.59 Gbits/sec
[  9]  0.0-10.0 sec  4.49 GBytes  3.86 Gbits/sec
[ 12]  0.0-10.0 sec  3.91 GBytes  3.36 Gbits/sec
[ 11]  0.0-10.0 sec  3.84 GBytes  3.30 Gbits/sec
[  6]  0.0-10.0 sec  4.07 GBytes  3.50 Gbits/sec
[ 10]  0.0-10.0 sec  3.95 GBytes  3.40 Gbits/sec
[  8]  0.0-10.0 sec  4.37 GBytes  3.75 Gbits/sec
[  7]  0.0-10.0 sec  4.02 GBytes  3.45 Gbits/sec
[  4]  0.0-10.0 sec  3.94 GBytes  3.39 Gbits/sec
[  3]  0.0-10.0 sec  4.10 GBytes  3.53 Gbits/sec
[  5]  0.0-10.0 sec  3.94 GBytes  3.39 Gbits/sec
[SUM]  0.0-10.0 sec  65.6 GBytes  56.4 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 16 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ 19] local 192.168.10.26 port 49429 connected with 192.168.10.27 port 5001
[ 18] local 192.168.10.26 port 49428 connected with 192.168.10.27 port 5001
[ 17] local 192.168.10.26 port 49427 connected with 192.168.10.27 port 5001
[ 15] local 192.168.10.26 port 49426 connected with 192.168.10.27 port 5001
[ 12] local 192.168.10.26 port 49423 connected with 192.168.10.27 port 5001
[ 13] local 192.168.10.26 port 49424 connected with 192.168.10.27 port 5001
[ 11] local 192.168.10.26 port 49422 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49420 connected with 192.168.10.27 port 5001
[ 10] local 192.168.10.26 port 49421 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49418 connected with 192.168.10.27 port 5001
[ 14] local 192.168.10.26 port 49425 connected with 192.168.10.27 port 5001
[  8] local 192.168.10.26 port 49419 connected with 192.168.10.27 port 5001
[  6] local 192.168.10.26 port 49417 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49416 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49415 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49414 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 19]  0.0-10.0 sec  4.55 GBytes  3.91 Gbits/sec
[ 18]  0.0-10.0 sec  4.38 GBytes  3.77 Gbits/sec
[ 17]  0.0-10.0 sec  4.06 GBytes  3.49 Gbits/sec
[ 15]  0.0-10.0 sec  3.93 GBytes  3.38 Gbits/sec
[ 12]  0.0-10.0 sec  4.22 GBytes  3.63 Gbits/sec
[ 13]  0.0-10.0 sec  3.91 GBytes  3.36 Gbits/sec
[ 11]  0.0-10.0 sec  4.15 GBytes  3.57 Gbits/sec
[  9]  0.0-10.0 sec  3.92 GBytes  3.37 Gbits/sec
[ 10]  0.0-10.0 sec  4.11 GBytes  3.53 Gbits/sec
[  7]  0.0-10.0 sec  3.96 GBytes  3.40 Gbits/sec
[ 14]  0.0-10.0 sec  3.92 GBytes  3.37 Gbits/sec
[  8]  0.0-10.0 sec  3.99 GBytes  3.43 Gbits/sec
[  6]  0.0-10.0 sec  4.26 GBytes  3.66 Gbits/sec
[  5]  0.0-10.0 sec  4.37 GBytes  3.76 Gbits/sec
[  4]  0.0-10.0 sec  3.90 GBytes  3.35 Gbits/sec
[  3]  0.0-10.0 sec  3.96 GBytes  3.40 Gbits/sec
[SUM]  0.0-10.0 sec  65.6 GBytes  56.4 Gbits/sec
[ 17] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53716
[ 15] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53715
[ 12] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53718
[ 13] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53717
[ 11] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53720
[ 18] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53719
[  7] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53721
[  8] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53722
[  9] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53723
[  6] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53724
[ 10] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53725
[  3] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53727
[  4] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53726
[  5] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53728
[ 14] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53730
[ 19] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53729
[ 17]  0.0-10.0 sec  3.60 GBytes  3.08 Gbits/sec
[ 15]  0.0-10.0 sec  3.62 GBytes  3.10 Gbits/sec
[ 12]  0.0-10.0 sec  5.45 GBytes  4.67 Gbits/sec
[ 13]  0.0-10.0 sec  4.24 GBytes  3.63 Gbits/sec
[ 11]  0.0-10.0 sec  4.74 GBytes  4.06 Gbits/sec
[ 18]  0.0-10.0 sec  5.31 GBytes  4.55 Gbits/sec
[  7]  0.0-10.0 sec  3.63 GBytes  3.11 Gbits/sec
[  8]  0.0-10.0 sec  3.55 GBytes  3.04 Gbits/sec
[  9]  0.0-10.0 sec  3.55 GBytes  3.04 Gbits/sec
[  6]  0.0-10.0 sec  3.61 GBytes  3.09 Gbits/sec
[ 10]  0.0-10.0 sec  3.64 GBytes  3.11 Gbits/sec
[  3]  0.0-10.0 sec  3.56 GBytes  3.04 Gbits/sec
[  4]  0.0-10.0 sec  3.62 GBytes  3.10 Gbits/sec
[  5]  0.0-10.0 sec  3.68 GBytes  3.15 Gbits/sec
[ 14]  0.0-10.0 sec  3.57 GBytes  3.06 Gbits/sec
[ 19]  0.0-10.0 sec  5.13 GBytes  4.39 Gbits/sec
[SUM]  0.0-10.0 sec  64.5 GBytes  55.2 Gbits/sec

C:\iperf\iperf-2.0.9-win64>iperf.exe -c 192.168.10.27 -P 8 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.10.27, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  6] local 192.168.10.26 port 49441 connected with 192.168.10.27 port 5001
[ 11] local 192.168.10.26 port 49445 connected with 192.168.10.27 port 5001
[ 10] local 192.168.10.26 port 49444 connected with 192.168.10.27 port 5001
[  9] local 192.168.10.26 port 49443 connected with 192.168.10.27 port 5001
[  3] local 192.168.10.26 port 49438 connected with 192.168.10.27 port 5001
[  7] local 192.168.10.26 port 49442 connected with 192.168.10.27 port 5001
[  5] local 192.168.10.26 port 49440 connected with 192.168.10.27 port 5001
[  4] local 192.168.10.26 port 49439 connected with 192.168.10.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  8.87 GBytes  7.62 Gbits/sec
[ 11]  0.0-10.0 sec  7.57 GBytes  6.50 Gbits/sec
[ 10]  0.0-10.0 sec  8.09 GBytes  6.95 Gbits/sec
[  9]  0.0-10.0 sec  8.18 GBytes  7.03 Gbits/sec
[  3]  0.0-10.0 sec  8.49 GBytes  7.29 Gbits/sec
[  7]  0.0-10.0 sec  9.07 GBytes  7.79 Gbits/sec
[  5]  0.0-10.0 sec  8.16 GBytes  7.01 Gbits/sec
[  4]  0.0-10.0 sec  7.53 GBytes  6.47 Gbits/sec
[SUM]  0.0-10.0 sec  66.0 GBytes  56.7 Gbits/sec
[  3] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53736
[  5] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53735
[  6] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53734
[  4] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53733
[  7] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53737
[  9] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53738
[ 10] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53740
[ 11] local 192.168.10.26 port 5001 connected with 192.168.10.27 port 53739
[  3]  0.0-10.0 sec  7.62 GBytes  6.54 Gbits/sec
[  5]  0.0-10.0 sec  7.21 GBytes  6.19 Gbits/sec
[  6]  0.0-10.0 sec  7.25 GBytes  6.22 Gbits/sec
[  4]  0.0-10.0 sec  10.5 GBytes  9.03 Gbits/sec
[  7]  0.0-10.0 sec  7.23 GBytes  6.21 Gbits/sec
[  9]  0.0-10.0 sec  10.1 GBytes  8.66 Gbits/sec
[ 10]  0.0-10.0 sec  7.26 GBytes  6.23 Gbits/sec
[ 11]  0.0-10.0 sec  7.18 GBytes  6.16 Gbits/sec
[SUM]  0.0-10.0 sec  64.4 GBytes  55.2 Gbits/sec


This is on an X11DPI-NT, no Bios optimization (in fact I just reset it after a bios update to 3.1)
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
I am playing around with this, but still no magic bullets here. I haven't tried jumbo frames, but I kind of don't want to go there. I'd like to enable TOE, but that jacks up NFS for as yet undetermined reasons. One thing that I tried while playing was disabling TSO, and that had a HUGE negative impact. I went from ~=22G down to ~=10G throughput. Extra threads on iperf3 (-P8 for the 8 physical cores) doesn't seem to help any. I can certainly brush up against 20G pretty consistently, but I still feel a bit unsatisfied by that.
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
iperf3 wont do P8 properly iirc, try iperf 2

Not sure why u are using iperf3 anyway, the binary on FreeNas is iperf2?

Code:
root@freenas[~]# which iperf

/usr/local/bin/iperf

root@freenas[~]# /usr/local/bin/iperf -v

iperf version 2.0.10 (11 Aug 2017) pthreads

root@freenas[~]#
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
Rand said:
iperf3 won't do P8 properly iirc, try iperf 2
Click to expand...
I hadn't tried that, but I got ~=37G from iperf2. I wouldn't have expected such a striking difference, but it is certainly there. 3 iterations with 4M window size and 8 threads, and I got 37.7G on both systems. Now that is pretty good!
Rand said:
Not sure why u are using iperf3 anyway, the binary on FreeNas is iperf2?
Click to expand...
It has both. I guess I foolishly assumed that iperf3 would be better than iperf2, but it would appear not. Then again, maybe it is just telling me what I want to hear....
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Some performance tests...
Windows 2012 to a Optane4800x mirror, SMB, async, noatime, 128K recordsize
1T1J ->879MB/s
16T16J ->3645MB/s
64T64J (cancelled) -> 3510MB/s
edit: 16T16J w 256K blocksize in fio -> 4281MB/s

Note that the FreeNas box has a significant amount of memory so will have cached a bit of that (But thats fine since we are looking into network performance here).
gstat showed only ~55% utilization of the drives though, not sure what was limiting them on higher T/J counts

Code:
Z:\>fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio  --size=100G --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=fio.test -name win1
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
win1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=windowsaio, iodepth=1
fio-3.16
Starting 1 thread
win1: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=817MiB/s][w=6533 IOPS][eta 00m:00s]
win1: (groupid=0, jobs=1): err= 0: pid=5284: Sun Dec 29 19:36:16 2019
  write: IOPS=6708, BW=839MiB/s (879MB/s)(100GiB/122116msec)
    slat (usec): min=15, max=604377, avg=113.73, stdev=671.36
    clat (nsec): min=241, max=391319, avg=8071.63, stdev=4381.48
     lat (usec): min=88, max=604394, avg=121.80, stdev=671.44
    clat percentiles (nsec):
     |  1.00th=[ 6176],  5.00th=[ 6368], 10.00th=[ 6368], 20.00th=[ 6432],
     | 30.00th=[ 6496], 40.00th=[ 6496], 50.00th=[ 6560], 60.00th=[ 6688],
     | 70.00th=[ 6944], 80.00th=[ 7328], 90.00th=[13248], 95.00th=[19328],
     | 99.00th=[24960], 99.50th=[29312], 99.90th=[39680], 99.95th=[46336],
     | 99.99th=[59136]
   bw (  KiB/s): min=751904, max=962304, per=100.00%, avg=861146.40, stdev=48995.71, samples=243
   iops        : min= 5874, max= 7518, avg=6727.46, stdev=382.73, samples=243
  lat (nsec)   : 250=0.01%, 500=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.05%, 10=87.80%, 20=7.24%, 50=4.86%
  lat (usec)   : 100=0.03%, 500=0.01%
  cpu          : usr=19.65%, sys=16.38%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,819200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=839MiB/s (879MB/s), 839MiB/s-839MiB/s (879MB/s-879MB/s), io=100GiB (107GB), run=122116-122116msec

Z:\>fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio  --size=100G --bs=128k --iodepth=16 --numjobs=16 --rw=write --filename=fio.test -name win1
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
win1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=windowsaio, iodepth=16
...
fio-3.16
Starting 16 threads
Jobs: 16 (f=16): [W(16)][99.8%][w=3307MiB/s][w=26.5k IOPS][eta 00m:01s]
win1: (groupid=0, jobs=16): err= 0: pid=6004: Sun Dec 29 19:45:30 2019
  write: IOPS=27.8k, BW=3476MiB/s (3645MB/s)(1600GiB/471327msec)
    slat (usec): min=3, max=4089, avg=11.43, stdev= 6.29
    clat (usec): min=110, max=362591, avg=9160.72, stdev=2262.43
     lat (usec): min=125, max=362602, avg=9172.15, stdev=2262.35
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 8160], 10.00th=[ 8291], 20.00th=[ 8455],
     | 30.00th=[ 8586], 40.00th=[ 8717], 50.00th=[ 8848], 60.00th=[ 9110],
     | 70.00th=[ 9241], 80.00th=[ 9634], 90.00th=[10421], 95.00th=[11076],
     | 99.00th=[12387], 99.50th=[12911], 99.90th=[13566], 99.95th=[13829],
     | 99.99th=[14746]
   bw (  MiB/s): min= 1406, max= 3969, per=99.99%, avg=3475.80, stdev=10.18, samples=15072
   iops        : min=11254, max=31755, avg=27805.70, stdev=81.47, samples=15072
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=84.75%, 20=15.22%, 50=0.01%
  lat (msec)   : 500=0.01%
  cpu          : usr=5.16%, sys=1.91%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13107200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=3476MiB/s (3645MB/s), 3476MiB/s-3476MiB/s (3645MB/s-3645MB/s), io=16
00GiB (1718GB), run=471327-471327msec

Z:\>fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio  --size=100G --bs=128k --iodepth=64 --numjobs=64 --rw=write --filename=fio.test -name win1
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
win1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=windowsaio, iodepth=64
...
fio-3.16
Starting 64 threads
Jobs: 64 (f=64): [W(64)][13.9%][w=3352MiB/s][w=26.8k IOPS][eta 28m:11s]
fio: terminating on signal 2

win1: (groupid=0, jobs=64): err= 0: pid=5252: Sun Dec 29 19:51:50 2019
  write: IOPS=26.8k, BW=3347MiB/s (3510MB/s)(891GiB/272726msec)
    slat (usec): min=3, max=70025, avg=14.82, stdev=49.38
    clat (usec): min=168, max=303734, avg=152738.44, stdev=5037.43
     lat (usec): min=423, max=303745, avg=152753.26, stdev=5027.99
    clat percentiles (msec):
     |  1.00th=[  148],  5.00th=[  148], 10.00th=[  150], 20.00th=[  150],
     | 30.00th=[  153], 40.00th=[  153], 50.00th=[  153], 60.00th=[  155],
     | 70.00th=[  155], 80.00th=[  155], 90.00th=[  157], 95.00th=[  157],
     | 99.00th=[  159], 99.50th=[  161], 99.90th=[  163], 99.95th=[  167],
     | 99.99th=[  268]
   bw (  MiB/s): min= 2742, max= 4828, per=100.00%, avg=3349.49, stdev= 4.24, samples=34837
   iops        : min=21935, max=38625, avg=26795.24, stdev=33.89, samples=34837
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.05%
  lat (msec)   : 100=0.04%, 250=99.87%, 500=0.02%
  cpu          : usr=1.42%, sys=0.41%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,7302503,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=3347MiB/s (3510MB/s), 3347MiB/s-3347MiB/s (3510MB/s-3510MB/s), io=891GiB (957GB), run=272726-272726msec
 
Last edited:
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
I had an unpleasant wrinkle yesterday. My primary FreeNAS had an unexpected reboot. I undid all my T5 specific tuning, and reset the T5 cards to default in their BIOS on both boxes. I am going to let them run a day or so and make sure I don't see any more furriness. On a positive note, I could still get 37.7G throughput between the two boxes with iperf. I guess the tuning wasn't really required. Ugh.
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
T5 cards have their own bios? Do T6 also? never noticed that:p

Hope its not going to cause you any more trouble...
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
Rand said:
T5 cards have their own bios? Do T6 also? never noticed that
Click to expand...
The T5's definitely have a BIOS. They can do PXE, iSCSI, or FCoE boot. No idea on T6's as I have never had my hands on one.
Rand said:
Hope its not going to cause you any more trouble...
Click to expand...
No kidding! Everything had been extremely stable. I thought I had some problem on the network side, and it turns out it was just iperf3 giving me inaccurate results. Ugh....
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Elliot Dierksen said:
The T5's definitely have a BIOS. They can do PXE, iSCSI, or FCoE boot. No idea on T6's as I have never had my hands on one.
Click to expand...
Well that wouldnt necessarily mean that you can enter that bios and change the config as you implied? Or have I misunderstood you there?
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
Here is what I get during the BIOS/POST.
1577737127240.png

When I hit Alt-C, it prompts me to select adapters (I only have one).
1577737163054.png

Here are the options it gives me once I select the adapter.
1577737189930.png

I had changed the EDD selection from 2.1 to 3.0 the other day. I don't know if that is what caused the errors, but I am pretty sure that was the only option that I changed. Below are the system event messages from when it rebooted.
1577737254330.png

I am sure that is the T580 because that matches the info I get from a pciconf -lv
Code:
t5iov0@pci0:131:0:0:    class=0x020000 card=0x00001425 chip=0x50101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov1@pci0:131:0:1:    class=0x020000 card=0x00001425 chip=0x50101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov2@pci0:131:0:2:    class=0x020000 card=0x00001425 chip=0x50101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov3@pci0:131:0:3:    class=0x020000 card=0x00001425 chip=0x50101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5nex0@pci0:131:0:4:    class=0x020000 card=0x00001425 chip=0x54101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
none42@pci0:131:0:5:    class=0x010000 card=0x00001425 chip=0x55101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Storage Controller'
    class      = mass storage
    subclass   = SCSI
none43@pci0:131:0:6:    class=0x0c0400 card=0x00001425 chip=0x56101425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T580-LP-CR Unified Wire Storage Controller'
    class      = serial bus
    subclass   = Fibre Channel
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Wow thats some real ancient stuff you can config there :O
=> https://lwn.net/Articles/9042/

Thanks for showing, doesn't look like i'd need to look for that on the T6's. Likely all related to remote disk booting from the looks
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
Rand said:
Wow thats some real ancient stuff you can config there
Click to expand...
Honestly I didn't even know what that was, and I am a bit of a grey beard.
Rand said:
Thanks for showing, doesn't look like i'd need to look for that on the T6's. Likely all related to remote disk booting from the looks
Click to expand...
It could also be that the card you have is only a NIC. I think a bunch of the stuff I have is because it is a unified wire adapter.
 
R

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Nah the 62100-LP-CR is also one , I just never looked at it booting ;)
 
You must log in or register to reply here.

Similar threads

pclausen
  • Locked
Performance issue with 10Gbps network
2 3 4
Replies
78
Views
19K
jgreco
jgreco
E
Cannot get full 10Gbe on Tx
Replies
11
Views
3K
VolumeTank
VolumeTank
D
  • Locked
Tweaking FreeNAS and ESXi for 10Gbps
Replies
2
Views
1K
bigphil
bigphil
F
10GbE only after reboot? Then drops to 2Gb/s...
Replies
4
Views
2K
fraenki
F
Brosif_My_Nif
X540-T2 10GB Very Slow - 700 to 900 KB/s Transfer
2
Replies
25
Views
10K
{HD}
{HD}
Top