Help understanding the bottleneck 100Gb/s

jarble · Jun 27, 2023

Hello all. I recently jumped from 10Gb/s networking to 100Gb/s networking and I am running into the weirdest issue and I am not sure where my problem is located. To start the problem is that I can only transfer at 10Gb/s read and 20Gb/s write on the network. This is a consistent speed limit and I can't understand why. I have tested both my striped nvme array and my spinning rust array and they both return the same results. More perplexing is that if I run iperf on the machines I can validate at 80Gb/s so I know the sum of all the parts are working correctly but there is something in the actual file transfer that is broken. I have tested in both iscsi and smb and they return the exact same limit. I am grasping at straws here as I dont know where to focus my efforts as there are a litany of possible causes so anyone with some experience that could shed some light on the matter would be most appreciated.

Truenas system
EPYC 7642 CPU's
Supermicro H12DSi-N6
512GB RAM
16 - 16TB HDD in four raid z1 volumes
4 - 1TB NVME in a stripe
Mellanox 100GB nic

Workstation
Threadriper 3970x
Gigabyte aurous extreme
128GB ram
8TB sabrent rocket pcie4.0 drive
Mellanox 100GB nic

Switch
MikroTik CRS504-4XQ-IN

Arwen · Jun 28, 2023

I have seen asymmetrical NIC routing. Where traffic goes out 1 NIC, but comes back through another. If their are different speeds for the NICs, that could be the problem.

Just to confirm, you have exactly 1 NIC active on both NAS & client?
And that NIC is the Mellonox 100Gbit/ps?

Beyond that, this is outside my skill set.

jarble · Jun 28, 2023

Arwen said:
I have seen asymmetrical NIC routing. Where traffic goes out 1 NIC, but comes back through another. If their are different speeds for the NICs, that could be the problem.

Just to confirm, you have exactly 1 NIC active on both NAS & client?
And that NIC is the Mellonox 100Gbit/ps?

Beyond that, this is outside my skill set.

Thanks for the reply. To confirm I only have the 100Gbit/s nic in both the client and server. No other nic's are active save for ipmi but that is on a different network entirely.

Arwen · Jun 28, 2023

Thanks for the confirmation. Someone else will have to take over... out of my skill depth.

indy · Jun 28, 2023

Can you post your iperf3 results in full?

jarble · Jun 28, 2023

I get the best results with iperf2 so here is client and server. I can also run a iperf3 if needed I just dont know the best commands on it.

morganL · Jun 28, 2023

What is the client OS and what are the specific tests you are performing on Client to measure throughput?

FYI.. we typically run 10 clients to test a 100Gbe NAS.

NickF · Jun 28, 2023

When we are talking about speeds THIS high, your single stream performance isn't going to be illustrative of your maximum performance with multiple clients. To set expectations here, this also likely translates to being unable to saturate a 100 Gigabit link from a single client in terms of file sharing, block devices, or whatever you are trying to do, even if the backing pool can.
You should run your iperf3 client with parameters to test multiple streams. This documentation will confirm this. https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/#:~:text=To run multiple iperf3 processes for a testing,iperf3 -c hostname -T s3 -p 5103 &;

For example, my production homelab SCALE box has a single 100 Gigabit Mellanox ConnectX4, plugged into a 40 gigabit port with a 40 gigabit DAC. I have an Epyc 7282 CPU so I'm not far off from you. My workstation has a EPYC 73F2 with a 25 Gigabit Mellanox ConnectX4 card, but I am using 10 Gigabit optics. It's also going through another switch, FWIW, so a bit of extra latency.

With only a single stream, I get:

Code:

C:\Users\nickf.FUSCO\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.57 port 49608 connected to 10.69.10.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   854 MBytes  7.17 Gbits/sec
[  4]   1.00-2.00   sec   874 MBytes  7.33 Gbits/sec
[  4]   2.00-3.00   sec   840 MBytes  7.04 Gbits/sec
[  4]   3.00-4.00   sec   815 MBytes  6.83 Gbits/sec
[  4]   4.00-5.00   sec   865 MBytes  7.26 Gbits/sec
[  4]   5.00-6.00   sec   865 MBytes  7.25 Gbits/sec
[  4]   6.00-7.00   sec   845 MBytes  7.09 Gbits/sec
[  4]   7.00-8.00   sec   811 MBytes  6.80 Gbits/sec
[  4]   8.00-9.00   sec   827 MBytes  6.94 Gbits/sec
[  4]   9.00-10.00  sec   849 MBytes  7.12 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  8.25 GBytes  7.08 Gbits/sec                  sender
[  4]   0.00-10.00  sec  8.25 GBytes  7.08 Gbits/sec                  receiver

iperf Done.

C:\Users\nickf.FUSCO\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>

But if I set the streams to 10, as @morganL suggests for 10 concurrent clients, I get closer to 10Gbit/sec

Code:

C:\Users\nickf.FUSCO\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -c prod.fusco.me -P 10
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.57 port 49690 connected to 10.69.10.8 port 5201
[  6] local 10.69.10.57 port 49691 connected to 10.69.10.8 port 5201
[  8] local 10.69.10.57 port 49692 connected to 10.69.10.8 port 5201
[ 10] local 10.69.10.57 port 49693 connected to 10.69.10.8 port 5201
[ 12] local 10.69.10.57 port 49694 connected to 10.69.10.8 port 5201
[ 14] local 10.69.10.57 port 49695 connected to 10.69.10.8 port 5201
[ 16] local 10.69.10.57 port 49696 connected to 10.69.10.8 port 5201
[ 18] local 10.69.10.57 port 49697 connected to 10.69.10.8 port 5201
[ 20] local 10.69.10.57 port 49698 connected to 10.69.10.8 port 5201
[ 22] local 10.69.10.57 port 49699 connected to 10.69.10.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  89.2 MBytes   749 Mbits/sec
[  6]   0.00-1.00   sec   127 MBytes  1.07 Gbits/sec
[  8]   0.00-1.00   sec   121 MBytes  1.01 Gbits/sec
[ 10]   0.00-1.00   sec  49.6 MBytes   416 Mbits/sec
[ 12]   0.00-1.00   sec  71.9 MBytes   603 Mbits/sec
[ 14]   0.00-1.00   sec   123 MBytes  1.03 Gbits/sec
[ 16]   0.00-1.00   sec   109 MBytes   911 Mbits/sec
[ 18]   0.00-1.00   sec   138 MBytes  1.16 Gbits/sec
[ 20]   0.00-1.00   sec  94.4 MBytes   792 Mbits/sec
[ 22]   0.00-1.00   sec   118 MBytes   993 Mbits/sec
[SUM]   0.00-1.00   sec  1.02 GBytes  8.74 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  91.1 MBytes   764 Mbits/sec
[  6]   1.00-2.00   sec   104 MBytes   877 Mbits/sec
[  8]   1.00-2.00   sec   117 MBytes   984 Mbits/sec
[ 10]   1.00-2.00   sec   106 MBytes   886 Mbits/sec
[ 12]   1.00-2.00   sec  93.4 MBytes   783 Mbits/sec
[ 14]   1.00-2.00   sec   110 MBytes   923 Mbits/sec
[ 16]   1.00-2.00   sec   126 MBytes  1.05 Gbits/sec
[ 18]   1.00-2.00   sec   132 MBytes  1.10 Gbits/sec
[ 20]   1.00-2.00   sec   114 MBytes   957 Mbits/sec
[ 22]   1.00-2.00   sec   116 MBytes   976 Mbits/sec
[SUM]   1.00-2.00   sec  1.08 GBytes  9.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  91.0 MBytes   763 Mbits/sec
[  6]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  8]   2.00-3.00   sec   107 MBytes   900 Mbits/sec
[ 10]   2.00-3.00   sec   112 MBytes   938 Mbits/sec
[ 12]   2.00-3.00   sec   117 MBytes   984 Mbits/sec
[ 14]   2.00-3.00   sec   116 MBytes   973 Mbits/sec
[ 16]   2.00-3.00   sec   116 MBytes   974 Mbits/sec
[ 18]   2.00-3.00   sec   122 MBytes  1.03 Gbits/sec
[ 20]   2.00-3.00   sec   100 MBytes   839 Mbits/sec
[ 22]   2.00-3.00   sec  96.2 MBytes   807 Mbits/sec
[SUM]   2.00-3.00   sec  1.07 GBytes  9.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   147 MBytes  1.23 Gbits/sec
[  6]   3.00-4.00   sec   108 MBytes   904 Mbits/sec
[  8]   3.00-4.00   sec  94.0 MBytes   789 Mbits/sec
[ 10]   3.00-4.00   sec   118 MBytes   988 Mbits/sec
[ 12]   3.00-4.00   sec   103 MBytes   862 Mbits/sec
[ 14]   3.00-4.00   sec   115 MBytes   962 Mbits/sec
[ 16]   3.00-4.00   sec   114 MBytes   955 Mbits/sec
[ 18]   3.00-4.00   sec  87.6 MBytes   735 Mbits/sec
[ 20]   3.00-4.00   sec   102 MBytes   860 Mbits/sec
[ 22]   3.00-4.00   sec  92.0 MBytes   772 Mbits/sec
[SUM]   3.00-4.00   sec  1.05 GBytes  9.06 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  94.9 MBytes   796 Mbits/sec
[  6]   4.00-5.00   sec  94.1 MBytes   789 Mbits/sec
[  8]   4.00-5.00   sec   106 MBytes   888 Mbits/sec
[ 10]   4.00-5.00   sec  96.9 MBytes   813 Mbits/sec
[ 12]   4.00-5.00   sec   114 MBytes   955 Mbits/sec
[ 14]   4.00-5.00   sec   118 MBytes   990 Mbits/sec
[ 16]   4.00-5.00   sec  99.0 MBytes   830 Mbits/sec
[ 18]   4.00-5.00   sec   132 MBytes  1.10 Gbits/sec
[ 20]   4.00-5.00   sec   111 MBytes   930 Mbits/sec
[ 22]   4.00-5.00   sec   118 MBytes   988 Mbits/sec
[SUM]   4.00-5.00   sec  1.06 GBytes  9.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec  98.0 MBytes   822 Mbits/sec
[  6]   5.00-6.00   sec  89.9 MBytes   754 Mbits/sec
[  8]   5.00-6.00   sec  78.5 MBytes   659 Mbits/sec
[ 10]   5.00-6.00   sec   116 MBytes   972 Mbits/sec
[ 12]   5.00-6.00   sec   112 MBytes   944 Mbits/sec
[ 14]   5.00-6.00   sec   108 MBytes   903 Mbits/sec
[ 16]   5.00-6.00   sec   143 MBytes  1.20 Gbits/sec
[ 18]   5.00-6.00   sec   110 MBytes   921 Mbits/sec
[ 20]   5.00-6.00   sec   114 MBytes   957 Mbits/sec
[ 22]   5.00-6.00   sec   139 MBytes  1.17 Gbits/sec
[SUM]   5.00-6.00   sec  1.08 GBytes  9.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  86.9 MBytes   729 Mbits/sec
[  6]   6.00-7.00   sec   115 MBytes   967 Mbits/sec
[  8]   6.00-7.00   sec   125 MBytes  1.05 Gbits/sec
[ 10]   6.00-7.00   sec  99.9 MBytes   838 Mbits/sec
[ 12]   6.00-7.00   sec   103 MBytes   864 Mbits/sec
[ 14]   6.00-7.00   sec   109 MBytes   911 Mbits/sec
[ 16]   6.00-7.00   sec   102 MBytes   853 Mbits/sec
[ 18]   6.00-7.00   sec   116 MBytes   976 Mbits/sec
[ 20]   6.00-7.00   sec   129 MBytes  1.08 Gbits/sec
[ 22]   6.00-7.00   sec   112 MBytes   943 Mbits/sec
[SUM]   6.00-7.00   sec  1.07 GBytes  9.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  96.8 MBytes   811 Mbits/sec
[  6]   7.00-8.00   sec  86.8 MBytes   727 Mbits/sec
[  8]   7.00-8.00   sec   116 MBytes   973 Mbits/sec
[ 10]   7.00-8.00   sec   107 MBytes   899 Mbits/sec
[ 12]   7.00-8.00   sec   143 MBytes  1.20 Gbits/sec
[ 14]   7.00-8.00   sec  81.4 MBytes   682 Mbits/sec
[ 16]   7.00-8.00   sec   115 MBytes   964 Mbits/sec
[ 18]   7.00-8.00   sec  91.1 MBytes   764 Mbits/sec
[ 20]   7.00-8.00   sec   111 MBytes   933 Mbits/sec
[ 22]   7.00-8.00   sec   132 MBytes  1.11 Gbits/sec
[SUM]   7.00-8.00   sec  1.06 GBytes  9.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   109 MBytes   911 Mbits/sec
[  6]   8.00-9.00   sec   114 MBytes   959 Mbits/sec
[  8]   8.00-9.00   sec  99.6 MBytes   836 Mbits/sec
[ 10]   8.00-9.00   sec   111 MBytes   928 Mbits/sec
[ 12]   8.00-9.00   sec   102 MBytes   854 Mbits/sec
[ 14]   8.00-9.00   sec   127 MBytes  1.07 Gbits/sec
[ 16]   8.00-9.00   sec   120 MBytes  1.01 Gbits/sec
[ 18]   8.00-9.00   sec   121 MBytes  1.01 Gbits/sec
[ 20]   8.00-9.00   sec   108 MBytes   902 Mbits/sec
[ 22]   8.00-9.00   sec   106 MBytes   887 Mbits/sec
[SUM]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   126 MBytes  1.06 Gbits/sec
[  6]   9.00-10.00  sec   120 MBytes  1.00 Gbits/sec
[  8]   9.00-10.00  sec   121 MBytes  1.02 Gbits/sec
[ 10]   9.00-10.00  sec  93.2 MBytes   782 Mbits/sec
[ 12]   9.00-10.00  sec   121 MBytes  1.01 Gbits/sec
[ 14]   9.00-10.00  sec   119 MBytes   995 Mbits/sec
[ 16]   9.00-10.00  sec   109 MBytes   913 Mbits/sec
[ 18]   9.00-10.00  sec   116 MBytes   974 Mbits/sec
[ 20]   9.00-10.00  sec  94.1 MBytes   790 Mbits/sec
[ 22]   9.00-10.00  sec   103 MBytes   861 Mbits/sec
[SUM]   9.00-10.00  sec  1.10 GBytes  9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.01 GBytes   864 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.01 GBytes   864 Mbits/sec                  receiver
[  6]   0.00-10.00  sec  1.05 GBytes   899 Mbits/sec                  sender
[  6]   0.00-10.00  sec  1.05 GBytes   899 Mbits/sec                  receiver
[  8]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec                  sender
[  8]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec  1009 MBytes   846 Mbits/sec                  sender
[ 10]   0.00-10.00  sec  1009 MBytes   846 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec  1.06 GBytes   906 Mbits/sec                  sender
[ 12]   0.00-10.00  sec  1.06 GBytes   906 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  sender
[ 14]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec  1.13 GBytes   966 Mbits/sec                  sender
[ 16]   0.00-10.00  sec  1.13 GBytes   966 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec  1.14 GBytes   977 Mbits/sec                  sender
[ 18]   0.00-10.00  sec  1.14 GBytes   977 Mbits/sec                  receiver
[ 20]   0.00-10.00  sec  1.05 GBytes   904 Mbits/sec                  sender
[ 20]   0.00-10.00  sec  1.05 GBytes   904 Mbits/sec                  receiver
[ 22]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec                  sender
[ 22]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.7 GBytes  9.17 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  10.7 GBytes  9.17 Gbits/sec                  receiver

You can parallelize the server side as well, see the link I posted above. The scaling for 100 gigabit will indefinitely be higher than my results. But even with parallelization on both ends, I don't think you'll see 100 gigabit in this test. That statement is regardless of the OS, TrueNAS or otherwise...

On the storage side of this conversation:
I have an Optane pool with 2 960GB mirrors (4 drives). Even that system - best case scenario in theoretical-land and not reality-ville - is probably less than 50 Gigabit's worth of pool.

jarble · Jun 29, 2023

morganL said:
What is the client OS and what are the specific tests you are performing on Client to measure throughput?

FYI.. we typically run 10 clients to test a 100Gbe NAS.

Thanks for the reply. The client is running windows 11pro and for test I have done both simple file copies and crystaldiskmark with both returning the same odd 10 down 20 up scenario.

jarble · Jun 29, 2023

NickF said:

When we are talking about speeds THIS high, your single stream performance isn't going to be illustrative of your maximum performance with multiple clients. To set expectations here, this also likely translates to being unable to saturate a 100 Gigabit link from a single client in terms of file sharing, block devices, or whatever you are trying to do, even if the backing pool can.
You should run your iperf3 client with parameters to test multiple streams. This documentation will confirm this. https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/#:~:text=To run multiple iperf3 processes for a testing,iperf3 -c hostname -T s3 -p 5103 &;

For example, my production homelab SCALE box has a single 100 Gigabit Mellanox ConnectX4, plugged into a 40 gigabit port with a 40 gigabit DAC. I have an Epyc 7282 CPU so I'm not far off from you. My workstation has a EPYC 73F2 with a 25 Gigabit Mellanox ConnectX4 card, but I am using 10 Gigabit optics. It's also going through another switch, FWIW, so a bit of extra latency.

With only a single stream, I get:

Code:

C:\Users\nickf.FUSCO\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.57 port 49608 connected to 10.69.10.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   854 MBytes  7.17 Gbits/sec
[  4]   1.00-2.00   sec   874 MBytes  7.33 Gbits/sec
[  4]   2.00-3.00   sec   840 MBytes  7.04 Gbits/sec
[  4]   3.00-4.00   sec   815 MBytes  6.83 Gbits/sec
[  4]   4.00-5.00   sec   865 MBytes  7.26 Gbits/sec
[  4]   5.00-6.00   sec   865 MBytes  7.25 Gbits/sec
[  4]   6.00-7.00   sec   845 MBytes  7.09 Gbits/sec
[  4]   7.00-8.00   sec   811 MBytes  6.80 Gbits/sec
[  4]   8.00-9.00   sec   827 MBytes  6.94 Gbits/sec
[  4]   9.00-10.00  sec   849 MBytes  7.12 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  8.25 GBytes  7.08 Gbits/sec                  sender
[  4]   0.00-10.00  sec  8.25 GBytes  7.08 Gbits/sec                  receiver

iperf Done.

C:\Users\nickf.FUSCO\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>

But if I set the streams to 10, as @morganL suggests for 10 concurrent clients, I get closer to 10Gbit/sec

Code:

C:\Users\nickf.FUSCO\Downloads\iperf-3.1.3-win64 (1)\iperf-3.1.3-win64>iperf3 -c prod.fusco.me -P 10
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.57 port 49690 connected to 10.69.10.8 port 5201
[  6] local 10.69.10.57 port 49691 connected to 10.69.10.8 port 5201
[  8] local 10.69.10.57 port 49692 connected to 10.69.10.8 port 5201
[ 10] local 10.69.10.57 port 49693 connected to 10.69.10.8 port 5201
[ 12] local 10.69.10.57 port 49694 connected to 10.69.10.8 port 5201
[ 14] local 10.69.10.57 port 49695 connected to 10.69.10.8 port 5201
[ 16] local 10.69.10.57 port 49696 connected to 10.69.10.8 port 5201
[ 18] local 10.69.10.57 port 49697 connected to 10.69.10.8 port 5201
[ 20] local 10.69.10.57 port 49698 connected to 10.69.10.8 port 5201
[ 22] local 10.69.10.57 port 49699 connected to 10.69.10.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  89.2 MBytes   749 Mbits/sec
[  6]   0.00-1.00   sec   127 MBytes  1.07 Gbits/sec
[  8]   0.00-1.00   sec   121 MBytes  1.01 Gbits/sec
[ 10]   0.00-1.00   sec  49.6 MBytes   416 Mbits/sec
[ 12]   0.00-1.00   sec  71.9 MBytes   603 Mbits/sec
[ 14]   0.00-1.00   sec   123 MBytes  1.03 Gbits/sec
[ 16]   0.00-1.00   sec   109 MBytes   911 Mbits/sec
[ 18]   0.00-1.00   sec   138 MBytes  1.16 Gbits/sec
[ 20]   0.00-1.00   sec  94.4 MBytes   792 Mbits/sec
[ 22]   0.00-1.00   sec   118 MBytes   993 Mbits/sec
[SUM]   0.00-1.00   sec  1.02 GBytes  8.74 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  91.1 MBytes   764 Mbits/sec
[  6]   1.00-2.00   sec   104 MBytes   877 Mbits/sec
[  8]   1.00-2.00   sec   117 MBytes   984 Mbits/sec
[ 10]   1.00-2.00   sec   106 MBytes   886 Mbits/sec
[ 12]   1.00-2.00   sec  93.4 MBytes   783 Mbits/sec
[ 14]   1.00-2.00   sec   110 MBytes   923 Mbits/sec
[ 16]   1.00-2.00   sec   126 MBytes  1.05 Gbits/sec
[ 18]   1.00-2.00   sec   132 MBytes  1.10 Gbits/sec
[ 20]   1.00-2.00   sec   114 MBytes   957 Mbits/sec
[ 22]   1.00-2.00   sec   116 MBytes   976 Mbits/sec
[SUM]   1.00-2.00   sec  1.08 GBytes  9.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  91.0 MBytes   763 Mbits/sec
[  6]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  8]   2.00-3.00   sec   107 MBytes   900 Mbits/sec
[ 10]   2.00-3.00   sec   112 MBytes   938 Mbits/sec
[ 12]   2.00-3.00   sec   117 MBytes   984 Mbits/sec
[ 14]   2.00-3.00   sec   116 MBytes   973 Mbits/sec
[ 16]   2.00-3.00   sec   116 MBytes   974 Mbits/sec
[ 18]   2.00-3.00   sec   122 MBytes  1.03 Gbits/sec
[ 20]   2.00-3.00   sec   100 MBytes   839 Mbits/sec
[ 22]   2.00-3.00   sec  96.2 MBytes   807 Mbits/sec
[SUM]   2.00-3.00   sec  1.07 GBytes  9.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   147 MBytes  1.23 Gbits/sec
[  6]   3.00-4.00   sec   108 MBytes   904 Mbits/sec
[  8]   3.00-4.00   sec  94.0 MBytes   789 Mbits/sec
[ 10]   3.00-4.00   sec   118 MBytes   988 Mbits/sec
[ 12]   3.00-4.00   sec   103 MBytes   862 Mbits/sec
[ 14]   3.00-4.00   sec   115 MBytes   962 Mbits/sec
[ 16]   3.00-4.00   sec   114 MBytes   955 Mbits/sec
[ 18]   3.00-4.00   sec  87.6 MBytes   735 Mbits/sec
[ 20]   3.00-4.00   sec   102 MBytes   860 Mbits/sec
[ 22]   3.00-4.00   sec  92.0 MBytes   772 Mbits/sec
[SUM]   3.00-4.00   sec  1.05 GBytes  9.06 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  94.9 MBytes   796 Mbits/sec
[  6]   4.00-5.00   sec  94.1 MBytes   789 Mbits/sec
[  8]   4.00-5.00   sec   106 MBytes   888 Mbits/sec
[ 10]   4.00-5.00   sec  96.9 MBytes   813 Mbits/sec
[ 12]   4.00-5.00   sec   114 MBytes   955 Mbits/sec
[ 14]   4.00-5.00   sec   118 MBytes   990 Mbits/sec
[ 16]   4.00-5.00   sec  99.0 MBytes   830 Mbits/sec
[ 18]   4.00-5.00   sec   132 MBytes  1.10 Gbits/sec
[ 20]   4.00-5.00   sec   111 MBytes   930 Mbits/sec
[ 22]   4.00-5.00   sec   118 MBytes   988 Mbits/sec
[SUM]   4.00-5.00   sec  1.06 GBytes  9.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec  98.0 MBytes   822 Mbits/sec
[  6]   5.00-6.00   sec  89.9 MBytes   754 Mbits/sec
[  8]   5.00-6.00   sec  78.5 MBytes   659 Mbits/sec
[ 10]   5.00-6.00   sec   116 MBytes   972 Mbits/sec
[ 12]   5.00-6.00   sec   112 MBytes   944 Mbits/sec
[ 14]   5.00-6.00   sec   108 MBytes   903 Mbits/sec
[ 16]   5.00-6.00   sec   143 MBytes  1.20 Gbits/sec
[ 18]   5.00-6.00   sec   110 MBytes   921 Mbits/sec
[ 20]   5.00-6.00   sec   114 MBytes   957 Mbits/sec
[ 22]   5.00-6.00   sec   139 MBytes  1.17 Gbits/sec
[SUM]   5.00-6.00   sec  1.08 GBytes  9.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  86.9 MBytes   729 Mbits/sec
[  6]   6.00-7.00   sec   115 MBytes   967 Mbits/sec
[  8]   6.00-7.00   sec   125 MBytes  1.05 Gbits/sec
[ 10]   6.00-7.00   sec  99.9 MBytes   838 Mbits/sec
[ 12]   6.00-7.00   sec   103 MBytes   864 Mbits/sec
[ 14]   6.00-7.00   sec   109 MBytes   911 Mbits/sec
[ 16]   6.00-7.00   sec   102 MBytes   853 Mbits/sec
[ 18]   6.00-7.00   sec   116 MBytes   976 Mbits/sec
[ 20]   6.00-7.00   sec   129 MBytes  1.08 Gbits/sec
[ 22]   6.00-7.00   sec   112 MBytes   943 Mbits/sec
[SUM]   6.00-7.00   sec  1.07 GBytes  9.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  96.8 MBytes   811 Mbits/sec
[  6]   7.00-8.00   sec  86.8 MBytes   727 Mbits/sec
[  8]   7.00-8.00   sec   116 MBytes   973 Mbits/sec
[ 10]   7.00-8.00   sec   107 MBytes   899 Mbits/sec
[ 12]   7.00-8.00   sec   143 MBytes  1.20 Gbits/sec
[ 14]   7.00-8.00   sec  81.4 MBytes   682 Mbits/sec
[ 16]   7.00-8.00   sec   115 MBytes   964 Mbits/sec
[ 18]   7.00-8.00   sec  91.1 MBytes   764 Mbits/sec
[ 20]   7.00-8.00   sec   111 MBytes   933 Mbits/sec
[ 22]   7.00-8.00   sec   132 MBytes  1.11 Gbits/sec
[SUM]   7.00-8.00   sec  1.06 GBytes  9.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   109 MBytes   911 Mbits/sec
[  6]   8.00-9.00   sec   114 MBytes   959 Mbits/sec
[  8]   8.00-9.00   sec  99.6 MBytes   836 Mbits/sec
[ 10]   8.00-9.00   sec   111 MBytes   928 Mbits/sec
[ 12]   8.00-9.00   sec   102 MBytes   854 Mbits/sec
[ 14]   8.00-9.00   sec   127 MBytes  1.07 Gbits/sec
[ 16]   8.00-9.00   sec   120 MBytes  1.01 Gbits/sec
[ 18]   8.00-9.00   sec   121 MBytes  1.01 Gbits/sec
[ 20]   8.00-9.00   sec   108 MBytes   902 Mbits/sec
[ 22]   8.00-9.00   sec   106 MBytes   887 Mbits/sec
[SUM]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   126 MBytes  1.06 Gbits/sec
[  6]   9.00-10.00  sec   120 MBytes  1.00 Gbits/sec
[  8]   9.00-10.00  sec   121 MBytes  1.02 Gbits/sec
[ 10]   9.00-10.00  sec  93.2 MBytes   782 Mbits/sec
[ 12]   9.00-10.00  sec   121 MBytes  1.01 Gbits/sec
[ 14]   9.00-10.00  sec   119 MBytes   995 Mbits/sec
[ 16]   9.00-10.00  sec   109 MBytes   913 Mbits/sec
[ 18]   9.00-10.00  sec   116 MBytes   974 Mbits/sec
[ 20]   9.00-10.00  sec  94.1 MBytes   790 Mbits/sec
[ 22]   9.00-10.00  sec   103 MBytes   861 Mbits/sec
[SUM]   9.00-10.00  sec  1.10 GBytes  9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.01 GBytes   864 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.01 GBytes   864 Mbits/sec                  receiver
[  6]   0.00-10.00  sec  1.05 GBytes   899 Mbits/sec                  sender
[  6]   0.00-10.00  sec  1.05 GBytes   899 Mbits/sec                  receiver
[  8]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec                  sender
[  8]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec  1009 MBytes   846 Mbits/sec                  sender
[ 10]   0.00-10.00  sec  1009 MBytes   846 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec  1.06 GBytes   906 Mbits/sec                  sender
[ 12]   0.00-10.00  sec  1.06 GBytes   906 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  sender
[ 14]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec  1.13 GBytes   966 Mbits/sec                  sender
[ 16]   0.00-10.00  sec  1.13 GBytes   966 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec  1.14 GBytes   977 Mbits/sec                  sender
[ 18]   0.00-10.00  sec  1.14 GBytes   977 Mbits/sec                  receiver
[ 20]   0.00-10.00  sec  1.05 GBytes   904 Mbits/sec                  sender
[ 20]   0.00-10.00  sec  1.05 GBytes   904 Mbits/sec                  receiver
[ 22]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec                  sender
[ 22]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.7 GBytes  9.17 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  10.7 GBytes  9.17 Gbits/sec                  receiver

You can parallelize the server side as well, see the link I posted above. The scaling for 100 gigabit will indefinitely be higher than my results. But even with parallelization on both ends, I don't think you'll see 100 gigabit in this test. That statement is regardless of the OS, TrueNAS or otherwise...

On the storage side of this conversation:
I have an Optane pool with 2 960GB mirrors (4 drives). Even that system - best case scenario in theoretical-land and not reality-ville - is probably less than 50 Gigabit's worth of pool.

Using iperf3 default single stream I am getting right at the 20Gbits/s

Edit: After some thought I decided to test the other direction with the truenas as the client and the windows machine as the server and low and behold I cant break 10Gbits/s in iperf or iperf3.

NickF · Jun 29, 2023

jarble said:
Using iperf3 default single stream I am getting right at the 20Gbits/s

sure, but the point I was making was that you NEED to leverage parallel operations to actually utilize your link. What are your results with -P 10 like in my example above?

jarble · Jun 29, 2023

NickF said:
sure, but the point I was making was that you NEED to leverage parallel operations to actually utilize your link. What are your results with -P 10 like in my example above?

Right at 20Gbit/s

NickF · Jun 29, 2023

Can you tmux or open another session and run htop while the test runs, then send another screen shot?

jarble · Jun 29, 2023

Not sure if I am capturing what you are looking for but I made a quick snip of htop running while the iperf3 is connected

NickF · Jun 29, 2023

If you'll notice, iPerf3 is hitting over 54% of your CPU usage during that test.

k: %CPU -- CPU usage
The task's share of the elapsed CPU time since the last screen
update, expressed as a percentage of total CPU time.

I think you may be hitting a wall due to the relatively low CPU clock speed of that multicore monster. This will probably partially explain why your workstation was faster as the server, since the TR is higher frequency.

Patrick M. Hausen · Jun 29, 2023

I wrote this on the OPNsense forum. TrueNAS is not a router/firewall but you might want to try those other tools to check if you really have a network bottleneck or are rather CPU bound as @NickF suggested.

Kristof Provost, one of the main current network developers for FreeBSD argued that iperf is not suitable to measure packet forwarding performance because at 10G and above you are more likely to max out ipferf itself.

He recommends pkt-gen/netmap or DPDK.

Elliot Dierksen · Jun 30, 2023

Have you looked at the resource on 10G and higher networking from @jgreco ? If wasn't able to get my systems to get full performance on a 40G network until I did some the the suggested tuning in that resource. Afterwards I was able to get in excess of 39Gb using iperf3.

jarble · Jun 30, 2023

Elliot Dierksen said:
Have you looked at the resource on 10G and higher networking from @jgreco ? If wasn't able to get my systems to get full performance on a 40G network until I did some the the suggested tuning in that resource. Afterwards I was able to get in excess of 39Gb using iperf3.

I am trying to follow the guide I found that they wrote here https://www.truenas.com/community/r...ng-to-maximize-your-10g-25g-40g-networks.207/ but I am struggling with where to add the flags after setting the algrithum to cubic.

indy · Jul 2, 2023

Try
iperf3 -c <your-ip>
iperf3 -c <your-ip> -P 8
iperf3 -c <your-ip> -R
iperf3 -c <your-ip> -R -P 8

iperf3 will show if packets are dropped, I am not sure iperf2 does.

jarble · Jul 2, 2023

indy said:
Try
iperf3 -c <your-ip>
iperf3 -c <your-ip> -P 8
iperf3 -c <your-ip> -R
iperf3 -c <your-ip> -R -P 8

iperf3 will show if packets are dropped, I am not sure iperf2 does.

Code:

C:\Users\Jarble\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64>iperf3.exe -c 10.40.0.68 -P8 -R
Connecting to host 10.40.0.68, port 5201
Reverse mode, remote host 10.40.0.68 is sending
[  4] local 10.40.0.133 port 55883 connected to 10.40.0.68 port 5201
[  6] local 10.40.0.133 port 55884 connected to 10.40.0.68 port 5201
[  8] local 10.40.0.133 port 55885 connected to 10.40.0.68 port 5201
[ 10] local 10.40.0.133 port 55886 connected to 10.40.0.68 port 5201
[ 12] local 10.40.0.133 port 55887 connected to 10.40.0.68 port 5201
[ 14] local 10.40.0.133 port 55888 connected to 10.40.0.68 port 5201
[ 16] local 10.40.0.133 port 55889 connected to 10.40.0.68 port 5201
[ 18] local 10.40.0.133 port 55890 connected to 10.40.0.68 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   138 MBytes  1.16 Gbits/sec
[  6]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec
[  8]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec
[ 10]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec
[ 12]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec
[ 14]   0.00-1.00   sec   139 MBytes  1.17 Gbits/sec
[ 16]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec
[ 18]   0.00-1.00   sec   142 MBytes  1.19 Gbits/sec
[SUM]   0.00-1.00   sec  1.10 GBytes  9.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec   145 MBytes  1.22 Gbits/sec
[  6]   1.00-2.00   sec   147 MBytes  1.23 Gbits/sec
[  8]   1.00-2.00   sec   147 MBytes  1.23 Gbits/sec
[ 10]   1.00-2.00   sec   147 MBytes  1.23 Gbits/sec
[ 12]   1.00-2.00   sec   147 MBytes  1.24 Gbits/sec
[ 14]   1.00-2.00   sec   144 MBytes  1.21 Gbits/sec
[ 16]   1.00-2.00   sec   148 MBytes  1.24 Gbits/sec
[ 18]   1.00-2.00   sec   148 MBytes  1.24 Gbits/sec
[SUM]   1.00-2.00   sec  1.15 GBytes  9.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec   143 MBytes  1.20 Gbits/sec
[  6]   2.00-3.00   sec   146 MBytes  1.22 Gbits/sec
[  8]   2.00-3.00   sec   145 MBytes  1.22 Gbits/sec
[ 10]   2.00-3.00   sec   146 MBytes  1.22 Gbits/sec
[ 12]   2.00-3.00   sec   146 MBytes  1.22 Gbits/sec
[ 14]   2.00-3.00   sec   142 MBytes  1.19 Gbits/sec
[ 16]   2.00-3.00   sec   146 MBytes  1.22 Gbits/sec
[ 18]   2.00-3.00   sec   146 MBytes  1.22 Gbits/sec
[SUM]   2.00-3.00   sec  1.13 GBytes  9.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   143 MBytes  1.20 Gbits/sec
[  6]   3.00-4.00   sec   145 MBytes  1.22 Gbits/sec
[  8]   3.00-4.00   sec   145 MBytes  1.22 Gbits/sec
[ 10]   3.00-4.00   sec   145 MBytes  1.22 Gbits/sec
[ 12]   3.00-4.00   sec   146 MBytes  1.22 Gbits/sec
[ 14]   3.00-4.00   sec   142 MBytes  1.19 Gbits/sec
[ 16]   3.00-4.00   sec   146 MBytes  1.22 Gbits/sec
[ 18]   3.00-4.00   sec   146 MBytes  1.22 Gbits/sec
[SUM]   3.00-4.00   sec  1.13 GBytes  9.71 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec   146 MBytes  1.22 Gbits/sec
[  6]   4.00-5.00   sec   147 MBytes  1.23 Gbits/sec
[  8]   4.00-5.00   sec   148 MBytes  1.24 Gbits/sec
[ 10]   4.00-5.00   sec   147 MBytes  1.24 Gbits/sec
[ 12]   4.00-5.00   sec   148 MBytes  1.24 Gbits/sec
[ 14]   4.00-5.00   sec   145 MBytes  1.22 Gbits/sec
[ 16]   4.00-5.00   sec   147 MBytes  1.23 Gbits/sec
[ 18]   4.00-5.00   sec   147 MBytes  1.23 Gbits/sec
[SUM]   4.00-5.00   sec  1.15 GBytes  9.86 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec   145 MBytes  1.22 Gbits/sec
[  6]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec
[  8]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec
[ 10]   5.00-6.00   sec   147 MBytes  1.24 Gbits/sec
[ 12]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec
[ 14]   5.00-6.00   sec   145 MBytes  1.21 Gbits/sec
[ 16]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec
[ 18]   5.00-6.00   sec   147 MBytes  1.24 Gbits/sec
[SUM]   5.00-6.00   sec  1.15 GBytes  9.86 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec   146 MBytes  1.22 Gbits/sec
[  6]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec
[  8]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec
[ 10]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec
[ 12]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec
[ 14]   6.00-7.00   sec   148 MBytes  1.24 Gbits/sec
[ 16]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec
[ 18]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec
[SUM]   6.00-7.00   sec  1.17 GBytes  10.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec   146 MBytes  1.22 Gbits/sec
[  6]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec
[  8]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec
[ 10]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec
[ 12]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec
[ 14]   7.00-8.00   sec   148 MBytes  1.24 Gbits/sec
[ 16]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec
[ 18]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec
[SUM]   7.00-8.00   sec  1.16 GBytes  10.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   145 MBytes  1.21 Gbits/sec
[  6]   8.00-9.00   sec   147 MBytes  1.23 Gbits/sec
[  8]   8.00-9.00   sec   147 MBytes  1.23 Gbits/sec
[ 10]   8.00-9.00   sec   147 MBytes  1.23 Gbits/sec
[ 12]   8.00-9.00   sec   147 MBytes  1.23 Gbits/sec
[ 14]   8.00-9.00   sec   144 MBytes  1.21 Gbits/sec
[ 16]   8.00-9.00   sec   147 MBytes  1.23 Gbits/sec
[ 18]   8.00-9.00   sec   146 MBytes  1.23 Gbits/sec
[SUM]   8.00-9.00   sec  1.14 GBytes  9.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   142 MBytes  1.19 Gbits/sec
[  6]   9.00-10.00  sec   144 MBytes  1.21 Gbits/sec
[  8]   9.00-10.00  sec   143 MBytes  1.20 Gbits/sec
[ 10]   9.00-10.00  sec   144 MBytes  1.20 Gbits/sec
[ 12]   9.00-10.00  sec   144 MBytes  1.21 Gbits/sec
[ 14]   9.00-10.00  sec   141 MBytes  1.19 Gbits/sec
[ 16]   9.00-10.00  sec   144 MBytes  1.21 Gbits/sec
[ 18]   9.00-10.00  sec   144 MBytes  1.21 Gbits/sec
[SUM]   9.00-10.00  sec  1.12 GBytes  9.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.41 GBytes  1.21 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  1.41 GBytes  1.21 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec    0             sender
[  6]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec    0             sender
[  8]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec    0             sender
[ 10]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec    0             sender
[ 12]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  1.41 GBytes  1.21 Gbits/sec    0             sender
[ 14]   0.00-10.00  sec  1.41 GBytes  1.21 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec    0             sender
[ 16]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec    0             sender
[ 18]   0.00-10.00  sec  1.43 GBytes  1.23 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  11.4 GBytes  9.79 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  11.4 GBytes  9.79 Gbits/sec                  receiver

iperf Done.

Important Announcement for the TrueNAS Community.

Help understanding the bottleneck 100Gb/s

Dabbler

MVP

Dabbler

MVP

Patron

Dabbler

Attachments

Captain Morgan

Guru

Dabbler

Dabbler

Attachments

Guru

Dabbler

Attachments

Guru

Dabbler

Attachments

Guru

Hall of Famer

Guru

Dabbler

Patron

Dabbler

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Help understanding the bottleneck 100Gb/s"

Similar threads