10GbE performance issue in one direction

[mimesot]

Dear community members,
first off I want to say thank you for all the great stuff I was already allowed to read on this forum. It helped me a lot in many cases, but now unfortunately I am stuck and need to ask you directly for help.

In short: I can upload stuff to my FreeNAS-server with almost 10Gbit/s but when downloading stuff I never exceed 3Gbit/s.

Note: Is seems to be most likely no ZFS or zpool issue since i am testing with iperf.

Hardware of the FreeNAS-server

• Intel Xeon E3-1245 v6 with 4x 3.7GHz plus Hyperthreading
• 64GB DDR4 RAM in 4 slots
• Supermicro X11SAT-F board
• LSI 9211-8i SATA-Controller in IT-Mode
• 9x WD Red 4TB ATA WDC WD40EFRX-68N harddrives with CMR (approx 135MB/s)
  (run as my main data-pool "hdd-pool" in a RAID 10 configuration plus one hot spare)
• 1x Optane 900P as a SLOG device (once I have my virtual maschines and or databases on the pool)
• 1x Samsung NVME SSD 256GB (as my test-pool just for performance tests)
• 1x Samsung SM951 SATA SSD 240GB as my OS-drive
• Intel X550-T2 ethernet card
• Seasonic Platinum Series 660W ATX power supply
• 4U Chassis with abundant cooling

Freenas configuration

• Operating System: Freenas 11.3-U3.2
• Active Services: Active Directory, SSH, SMB, SMART
• Networking: using ethernet port ix0, created vlan interface, vlan called "Server-VLAN", vlan interface has a static IP, ix0 has no configuration

HDD-pool confoguration
Even though it turned out that the pool is not involved in my network issues I wanna give you my config here. It is a Raid-10 configuration, which i chose because i want to have fast resilvering times.

• mirror
  • ATA WDC WD40EFRX-68N
  • ATA WDC WD40EFRX-68N
• mirror
  • ATA WDC WD40EFRX-68N
  • ATA WDC WD40EFRX-68N
• mirror
  • ATA WDC WD40EFRX-68N
  • ATA WDC WD40EFRX-68N
• mirror
  • ATA WDC WD40EFRX-68N
  • ATA WDC WD40EFRX-68N
• spare
  • ATA WDC WD40EFRX-68N
• log
  • NVME Intel Optant 900P

Hardware of the surrounding

• Cabeling: Cat6A S/FTP
• Switch: TP-Link T1700X with 12x 10GbE and 4x SFP+
• Server1 (Virtualization): Similar to the FreeNAS-Server, far less storage but including NVME, Xeon E3-1240L v5, 32GB RAM, Supermicro X11SSH-CTF, comes with 2x Intel X550T, using VLAN 802.1Q tagged "Server-VLAN", operating system Proxmox 6.2 with Debian 10 as a basis
• Server2 (Workstation): Similar to the FreeNSA-Server, far less storage but including NVME, Xeon E3-1245 v6, 32GB RAM, MSI 236a Workstation, Intel X550-T2, using VLAN 802.1Q tagged "Server-VLAN", operating system Proxmox 6.2 with Debian 10 as a basis
• Server3 (FreeNAS-Test): This is essentially the same hardware like the original FreeNAS-Server but with no pool. Purely set up for Testing.

Narrowing down the problem

• Started with a simple SMB share on the FreeNAS-Server. Tested the download from the FreeNAS-servers hdd-pool to my Server2 (Workstation). Therefore I created a 10GByte random bytes file.
  Downloaded it with max. 150MB/s, uploaded it with 850MB/s. Weird!
• Repeated this with the nvme-pool ... same result.
• Repeated it with the srv1 ... same result.
• Repeated it over SSH (with scp) ... same results.
• I made an internal copy from the hdd-pool to the nvme-pool. Worked perfectly. So it was not the pools.
• Switched to iperf for further testing, because as far as I know that doesn't involve the storage. Guess what ... same results:
  Upload to the FreeNAS-Server: around 9,3Gbit/s
  Download from the FreeNAS-Server: 1-3Gbit/s ... we are definitely exceeding 1Gbit/s here.
• Checked the Ports: No failed auto-negotiation. All say 10GbE full-duplex
• Checked the system load: the cpu cores do not shed a single droplet of sweat
• Performed iperf in both directions between Server1 and Server2. They managed about 9.3 to 9.4 Gbit/s in both directions.
  --> The problem has to be in between the FreeNAS-Server and the Switch.
• Changed the switch port
• Changed the cable
• Changed the network card port
• Changed the network card (to another X550-T2)
• Changed the PCIe port of the network card
  ... no changes in the results
• Installed Debian 10.4 on the OS-SSD ... i get 9.3 Gbit/s up and dows
  --> it has to be the FreeNAS-OS itself or in combination with my hardware
• Update: Tried to enable Jumbo Frames MTU 9000 at all devices involved. Performance increased by a factor of 2-3, but the core problem persisted. It hints at the package processing rate at the server is limited by some unknown factor.
• Update: I created Server3 with basically the same harware and default config and tested FreeNA-to-FreeNAS. The same asymmetry occurs -> it is not an Issue in between different OSes.

Tunbables
After a bit or research I read that the FreeNAS 11 tunables might not be suited for optimal performance of a 10GbE network connection (which i cannot really believe but, still). I tried this (source: https://gist.github.com/bjornbouetsmith/2c8201c9e71ef33c0087010a3cf1e186):

Code:

#Add these lines to the file /etc/sysctl.conf
#10GBE tuning
kern.ipc.soacceptqueue=2048
kern.ipc.somaxconn=2048
kern.ipc.maxsockbuf=33554432
net.inet.tcp.recvbuf_inc=2097152    # (default 16384)
net.inet.tcp.recvbuf_max=16777216  # (default 2097152)
net.inet.tcp.recvspace=4194304      # (default 65536)
net.inet.tcp.sendbuf_inc=2097152    # (default 8192)
net.inet.tcp.sendbuf_max=16777216  # (default 2097152)
net.inet.tcp.sendspace=4194304      # (default 32768)
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1

#Add these lines to /boot/loader.conf
net.isr.maxthreads="-1"
net.isr.bindthreads="1"

In addition I have this one active as well:

Code:

net.route.netisr_maxqlen=2048

But that didn't change a thing either.

As you can see I am running out of ideas.
Can anybody please help me?
In order to help me, do you need any further informations (e.g. config-files)?
Thanks in advance and kind regards
mimesot

 

[Kcaj]

How big is the file you are uploading? Bigger than the 64GB? Sounds like you are just writing to your RAM.

Edit: I see you made a 10GB file, try a 128GB file to test as you are just writing your test file into your RAM and thinking this is your true write speed.

3Gb/s read sounds excellent, WD Red has a read of about 50MB/s from memory (dont quote me on that - look it up ;))

 

[HoneyBadger]

WD Red 4TB

Full model number please. Hopefully not the EFAX SMR ones.

 

[mimesot]

Hi Kcaj,
Thank you for your answer.

Edit: I see you made a 10GB file, try a 128GB file to test as you are just writing your test file into your RAM and thinking this is your true write speed.

Yes, if I really go on testing my hdd-pool performance i will use a 128GB file. But my problem does not involve the hdd-pool. I can internally copy stiff from the hdd-pool to my nvme drive with almost 1GByte/s. The issue starts when I send data to the network. In order to rule out hdd-pool issues I used the command iper for testing.

3Gb/s read sounds excellent, WD Red has a read of about 50MB/s from memory (dont quote me on that - look it up ;))

Well, the reason I am unhappy with the 3Gbit/s is because I had almost 1GByte/s with Linux. The pool consits of 8 equal Harddrives and each of them is capable of 170MB/s sequential download (in WD dreams) which is realistically about 135MB/s (tested) which still reaults in over 1GByte/s if a file is properly distributed over all 8 disks. And I actulally managed to get those results with the same drives using Debian 10 + Proxmox 6 (the cernal can do ZFS natively) + Samba (with a Windows client). So I am 100% sure this is an freenas network issue.

 

[mimesot]

Dear HoneyBadger,
thank you for your reply.

Full model number please. Hopefully not the EFAX SMR ones.

The model is ATA WDC WD40EFRX-68N. According to https://nascompares.com/answer/how-...en-dm-smr-and-non-smr-cmr-drives-hdd-compare/ there have usually been CMR and there are SMR ones in existnace as well. When I bought them I always made sure that is says CMR and not SMR. I am aware that SMR would negativley my writes on the hdd-pool, but my problem is the read from FreeNAS.

Anyway, as I stated above the harddisks are not related to my problem, because the hdd-pool works (tested) with 4 times the performance on Debian 10 + Proxmox 6 and my problems exclusively occur with FreeNAS. Addditionally I get the same bad performance with my high speed NVME drive and also over operf, which doesn't use drives for the test at all.

Thabk you and greetings
mimesot

 

[Kcaj]

Maybe its not your pool but can you please just tell us what your pool arrangement is? You said RAID 10 but what is it in ZFS terms?

 

[mimesot]

Maybe its not your pool but can you please just tell us what your pool arrangement is? You said RAID 10 but what is it in ZFS terms?

Sure! I added it to the initial post.

 

[Samuel Tai]

Have you tried enabling jumbo frames?

 

[mimesot]

Have you tried enabling jumbo frames?

Hi,
thanks for the hint.
Yes I tried that. No effect. Ill add that info to the initial post.

 

[Samuel Tai]

Does the same asymmetry manifest with unbuffered transfers, e.g., with robocopy -j? I.e., this may be a bufferbloat issue on one side.

 

[mimesot]

Does the same asymmetry manifest with unbuffered transfers, e.g., with robocopy -j? I.e., this may be a bufferbloat issue on one side.

I can try that.

Is bufferbloat even a possible issue when imuse iperf?

 

[Samuel Tai]

Is bufferbloat even a possible issue when imuse iperf?

iperf, so far as I know, bypasses any OS buffers to get true NIC performance.

 

[mimesot]

iperf, so far as I know, bypasses any OS buffers to get true NIC performance.

So you see, my problem has nothing to do with the storage. It has only something to do with how FreeNAS handles my Intel X550-T2 network adapter or any influences on that communication. I think I ruled out any other options. Or can you think of any I haven't mentioned in my initial post?

 

[mimesot]

Hello, can anybody please have a llok at my issue. Tanks and kind regards
mimesot

 

[Tony-1971]

Did you try zpool iostat command? Something like:

Code:

zpool iostat -v <pool-name> 2

You can see the read / write performance for each drive while you are copying file.

See also https://www.youtube.com/watch?v=oDbGj4YJXDw&t=117s : old but interesting

How much RAM you had? The best results are when all data can be readed from ARC cache: so if data are in ARC and read is slow the problem is in network interface (to fill ARC you must read data multiple times).

You can see ARC status in http://<ip-freenas>/reportsdashboard/zfs

Best Regards,
Antonio

 

[mimesot]

Hi,
thank you for your reply. But excuse me, because maybe I still don't understand something. I am using iperf foer performance measuring and that one creates the data in flight, which means without touching ZFS of my zpool.

Did you try zpool iostat command? Something like:

Code:

zpool iostat -v <pool-name> 2

Thanks I did. And I also tested transferring Data from my NVME-drive to the HDD-pool. 27GB in about 50 Sekonds is totally fine.

How much RAM you had? The best results are when all data can be readed from ARC cache: so if data are in ARC and read is slow the problem is in network interface (to fill ARC you must read data multiple times).

I have 64GB of RAM. My ARC currently measures 53GB and has a hit rate of about 93%. And yes indeed, I highly assume the poblem is network interface related.

Thank you, and kind regards
mimesot

 

[Tony-1971]

Hi,

Can you share your iperf3 test?
This is my system:

Code:

root@freenas-sm[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.200.10, port 63524
[  5] local 192.168.200.20 port 5201 connected to 192.168.200.10 port 55933
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   889 MBytes  7.46 Gbits/sec
[  5]   1.00-2.00   sec   999 MBytes  8.38 Gbits/sec
[  5]   2.00-3.00   sec  1023 MBytes  8.58 Gbits/sec
[  5]   3.00-4.00   sec  1023 MBytes  8.58 Gbits/sec
[  5]   4.00-5.00   sec  1.04 GBytes  8.94 Gbits/sec
[  5]   5.00-6.00   sec   980 MBytes  8.22 Gbits/sec
[  5]   6.00-7.00   sec  1.06 GBytes  9.09 Gbits/sec
[  5]   7.00-8.00   sec  1.02 GBytes  8.78 Gbits/sec
[  5]   8.00-9.00   sec  1.02 GBytes  8.80 Gbits/sec
[  5]   9.00-10.00  sec  1.08 GBytes  9.24 Gbits/sec
[  5]  10.00-10.00  sec   204 KBytes  8.92 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.0 GBytes  8.61 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Using this tunables:

Code:

kern.ipc.maxsockbuf                16777216    sysctl    yes
net.inet.ip.intr_queue_maxlen    2048        sysctl    yes
net.inet.tcp.recvbuf_inc        524288        sysctl    yes
net.inet.tcp.recvbuf_max        16777216    sysctl    yes
net.inet.tcp.recvspace            4194304        sysctl    yes
net.inet.tcp.sendbuf_inc        32768        sysctl    yes
net.inet.tcp.sendbuf_max        16777216    sysctl    yes
net.inet.tcp.sendspace            2097152        sysctl    yes
net.route.netisr_maxqlen        2048        sysctl    yes

Best Regards,
Antonio

 

[mimesot]

Can you share your iperf3 test?

Sure!

This is one direction:

Code:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.18.2.10, port 44278
[  5] local 172.18.2.106 port 5201 connected to 172.18.2.10 port 44280
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.09 GBytes  9.35 Gbits/sec
[  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   3.00-4.00   sec  1.09 GBytes  9.38 Gbits/sec
[  5]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   5.00-6.00   sec  1.09 GBytes  9.38 Gbits/sec
[  5]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec
[  5]   7.00-8.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec
[  5]  10.00-10.00  sec   778 KBytes  8.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

And this is the other direction:

Code:

iperf3 -c 172.18.2.10
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 40698 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   309 MBytes  2.59 Gbits/sec   11    101 KBytes
[  5]   1.00-2.00   sec   357 MBytes  2.99 Gbits/sec    4    104 KBytes
[  5]   2.00-3.00   sec   385 MBytes  3.23 Gbits/sec    6    110 KBytes
[  5]   3.00-4.00   sec   352 MBytes  2.95 Gbits/sec    5    117 KBytes
[  5]   4.00-5.00   sec   319 MBytes  2.68 Gbits/sec    6   55.6 KBytes
[  5]   5.00-6.00   sec   294 MBytes  2.46 Gbits/sec    6   94.1 KBytes
[  5]   6.00-7.00   sec   343 MBytes  2.88 Gbits/sec    5   81.2 KBytes
[  5]   7.00-8.00   sec   369 MBytes  3.09 Gbits/sec    5    151 KBytes
[  5]   8.00-9.00   sec   445 MBytes  3.73 Gbits/sec    3    151 KBytes
[  5]   9.00-10.00  sec   441 MBytes  3.70 Gbits/sec    4   82.7 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.53 GBytes  3.03 Gbits/sec   55             sender
[  5]   0.00-10.00  sec  3.52 GBytes  3.03 Gbits/sec                  receiver

iperf Done.

And finally I also did some iperf on myself. I guess this bypasses the actual network connector but perhaps indicates how many packages the System is able to create within a certain amount of time.

Code:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.18.2.106, port 29208
[  5] local 172.18.2.106 port 5201 connected to 172.18.2.106 port 13391
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.08 GBytes  43.6 Gbits/sec
[  5]   1.00-2.00   sec  5.52 GBytes  47.4 Gbits/sec
[  5]   2.00-3.00   sec  5.53 GBytes  47.5 Gbits/sec
[  5]   3.00-4.00   sec  5.52 GBytes  47.4 Gbits/sec
[  5]   4.00-5.00   sec  5.51 GBytes  47.3 Gbits/sec
[  5]   5.00-6.00   sec  5.47 GBytes  47.0 Gbits/sec
[  5]   6.00-7.00   sec  5.52 GBytes  47.4 Gbits/sec
[  5]   7.00-8.00   sec  5.51 GBytes  47.3 Gbits/sec
[  5]   8.00-9.00   sec  5.50 GBytes  47.2 Gbits/sec
[  5]   9.00-10.00  sec  5.49 GBytes  47.2 Gbits/sec
[  5]  10.00-10.00  sec  5.38 MBytes  47.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  54.7 GBytes  46.9 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Using this tunables:

I am currently using all of them, occasionally with higher numbers. And I have sendbuf auto and recvbuf auto activated.

Any conclusions?

 

[Tony-1971]

Hi,

So the second "direction" is the one with problem.
The "Retr" column mean congested network or corrupted data (not sure if this is the problem but is something to start understanding why)
When you perform

Code:

iperf3 -c 172.18.2.10

on server side there are problems?

Best Regards,
Antonio

 

[Samuel Tai]

The problematic direction pretty much confirms my hypothesis of bufferbloat. The transfer rate is lower due to retransmits and TCP congestion window throttling.

StackPath