10Gbe speed drops

[DrA]

Hi

I've been reading a lot on 10Gbe, but the amount of details is a bit overwhelming.

Hoping for someone to point me in the right direction.

I have a x9 dual xeon 192GB supermicro box with a 2-vdev pool both 10 wide z3 w 3TB enterprise disks.

Direct attach to my main win10 workstation. Both boxes with Intel X540-T2 cards

I have done no tuning what so ever. Simple SMB share. I've transferred 15TB of data from the workstation -> freenas with a pretty constant transfer rate of about 300MB/s.

As part of a backup job I then needed to transfer the same mixed file set the other way, but the transfer speed is regularly changing in a constant pattern, attached. It's going back to the same raided disk config on the workstation (4 12TBs in software RAID10)

I was hoping someone instantly recognised this speed graph as a symptom of something I can read up on and fix, maybe with some tunables?

Any help appreciated.

 

[Samuel Tai]

This looks like classic TCP saw-tooth behavior. On the FreeNAS side, try changing the TCP congestion control behavior. On my FreeNAS system, I'm using cubic TCP congestion control (same as current Linux and Windows) via the following tunables:

• Loader tunable cc_cubic_load="YES"
• Sysctl tunable net.inet.tcp.cc.algorithm=cubic

Reboot to have these take effect.

 

[DrA]

Thanks a bunch, Samuel!

I'll have to wait for the current job to finish, but will try tomorrow and report back.

 

[Samuel Tai]

Also try setting MTU 9000 on both sides of this direct link.

 

[jims23211]

Where are those tunables in the GUI, or do I have to edit the file directly?

 

[Samuel Tai]

System->Tunables.

 

[jims23211]

Okay, I decided to take it a bit further and follow this blog on tuning freebsd for 1G and 10G networks. Following the blog, I did the changes to both loader.conf and the sysctl.conf. Rebooted and now I am seeing full performance on my 10G network where before I would only get 1/3 my bandwidth going from my Proxmox to Freenas. Clearly Freenas is not tuned for 10G networks.

Here is the link to the blog

FreeBSD Network Performance Tuning @ Calomel.org