Very slow scp transfer speeds to baremetal FreeNAS but fast to VM in FreeNAS (bhyve)

[ljdelight]

Hi I'm seeing very slow scp transfer speeds to FreeNAS baremetal install. Using a 1Gbps network and iperf shows raw speeds are normal. This is a basic setup with scp moving iso on nodeA to freenas and vmInFreenas, where "freenas" is baremetal and "vmInFreenas" is fedora-server-26 in bhyve (on FreeNAS baremetal). I'm seeing 30MB/s to FreeNAS, but a VM in the same FreeNAS host transfers at 112MB/s. Initially FreeNAS speeds were 2MB/s (_really_ slow, eh) a few sysctls helped move it up to 30MB/s.


The list of things that I have tried:

• Clean install of FreeNAS 11 and only configured ssh for scp.
• Clean install of FreeNAS 9u4. Slow as well.
• Cleared settings on a spare router and configured clean FreeNAS and nodeA with 192.168.1.0/24 subnet and DHCP, and only two devices on this subnet. No change.
• Set some tunables; tried autotune; a mixture of both. See attached screenshot of sysctls to move from 1MB/s to 30MB/s.
• Captured iperf and scp timings. See below. I used strace and compared between the slow and fast transfers; both have about the same number of write calls, just much different in how long it takes to write.

Any ideas on what is wrong with the setup? It doesn't appear to be hardware.


Memory: Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Memory: Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Board: Supermicro X10SDV-4C+-TLN4F-O Mini-ITX Intel Xeon D-1518 DDR4 Motherboard and CPU
Network controller: Intel i350-AM2 Gigabit Ethernet

nodeA to vmInFreenas
IPERF
[ljdelight@inception ~]$ iperf -c 10.10.0.79 -r
[ 5] local 10.10.0.165 port 51074 connected with 10.10.0.79 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
[ 4] local 10.10.0.165 port 5001 connected with 10.10.0.79 port 58952
[ 4] 0.0-10.0 sec 1.06 GBytes 909 Mbits/sec

SCP
[ljdelight@inception ~]$ scp Fedora-Server-dvd-x86_64-26-1.5.iso ljdelight@10.10.0.79:/dev/null
Fedora-Server-dvd-x86_64-26-1.5.iso 100% 2290MB 111.8MB/s 00:20

nodeA to FreeNAS
IPERF
[ljdelight@inception ~]$ iperf -c 10.10.0.205 -r
[ 5] local 10.10.0.165 port 39918 connected with 10.10.0.205 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.09 GBytes 940 Mbits/sec
[ 4] local 10.10.0.165 port 5001 connected with 10.10.0.205 port 44958
[ 4] 0.0-10.1 sec 1.10 GBytes 938 Mbits/sec

SCP
ljdelight@inception ~]$ scp Fedora-Server-dvd-x86_64-26-1.5.iso ljdelight@10.10.0.205:/dev/null
Fedora-Server-dvd-x86_64-26-1.5.iso 100% 2290MB 27.9MB/s 01:22

 

[m0nkey_]

Turn off autotune and remove any tuneables it created.

Autotune is known to slow down systems and there are a very few rare cases where it should be turned on. I've asked for this feature to be removed or a HUGE RED WARNING sating the known issues with it.

https://bugs.freenas.org/issues/24112

 

[ljdelight]

Sure, I started without autotune and the system was at 1MB/s. I have a strong dislike of autotune magic because the majority of ctls I can't find documentation. I support its removal.

Does everyone see 30MB/s speeds on 1Gbps with FreeNAS? I've used it for 7+ years and always assumed it was my (shit) hardware but I've upgraded to the fancy components listed above.
Looking for any suggestions to figure this out.

 

[m0nkey_]

One theory I have is the OpenSSH version that ships with FreeNAS is not utilizing AES-NI. It'll need more investigation.

 

[ljdelight]

One theory I have is the OpenSSH version that ships with FreeNAS is not utilizing AES-NI. It'll need more investigation.

I attempted to disable data encryption with `scp -c none ...` but unfortunately the client side doesn't show the selected cipher. And the freebsd sshd_config manual makes it sound like the none cipher isn't available.

EDIT: Oops, I didn't finish the thought. I'm thinking it's a mixture of missing sysctls and ssh.

 

[ljdelight]

I found that this is an openssh issue with the chacha20-poly1305@openssh.com cipher. At least, that's the best I am able to assist solo.
I opened a ticket to track this issue https://bugs.freenas.org/issues/25383

Workaround:

• Remove chacha20-poly1305@openssh.com from the openssh cipher list.
  • Edit the ssh service, advanced options, and use 'Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com'.
• Removing that cipher increases transfer rate to 50MB/s. Use these systctls to get the FULL 1Gbps rates:
  
  Code:

  sysctl kern.ipc.maxsockbuf=33554432
  sysctl net.inet.tcp.recvbuf_auto=1
  sysctl net.inet.tcp.sendbuf_auto=1
  sysctl net.inet.tcp.sendbuf_max=33554432
  sysctl net.inet.tcp.recvbuf_max=33554432
  sysctl net.inet.tcp.recvspace=4194304
  sysctl net.inet.tcp.sendspace=4194304

 

[bill_mcgonigle]

Code:

sysctl net.inet.tcp.recvbuf_auto=1
sysctl net.inet.tcp.sendbuf_auto=1

FWIW,these reduced my scp throughput from 330Mbps (defaults) to 40Mbps with a realtek NIC, 11.0-STABLE kernel.

 

[Aitor]

Hello:

Thank you very much ljdeligh. I had the same problem after upgrade to Freenas 11.1. Also with the nightly updates from yesterday. After edit the ciphers list in Advance options of SSH service it works at 1Gbps again.

 

[MMartinez]

Hi,

Thanks Ijdelight, I've also had this problem with a 10GbE NIC on 11.1. I've done a lot of tests before reading your quick solution!

Best regards,

Manuel Martínez