Hi I'm seeing very slow scp transfer speeds to FreeNAS baremetal install. Using a 1Gbps network and iperf shows raw speeds are normal. This is a basic setup with scp moving iso on nodeA to freenas and vmInFreenas, where "freenas" is baremetal and "vmInFreenas" is fedora-server-26 in bhyve (on FreeNAS baremetal). I'm seeing 30MB/s to FreeNAS, but a VM in the same FreeNAS host transfers at 112MB/s. Initially FreeNAS speeds were 2MB/s (_really_ slow, eh) a few sysctls helped move it up to 30MB/s.
The list of things that I have tried:
Any ideas on what is wrong with the setup? It doesn't appear to be hardware.
Memory: Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Memory: Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Board: Supermicro X10SDV-4C+-TLN4F-O Mini-ITX Intel Xeon D-1518 DDR4 Motherboard and CPU
Network controller: Intel i350-AM2 Gigabit Ethernet
nodeA to vmInFreenas
IPERF
[ljdelight@inception ~]$ iperf -c 10.10.0.79 -r
[ 5] local 10.10.0.165 port 51074 connected with 10.10.0.79 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
[ 4] local 10.10.0.165 port 5001 connected with 10.10.0.79 port 58952
[ 4] 0.0-10.0 sec 1.06 GBytes 909 Mbits/sec
SCP
[ljdelight@inception ~]$ scp Fedora-Server-dvd-x86_64-26-1.5.iso ljdelight@10.10.0.79:/dev/null
Fedora-Server-dvd-x86_64-26-1.5.iso 100% 2290MB 111.8MB/s 00:20
nodeA to FreeNAS
IPERF
[ljdelight@inception ~]$ iperf -c 10.10.0.205 -r
[ 5] local 10.10.0.165 port 39918 connected with 10.10.0.205 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.09 GBytes 940 Mbits/sec
[ 4] local 10.10.0.165 port 5001 connected with 10.10.0.205 port 44958
[ 4] 0.0-10.1 sec 1.10 GBytes 938 Mbits/sec
SCP
ljdelight@inception ~]$ scp Fedora-Server-dvd-x86_64-26-1.5.iso ljdelight@10.10.0.205:/dev/null
Fedora-Server-dvd-x86_64-26-1.5.iso 100% 2290MB 27.9MB/s 01:22
The list of things that I have tried:
- Clean install of FreeNAS 11 and only configured ssh for scp.
- Clean install of FreeNAS 9u4. Slow as well.
- Cleared settings on a spare router and configured clean FreeNAS and nodeA with 192.168.1.0/24 subnet and DHCP, and only two devices on this subnet. No change.
- Set some tunables; tried autotune; a mixture of both. See attached screenshot of sysctls to move from 1MB/s to 30MB/s.
- Captured iperf and scp timings. See below. I used strace and compared between the slow and fast transfers; both have about the same number of write calls, just much different in how long it takes to write.
Any ideas on what is wrong with the setup? It doesn't appear to be hardware.
Memory: Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Memory: Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Board: Supermicro X10SDV-4C+-TLN4F-O Mini-ITX Intel Xeon D-1518 DDR4 Motherboard and CPU
Network controller: Intel i350-AM2 Gigabit Ethernet
nodeA to vmInFreenas
IPERF
[ljdelight@inception ~]$ iperf -c 10.10.0.79 -r
[ 5] local 10.10.0.165 port 51074 connected with 10.10.0.79 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
[ 4] local 10.10.0.165 port 5001 connected with 10.10.0.79 port 58952
[ 4] 0.0-10.0 sec 1.06 GBytes 909 Mbits/sec
SCP
[ljdelight@inception ~]$ scp Fedora-Server-dvd-x86_64-26-1.5.iso ljdelight@10.10.0.79:/dev/null
Fedora-Server-dvd-x86_64-26-1.5.iso 100% 2290MB 111.8MB/s 00:20
nodeA to FreeNAS
IPERF
[ljdelight@inception ~]$ iperf -c 10.10.0.205 -r
[ 5] local 10.10.0.165 port 39918 connected with 10.10.0.205 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.09 GBytes 940 Mbits/sec
[ 4] local 10.10.0.165 port 5001 connected with 10.10.0.205 port 44958
[ 4] 0.0-10.1 sec 1.10 GBytes 938 Mbits/sec
SCP
ljdelight@inception ~]$ scp Fedora-Server-dvd-x86_64-26-1.5.iso ljdelight@10.10.0.205:/dev/null
Fedora-Server-dvd-x86_64-26-1.5.iso 100% 2290MB 27.9MB/s 01:22
Attachments
Last edited by a moderator: