Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

What is holding the transfer speed?

melloa

FreeNAS Expert
Joined
May 22, 2016
Messages
1,645
Thanks
317
#1
Folks,

While waiting on my switch to get here, i run some tests and noticed the rsync transfer speed just a little bit faster over 10gb than over 1gb.

Both servers are using Chelsio 10GB 2-Port PCI-e Card PCI-E 110-1088-30, connected with a Cisco SFP-H10GB-CU1M 1 Meter Twinax 10GB Cable, both servers are using X9DRi-LN4F+/192GB, and the Chelsios are passed through to the ESXi 6.5 VMs.

1Gb NIC is on 10.10.10.0/24 and the 10Gb NIC is on 10.10.99.0/24:

iperf test run from server 1 to server 2
Code:
iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 10.10.99.2 port 5001 connected with 10.10.99.1 port 26554
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  11.5 GBytes  9.90 Gbits/sec
[  5] local 10.10.10.220 port 5001 connected with 10.10.10.200 port 48511
[  5]  0.0-10.0 sec  1.10 GBytes   942 Mbits/sec


rsync between both through the 10Gb NICs:
Code:
4,908,552,205 100%   46.84MB/s	0:01:39 (xfr#1, ir-chk=2422/79826)


dd on receiver side server:
Code:
dd bs=256k if=/dev/da0 of=/dev/null
131072+0 records in
131072+0 records out
34359738368 bytes transferred in 41.311843 secs (831716427 bytes/sec)


rsync command:
rsync --delete --progress -rltuv root@10.10.99.1:/mnt ./


Understand rsync does more than copy, so I'd expect not get 10Gb transfer, but it is the 46MB/s normal?
 
Joined
Jul 3, 2015
Messages
491
Thanks
180
#2

sretalla

FreeNAS Expert
Joined
Jan 1, 2016
Messages
1,246
Thanks
321
#3
rsync --delete --progress -rltuv root@10.10.99.1:/mnt ./
You're using rsync over ssh here (because of the colon in the command), which means the comms are encrypted... perhaps contributing something to the slowness.

Try it without the colon...

Like this
rsync --delete --progress -rltuv root@10.10.99.1/mnt ./
 

melloa

FreeNAS Expert
Joined
May 22, 2016
Messages
1,645
Thanks
317
#4
rsync is notoriously slow.
Agree. Was just expecting a little improvement.

I would do a cp from the same systems and compare the speed differences.
There is no comparison. A 1.4 GB files rsync'ed in 29 secs. The same file was copied internally in less than 2 secs, but that's also expected as cp copied locally.

Code:
1,406,838,780 100%   45.68MB/s	0:00:29 (xfr#12, ir-chk=2591/79876)


Try it without the colon...
Never saw that syntax. The shell didn't like is either :) It seems to "think" is copying within the same machine:

Code:
rsync --delete --progress -rltuv root@10.10.99.1/mnt ./
sending incremental file list
rsync: change_dir "/root@10.10.99.1" failed: No such file or directory (2)

sent 20 bytes  received 12 bytes  64.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1180) [sender=3.1.2]
 

Arwen

FreeNAS Expert
Joined
May 17, 2014
Messages
1,112
Thanks
547
#5
If they are on local network, why not setup a Rsync share on the source, and use something like this on the destination;

rsync -aHSXx --delete --progress rsync://SOURCE_IP/SHARE_NAME/ /mnt/NEW/LOCATION/

Yes, Rsync is slower than straight copy. Especially over SSH.

But, for me, I like the fact that Rsync can copy an entire tree of files, but interruptable. Using cp -r or mv are not interruptable. Meaning they leave either the destination, (for cp -r), or both source and destination, (for mv), in an undetermined state if interrupted.
 

melloa

FreeNAS Expert
Joined
May 22, 2016
Messages
1,645
Thanks
317
#6
If they are on local network, why not setup a Rsync share on the source
Never used. Will test it, thanks.

But, for me, I like the fact that Rsync can copy an entire tree of files, but interruptable.
Yes, agree, great tool. If it only was faster ;)
 
Joined
Jan 18, 2017
Messages
10
Thanks
0
#7
Hi, having the same issue, 1gbps performance over a 10gbps link, FreeNAS on a side, QNAP on the other.
Iperf shows
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.1 sec 887 MBytes 6.49 Gbits/sec

Yet rsync is waaay slower than that.

Are there any updates or work around for this?

thx
 
Joined
Jan 1, 2016
Messages
1,246
Thanks
321
#8
Top