Multithread rsync transfers

Status
Not open for further replies.

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
Hi all,

I am currently in the process of copying over my zpool to a new server. I am using rsync, but I am getting the painfully slow rsync blues. I am transferring from a 8x8TB E5-2430 v2 48GB server to a 8x10TB E3-1230 v2 16GB server over a 10Gbe link. Both servers are running FreeNAS 11.2 U6. I am using regular rsync, as opposed to rsync over SSH with the following command:

rsync -avp --progress --inplace /mnt/RAIDZ1Pool/RAIDZ1Pool/ airwolf@10.10.0.56::a/a/

With this command, I get anywhere from 50 MB/s to about 90 MB/s, with an average of about 50 MB/s when the transfers finish on large files. Both arrays can read and write at 600+ MB/s and the CPU usage is not maxing out on either server. If I open new SSH sessions and run 10 rsync transfers simultaneously, the aggregate bandwidth will be about 250 MB/s.

The problem with this option is that the smaller directories finish and I end up with 1 or 2 very large directories with 1000+ folders in them eventually being the only ones transferring, so I have to wait for those to complete at 50 MB/s, which takes days. I was hoping to use a parallel command to speed up the transfer, but it appears that FreeNAS does not offer this ability:

ls /mnt/RAIDZ1Pool/RAIDZ1Pool/Test/ | parallel -j 10 -i rsync -avp --progress --inplace {} airwolf@10.10.0.56::a/a/ parallel: Command not found.

I have been looking around, but I do not see anything else I can use in place of parallel. I was hoping somebody on here could advise on a way to speed up these transfers. Any help would be appreciated.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I am currently in the process of copying over my zpool to a new server. I am using rsync
You are using the wrong tool.
Have a look at the discussion here:
https://forums.freenas.org/index.php?threads/rsync-speed-between-local-pools.71613/#post-495760

If there is any way that you can rig it, you should connect all the drives to one system temporarily so you can do a local copy instead of doing it across the network. It is so much faster, you could copy all the data in a matter of hours instead of days.
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
You are using the wrong tool.
Have a look at the discussion here:
https://forums.freenas.org/index.php?threads/rsync-speed-between-local-pools.71613/#post-495760

If there is any way that you can rig it, you should connect all the drives to one system temporarily so you can do a local copy instead of doing it across the network. It is so much faster, you could copy all the data in a matter of hours instead of days.

I appreciate the response. I do not have the ability to hook up the drives to the same machine at the same time, but I am interested in trying out ZFS send. I have been Googling around and I am having some trouble with documentation on how to setup a ZFS send over a network. Do you by chance know how to do this or can point me to a resource for this?
 

toadman

Guru
Joined
Jun 4, 2013
Messages
619
Probably many resources online for zfs send/recv between two servers. Here is what the official Oracle docs say (yes, I know these docs don't always match openzfs, but in this case should be good). Basically you pipe zfs send through ssh to the remote system.

Remote Replication of ZFS Data
You can use the zfs send and zfs recv commands to remotely copy a snapshot stream representation from one system to another system. For example:
# zfs send tank/cindy@today | ssh newsys zfs recv sandbox/restfs@today
This command sends the tank/cindy@today snapshot data and receives it into the sandbox/restfs file system. The command also creates a restfs@today snapshot on the newsys system. In this example, the user has been configured to use ssh on the remote system.

Also, since you are running FreeNAS on both you can setup replication via the FreeNAS docs and the new system will receive the snapshot automatically. Then promote the snapshot on the new system to be the dataset data (essentially a rollback to the snapshot on the new system).

https://www.ixsystems.com/documentation/freenas/11/storage.html#replication-tasks
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
I got ZFS send over SSH working and it was actually slower than rsync >_<

I also tried mounting the other server as an NFS share and sent via rsync. Same speed as normal rsync. So back to square one.
 

PnoT

Dabbler
Joined
Apr 12, 2017
Messages
41
Don't use zfs send on it's own as it's slow compared to other methods. What you really should look into is using bbcp to make the transfers as it will do line speed with the right options IF your storage can handle it on both ends. I have 2 FreeNAS boxes with huge pools that can easily handle 1.2Gbps with bbcp.

You can read here for more info: https://forums.freenas.org/index.ph...ove-freenas-11-replication.61207/#post-493788
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
Don't use zfs send on it's own as it's slow compared to other methods. What you really should look into is using bbcp to make the transfers as it will do line speed with the right options IF your storage can handle it on both ends. I have 2 FreeNAS boxes with huge pools that can easily handle 1.2Gbps with bbcp.

You can read here for more info: https://forums.freenas.org/index.ph...ove-freenas-11-replication.61207/#post-493788

I really appreciate it. I will give this a try next time. My rsync transfer finished after like 3 days. I guess I would need to install this in a jail to make it persistent.
 
Status
Not open for further replies.
Top