Wishbringer
Cadet
- Joined
- Aug 22, 2014
- Messages
- 5
After assembling an 6bay NAS with an SuperMicro ASAi-2750 (Avoton 8 cores),
I experienced poor rsync performance.
When running for example three rsync tasks (Backups, Videos, Musik) to another NAS,
I only got around 300MBit/sec. over network.
After research, I found that even when CPU is only uses around 34%, rsync threads topped single cores at 100%.
So rsync is single-threaded and don't use more than one core.
On energy-efficent NASes (C2750, J1900) this is a drawback.
Possible solution (found in www):
https://wiki.ncsa.illinois.edu/display/~wglick/Parallel+Rsync
over ssh:
But I am not a programmer... so posting here as a suggestion.
I experienced poor rsync performance.
When running for example three rsync tasks (Backups, Videos, Musik) to another NAS,
I only got around 300MBit/sec. over network.
After research, I found that even when CPU is only uses around 34%, rsync threads topped single cores at 100%.
So rsync is single-threaded and don't use more than one core.
On energy-efficent NASes (C2750, J1900) this is a drawback.
Possible solution (found in www):
https://wiki.ncsa.illinois.edu/display/~wglick/Parallel+Rsync
/bin/bash
# SETUP OPTIONS
export SRCDIR="/folder/path"
export DESTDIR="/folder2/path"
export THREADS="8"
# RSYNC TOP LEVEL FILES AND DIRECTORY STRUCTURE
rsync -lptgoDvzd $SRCDIR/ /$DESTDIR/
# FIND ALL FILES AND PASS THEM TO MULTIPLE RSYNC PROCESSES
cd $SRCDIR; find . -type f | xargs -n1 -P$THREADS -I% rsync -az % /$DESTDIR/%
# IF YOU WANT TO LIMIT THE IO PRIORITY,
# PREPEND THE FOLLOWING TO THE rsync & cd/find COMMANDS ABOVE:
# ionice -c2
# SETUP OPTIONS
export SRCDIR="/folder/path"
export DESTDIR="/folder2/path"
export THREADS="8"
# RSYNC TOP LEVEL FILES AND DIRECTORY STRUCTURE
rsync -lptgoDvzd $SRCDIR/ /$DESTDIR/
# FIND ALL FILES AND PASS THEM TO MULTIPLE RSYNC PROCESSES
cd $SRCDIR; find . -type f | xargs -n1 -P$THREADS -I% rsync -az % /$DESTDIR/%
# IF YOU WANT TO LIMIT THE IO PRIORITY,
# PREPEND THE FOLLOWING TO THE rsync & cd/find COMMANDS ABOVE:
# ionice -c2
over ssh:
rsync -lptgoDvzd -e 'ssh -c arcfour' $SRCDIR/ remotehost:/$DESTDIR/
cd $SRCDIR; find . -type f | xargs -n1 -P$THREADS -I% rsync -az -e 'ssh -c arcfour' % remotehost:/$DESTDIR/%
cd $SRCDIR; find . -type f | xargs -n1 -P$THREADS -I% rsync -az -e 'ssh -c arcfour' % remotehost:/$DESTDIR/%
But I am not a programmer... so posting here as a suggestion.