Using BBCP to improve FreeNAS 11 Replication

[Glen Baars]

Hello,

As a long time user of FreeNAS replication we have always had a few issues with it.

1. Only replicates one job at a time.
2. Slow over WAN links with > 15ms latency

We have been using Solaris or FreeBSD with zrep/bbcp to overcome these issue on other installations. The logical thing to do was to port these changes across to FreeNAS.

At this stage I have this running on a few installs and it is fine. I hope that FreeNAS Dev team will implement these changes as a gui option :) If anyone has any code update for it, let me know.

The reason that BBCP is faster then the current SSH model is that it uses 12 threads to copy the data instead of 1. We went from 20-30mbit/s over 85ms latency to 600mbit/s when these changes.

To install the patch
download attached zip file
Copy autorepl.py.new to /usr/local/www/freenasUI/tools/autorepl.py on source FreeNAS.
Google bbcp-20120520 and download a copy of it. It has to be this old version as FreeBSD IPv6 stack has issues with new versions. I can't attach a copy due to the forum rules.
Copy bbcp to /sbin/bbcp on BOTH servers
Setup replication as per FreeNAS normal

Some important notes!
###does not support encryption###
Always compresses the data regardless of the settings in FreeNAS.
If you want to change the level of compression then find /sbin/bbcp %s -c 1 -s 12 in autorepl.py and increase the -c number at a cost of CPU cycles.
Requires all the BBCP ports to be open between the servers.

Glen

[Moderator's note: the attached zip file included a binary of bbcp. We do not redistribute unknown binaries here because of the security implications. An archive without that binary has been requested from the author.]

I have now attached a copy without the binary file required.

 

[vinca]

Hi Glen, I'm struggling right now to setup BBCP with ZREP. Which version of BBCP do you use?

Mine is failing because of IPv6 to IPv4 conversion and it's impossible to force the IPv4 for an unknown reason.

 

[Glen Baars]

I had the exact same issue. Ended up going back to bbcp-20120520 and all was well.

Google will help as i'm not allowed to attach a binary :D

 

[Glen Baars]

Also, I would not use zrep on the FreeNAS system. The inbuilt FreeNAS replication module is good and supports all the features you need. With the changes that I listed above you can just setup the replication through the GUI.

 

[vinca]

Also, I would not use zrep on the freenas system. The inbuilt freenas replication module is good and supports all the features you need. With the changes that I listed above you can just setup the replication through the GUI.

Well, I have only 1 FreeNAS system and I'm replicating data from other OSs. ZREP is perfect fit, although the installation is little bit clumsy. The FreeNAS replication is useless for me, until it supports pull tasks.

 

[PnoT]

I came across this post by searching google and had a few things to add if you want to compile the source yourself.

This post talks about using replication with bbcp and you can choose from 3 options:

1. Download a copy of the binary that he's compiled on 10.3
2. Download the make files to compile it yourself
3. A link to the exact repository

I chose to download the link with the Makefiles and it worked perfect on 11.1-U6.

A quick question on this replace file... do you see anything different in the replication screen via the GUI to know that the .py is working? I'm still using the old GUI style as well vs. the new one if that makes any difference.

 

[GEOptic]

hmmm… trying bbcp now… still 150Mbits … on a 1Gb direct connection.
CPU usage 20%. Disk sleeping 10% busy.


igb1 in 18.930 MB/s 19.632 MB/s 68.587 GB
out 451.480 KB/s 467.913 KB/s 1.697 GB


Replication between 2 iXSystems hardware (11.1U5, 2U, 64Gb Ram, Xeon 1.7Gb, RaidZ2 12x2TB)

I would not mind that speed for small replication, but 2.5TB will take more than 40 hours…

I did not recreate the replication task after installing bbcp and autorepl.sh on the systems. I just updated it (change start hour). I used the precompiled bbcp.


Ideas? Explications?

 

[PnoT]

Are you testing bbcp like some of the examples which go from /dev/zero -> /dev/null on another host and pulling these numbers? What options are you using with bbcp?

 

[GEOptic]

Default options. Did not change anything. Just install bbcp in /sbin and replaced autorepl.sh (have a backup of the file ;")

BBCP process command on source FreeNas :
/sbin/bbcp -c 1 -s 12 -N io -T /usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 10.1.1.202 /sbin/bbcp /sbin/zfs send -V Vol1/ge1-home@auto-20181105.2000-1m root@10.1.1.202:/sbin/zfs receive -F -d RepVol1

ZFS process running :

From source FreeNas
​


root@stk01:~ # ps -aux | grep zfs
root 10958 8.9 0.0 9912 2660 - S 20:00 28:51.91 zfs: sending Vol1/ge1-home@auto-20181105.2000-1m (46%: ...) (zfs)


From target FreeNas
root 60909 8.0 0.0 7808 2176 - S 20:00 36:52.60 /sbin/zfs receive -F -d RepVol1


The numbers come from

systat -ifstat

And they concur the UI graph (150Mbits/s) !

Source FreeNas 11.1u5 (Xeon E5-2603 v4, 64Gb Ram, 12x2TB) (CPU 20% avg)
Target FreeNas 11.1u6 (Atom C2750, 32Gb Ram, 8x2TB) (Mini XL) (CPU 3% avg)

I feel network performance should be way faster.

 

[PnoT]

Why not start with using just bbcp for a file transfer from one box to another and remove the replication pieces and the additional .py script?

Code:

bbcp -P 2 -s 4 local_file remoteIP:/folderpath/

A few of the options that may speed up or down would be more/less streams (-s option) or add the -w 1M option. On my 10Gbps I can easily get 800MB/sec+ with -s 4 -w 1M

 

[GEOptic]

Ok … did a few test with bbcp.


root@stk:/var/tmp # bbcp -P 2 -s 4 /var/tmp/X16-32010.exe root@192.168.1.2:/var/tmp/speedtest.txt
bbcp: Creating /var/tmp/speedtest.txt
bbcp: 181121 09:49:21 21% done; 111.9 MB/s
bbcp: 181121 09:49:23 58% done; 112.0 MB/s
bbcp: 181121 09:49:25 95% done; 112.0 MB/s


This is very good speed on a 1Gb nic.

So, something is going on with replication!

EDIT : Just tried without encryption. Worst !