Help with replicating to 2 targets, maybe with script?

[yanir99]

Hi All,
I'm using version 9.2.1.9 on 3 FreeNAS machines.
2 of them are in-house on different floors, with replication of 2 datasets working great.
When I added the 3rd machine, which intended to be moved eventually to a DR site, I've added another 2 replication jobs for the same datasets to the new machine.
I quickly realized that because of the way FreeNAS marks the snapshost (-,NEW,LATEST) a replication to 2 targets is basically impossible.
My current work around is basically have the replication jobs for the 3rd machine disabled, and every couple of days I:
1. Disable the replication jobs to the 2nd machine
2. Use the CLI to find the last snapshot to the 3rd machine
3. change the freenas:state of that snapshots to LATEST
4. change all following snapshots freenas:state to NEW
5. enable the replication jobs for the 3rd machine
6. wait for the replication to finish successfully
7. disable the replication jobs for the 3rd machine
8. re-enable the replication jobs to the 2nd machine

Is there anyway to automate this?? with cron job maybe? or even just a script that I'll run manually?
Is there a way to disable / enable replication jobs (of the GUI) via CLI?

Thanks!

 

[Robert Trevellyan]

What if you were to automatically replicate from the 2nd machine to the 3rd?

 

[yanir99]

I"ve thought of that, but there is no way to create a replication job without scheduled snapshots, and all the snapshots on the 2nd machine are from the 1st machine so there is no scheduled snapshots defined on the 2nd...

 

[Robert Trevellyan]

OK, well replication works via the zfs send and zfs recv commands, so maybe you could create a CRON job that uses those. The basic idea (without the CRON job) is described here. I'm sure you'll need to play around with it on the command line first to get all the parts working together.

 

[yanir99]

for this I will have to still figure out from the 3rd machine what was the last snapshot it got, and send incremantly all the snapshots from that point forward... which I have no idea how to do automatically.

Do you have any idea if changing the status of the replication jobs on the gui, from enable to disable, via CLI is possible?

Thanks for the answers so far :)

 

[Robert Trevellyan]

for this I will have to still figure out from the 3rd machine what was the last snapshot it got, and send incremantly all the snapshots from that point forward...

Really? Doesn't sending an incremental snapshot take care of the in-betweens automatically? Honestly I haven't tried, it's just how I assume it would work.

Do you have any idea if changing the status of the replication jobs on the gui, from enable to disable, via CLI is possible?

No doubt with sufficient time and energy someone could figure out a way, but it sounds risky and it isn't something I would attempt.

 

[yanir99]

Really? Doesn't sending an incremental snapshot take care of the in-betweens automatically? Honestly I haven't tried, it's just how I assume it would work.

No doubt with sufficient time and energy someone could figure out a way, but it sounds risky and it isn't something I would attempt.

I have no idea.. this is why I asked for help.
I don't want to risk messing the current config because it's about 10TB of data and takes more than a couple of days resend everything.
with the send receive option I don't have any indication of the process or it's progress.
If using the send receive commands will definitely take care of all the snapshots in between I will try to get a cron job set for that, but I need a definite answer about that before testing on my machines..

 

[Robert Trevellyan]

If I understand this page correctly, you're correct, you have to specify two snapshots when sending an incremental snapshot. But keeping track of the last snapshot successfully sent would be sufficient. After a one-time initialization, and with the right script, this seems doable. You could even do it manually outside of your automated replication, without having to shut down that task the way you do now. Unless I'm missing something, it would greatly simplify your existing manual process.

Note that the zfs send and zfs recv commands have a -n option for dry-run and a -v option that will report progress (you would only use -v on the send in this scenario). (edited to add details).

 

[Roger Wilco]

If I understand this page correctly, you're correct, you have to specify two snapshots when sending an incremental snapshot. But keeping track of the last snapshot successfully sent would be sufficient. After a one-time initialization, and with the right script, this seems doable. <snip>

Hi,
yes it is doable but in shell you'll end up with a quite big and not very nice script. If you decide then to rewrite it in a "proper language" you'll probably choose Python on FreeNAS, and then you're basically rewriting the replication mechanism iXSystems have already implemented. I've had a similar requirement and wrote a Feature proposal about it. You can have a look at it here.

 

[Robert Trevellyan]

I've had a similar requirement and wrote a Feature proposal about it.

I have no argument with your proposal, but in the meantime it seems to me the OP could revise their existing manual procedure from 8 steps to 4:

1. Consult paper notes to find the last snapshot to the 3rd machine.
2. Execute zfs send | zfs recv command.
3. Wait for the replication to finish successfully.
4. Update paper notes with new "last snapshot".

Since this is something they do every "couple of days", the concern about having to look up the ZFS commands probably doesn't apply.

 

[yanir99]

Hi,
yes it is doable but in shell you'll end up with a quite big and not very nice script. If you decide then to rewrite it in a "proper language" you'll probably choose Python on FreeNAS, and then you're basically rewriting the replication mechanism iXSystems have already implemented. I've had a similar requirement and wrote a Feature proposal about it. You can have a look at it here.

I guess there was no reply for your feature request...

Is there a way to send all snapshots in between 2 snapshots, lets say I have 3 snapshots, can I send with 1 command snapshots 2 and 3 (assuming snapshot 1 is already on both)?

 

[yanir99]

Never mind, found it :)
I can actually get the name of the last snapshot sent to the 3rd FreeNAS by using:

Code:

/usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 <IP of 3rd freenas> "zfs list -Hr -o name -t snapshot -d 1 <VolumeName> | tail -n 1 | cut -d@ -f2"

and get the last snapshot created on the 1st FreeNAS with:

Code:

zfs list -Hr -o name -t snapshot -d 1 <Volume Name> | tail -n 1 | cut -d@ -f2"

and just use these as variables to send with -R and -I

I think I can setup a script with a cron job with that.

Thanks for the help and ideas!!!

 

[yanir99]

OK I made a script, called it man_repl.sh and the usage is: man_repl.sh <Volume Name Source> <Remote Machine IP> <Remote Volume Target>
You can probably change it a bit to fit your needs if you want to use it also, I'm going to run it as cron job.

Code:

#!bin/sh
echo "Script start at" $(date)
vol_from=$1
echo "Going to replicate from volume:" $vol_from
machine_toIP=$2
echo "To machine with IP:" $machine_toIP
vol_to=$3
echo "With volume name:" $vol_to
last_sent=$(/usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 $machine_toIP "zfs list -Hr -o name -t snapshot -d 1 $vol_to | tail -n 1 | cut -d@ -f2")
echo "The latest snapshot replicated to:" $machine_toIP "was" $last_sent
last_created=$(zfs list -Hr -o name -t snapshot -d 1 $vol_from | tail -n 1 | cut -d@ -f2)
echo "The last created snapshot on this machine was:" $last_created
echo "Starting  to send incremental snapshots recursivly between:" $last_sent "and" $last_created
zfs send -R -I $vol_from@$last_sent $vol_from@$last_created | ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 $machine_toIP "zfs receive -vF -d $vol_to"
echo "Finished running at" $(date)

 

[Robert Trevellyan]

OK I made a script, called it man_repl.sh and the usage is: man_repl.sh <Volume Name Source> <Remote Machine IP> <Remote Volume Target>

Excellent!

 

[cyberjock]

Just want to add one thing. In FreeNAS 9.2.1+, the .system dataset must exist, be writeable, etc. If you do replication from local:/mnt/tank to remote:/mnt/tank you *will* fubar the .system dataset on the destination server. This will literally bring down the destination server. FreeNAS 9.3 after some build from a month or two ago will *not* automatically replicate the .system dataset to prevent this. Do be warned that if you use this script to move a whole pool, that script *will* include any snapshots of the .system dataset and it will happily replicate those to the destination, thereby fubaring the destination server.

So use at your own risk (or benefit).

 

[Ericloewe]

Just want to add one thing. In FreeNAS 9.2.1+, the .system dataset must exist, be writeable, etc. If you do replication from local:/mnt/tank to remote:/mnt/tank you *will* fubar the .system dataset on the destination server. This will literally bring down the destination server. FreeNAS 9.3 after some build from a month or two ago will *not* automatically replicate the .system dataset to prevent this. Do be warned that if you use this script to move a whole pool, that script *will* include any snapshots of the .system dataset and it will happily replicate those to the destination, thereby fubaring the destination server.

So use at your own risk (or benefit).

9.2.1.6+, no?

 

[cyberjock]

Nope. It existed in 9.2.1.0. Samba4 really pushed the need for the .system dataset. 9.2.1.6 let you do things like store logs and rrd data. ;)

 

[ser_rhaegar]

Just want to add one thing. In FreeNAS 9.2.1+, the .system dataset must exist, be writeable, etc. If you do replication from local:/mnt/tank to remote:/mnt/tank you *will* fubar the .system dataset on the destination server. This will literally bring down the destination server.

Mine was trashed in 9.2.1.3 from replication and the system kept chugging along. Just complained about no log files. Upgrading to 9.3 worked on the same pool. It's still gimped and runs today - it's one of my lab servers/pools, a virtual throwaway (no my prod server isn't virtual) before you tell me I'm risking data loss. I wouldn't recommend it (who wants a system without logs?) but the system does run and the lab hasn't lost data almost a year later.

Code:

cp: /var/log/mount.today: No such file or directory
diff: /var/log/mount.today: No such file or directory
cp: /var/log/pf.today: No such file or directory
diff: /var/log/pf.today: No such file or directory
cp: /var/log/dmesg.today: No such file or directory
diff: /var/log/dmesg.today: No such file or directory

-- End of security output --

Edit: fixed version typo

 

[cyberjock]

They definitely don't run anymore. I see that problem almost daily. I never really had experience with the early 9.2.1.x releases, so they may be more forgiving than later releases.

9.3 poops all over itself then does some fecal smearing for good measure without a .system dataset. That's what lead to the protection mechanism for the .system dataset. It is also why the .system dataset doesn't mount in the pool anymore (it goes in /var/something).

 

[yanir99]

Just want to add one thing. In FreeNAS 9.2.1+, the .system dataset must exist, be writeable, etc. If you do replication from local:/mnt/tank to remote:/mnt/tank you *will* fubar the .system dataset on the destination server. This will literally bring down the destination server. FreeNAS 9.3 after some build from a month or two ago will *not* automatically replicate the .system dataset to prevent this. Do be warned that if you use this script to move a whole pool, that script *will* include any snapshots of the .system dataset and it will happily replicate those to the destination, thereby fubaring the destination server.

So use at your own risk (or benefit).

I've used this script for a couple of days now, and it indeed replicate the .system snapshost along, but the 3rd FreeNAS (the one I'm replicating to) is working fine and all data accessible...
Am I missing something??