Trouble automating replication of freenas snapshots to zfs on Debian

[Andy Fraser]

Hello all,

I've named my ixsystems mini "daneeka" (the cautious MD in Catch-22).
I have set up a ZFS disk on my Debian system, "cathcart". I want to automatically and periodically (period of one hour, say) replicate snapshots from daneeka to cathcart.

After reading the documentation and several google searches, I've gotten my data from daneeka replicated on cathcart. Here are the key commands from the history (note that first incremental had -F on recv):

Code:

29  zfs send FiveT/for_linux@auto-20161205.0737-2h | ssh -p 81 -i /data/ssh/replication cathcart zfs recv zfsbackup/for_linux@auto-20161205.0737-2h

57  zfs send -i auto-20161205.0727-2h FiveT/for_linux@auto-20161205.0737-2h | ssh -p 81 -i /data/ssh/replication cathcart zfs recv -F zfsbackup/for_linux
58  zfs send -i auto-20161205.0737-2h FiveT/for_linux@auto-20161205.0747-2h | ssh -p 81 -i /data/ssh/replication cathcart zfs recv zfsbackup/for_linux
59  zfs send -i auto-20161205.0747-2h FiveT/for_linux@auto-20161205.0849-2d | ssh -p 81 -i /data/ssh/replication cathcart zfs recv zfsbackup/for_linux

Trying to get the same effect from the FreeNAS GUI produces email like the following:

From: XXX
Subject: daneeka.local: Critical Alerts
To: YYY
Date: Tue, 06 Dec 2016 01:10:15 -0000 (4 hours, 35 minutes, 46 seconds ago)

Replication FiveT/for_linux -> cathcart:zfsbackup/for_linux failed:
Failed: FiveT/for_linux (auto-20161205.1803-2d)

Here are the values I have in the GUI for snapshots and replication:

Code:

  Snapshot:

  Volume/Dataset: FiveT/for_linux
  Recursive: True
  When: always
  Frequency: every hour
  Keep snapshot for: 2 day
  VMware Sync: false
  Enabled true
 
  Replication:

  Volume/Dataset:  FiveT/for_linuxR
  Remote ZFS Volume/Dataset: zfsbackup/for_linux
  Recursively replicate child dataset's snapshots: check
  Delete stale snapshots on remote system: check
  Replication Stream Compression: lz4
  Limit (kB/s): 0
  Begin: 
  End: 
  Enabled: 
  Remote hostname: cathcart
  Remote port: 81
  Dedicated User Enabled: unchecked
  Dedicated User: 
  Encryption Cipher: Fast
  Remote hostkey: Public Key from cathcart

I'd be grateful for any advice on how to get freenas to automate what I can do from the command line.

 

[depasseg]

Did you try clearing out all the test snapshots?
http://doc.freenas.org/9.10/storage.html#troubleshooting-replication

If the zfs send fails, open Shell on PULL and use the zfs destroy -R volume_name@snapshot_name command to delete the stuck snapshot. You can then use the zfs list -t snapshot on PULL to confirm if the snapshot successfully replicated.

 

[wblock]

Why port 81 on the receiving system but not on the sending one? Also, the labels on some of those fields are wildly different from the current ones. Which version of FreeNAS is it?

 

[Andy Fraser]

I am not sure if I did that before posting, but I tried it just now
and the replication is still failing. Repeating: PUSH is daneeka and
PULL is cathcart. After "zfs destroy" on cathcart, I get

Code:

->sudo zfs list -t snapshot
no datasets available

and on daneeka, I get

Code:

->zfs list -t snapshot
NAME  USED  AVAIL  REFER  MOUNTPOINT
FiveT/for_linux@auto-20161205.1803-2d  2.03M  -  121G  -
FiveT/for_linux@auto-20161205.1903-2d  186K  -  121G  -
FiveT/for_linux@auto-20161205.2003-2d  186K  -  121G  -
FiveT/for_linux@auto-20161205.2103-2d  186K  -  121G  -
FiveT/for_linux@auto-20161205.2203-2d  895K  -  121G  -
FiveT/for_linux@auto-20161206.0700-2d  430K  -  121G  -
FiveT/for_linux@auto-20161206.0729-2d  1.54M  -  121G  -
freenas-boot/ROOT/9.10.1-U4@2016-11-23-09:41:09  4.24M  -  827M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-01-15:31:23  384K  -  828M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-02-21:53:04  1.84M  -  830M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-02-22:25:56  2.07M  -  830M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-03-21:05:19  216K  -  831M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-03-21:12:54  216K  -  831M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-03-21:55:06  224K  -  831M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-03-22:04:22  216K  -  831M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-03-22:11:18  136K  -  831M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-04-18:35:38  2.79M  -  831M  -

Re-enabling the replication task yields email that says:

From: X
Subject: Replication failed when sending FiveT/for_linux@auto-20161205.1803-2d
To: Y
Date: Tue, 06 Dec 2016 14:30:05 -0000 (8 minutes, 6 seconds ago)


Hello,
The replication failed for the local ZFS FiveT/for_linux while attempting to
send snapshot auto-20161205.1803-2d to cathcart

 

[Andy Fraser]

Why port 81 on the receiving system but not on the sending one? Also, the labels on some of those fields are wildly different from the current ones. Which version of FreeNAS is it?

I use different ports for ssh on each of my Linux boxes at home. I
have the NAT router forward them. Then from the outside, I can ssh to
each Linux box separately using the same IP address with different ports.
Daneeka, the freenas box, uses port 22 and is not accessible from
outside.

Here is a cut and paste from "System" in the freenas GUI:

Hostname daneeka.local
Build FreeNAS-9.10.1-U4 (ec9a7d3)

 

[wblock]

Please use code tags for the output so it is readable.

Does the receiving system have a volume called zfsbackup and a dataset called for_linux as a place to store the incoming snapshots?

 

[wblock]

Also, your first post says the replication task is not enabled.

 

[Andy Fraser]

Also, your first post says the replication task is not enabled.

Forty minutes ago I tried a new replication task from the FreeNas GUI that worked. The key change was to turn off "Replication Stream Compression". I did that after I found

Code:

Dec 6 21:46:05 daneeka autorepl.py: [tools.autorepl:157] Replicationresult: bash: /usr/local/bin/lz4c: No such file or directory

in /var/log/debuglog on the FreeNas system. On Debian the executable is in

Code:

/usr/bin/lz4c

Thanks to all who replied. After I get this working satisfactorily, I'll post a summary.

 

[wblock]

Aha, this has already been recognized and will be fixed in the next release: https://bugs.freenas.org/issues/17467.

Thanks to @William Grzybowski for pointing this out!

 

[Andy Fraser]

I made a fresh start on cathcart, the PULL Debian system, and modified the replication task on daneeka, the PUSH FreeNAS system. The following list explains the objections I had with my previous setup and the solutions that I implemented:

• C Backup mounted as /daneekaBU/for_linux/for_linux: Solution: Don't create daneekaBU/for_linux
• C Backup not compressed: Solution: Set compression=lz4 for pool
• C sshd uses port 81 exclusively: Solution: Add port 22 to /etc/ssh/sshd_config
• D Replication fails: Solution: Turn off stream compression

I implemented the solutions as follows:

On cathcart issue

Code:

root@cathcart:~# zpool destroy daneekaBU
root@cathcart:~# zpool create daneekaBU /dev/sdb
root@cathcart:~# zfs set compression=lz4 daneekaBU

and edit /etc/ssh/sshd_config to contain:

Code:

# What ports, IPs and protocols we listen for
Port 22
Port 81

See
http://superuser.com/questions/877186/zfs-send-recieve-over-ssh-on-linux-without-allowing-root-login
for why one can't have a dedicated user and how to use the root account on cathcart.

From the FreeNAS Replication GUI on daneeka:

Code:

Volume/Dataset:													FiveT/for_linux
  Remote Hostname:												cathcart
  Remote ZFS Volume/Dataset:								daneekaBU
  Recursively replicate child dataset's snapshots:	 true
  Delete stale snapshots on remote system:			  true
  Replication Stream Compression:						  false
  Limit (kB/s):														  0
  Begin:																   00:00:00
  End:																	  23:59:00
  Enabled:																true
  Remote hostname:												cathcart
  Remote port:														 22
  Dedicated User Enabled:									  false
  Dedicated User:  
  Encryption Cipher:											  Standard
  Remote hostkey:												 Public Key from cathcart​

​

From the command line on daneeka put the IP address for cathcart in /etc/hosts and ensure I can connect with:

Code:

->ssh cathcart whoami
The authenticity of host 'cathcart (192.168.1.6)' can't be established.
ECDSA key fingerprint is SHA256:6snp/JxNbIh/Kn5QDla/o/1VEhDvg/K/oFRfi9kLq6U.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cathcart' (ECDSA) to the list of known hosts.
andy
->sudo ssh cathcart whoami
The authenticity of host 'cathcart (192.168.1.6)' can't be established.
ECDSA key fingerprint is SHA256:6snp/JxNbIh/Kn5QDla/o/1VEhDvg/K/oFRfi9kLq6U.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cathcart' (ECDSA) to the list of known hosts.
root

Now the automated replication tasks work.