Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

how to properly backup (replicate) iocage jails for disaster recovery?

guermantes

FreeNAS Experienced
Joined
Sep 27, 2017
Messages
116
Thanks
8
#1
I have searched Oracle zfs docs and the forum but I am still unsure regarding how to properly backup my iocage jails to a different pool so as to have a backup if I mess up or otherwise get my jails destroyed.

Naively I began with a recursive snapshop task of the entire iocage/ tree and tried to replicate that to my secondary pool where I collect locally stored backups (not optimal but I also have disks offsite). Only the top iocage dataset was transferred but nothing of substance relating to my jails (which are nested further down in the iocage directory tree). As per this thread it seems that was to be expected as replication does not include child datasets. What a bummer!

So what are my options? Could I simply snapshot and replicate the e.g. tank/iocage/jails/MYJAILNAME-1/root and tank/iocage/jails/MYJAILNAME-2/root datasets? Is that how it should be done? Or are they dependent on files in other datasets in the iocage/ hierarchy? Do I also need to separately replicate all the datasets in the iocage/ tree so as to be able to clone back the iocage/ tree in its entirety?

Now, someone may say replication task, but I have tried that and I am not succeeding in setting up "localhost replication". I have tried twice and have lost several hours but still can't understand how to set it up even after looking at threads discussing localhost replication here on the forums. (I get an "ECDSA key error").
 

dlavigne

Storage Engineering
Administrator
Moderator
iXsystems
Joined
May 24, 2011
Messages
11,502
Thanks
992
#2
That is a very old thread and a bug that was fixed over 4 years ago.

Did you get any further with this? If not, are you running 11.2-U3 (as in your signature) or are you up-to-date with 11.2-U5?
 

guermantes

FreeNAS Experienced
Joined
Sep 27, 2017
Messages
116
Thanks
8
#3
That is a very old thread and a bug that was fixed over 4 years ago.

Did you get any further with this? If not, are you running 11.2-U3 (as in your signature) or are you up-to-date with 11.2-U5?
No, I haven't got any further regrettably. I tried incorporating the "root" dataset for one of my iocage jails in my nightly bash/cron script that replicates snapshots of my important datasets locally to a different pool, but after successful initial replication night 1, on night 2 it failed during the incremental replication of that dataset with an error none of the other datasets in the script have ever thrown (warning: cannot send 'TANK/iocage/jails/jailwebdev/root@auto-20190907.0200-3m': signal received). Since then I have work to do so I haven't been able to investigate.

I'm still on 11.2 U3. Perhaps U5 with the fix "Do not crash replication script on misconfigured SSH connection" will allow me to go down the replication route instead? Will try to find time to look at the during the coming week-end.
 
Joined
Jul 3, 2016
Messages
18
Thanks
1
#4
I have a recursive replication setup for work that hasn't given me any trouble, but that isn't trying to replicate jailed datasets. My guess would be that your issue may be related to that, but I could be mis-remembering what I read about jailed datasets.
 

guermantes

FreeNAS Experienced
Joined
Sep 27, 2017
Messages
116
Thanks
8
#5
From the log:
cannot receive incremental stream: destination GOLD/sendrecv/iocage/jails/jailwebdev/root has been modified since most recent snapshot.

So the destination has been modified? I would expect this if atime was on but it is not. Not for any of the datasets in the hierarchy.

Does it have to do with the fact that the jail is running when the snapshot is taken?
 
Joined
Jul 3, 2016
Messages
18
Thanks
1
#6
OK so the problem is not related to the dataset being jailed.

You are reading the that message from the log correctly, the destination dataset has been modified. This would not be caused by a running jail when the snapshot is taken.

Are you sharing (NFS or CIFS/SMB) the destination dataset? Even if the destination dataset is read only a sharing process can lock the dataset and cause the error you are seeing. The read only property in ZFS (on a dataset/zvol not file or folder) prevents data from changing, but won't stop ZFS replication or process locks; based on my experience.
 

guermantes

FreeNAS Experienced
Joined
Sep 27, 2017
Messages
116
Thanks
8
#7
No. None of the datasets in pool GOLD are being shared. I am completely flummoxed by this.
 
Joined
Jul 3, 2016
Messages
18
Thanks
1
#8
Code:
zfs get all <dataset>

I wonder if a difference in dataset version could be an issue.

I'm thinking you may need to write a script to get the outcome you're looking for. If you use the built-in replication tool, it will flatten the dataset hierarchy.
 

guermantes

FreeNAS Experienced
Joined
Sep 27, 2017
Messages
116
Thanks
8
#9
The versions seem to be the same (5).

source:
Code:
root@freenas:~ # zfs get all TANK/iocage/jails/jailwebdev/root
NAME                               PROPERTY                 VALUE                                              SOURCE
TANK/iocage/jails/jailwebdev/root  type                     filesystem                                         -
TANK/iocage/jails/jailwebdev/root  creation                 Fri Apr  5 20:08 2019                              -
TANK/iocage/jails/jailwebdev/root  used                     633M                                               -
TANK/iocage/jails/jailwebdev/root  available                5.46T                                              -
TANK/iocage/jails/jailwebdev/root  referenced               978M                                               -
TANK/iocage/jails/jailwebdev/root  compressratio            1.89x                                              -
TANK/iocage/jails/jailwebdev/root  mounted                  yes                                                -
TANK/iocage/jails/jailwebdev/root  origin                   TANK/iocage/releases/11.2-RELEASE/root@jailwebdev  -
TANK/iocage/jails/jailwebdev/root  quota                    none                                               default
TANK/iocage/jails/jailwebdev/root  reservation              none                                               default
TANK/iocage/jails/jailwebdev/root  recordsize               128K                                               default
TANK/iocage/jails/jailwebdev/root  mountpoint               /mnt/TANK/iocage/jails/jailwebdev/root             default
TANK/iocage/jails/jailwebdev/root  sharenfs                 off                                                default
TANK/iocage/jails/jailwebdev/root  checksum                 on                                                 default
TANK/iocage/jails/jailwebdev/root  compression              lz4                                                inherited from TANK/iocage/jails
TANK/iocage/jails/jailwebdev/root  atime                    off                                                inherited from TANK
TANK/iocage/jails/jailwebdev/root  devices                  on                                                 default
TANK/iocage/jails/jailwebdev/root  exec                     on                                                 default
TANK/iocage/jails/jailwebdev/root  setuid                   on                                                 default
TANK/iocage/jails/jailwebdev/root  readonly                 off                                                default
TANK/iocage/jails/jailwebdev/root  jailed                   off                                                default
TANK/iocage/jails/jailwebdev/root  snapdir                  hidden                                             default
TANK/iocage/jails/jailwebdev/root  aclmode                  passthrough                                        inherited from TANK/iocage/jails
TANK/iocage/jails/jailwebdev/root  aclinherit               passthrough                                        inherited from TANK/iocage/jails
TANK/iocage/jails/jailwebdev/root  canmount                 on                                                 default
TANK/iocage/jails/jailwebdev/root  xattr                    off                                                temporary
TANK/iocage/jails/jailwebdev/root  copies                   1                                                  default
TANK/iocage/jails/jailwebdev/root  version                  5                                                  -
TANK/iocage/jails/jailwebdev/root  utf8only                 off                                                -
TANK/iocage/jails/jailwebdev/root  normalization            none                                               -
TANK/iocage/jails/jailwebdev/root  casesensitivity          sensitive                                          -
TANK/iocage/jails/jailwebdev/root  vscan                    off                                                default
TANK/iocage/jails/jailwebdev/root  nbmand                   off                                                default
TANK/iocage/jails/jailwebdev/root  sharesmb                 off                                                default
TANK/iocage/jails/jailwebdev/root  refquota                 none                                               default
TANK/iocage/jails/jailwebdev/root  refreservation           none                                               default
TANK/iocage/jails/jailwebdev/root  primarycache             all                                                default
TANK/iocage/jails/jailwebdev/root  secondarycache           all                                                default
TANK/iocage/jails/jailwebdev/root  usedbysnapshots          101M                                               -
TANK/iocage/jails/jailwebdev/root  usedbydataset            531M                                               -
TANK/iocage/jails/jailwebdev/root  usedbychildren           0                                                  -
TANK/iocage/jails/jailwebdev/root  usedbyrefreservation     0                                                  -
TANK/iocage/jails/jailwebdev/root  logbias                  latency                                            default
TANK/iocage/jails/jailwebdev/root  dedup                    off                                                default
TANK/iocage/jails/jailwebdev/root  mlslabel                                                                    -
TANK/iocage/jails/jailwebdev/root  sync                     standard                                           default
TANK/iocage/jails/jailwebdev/root  refcompressratio         1.88x                                              -
TANK/iocage/jails/jailwebdev/root  written                  8.76M                                              -
TANK/iocage/jails/jailwebdev/root  logicalused              848M                                               -
TANK/iocage/jails/jailwebdev/root  logicalreferenced        1.38G                                              -
TANK/iocage/jails/jailwebdev/root  volmode                  default                                            default
TANK/iocage/jails/jailwebdev/root  filesystem_limit         none                                               default
TANK/iocage/jails/jailwebdev/root  snapshot_limit           none                                               default
TANK/iocage/jails/jailwebdev/root  filesystem_count         none                                               default
TANK/iocage/jails/jailwebdev/root  snapshot_count           none                                               default
TANK/iocage/jails/jailwebdev/root  redundant_metadata       all                                                default
TANK/iocage/jails/jailwebdev/root  org.freebsd.ioc:active   yes                                                inherited from TANK
TANK/iocage/jails/jailwebdev/root  org.freenas:description                                                     inherited from TANK


destination:
Code:
root@freenas:~ # zfs get all GOLD/sendrecv/iocage/jails/jailwebdev/root
NAME                                        PROPERTY                 VALUE                                            SOURCE
GOLD/sendrecv/iocage/jails/jailwebdev/root  type                     filesystem                                       -
GOLD/sendrecv/iocage/jails/jailwebdev/root  creation                 Sat Sep 14  3:30 2019                            -
GOLD/sendrecv/iocage/jails/jailwebdev/root  used                     804M                                             -
GOLD/sendrecv/iocage/jails/jailwebdev/root  available                4.23T                                            -
GOLD/sendrecv/iocage/jails/jailwebdev/root  referenced               804M                                             -
GOLD/sendrecv/iocage/jails/jailwebdev/root  compressratio            1.88x                                            -
GOLD/sendrecv/iocage/jails/jailwebdev/root  mounted                  yes                                              -
GOLD/sendrecv/iocage/jails/jailwebdev/root  quota                    none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  reservation              none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  recordsize               128K                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  mountpoint               /mnt/GOLD/sendrecv/iocage/jails/jailwebdev/root  default
GOLD/sendrecv/iocage/jails/jailwebdev/root  sharenfs                 off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  checksum                 on                                               default
GOLD/sendrecv/iocage/jails/jailwebdev/root  compression              lz4                                              inherited from GOLD
GOLD/sendrecv/iocage/jails/jailwebdev/root  atime                    off                                              inherited from GOLD/sendrecv
GOLD/sendrecv/iocage/jails/jailwebdev/root  devices                  on                                               default
GOLD/sendrecv/iocage/jails/jailwebdev/root  exec                     on                                               default
GOLD/sendrecv/iocage/jails/jailwebdev/root  setuid                   on                                               default
GOLD/sendrecv/iocage/jails/jailwebdev/root  readonly                 off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  jailed                   off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  snapdir                  hidden                                           default
GOLD/sendrecv/iocage/jails/jailwebdev/root  aclmode                  passthrough                                      inherited from GOLD/sendrecv/iocage/jails/jailwebdev
GOLD/sendrecv/iocage/jails/jailwebdev/root  aclinherit               passthrough                                      inherited from GOLD
GOLD/sendrecv/iocage/jails/jailwebdev/root  canmount                 on                                               default
GOLD/sendrecv/iocage/jails/jailwebdev/root  xattr                    off                                              temporary
GOLD/sendrecv/iocage/jails/jailwebdev/root  copies                   1                                                inherited from GOLD/sendrecv/iocage/jails/jailwebdev
GOLD/sendrecv/iocage/jails/jailwebdev/root  version                  5                                                -
GOLD/sendrecv/iocage/jails/jailwebdev/root  utf8only                 off                                              -
GOLD/sendrecv/iocage/jails/jailwebdev/root  normalization            none                                             -
GOLD/sendrecv/iocage/jails/jailwebdev/root  casesensitivity          sensitive                                        -
GOLD/sendrecv/iocage/jails/jailwebdev/root  vscan                    off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  nbmand                   off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  sharesmb                 off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  refquota                 none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  refreservation           none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  primarycache             all                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  secondarycache           all                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  usedbysnapshots          152K                                             -
GOLD/sendrecv/iocage/jails/jailwebdev/root  usedbydataset            804M                                             -
GOLD/sendrecv/iocage/jails/jailwebdev/root  usedbychildren           0                                                -
GOLD/sendrecv/iocage/jails/jailwebdev/root  usedbyrefreservation     0                                                -
GOLD/sendrecv/iocage/jails/jailwebdev/root  logbias                  latency                                          default
GOLD/sendrecv/iocage/jails/jailwebdev/root  dedup                    off                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  mlslabel                                                                  -
GOLD/sendrecv/iocage/jails/jailwebdev/root  sync                     standard                                         default
GOLD/sendrecv/iocage/jails/jailwebdev/root  refcompressratio         1.88x                                            -
GOLD/sendrecv/iocage/jails/jailwebdev/root  written                  152K                                             -
GOLD/sendrecv/iocage/jails/jailwebdev/root  logicalused              1.38G                                            -
GOLD/sendrecv/iocage/jails/jailwebdev/root  logicalreferenced        1.38G                                            -
GOLD/sendrecv/iocage/jails/jailwebdev/root  volmode                  default                                          default
GOLD/sendrecv/iocage/jails/jailwebdev/root  filesystem_limit         none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  snapshot_limit           none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  filesystem_count         none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  snapshot_count           none                                             default
GOLD/sendrecv/iocage/jails/jailwebdev/root  redundant_metadata       all                                              default
GOLD/sendrecv/iocage/jails/jailwebdev/root  org.freenas:description                                                   inherited from GOLD/sendrecv


I am thinking maybe I should just call it quits trying to backup my jails like this, because it occurs to me that the jails themselves are pretty static. jailwebdev has it's code parts in a different dataset which is mounted into the jail but already being snapshotted successfully from its "proper" dataset, and for the nextcloud jail I suppose I could cron-backup the database instead of the entire jail, and send the database to a location where it will be backed up.
 

guermantes

FreeNAS Experienced
Joined
Sep 27, 2017
Messages
116
Thanks
8
#10
I'm thinking you may need to write a script to get the outcome you're looking for. If you use the built-in replication tool, it will flatten the dataset hierarchy.
I am actually using a homebrewed script already. Super elementary I am sure since it's pretty much my first that is longer than ten lines.

I am posting it below, but I am in no way asking anyone to review it unless they take pleasure in doing so.

Code:
#!/bin/bash

# scp sendrecv.sh root@10.0.0.50:/mnt/TANK/sysadmin/sendrecv/
# This script performs incremental zfs send/recv within the same Freenas system, for those who do not have the luxury of having a second system and thus cannot set up standard replication tasks. The script itself is intended to be run by cron, and supports incremental send/recv as well as a first send/recv operation the first time the script is run on a particular dataset in order to kick-off the process of sending incrementally. To start anew, delete the file 'previousNNsnapshot' in assets folder and empty the corresponding dataset on GOLD.
# PREREQUISITE: snapshots to be sent must be created automatically in order to be accessible to the script.
# The second grep filters out snaps that only live a few hours and thus are prone to disappear before the next send which leads to broken incremental


# Backup function (incremental)
backup() {
    zfs list -t snapshot -o name | grep $snapshotPrefix$dataset@auto | grep '[^h]$' > $assetPath$dataset"snapshots"
    currentSnap=$( tail -n 1 $assetPath$dataset"snapshots" )
    previousSnap=$( head -n 1 $assetPath"previous"$dataset"snapshot" )
    if [ $currentSnap = $previousSnap ]
    then
        echo $(date '+%F %T') "WARNING:" $dataset"@auto: previous and current snapshots are the same. Nothing to do, aborting..." >> $logFile
    else
        zfs send -i $previousSnap $currentSnap | zfs recv $destinationPath$dataset 2> $assetPath$dataset"-zfserror"
        if [ $? -eq 0 ]
        then
            echo $currentSnap > $assetPath"previous"$dataset"snapshot"
            echo $(date '+%F %T') "SUCCESS:" $dataset"@auto: zfs send/recv successful." >> $logFile
        else
            echo $(date '+%F %T') "FAILURE:" $dataset"@auto: zfs send/recv failed while sending or receiving." >> $logFile
            cat $assetPath$dataset"-zfserror" >> $logFile
        fi
        rm $assetPath$dataset"-zfserror"
    fi
}


# Send initial snapshot, executes if script has not run before for this dataset
initial_send() {
    zfs list -t snapshot -o name | grep $snapshotPrefix$dataset@auto | grep '[^h]$' > $assetPath$dataset"snapshots"
    currentSnap=$( tail -n 1 $assetPath$dataset"snapshots" )
    zfs send $currentSnap | zfs recv -F $destinationPath$dataset 2> $assetPath$dataset"-zfserror"
    if [ $? -eq 0 ]
    then
        echo $currentSnap > $assetPath"previous"$dataset"snapshot"
        echo $(date '+%F %T') "SUCCESS:" $dataset"@auto: initial send successful." >> $logFile
    else
        echo $(date '+%F %T') "FAILURE:" $dataset"@auto: initial send failed." >> $logFile
        cat $assetPath$dataset"-zfserror" >> $logFile
    fi
    rm $assetPath$dataset"-zfserror"
}


# Checks whether script has run before on dataset in question and chooses appropriate action
execute_script() {
    if [ -a $assetPath"previous"$dataset"snapshot" ]
    then
        backup
    else
        initial_send
    fi
}



# Backup function (incremental)
backup_jail() {
    zfs list -t snapshot -o name | grep $snapshotPrefix$dataset$datasetsuffix@auto | grep '[^h]$' > $assetPath$dataset"snapshots"
    currentSnap=$( tail -n 1 $assetPath$dataset"snapshots" )
    previousSnap=$( head -n 1 $assetPath"previous"$dataset"snapshot" )
    if [ $currentSnap = $previousSnap ]
    then
        echo $(date '+%F %T') "WARNING:" $dataset"@auto: previous and current snapshots are the same. Nothing to do, aborting..." >> $logFile
    else
        zfs send -i $previousSnap $currentSnap | zfs recv $destinationPathJail$dataset$datasetsuffix 2> $assetPath$dataset"-zfserror"
        if [ $? -eq 0 ]
        then
            echo $currentSnap > $assetPath"previous"$dataset"snapshot"
            echo $(date '+%F %T') "SUCCESS:" $dataset"@auto: zfs send/recv successful." >> $logFile
        else
            echo $(date '+%F %T') "FAILURE:" $dataset"@auto: zfs send/recv failed while sending or receiving." >> $logFile
            cat $assetPath$dataset"-zfserror" >> $logFile
        fi
        rm $assetPath$dataset"-zfserror"
    fi
}


# Send initial snapshot, executes if script has not run before for this dataset
initial_send_jail() {
    zfs list -t snapshot -o name | grep $snapshotPrefix$dataset$datasetsuffix@auto | grep '[^h]$' > $assetPath$dataset"snapshots"
    currentSnap=$( tail -n 1 $assetPath$dataset"snapshots" )
    zfs send $currentSnap | zfs recv -F $destinationPathJail$dataset$datasetsuffix 2> $assetPath$dataset"-zfserror"
    if [ $? -eq 0 ]
    then
        echo $currentSnap > $assetPath"previous"$dataset"snapshot"
        echo $(date '+%F %T') "SUCCESS:" $dataset"@auto: initial send successful." >> $logFile
    else
        echo $(date '+%F %T') "FAILURE:" $dataset"@auto: initial send failed." >> $logFile
        cat $assetPath$dataset"-zfserror" >> $logFile
    fi
    rm $assetPath$dataset"-zfserror"
}


# Checks whether script has run before on dataset in question and chooses appropriate action
jail_execute_script() {
    if [ -a $assetPath"previous"$dataset"snapshot" ]
    then
        backup_jail
    else
        initial_send_jail
    fi
}



# Constants
assetPath="/mnt/TANK/sysadmin/sendrecv/assets/" # WARNING: changing this directory will break incremental replication, since previousfile will not be found in the new directory and as such a new sending with -F flag will occur the next time the script is run.
logFile="/mnt/TANK/sysadmin/sendrecv/sendrecv.log"
destinationPath="GOLD/sendrecv/" # dataset has to exist and be empty, and must be created with atime off, otherwise incremental is prone to break
destinationPathJail="GOLD/sendrecv/iocage/jails/" # dataset has to exist and be empty, and must be created with atime off, otherwise incremental is prone to break


# Later on we want to know what day it is...
today=$( LC_TIME=en_GB.UFT8 date +%A )

# Task 1
dataset="dev" # name of dataset that the snapshot to be sent refers to
snapshotPrefix="TANK/" # varies depending on whether snapshotted datasets are nested or not and REQUIRES TRAILING SLASH
execute_script
echo "DEV done!"

# Task 2
dataset="hibou"
snapshotPrefix="TANK/"
execute_script
echo "HIBOU done!"

# Task 3
dataset="home"
snapshotPrefix="TANK/"
execute_script
echo "HOME done!"

# Task 4
dataset="foto"
snapshotPrefix="TANK/"
execute_script
echo "FOTO done!"

# Task 5
dataset="nibelheim"
snapshotPrefix="TANK/"
execute_script
echo "NIBELHEIM done!"

# Task 6
dataset="sysadmin"
snapshotPrefix="TANK/"
execute_script
echo "SYSADMIN done!"

# Task 7


# Task 8
dataset="backup"
snapshotPrefix="TANK/"
execute_script
echo "BACKUP done!"
        
# Task 9 only runs on Sundays since Musica snapshots are weekly and I don't wan't to clutter the log with warnings during the week
if [ $today = "Sunday" ]
then
        dataset="musica"
        snapshotPrefix="TANK/bibliotek/"
        execute_script
        echo "MUSICA done!"
fi

# Task 10 only runs on Sundays since Video snapshots are weekly and I don't wan't to clutter the log with warnings during the week
if [ $today = "Sunday" ]
then
        dataset="bluray-musik"
        snapshotPrefix="TANK/bibliotek/video/"
        execute_script
        echo "VIDEO BLURAY-MUSIK done!"
fi

# Task 11 only runs on Sundays since Video snapshots are weekly and I don't wan't to clutter the log with warnings during the week
if [ $today = "Sunday" ]
then
        dataset="annat"
        snapshotPrefix="TANK/bibliotek/video/"
        execute_script
        echo "VIDEO ANNAT done!"
fi

# Jails
# Task 12
dataset="jailwebdev" # name of dataset that the snapshot to be sent refers to
snapshotPrefix="TANK/iocage/jails/" # REQUIRES TRAILING SLASH
datasetsuffix="/root"
jail_execute_script
echo "JAILWEBDEV done!"

# Task 13
dataset="nextcloud"
snapshotPrefix="TANK/iocage/jails/"
datasetsuffix="/root"
jail_execute_script
echo "NEXTCLOUD done!"
 
Top