noobe question: zfs volumes and pools....

[ghostwolf59]

Hi,
I have just upgraded to freenas 9.2.1.8 and slowly trying to get my head around zfs volumes.

I have so far always used ufs volumes and have a good grip on how I can manage backing up my data.

With zfs volumes I run into several questions in term of how I best can backup my data and also understand how I can restore data from a backup in case of a disk failure or other data loss issues

I don't plan to automate this - happy to take backups manually

so this is what I regularly do on my ufs drives...
rsync -pavt /mnt/<ufs_volume> /mnt/<ufs_backup_volume>

In the event of recover the complete drive I can simply fall back on the backup

How would I do this on zfs volumes

I guess I can do a rsync -pavt /mnt/<zfs_volume> /mnt/<zfs_backup_volume>
but that would not create a image copy of the original drive and its nested dataset(s)
So my question is simply this...
If I've done a rsync on my zfs volume and my drive dies, how could I then use my backup to re-create all original datasets on a new replacement drive ?

Do I need to create each pool on my backup first and then backup each zfs dataset separate or...? Don's seem to be able to get my head around this.

Please note that my aim is to be able to backup all my data and dataset on a secondary device (hdd) that I can mount at will.

cheers

 

[danb35]

Ordinarily you will have only one pool, which is your volume. With UFS, if you had a complete failure, you'd recreate the volume, then restore. With ZFS, it's the same--you recreate the volume (pool), and then restore your data. Your reference to "the original drive and its pools" sounds like you're misunderstanding what a pool is. What do you mean by that term? Is it possible you're referring to datasets instead?

To back up a ZFS pool, you can do rsync, just as you've always done. You can also use ZFS replication, which would likely be a better option.

 

[ghostwolf59]

your absolutely right - my reference to pool(s) was actually meant to be datasets within the volume. I have played around with various ways of backing up my datasets on the volume, but so far not even near a 100% success.
Partly because the dataset I play with (jails) have nested datasets within. When I try to backup jails I successfully create the root folders on each nested dataset but with empty content

This is what I tried...
Backup:
#jails
zfs snapshot -r STORE/jails@$(date +%Y%m%d)
zfs send STORE/jails@$(date +%Y%m%d) | zfs recv -d BAK/Backups

When I do this the data stored on the BAK/Backups holds all nested datasets from my jail, but with empty content (with some exceptions)

Then I tried to do a backup of each jail dataset (that comes across as silly - ought to be a simpler way acting on a root down structure), when I ran into issues such as (jails already exists exception being thrown)
zfs snapshot -r STORE/jails/.warden-template-pluginjail@$(date +%Y%m%d)
zfs send STORE/jails/.warden-template-pluginjail@$(date +%Y%m%d) | zfs recv -d BAK/Backups/jails

zfs snapshot -r STORE/jails/couchpotato_1@$(date +%Y%m%d)
zfs send STORE/jails/couchpotato_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups/jails

zfs snapshot -r STORE/jails/firefly_1@$(date +%Y%m%d)
zfs send STORE/jails/firefly_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups/jails

zfs snapshot -r STORE/jails/minidlna_1@$(date +%Y%m%d)
zfs send STORE/jails/minidlna_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups/jails

zfs snapshot -r STORE/jails/plexmediaserver_1@$(date +%Y%m%d)
zfs send STORE/jails/plexmediaserver_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups/jails

zfs snapshot -r STORE/jails/sabnzbd_1@$(date +%Y%m%d)
zfs send STORE/jails/sabnzbd_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups/jails

zfs snapshot -r STORE/jails/sickbeard_1@$(date +%Y%m%d)
zfs send STORE/jails/sickbeard_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups

zfs snapshot -r STORE/jails/transmission_1@$(date +%Y%m%d)
zfs send STORE/jails/transmission_1@$(date +%Y%m%d) | zfs recv -d BAK/Backups

 

[ghostwolf59]

rsync would backup my entire wolume, but my datasets within would get lost (other than preserved directory structure on the backup media)
Is there a way to backup a zfs volume and preserve all mapped datasets so it easily could be restored without being forced to manually recreate each dataset before copy the content back from the backup ?
That's really what I want.

Also want a easy way to backup every single dataset within the volume without having to worry about apps/plugins adding nested datasets within what I just created *(such as with jail plugins)

sorry if all of this comes across as somewhat dumb - but at the moment "dumb" is my middle name :)

 

[ghostwolf59]

Is this a common problem or...? 66 viewer (one kind attempt to answer my question with another question (legit though :) )

danb32 seem to have understood only part of my question. sure I can do a rsync on the complete volume, but that leave a manual task to try to recreate each defined zfs dataset as they were pre hardware failure.

What I'm looking for is the simplest possible straight forward way to backup zfs volumes and defined zfs datasets within, that then would allow me to restore the entire volume and it's zfs datasets or individual zfs dataset from backup with preserved access rights/permission.

My attempt to backup/restore my zfs jails dataset seem to be an example were installed plugins ends up creating nested zfs datasets within the defines jails zfs dataset - something I don't have control over or even know about (or really care about, but something I would be keen to re-create if needed


cheers

 

[Ericloewe]

Have you looked at ZFS replication? It should do everything you're looking for.

 

[rogerh]

Speaking as another newby, should you not use 'zfs send' with some flag to make it recursive?

Also, I hope your 'STORE' and 'BAK' really are on completely separate pools, you were a bit vague about pools, volumes and datasets above!

 

[ghostwolf59]

ok, so I have now played around with a manual backup/restore
The backup seem to work, but the restore seem somewhat problematic - not sure if this because I am testing this against the same drive where the backup where stored - keep getting broken pipe/IO error on the
"zfs send -R BACKUP@BACKUP-20141107 | zfs receive -vF BACKUP/" statement

so here's the scenario
1. used the script to backup my jail (STORE02x)
2 verified that both data and snapshots where created (both on STORE02X and BACKUP) named STORE02X-ccyymmdd and BACKUP-ccyymmdd
These snapshots is complete with all child storage

Then I try to do a restore of what I just backed up
enter backrest.sh restore BACKUP ccyymmdd <enter>
Where the "zfs send -R BACKUP@BACKUP-20141107 | zfs receive -vF BACKUP/" throws an IO error (broken pipe)

console snippet:
[root@freenas ~]# zfs send -R BACKUP@BACKUP-20141107 | zfs receive -vF BACKUP/
cannot receive: invalid name
warning: cannot send 'BACKUP@BACKUP-20141107': Broken pipe
cannot send 'BACKUP': I/O error



Before I sign off on this I'd like to get some conformation of that I also can restore what I (seemingly) could backup

here's the complete script (not that I never managed to get a true/false test to work by calling functions - hence falling back on testing the return (silly but that's the best I can do for now)
My backup drive name is static (BACKUP)
the rest is parameter driven
Pleeeeeease, do not get too technical with how I written this or how I can resolve the problem

zfs snapshot command to backup: backrest.sh backup <volumeName>
zfs snapshot command to restore: backrest.sh restore <volumeName> <ccyymmdd>
general rsync:
to image copy: backrest.sh sync <volumeName> [-f]
to restore: backrest.sh syncrestore <volumeName> [-f]

Code:

#!/bin/bash
#
# backup/restore syncronization
#
# https://forums.freenas.org/index.php?threads/zfs-send-to-external-backup-drive.17850/

#echo "Starting"

req="${1,,}" #convert to lower case request (backup or restore)
drive="${2^^}" #convert to upper case target/destination drive
dt="${3^^}" #convert to upper case date of a backup point to restore from
force="${3,,}" #convert flag setting to lower case
u="" #default force setting flag (rsync)

if [ "${force}" == "" ];
then
   force=""
fi

back="BACKUP" #default name of backup volume

status=0

# run if user hits control-c
function control_c() {
  echo -en "\n*** Ouch! Exiting ***\n"
  exit $?
}

# trap keyboard interrupt (control-c)
trap control_c SIGINT

#pause prompt
function pause(){
  read -p "$*"
}



#command line syntax error message
function error() {
   echo "command line error ${@}"
   echo " to backup zfs with snapshots:"
   echo "  /mnt/<drive>/backrest.sh backup <drive>"
   echo " or to restore zfs with snapshots:"
   echo "  /mnt/<drive>/backrest.sh restore <drive> <ccyymmdd> or sync <drive>"
   echo " or to do rsync do ..."
   echo "  backup drive to /mnt/${back}:"
   echo "  /mnt/<drive>/backrest.sh sync <drive>"
   echo "  restore from /mnt/${back}:"
   echo "  /mnt/<drive>/backrest.sh syncrestore <drive>"
   return
}

#check if the drive is mounted or even exists on the system
function is_mounted() {
   local FS=$(zfs get -H mounted "${@}")
   FS_REGEX="^${@}\s+mounted\s+yes\s+-"

   #check for errors
   if [ "${FS}" == "" ]; then
     status=$? #status from last function call
     echo "${@} not found/mounted"
     return 0 #evaluates to false - error condition
   fi
   if [[ $FS =~ $FS_REGEX ]]; then
     status=$? #status from last function call
     echo "${@} not mounted"
     return 0 #evaluates to false - error condition
   fi
  
   #drive checked out ok
   echo "${@} mounted"
   return 1 #evaluates to true - success
}

#wrapper to ensure BACKUP drive and nominated target/destination drive is mounted/exists
function checkMountPoints() {

   #checks if the backup/restore drive exists
  
   #check source drive
   is_mounted ${@}
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     echo "${@} checked out ok"
   else  
     return 0 #evalutes to false error condition
   fi
  
   #check backup drive
   is_mounted ${back}
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     echo "${back} checked out ok"
     return 1
   else  
     return 0 #evaluates to false - error condition
   fi
}


#rsync across the whole source volume to backup destination volume
function RyncFromSourceToDetachedMediaBackup() {
   #checkMountPoints ${drive}
   checkMountPoints ${1}
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     echo "rsync -pogavt${2} /mnt/${1} /mnt/${back}"
     rsync -pogavt${2} /mnt/${1} /mnt/${back}
     status=$? #status from last function call
     if [ $status == 0 ];
     then
       echo "rsync from /mnt/${1} to /mnt/${back}/${1} successful"
     else
       echo "rsync from /mnt/${1} to /mnt/${back}/${1} unsuccessful"
     fi
   else
     status=1
   fi
   return $status
}

#rsync across the whole backup volume to destination source volumne
function RsyncFromBackupToSourceRestore() {
   #checkMountPoints ${drive}
   checkMountPoints ${1}
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     echo "rsync -pogavt${2} /mnt/${back}/${1}/ /mnt/${1}"
     rsync -pogavt${2} /mnt/${back}/${1}/ /mnt/${1}
     status=$? #status from last function call
     if [ $status == 0 ];
     then
       echo "rsync from /mnt/${back}/${1}/ to /mnt/${1} successful"
     else
       echo "rsync from /mnt/${back}/${1}/ to /mnt/${1} unsuccessful"
     fi
   else
     status=1
   fi
   return $status
}


function takesnapshot() {
   #echo "start takesnapshot() of ${@} as ${@}-$(date +%Y%m%d)"

   zfs snapshot -r ${@}@${@}-$(date +%Y%m%d)
   status=$? #status from last function call
   if [ $status == 0 ];
   then
     status=1
   else
     status=0
   fi  
   #echo "chk outcome takesnapshot() $status"

   return $status
}

function renamesnapshot() {
   #echo "start renamesnapshot() ${@}@${@}-${date} to ${@}@${@}-$(date +%Y%m%d)-restored_$(date +%Y%m%d)"

   zfs rename -r ${@}@${@}-${date} ${@}@${@}-$(date +%Y%m%d)-restored_$(date +%Y%m%d)
   status=$? #status from last function call
   if [ $status == 0 ];
   then
     status=1
   else
     status=0
   fi  

   #echo "chk outcome renamesnapshot() $status"

   return $status
}

function deletesnapshot() {
   #echo "start deletesnapshot() ${@}@${@}-$(date +%Y%m%d)"
   zfs destroy -r ${@}@${@}-$(date +%Y%m%d)
   status=$? #status from last function call
   if [ $status == 0 ];
   then
     status=1
   else
     status=0
   fi  

   #echo "chk outcome deletesnapshot() $status"

   return $status
}

function sendsnapshot () {
   #echo "start sendsnapshot() ${@}@${@}-$(date +%Y%m%d) to ${back}/${@}-$(date +%Y%m%d)"

   zfs send -R ${@}@${@}-$(date +%Y%m%d) | zfs receive -vF ${back}/${@}-$(date +%Y%m%d)
   status=$? #status from last function call
   if [ $status == 0 ];
   then
     status=1
   else
     status=0
   fi  

   #echo "chk outcome sendsnapshot() $status"

   return $status
}  

#takes the backup of nominated drive, sending it to the BACKUP drive
function doBackup() {
   checkMountPoints ${drive}
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     echo "about to take snapshot of $drive"
     takesnapshot $drive
     status=$? #status from last function call
     if [ $status == 1 ];
     then
       echo "about to send snapshot of $drive to $back"
       sendsnapshot ${drive}
       status=$? #status from last function call
       if [ $status == 1 ];
       then
         echo "about to take snapshot of ${back}"
         takesnapshot ${back};
         status=$? #status from last function call
         if [ $status == 1 ];
         then
           echo "about to delete snapshot of ${drive}"
           deletesnapshot ${drive}
           status=$? #status from last function call
           if [ $status == 1 ];
           then
             #echo "about to delete snapshot of ${back}"
             #deletesnapshot ${back}
             #status=$? #status from last function call
             #if [ $status == 1 ];
             #then
               echo "Successfully backed up ${drive} onto ${back}"
               return
             #fi  
           fi  
         fi  
       fi  
     fi  
   fi
   status=$? #status from last function call
   error $status
   echo "Unsuccessful backup of ${@}"
   return
}

function validateDate() {
   # Script expecting a Date parameter in YYYYMMDD format as input
   echo ${dt} | grep '^[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]$'
   status=$? #status from last function call
   if [ $status -eq 0 ]; #valid date
   then
     status=0 #do nothing statement
   else
     #since previous test failed I will try to reverse the argument assuming a reverse input
     echo ${drive} | grep '^[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]$'
     status=$? #status from last function call
     if [ $status -eq 0 ]; #valid date
     then
       # parameters in reversed argument order - switch order of argument input
       tmpdrive="${dt}"
       dt="${drive}"
       drive="${tmpdrive}"
     fi  
   fi
  
   return $status #0=success, otherwise error
}

#some checks post restore a zfs volume and all datasets
function doRestore() {
   echo "Start Restore..."
   validateDate
   status=$? #status from last function call
   #validated date params
   if [ $status == 0 ];
   then
     checkMountPoints ${drive} #check if nominated target/destination drive/volume is mounted/available
     status=$? #status from last function call
     if [ $status == 1 ]; #target/destination drives(volumes) accessible
     then
       echo "Initiate restore of $drive from ${back}@${drive}-${dt}"
       execRestore ${drive} ${dt} #do the restore of drive for date (assuming it exists for set date)
       status=$? #status from last function call
       if [ $status == 0 ]; #success
       then
         echo "Successfully restored $drive from ${back}@${drive}-${dt}"
       else  
         echo "Unsuccessful restore of $drive from ${back}"
         error $status
       fi
     else  
       echo "${drive} or ${back} not mounted"
       error $status
     fi  
   else
     echo "${dt} is not a valid date"
     error $status
   fi
  
   return
}

function destroyDataset () {
   echo "about to delete dataset ${@}"
  
   if [ "${@}" == "" ];
   then
     echo "empty dataset passed"
     status=1
   else  
     #zfs destroy -R $back/$drive-$dt/${@}
     zfs destroy -R ${@}
     status=$? #status from last function call
     if [ $status == 0 ]; #successful rename
     then
       #echo "Successfully deleted dataset $back/$drive-$dt/${@}"
       echo "Successfully deleted dataset ${@}"
     else
       #echo "Faailed to delete dataset $back/$drive-$dt/${@}"
       echo "Failed to delete dataset ${@}"
     fi  
   fi

   return $status
}

function jailCleanExists () {
   echo "check if ${@} exists"

   zfs list ${@}
   status=$? #status from last function call
   if [ $status == 0 ]; #successful rename
   then
     status=1
   else
     status=0
   fi

   echo "status from call $status"
   return $status
}  

function removeJailDependencies () {

   exception=0
  
   jailCleanExists ${@}/jails/sickbeard_2
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     destroyDataset ${@}/jails/sickbeard_2
     status=$? #status from last function call
     if [ $status == 0 ];
     then
       echo "successfully removed ${@}/jails/jails/sickbeard_2"
     else
       echo "Warning: ${@}/jails/jails/sickbeard_2 not found"
       exception=1
     fi
   fi  

   jailCleanExists ${@}/jails/plexmediaserver_1
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     destroyDataset ${@}/jails/plexmediaserver_1
     status=$? #status from last function call
     if [ $status == 0 ];
     then
       echo "successfully removed ${@}/jails/plexmediaserver_1"
     else
       echo "Warning: ${@}/jails/plexmediaserver_1 not found"
       exception=1
     fi
   fi
  
   jailCleanExists ${@}/jails/firefly_1
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     destroyDataset ${@}/jails/firefly_1
     status=$? #status from last function call
     if [ $status == 0 ];
     then
       echo "successfully removed ${@}/jails/firefly_1"
     else
       echo "Warning: ${@}/jails/firefly_1 not found"
       exception=1
     fi  
   fi
  
   jailCleanExists ${@}/jails/.warden-template-pluginjail@clean
   status=$? #status from last function call
   if [ $status == 1 ];
   then
     destroyDataset ${@}/jails/.warden-template-pluginjail@clean
     status=$? #status from last function call
     if [ $status == 0 ];
     then
       echo "successfully removed ${@}/jails/.warden-template-pluginjail@clean"
     else
       echo "Warning: ${@}/jails/.warden-template-pluginjail@clean not found"
       exception=1
     fi  
   fi

   status=$exception
   return $status

}

#restore a zfs volume and all datasets
function execRestore(){
   #echo "Restoring ${drive} for ${dt} from ${back}/${drive}-${dt}..."
   #test
   echo "Restoring ${back}@${drive}-${dt} to ${drive}..."
  
   #assume the backup for set date exists - the backup name is based on previous backup whith inherited source volume name
   zfs send -R ${back}@${drive}-${dt} | zfs receive -vF ${drive}/
   status=$? #status from last function call
   if [ $status == 0 ]; #worked
   then
     #take a new snapshot from restored data
     takesnapshot ${drive}
     status=$? #status from last function call
     if [ $status == 0 ]; #snapshot successful
     then
       renamesnapshot ${drive} #rename previous snaphot on the destination volume to prevent accidental restore since the backup release superseeds this snapshot
       status=$? #status from last function call
       if [ $status == 0 ]; #successful rename
       then
         echo "about to delete snapshot of ${back}"
         deletesnapshot ${back} #remove the snapshot from the backup drive since it's now been restored (not sure this is the correct approach)
         status=$? #status from last function call
         if [ $status == 0 ]; #successful rename
         then
           echo "about to delete dataset ${back}@${drive}-${dt}"
           destroyDataset ${back}@${drive}-${dt}
           status=$? #status from last function call
           if [ $status == 0 ]; #successful rename
           then
             echo "check if ${drive}/jails/.warden-template-pluginjail@clean exists"
             jailCleanExists ${drive}/jails/.warden-template-pluginjail@clean
             status=$? #status from last function call
             if [ $status == 1 ]; #@clean exists
             then
               echo "remove dependencies for ${back}/${drive}-${dt}/jails/.warden-template-pluginjail@clean"
               removeJailDependencies ${back}/${drive}-${dt}
               status=$? #status from last function call
               if [ $status == 0 ]; #@cleaned
               then
                 echo "Dependencies successfully removed for ${back}/${drive}-${dt}/jails/.warden-template-pluginjail@clean"
                 status=1
               else
                 echo "Exception thrown while cleaning up dependant datasets for ${back}/${drive}-${dt}/jails/.warden-template-pluginjail@clean"
                 status=0
               fi
             else
               status=1
             fi
           else
             status=0
           fi
         fi
       fi  
     fi
   fi

   return $status   #status from last call returned (0=error, 1=success)
}


#main logic (backup or restore)
if [ "${drive}" == "" ];
then
   error
else
   if [ "${req}" == "backup" ];
   then
     doBackup $drive
   else
     if [ "${req}" == "restore" ];
     then
       doRestore $drive $dt
     else
       if [ "${req}" == "sync" ];
       then
         if [ "${force}" == "-f" ];
         then
           u="u"
         fi  
         echo "WARNING: You are about to rsync /mnt/${drive} onto /mnt/${back}/${drive}"
         pause "Press [Enter]key  to continue or CTRL+C to cancel"
         RyncFromSourceToDetachedMediaBackup ${drive} ${u}
         status=$? #status from last function call
         if [ $status == 0 ]; #successful rename
         then
           echo "ok"
         else
           error
         fi
       else
         if [ "${req}" == "syncrestore" ];
         then
           if [ "${force}" == "-f" ];
           then
             u="u"
           fi  
           echo "WARNING: You are about to rsync /mnt/${back}/${drive}/ onto /mnt/${drive}"
           pause "Press [Enter]key  to continue or CTRL+C to cancel"
           RsyncFromBackupToSourceRestore ${drive} ${u}
           status=$? #status from last function call
           if [ $status == 0 ]; #successful rename
           then
             echo "ok"
           else
             error
           fi
         else
           error
         fi  
       fi  
     fi  
   fi
fi
exit 0
#

 

[ghostwolf59]

Have you looked at ZFS replication? It should do everything you're looking for.

from what I can make out of the zfs replication and the initial pre-reqs of first creating a scheduled zfs snapshot repeat task the zfs replication expects that you have your backup volume mounted at all time - Adhoc replication don't appear to be supported. For a single mounted drive/volume that's fine I guess, but when you have 4-8 or more physical drives/volumes where the total volume capacity exceeds what you can store on a backup drive then I see little option than mounting the backup when you want to take your backup (for a specific volume) i.e run a script to backup/restore, similar to how I used to manage UFS backups via script.

So replication might do the job (technically) but when you don't have the backup volume mounted then this shits itself.

So, going back to how I used to do it with UFS, I rather take a manual backup when I want - AND if things goes down the gurgle, fall back on a manual restore from where I know the data last were backed up.

I've added my backup/restore script (as it currently stands) to my last post - the backup seem to work, but there seem to be some issues with recovering from a previous backup. - any ideas ????