Backup FreeNAS to USB Drive

Status
Not open for further replies.

Nik Ansell

Dabbler
Joined
Oct 28, 2014
Messages
11
Hello All,

Although this is a contentious area, I have been using this system for a few months now and think other users may find it useful. This is a simple shell script which can be scheduled to run every night.

Anyone with basic shell script knowledge should be able to tweak it as required, perhaps even replacing the
pushover notification with a sendmail command if you prefer.

Please feel free to provide me feedback or mods, I am always happy to learn new things :)

What the script does:
# Creates a mount point folder if required
# Mounts the USB drive if not already mounted
# Backs up files to USB drive
# Backs up freenas DB and plug-in config files
# Maintains 7 days of log files
# Sends a pushover message when complete

My Setup:
# I have two portable USB drives, both formatted using UFS
# The backup script runs every day
# I swap over HDDs every month or so, to allow for HDD failure etc - Run 'umount /mnt/usb1' before unplugging the USB HDD, or create a cron job with no schedule to run the command from the UI

To use:
# Setup the USB HDD(s)
# Copy the code below into a file called freenas_backup.sh
# Modify variables as required
# Create directories as defined in variables
# Setup a pushover account and register app on pushover site - Optional
# Modify calls to copy_file as required
# Modify calls to backup_files as required
# add --delete to the rsync call on backup_files function (delete extraneous files from destination dirs)

# -----------------------------------
# Setup disk using the ufs filesystem
# -----------------------------------

This is to prepare a USB HDD for use by freenas_backup.sh
It will destroy all partitions on the USB HDD
Then create a single ufs partition and format it for use

!!====================================================!!
# Make sure the disk is correct before doing this!!!!!
!!====================================================!!

1. Plugin your USB drive, viewing the console or running 'dmesg' should give you an idea of the disk name
These instructions assume that USB drive is /dev/da0

2. If you are still not sure you can check to see all the attached disks and partitions by running 'gpart show'

# Delete previous partitions etc on the USB drive
gpart delete -i 1 da0
gpart destroy da0
gpart show da0 (This should confirm no partitions exist on the USB drive)

# Create new ufs partition on the whole disk
gpart create -s gpt da0
gpart show da0 (This should show the new GPT partition)
gpart add -t freebsd-ufs da0
newfs /dev/da0s1 or newfs /dev/da0p1 (whichever is appropriate)

# Start of shell script, copy from here........

#! /bin/bash

# ===========================================
#
# freenas_backup.sh
# Nik Ansell
#
# Version History:
#
# 0.1 Early draft
# 0.2 12-Apr-15 Added backup_files and pushover functions
# 0.3 14-Apr-15 Added mount checks
# 0.4 27-Apr-15 Added cURL checks
# 0.5 05-May-15 Added use of logs directory
# 0.6 12-Jun-15 Added number of backed up files
# 0.7 07-Sep-15 Added backup of FreeNAS config file
# 0.8 12-Sep-15 Added jail config backup
# 0.9 12-Sep-15 Added copy_file function
# 0.91 15-Sep-15 Changed copy_file output to allow inclusion in file count routine
# 0.92 16-Sep-15 Change CURL command to use variables
# ===========================================

# Useful Commands:

# If you get the error below when trying to mount, run fsck -t ufs USB_PATH
# mount: /dev/da1p1: R/W mount of /mnt/usb1 denied. Filesystem is not clean - run fsck.: Operation not permitted

# Check a UFS filesystem for errors
# fsck -t ufs /dev/da0p1
# or possibly (Depending on where the USB drive is mounted)
# fsck -t ufs /dev/da1p1
#
# Should see something like below
# [root@freenas] /mnt/vol1/Backup# fsck -t ufs /dev/da1p1
# ** /dev/da1p1
# ** Last Mounted on /mnt/usb1
# ** Phase 1 - Check Blocks and Sizes
# ** Phase 2 - Check Pathnames
# ** Phase 3 - Check Connectivity
# ** Phase 4 - Check Reference Counts
# ** Phase 5 - Check Cyl groups
# 96781 files, 140911666 used, 95610172 free (5524 frags, 11950581 blocks, 0.0% fragmentation)
#
# ***** FILE SYSTEM MARKED CLEAN *****

# Initialise Variables
SRC_ROOT=/mnt/vol1
SRC_PATH_AUDIO=$SRC_ROOT/Audio
SRC_PATH_APERTURE=$SRC_ROOT/Images/ApertureExport
SRC_PATH_EBOOKS=$SRC_ROOT/eBooks
SRC_PATH_USERDATA=$SRC_ROOT/Users
SRC_PATH_HOMEMOVIES=$SRC_ROOT/Video/HomeMovies
SRC_PATH_MUSIC=$SRC_ROOT/Music
SRC_PATH_BACKUP=$SRC_ROOT/Backup
DEST_ROOT=/mnt/usb1
JAIL_ROOT=/mnt/vol1/jails_ds
CFG_BACKUP_ROOT=$DEST_ROOT/Backup/config_backup
USB_PATH=/dev/da1p1

PUSH_MESSAGE="Backup Job Finished"
PUSH_USER="YOUR_PUSHOVER_USERNAME"
PUSH_TOKEN="YOUR_PUSHOVER_TOKEN"

DATE_STR=$(date "+%Y-%m-%d")
LOGFILE=${SRC_ROOT}/Backup/logs/${DATE_STR}_backup.log

function output {
NOW=$(date "+%F %T")
echo $NOW $1
echo $NOW $1 >> $LOGFILE
}

function backup_files {
output "Backing up $1 to $DEST_ROOT"
rsync -arh -i -u --update -lK --log-file=$LOGFILE $1 $DEST_ROOT

# Check for error
if [ $? -eq 0 ]; then
output "Copy OK"
else
PUSH_MESSAGE="Error $? backing up $1"
output "$PUSH_MESSAGE"
fi
}

function copy_file {
output "Copying $1 to $2"
cp $1 $2

# Check for error
if [ $? -eq 0 ]; then
output "Copy OK >f+++++++++"
else
PUSH_MESSAGE="Error $? backing up $1"
output "$PUSH_MESSAGE"
fi

}

if [ -e $LOGFILE ];
then
LOGFILE=${SRC_ROOT}/Backup/logs/${DATE_STR}_backup_2.log
fi

output "Using Logfile: $LOGFILE"

# Delete previous log files
output "Deleting log files older than 7 days from $SRC_ROOT/Backup/logs"
find $SRC_ROOT/Backup/logs* -mtime +7 -exec rm {} \;

# Some code here to confirm the correct USB mount point
# Maybe including the mounting of the correct USB Drive

# Create Destination Root ($DEST_ROOT) if doesn't exist
# Error codes:
# 0 OK
# 1 File exists
output "Creating destination root folder ($DEST_ROOT) if doesnt exist"
mkdir -p $DEST_ROOT

if [ $? -gt 1 ] ; then
PUSH_MESSAGE="Error $? running mkdir -p $DEST_ROOT"
output "$PUSH_MESSAGE"
exit
fi

# Check if drive mounted
output "Checking if $USB_PATH is mounted"
mount | grep usb >$SRC_ROOT/Backup/m.usb

if grep -cq "usb" $SRC_ROOT/Backup/m.usb; then
output "Drive already mounted to $DEST_ROOT"
else

# Mount the USB Drive
# Error codes:
# 0 OK
# 1 Unknown file system / File system is not clean
# 64 No such file or directory
output "Mounting USB Drive ($USB_PATH) to $DEST_ROOT"
mount $USB_PATH $DEST_ROOT

if [ $? -gt 0 ] ; then
PUSH_MESSAGE="Error $? running mount $USB_PATH $DEST_ROOT"
output "$PUSH_MESSAGE"
exit
fi

fi

# Backup config files
output "Backing up config files"
copy_file /data/freenas-v1.db $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/couchpotato_1/var/db/couchpotato/settings.conf $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/sabnzbd_1/var/db/sabnzbd/sabnzbd.ini $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/sickrage_1/var/db/sickrage/config.ini $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/transmission_2/usr/pbi/transmission-amd64/etc/transmission/home/settings.json $CFG_BACKUP_ROOT/

# Backup files
backup_files $SRC_PATH_AUDIO
backup_files $SRC_PATH_APERTURE
backup_files $SRC_PATH_EBOOKS
backup_files $SRC_PATH_USERDATA
backup_files $SRC_PATH_HOMEMOVIES
backup_files $SRC_PATH_MUSIC
backup_files $SRC_PATH_BACKUP

# Check how many files were copied
num=$(cat $LOGFILE | grep "++" | wc -l)
# Remove leading spaces from wc output
NUM_FILES=$(echo $num)

PUSH_MESSAGE="Backup Job Completed. Files backed up:$NUM_FILES"
output "$PUSH_MESSAGE"

# Send Pushover message
output "Sending pushover notification using cURL"
curl -v --data "user=$PUSH_USER&token=$PUSH_TOKEN&title=FreeNasBackup&message=$PUSH_MESSAGE&type=notification" https://api.pushover.net/1/messages.json >>$LOGFILE 2>&1

# Add newline to log file, because the cURL output doesn't have one at the end
output ""

# Check for error
if [ $? -eq 0 ]; then
output "Pushover notification sent OK"
else
output "Error $? sending pushover notification"
fi

exit;
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
This could be helpful for others, thanks for sharing. Could you put the script in code tags? It can prevent malformatting caused by the forum software, unintended line breaks, etc.
 

Nik Ansell

Dabbler
Joined
Oct 28, 2014
Messages
11
Good point, here you go.

Code:
#! /bin/bash

# ===========================================
#
#    freenas_backup.sh
#    Nik Ansell
#
#    Creates a mount point folder if required
#    Mounts the USB drive if not already mounted
#    Backs up files to USB drive
#    Backs up freenas DB and plug-in config files
#    Maintains 7 days of log files
#    Sends a pushover message when complete
#
#    To use:
#    Modify variables as required
#    Create directories as defined in variables
#    Setup a pushover account and register app on pushover site - Optional
#    Modify calls to copy_file as required
#    Modify calls to backup_files as required
#   
#    Version History:
#
#    0.1        Early draft
#    0.2        12-Apr-15    Added backup_files and pushover functions
#    0.3        14-Apr-15    Added mount checks
#    0.4        27-Apr-15    Added cURL checks
#    0.5        05-May-15    Added use of logs directory
#    0.6        12-Jun-15    Added number of backed up files
#    0.7        07-Sep-15    Added backup of FreeNAS config file
#    0.8        12-Sep-15    Added jail config backup
#    0.9        12-Sep-15    Added copy_file function
#    0.91    15-Sep-15    Changed copy_file output to allow inclusion in file count routine
#    0.92    16-Sep-15    Change CURL command to use variables
# ===========================================

#    Useful Commands:

#    If you get the error below when trying to mount, run fsck -t ufs USB_PATH
#    mount: /dev/da1p1: R/W mount of /mnt/usb1 denied. Filesystem is not clean - run fsck.: Operation not permitted

#    Check a UFS filesystem for errors
#    fsck -t ufs /dev/da0p1
#    or possibly (Depending on where the USB drive is mounted)
#    fsck -t ufs /dev/da1p1
#
#    Should see something like below
#    [root@freenas] /mnt/vol1/Backup# fsck -t ufs /dev/da1p1
#    ** /dev/da1p1
#    ** Last Mounted on /mnt/usb1
#    ** Phase 1 - Check Blocks and Sizes
#    ** Phase 2 - Check Pathnames
#    ** Phase 3 - Check Connectivity
#    ** Phase 4 - Check Reference Counts
#    ** Phase 5 - Check Cyl groups
#    96781 files, 140911666 used, 95610172 free (5524 frags, 11950581 blocks, 0.0% fragmentation)
#   
#    ***** FILE SYSTEM MARKED CLEAN *****

# Initialise Variables
SRC_ROOT=/mnt/vol1
SRC_PATH_AUDIO=$SRC_ROOT/Audio
SRC_PATH_APERTURE=$SRC_ROOT/Images/ApertureExport
SRC_PATH_EBOOKS=$SRC_ROOT/eBooks
SRC_PATH_USERDATA=$SRC_ROOT/Users
SRC_PATH_HOMEMOVIES=$SRC_ROOT/Video/HomeMovies
SRC_PATH_MUSIC=$SRC_ROOT/Music
SRC_PATH_BACKUP=$SRC_ROOT/Backup
DEST_ROOT=/mnt/usb1
JAIL_ROOT=/mnt/vol1/jails_ds
CFG_BACKUP_ROOT=$DEST_ROOT/Backup/config_backup
USB_PATH=/dev/da1p1

PUSH_MESSAGE="Backup Job Finished"
PUSH_USER="YOUR_PUSHOVER_USERNAME"
PUSH_TOKEN="YOUR_PUSHOVER_TOKEN"

DATE_STR=$(date "+%Y-%m-%d")
LOGFILE=${SRC_ROOT}/Backup/logs/${DATE_STR}_backup.log

function output {
    NOW=$(date "+%F %T")
    echo $NOW $1
    echo $NOW $1 >> $LOGFILE
}

function backup_files {
    output "Backing up $1 to $DEST_ROOT"
    rsync -arh -i -u --update -lK --log-file=$LOGFILE $1 $DEST_ROOT
   
    #    Check for error
    if [ $? -eq 0 ]; then
        output "Copy OK"
        else
        PUSH_MESSAGE="Error $? backing up $1"
        output "$PUSH_MESSAGE"
    fi
}

function copy_file {
    output "Copying $1 to $2"
    cp $1 $2
   
    #    Check for error
    if [ $? -eq 0 ]; then
        output "Copy OK  >f+++++++++"
        else
        PUSH_MESSAGE="Error $? backing up $1"
        output "$PUSH_MESSAGE"
    fi
   
}

if [ -e $LOGFILE ];
then
    LOGFILE=${SRC_ROOT}/Backup/logs/${DATE_STR}_backup_2.log
fi

output "Using Logfile: $LOGFILE"

#    Delete previous log files
output "Deleting log files older than 7 days from $SRC_ROOT/Backup/logs"
find $SRC_ROOT/Backup/logs* -mtime +7 -exec rm {} \;

#    Some code here to confirm the correct USB mount point
#    Maybe including the mounting of the correct USB Drive

#    Create Destination Root ($DEST_ROOT) if doesn't exist
#    Error codes:
#    0    OK
#    1    File exists
output    "Creating destination root folder ($DEST_ROOT) if doesnt exist"
mkdir -p $DEST_ROOT

if [ $? -gt 1 ] ; then
    PUSH_MESSAGE="Error $? running mkdir -p $DEST_ROOT"
    output "$PUSH_MESSAGE"
    exit
fi

#    Check if drive mounted
output "Checking if $USB_PATH is mounted"
mount | grep usb >$SRC_ROOT/Backup/m.usb

if grep -cq "usb" $SRC_ROOT/Backup/m.usb; then
    output "Drive already mounted to $DEST_ROOT"
else

    #    Mount the USB Drive
    #    Error codes:
    #    0    OK
    #    1    Unknown file system / File system is not clean
    #    64    No such file or directory
    output    "Mounting USB Drive ($USB_PATH) to $DEST_ROOT"
    mount $USB_PATH $DEST_ROOT

    if [ $? -gt 0 ] ; then
        PUSH_MESSAGE="Error $? running mount $USB_PATH $DEST_ROOT"
        output "$PUSH_MESSAGE"
        exit
    fi
   
fi

#    Backup config files
output "Backing up config files"
copy_file /data/freenas-v1.db $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/couchpotato_1/var/db/couchpotato/settings.conf $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/sabnzbd_1/var/db/sabnzbd/sabnzbd.ini $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/sickrage_1/var/db/sickrage/config.ini $CFG_BACKUP_ROOT/
copy_file $JAIL_ROOT/transmission_2/usr/pbi/transmission-amd64/etc/transmission/home/settings.json $CFG_BACKUP_ROOT/

#    Backup files
backup_files $SRC_PATH_AUDIO
backup_files $SRC_PATH_APERTURE
backup_files $SRC_PATH_EBOOKS
backup_files $SRC_PATH_USERDATA
backup_files $SRC_PATH_HOMEMOVIES
backup_files $SRC_PATH_MUSIC
backup_files $SRC_PATH_BACKUP

#    Check how many files were copied
num=$(cat $LOGFILE | grep "++" | wc -l)
#    Remove leading spaces from wc output
NUM_FILES=$(echo $num)

PUSH_MESSAGE="Backup Job Completed. Files backed up:$NUM_FILES"
output "$PUSH_MESSAGE"

#    Send Pushover message
output "Sending pushover notification using cURL"
curl -v --data "user=$PUSH_USER&token=$PUSH_TOKEN&title=FreeNasBackup&message=$PUSH_MESSAGE&type=notification" https://api.pushover.net/1/messages.json >>$LOGFILE 2>&1

#    Add newline to log file, because the cURL output doesn't have one at the end
output ""

#    Check for error
if [ $? -eq 0 ]; then
    output "Pushover notification sent OK"
else
    output "Error $? sending pushover notification"
fi

exit;
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Moving to the how-to guides/scripts as this isn't useful for me, but there is an obvious demand.

And thanks for adding the disclaimer that it's not recommended. ;)
 
Status
Not open for further replies.
Top