Backup to external eSATA

[jwall]

I'm looking to semi-automate the backup of my new FreeNAS-8.3.1-RELEASE-p2-x64 server so that I can permanently connect an external eSATA dock and when I dock an HD every couple weeks I can initiate a script that will mount the drive and sync my volume. Does anyone else do this, or are there reasons not to handle backups like this? If not, how do others backup their data? Building a second FreeNAS server is not an option for me. I know I could use a utility like RoboCopy or Rsync on Windows to sync over the network to an external device attached to a Windows machine but I thought I'd see if the first method is being used by anyone.

 

[jgreco]

Yes. The words "external" and "eSATA" are distractions except to clarify that you're not trying this via the dreaded "USB".

My opinion is that FreeNAS is poorly suited to doing this via the built-in GUI, but you can readily do this via the command line without too much fuss. The GUI tends to populate stuff into the FreeNAS configuration database and the appearance/disappearance of transient devices is not something that has been carefully analyzed, tested, or understood (at least here in the forums) in that context. However, I'm comfortable saying that you should be able to appear/disappear non-FreeNAS-managed devices all day long without trouble.

With that in mind, things to consider:

1) It is probably better if your SATA port supports hot-swap,

2) If you are using multiple hard drives for the purpose, it is probably better to make sure you have physically labeled hard drives and some sort of rotation strategy (and you do want at least two, because what happens if something bad happens while doing a backup and your NAS and the backup disk both go byebye),

3) Make sure that you carefully consider the benefits/downsides of any given filesystem. I'm kind of thinking that UFS/FFS will tend to be easier for a transient backup filesystem and supporting scripting. It can be mounted async if you have lots of small files (and a UPS!) that would otherwise slow things down.

 

[jwall]

Ok this sounds promising. Given I have zero experience with scripting on unix, are you aware of any resources or quick start guides that would be useful in getting me started on this? I realize this is probably not something I can throw together in an hour but anything to guide me along would be helpful. With FreeNAS in particular, searching the net it seems there's more info on mounting a drive via the GUI, not so much on using the cmdline.

My ZFS pool is 4TB, once all my data is copied to it it'll use consume 2TB so in the interest of future expansion I think I'll need 2 x 2TB (they seem to be cheaper than 3TB drives currently) drives for backup, plus 2 more for backup rotation as you suggest. Since all my data won't fit onto one if these backup drives I'll need to split things up, will it be possible to detect which drive, by serial number or some other method, is docked and then copy the proper data (datasets) to it? I am leaning towards a dual bay dock, which I believe would use a SATA hub with 1 physical eSATA connection to the motherboard, if that changes things any.

 

[jgreco]

With 3TB drives being in the $110 price range and 4TB in the $150 range, you'd be SO much better off avoiding the complications and annoyances of trying to make two drives work together as part of a single backup volume. There are complications with SATA port multipliers as many of them are gimpy or dysfunctional, and you'd be better off spending a little more on larger drives than on a more complicated dock and more scripting work.

Your scripting might actually end up being fairly simple, but it will also be somewhat dependent on your hardware specifics. If we're talking a script that you run manually, it is maybe vaguely something like:

1) fsck the filesystem
2) mount the filesystem
3) rsync the filesystem
4) umount the filesystem
5) alert completion (email etc).

 

[lpittman]

Here are two scripts that I wrote for a CentOS server I was playing around with a few years ago. These will not work directly on a FreeNAS box, but they may help as a frame of reference for you or simply to give you some ideas.

Code:

#!/bin/sh

COMMAND="rsync -aS --delete /raid1/ /mnt/usbdock"

#check if rsync is already running or not

if ! ps aux | grep -v grep | grep "$COMMAND" > /dev/null
then

        if [ -e /dev/sde1 ]
        then
                echo "Backup Drive Detected"
                echo "Attempting to Mount Backup Drive"

                if mount /dev/sde1 /mnt/usbdock > /dev/null 2>&1
                then
                        echo "Backup Drive Mounted"
                        sleep 10

                        echo "---"
                        $COMMAND
                        echo "---"

                        sleep 10

                        echo "Backup Complete - Unmounting Drive"
                        if umount /mnt/usbdock > /dev/null 2>&1
                        then
                                echo "Backup Drive Unmounted"
                        else
                                echo "Error: Could Not Unmount Backup Drive"
                        fi

                else
                        echo "Error: Could Not Mount Backup Drive"

                fi
        fi
fi

Code:

#!/bin/sh
#
# This script checks for a file system on /dev/sde as /dev/sde1
# and will mount the drive and create a mirror of /raid1 to /mnt/usbdock.
# It will then create a file containing the date the last backup was performed
# and only perform another backup if it was over 7 days ago
#

COMMAND="rsync -aS --delete /raid1/ /mnt/usbdock"                       # backup command
LASTBCK=/mnt/usbdock/last-backup.txt                                    # file to store last backup date
TODAYSDATE=`date +%F`                                                   # get todays date in YYYY-MM-DD format

if [ -e /dev/sde1 ]; then

        echo "Backup Drive Detected"
        echo "Attempting to Mount Backup Drive"

        if mount /dev/sde1 /mnt/usbdock > /dev/null 2>&1                # Attempts to mount backup drive
        then
                echo "Backup Drive Mounted"
                sleep 10

                if [ ! -e $LASTBCK ]; then                              # Check if LASTFULLFILE exists
                        touch $LASTBCK                                  # It doesn't - create it
                        echo 1990-01-01 > $LASTBCK                      # Give it a super old date
                fi

                LASTBACKUPDATE=`cat $LASTBCK`                           #
                DATE1=`date +%s -d $LASTBACKUPDATE`                     # Calculate number of seconds since last backup
                DATE2=`date +%s -d $TODAYSDATE`                         #
                ((SECONDS_SINCE_BACKUP=DATE2-DATE1))                    #

                if [ $SECONDS_SINCE_BACKUP -ge 604800 ]; then

                        $COMMAND                                        # Execute backup command

                        sleep 10

                        echo "Backup Complete"

                        rm -f $LASTBCK                                  #
                        touch $LASTBCK                                  # Update last backup date file
                        echo $TODAYSDATE > $LASTBCK                     #

                else
                        echo "Backup Not Performed - Backup Performed Less Than 7 Days Ago"
                fi

                if umount /mnt/usbdock > /dev/null 2>&1
                then
                        echo "Backup Drive Unmounted"
                else
                        echo "Error: Could Not Unmount Backup Drive"
                fi
        fi
fi

 

[jwall]

Thanks for flowchart jgraco, and for the sample code lpittman. I have a single bay dock on the way when it arrives I'll get started on this, I'm certain I will have more questions too.