Resource icon

A script to run rclone on the FreeNAS server to backup NAS data to Backblaze B2 cloud storage. 1.0

Hazimil

Contributor
Joined
May 26, 2014
Messages
172
Hazimil submitted a new resource:

A script to run rclone on the FreeNAS server to backup NAS data to Backblaze B2 cloud storage. - rclone script to backup to Backblaze B2

A script to run rclone on the FreeNAS server to backup NAS data to Backblaze B2 cloud storage. Can easily be adapted for other cloud providers.

Script was adapted from the one posted in the FreeNAS community forums by Martin Aspeli: https://www.ixsystems.com/community...2-backup-solution-instead-of-crashplan.58423/

Why? Freenas (v11.1 onwards) now supports rclone nativetly and also has GUI entries for Cloudsync, however there is no option to use...

Read more about this resource...
 

Hazimil

Contributor
Joined
May 26, 2014
Messages
172
I have updated the script today to fix some syntax issues with the original code, for example I was referring to the log file by name instead of using the log_file variable.

I have also tweaked it to produce a user friendly email body, with the results of the backup, an example email body is below.

Code:
rclone backup started at: 2020-01-29 00:00:00 and finished at: 2020-01-29 00:43:02

======================================
The stats for this rclone backup were:
======================================
Transferred:           4.499G / 4.499 GBytes, 100%, 1.784 MBytes/s, ETA 0s
Errors:                 0
Checks:            421641 / 421641, 100%
Transferred:          261 / 261, 100%
Elapsed time:     43m2.7s


====================================
The following NEW files were copied:
====================================
Films/Brassed Off (1996) [15]/Brassed Off (1996).mkv
Films/Brassed Off (1996) [15]/Interviews/Interviews.mkv
Films/Christmas in a Day (2013) [PG]/Christmas in a Day (2013).mkv
Films/Tombstone (1993) [15]/Behind The Scenes/The Making Of.mkv
Films/Tombstone (1993) [15]/Tombstone (1993).mkv
Films/What Women Want (2000) [12]/Behind The Scenes/Behind The Scenes.mkv
Films/What Women Want (2000) [12]/Interviews/Cast and Crew Interviews.mkv
Films/What Women Want (2000) [12]/What Women Want (2000).mkv
Sysadmin/plex_backup/plex_backup_2020-01-28.tar.gz
Sysadmin/scripts/rclonebackup-v1.sh

==================================================
The following files were REPLACED with new copies:
==================================================
FamilyJonathan/Headed-Paper.docx
Sysadmin/scripts/rclonebackup.sh

=================================
The following files were DELETED:
=================================
Sysadmin/config_backups/FreeNAS_Config_2019-12-19.tar.gz
Sysadmin/config_backups/FreeNAS_Config_2019-12-22.tar.gz
Sysadmin/config_backups/FreeNAS_Config_2019-12-25.tar.gz
Sysadmin/config_backups/FreeNAS_Config_2019-12-28.tar.gz
Sysadmin/plex_backup/plex_backup_2019-12-22.tar.gz
Sysadmin/plex_backup/plex_backup_2019-12-25.tar.gz
Sysadmin/plex_backup/plex_backup_2019-12-28.tar.gz
Sysadmin/plex_backup/plex_backup_2019-12-31.tar.gz
TemporaryStuff/testing.txt

============================================
The following ERRORs and NOTICEs were found:
============================================


Hope you find this useful?

Jonathan
 

Hazimil

Contributor
Joined
May 26, 2014
Messages
172
I have updated a new version of the script, as per the following updates (update 1 done previously):

Update(2):
I've noticed that since upgrading to FreeNAS v11.3 that this script was no longer working. When you create the "rclone.conf" file from a shell prompt (or SSH connection) with the command "rclone config" it is stored at "/root/.config/rclone/rclone.conf" (you can check this by running the command "rclone config file").

However, when you then run this bash script as a cron task, for some reason it is looking for the config file at "/.config/rclone/rclone.conf"; a quick fix was to copy the "rclone.conf" file to this folder. However, it's not a good idea to have multiple copies of the "rclone.conf" config file as this could become confusing when you make changes. Therefore I decided to update the script to utilise the "--config" parameter of rclone to point to the config file created at "/root/.config/rclone/rclone.conf". I did log this as a bug (NAS-105088), but it was closed by IX as a script issue - however, I don't believe that is correct!

Update(3):
I have updated the script to be dual purpose, with regard to running a "rclone sync" to backup, and "rclone cryptcheck" to do a verification of your cloud based files and see if any files are missing as well as confirming the checksums of all encrypted files (for the paranoid out there!). You now need to run this script with an parameter, for example: "rclonebackup.sh sync" or "rclonebackup.sh cryptcheck", any other parameters or no parameter will result in an error email being sent. As this reuses code dynamically it means that any configuration changes in this script are only entered once and used for both sync and cryptcheck.
Further information about cryptcheck can be found at: https://rclone.org/commands/rclone_cryptcheck/
 

Hazimil

Contributor
Joined
May 26, 2014
Messages
172
02-04-2020: I have updated it as follows:
  • Tidied up the script;
  • Separated out user and system defined variables (to make editing easier);
  • Have moved from multiple "--exclude" statements to using "--exclude-from" and a separate file to hold excludes, which will make managing excludes much easier and saves editing this script.
  • Have changed the default email to root.
  • Have added in a third parameter option "check", which checks the files in the source and destination match.
  • Rewrote the email generation aspect of the script to remove the need for the separate email_attachments.sh script - all managed within the one script now; as well as moving from "echo" to "printf" commands for more control.
 

Hazimil

Contributor
Joined
May 26, 2014
Messages
172
Hi, not sure if anyone is actually using this, especially as the GUI has more functionality now, however I have done an update:

15-06-2020: I have updated it as follows:
  • Tidied up the script some more;
  • Have added another option to compress (gzip) the log file before being attached to the email (compressLog="yes"), this is set to "yes" by default.
  • You can also now request the script to keep a local copy (backup) of the unformatted log file (keepLog="no"), this is set to "no" by default, as well as how many logs to keep (amountBackups=31). Keeping the log was code I used to cut-n-paste in for my debugging, but decided to leave it in as an option now.
  • Have added in some checks that will verify that some of the user defined variables are valid.

Yours
Jonathan
 
Top