SOLVED Problems with scripting rclone - unexpected behaviour

[Hazimil]

Hi,

I'm in the process of moving to B2 for my cloud storage and plan to run via a script rather than the FreeNAS GUI as there are some additional parameters I wish to use. However, I'm getting an error with my script and was wondering if anyone could help (currently testing on one dataset [my young daughters data] at the moment). I'm executing the script via a Putty Shell connection. https://www.shellcheck.net/ shows no error on my code.

Code:

#!/bin/bash

### Notes
# A script to run rclone on the FreeNAS server to backup my NAS data to Backblaze B2 cloud storage
#
# FreeNAS: https://freenas.org/
# rclone: https://rclone.org/
#
# Script was adapted from the one posted in the FreeNAS community forums by Martin Aspeli
# https://www.ixsystems.com/community/threads/rclone-backblaze-b2-backup-solution-instead-of-crashplan.58423/
###

### Why?
# Freenas (v11.1 onwards) now supports rclone nativetly and also has GUI enteries for Cloudsync, however
# there is no option to use additional parameters within the GUI, thus I have adapted a script to be
# run instead via a cron job.
#
# You will need to configure rclone for Backblaze B2 as per: https://rclone.org/b2/
# If encrypting your data with rclone you need to configure as per: https://rclone.org/crypt/
#
# Note: The rclone config file save via the CLI is different to the one used in the FreeNAS GUI, so its
# safe to update this configuration file via "rclone config" without affecting anything you have configured
# in the FreeNAS GUI.
###

### What Parameters?
#I wish to use the following parameters:
#
# Number of file transfers to run in parallel. (default 4)
# Backblaze recommends that you do lots of transfers simultaneously for maximum speed.
# --transfers int
#
# Use recursive list if available. Uses more memory but fewer transactions (i.e. cheaper). Backblaze B2 supports.
# --fast-list
#
# Follow symlinks and copy the pointed to item.
# --copy-links
#
# Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
# --min-age 15m
#
# Log everything to a log file
# --log-file filename
#
# This sets the log level for rclone. The default log level is NOTICE.
# --log-level level
#
# Permanently delete files on remote removal, otherwise hide files. Backblaze B2 specific parameter.
# Note: This means no version control of files on backblaze (saves space & thus costs)
# --b2-hard-delete
#
# Exclude files matching pattern (https://rclone.org/filtering/), amke sure you put them within ""
# --exclude
# I wish to exclude: Thumbs.db, desktop.ini. AlbumArt*, .recycle, .windows
#
# For testing purposes only (i.e. don't actually sync files), so a trial run with no permanent changes
# --dry-run
###

### Define Parameters
src=/mnt/tank/FamilyHannah
dest=b2encrypt:FamilyHannah
email=me@email.address
log_file=/tmp/rclonelog.txt
log_level=INFO
min_age=15m
transfers=16
###

### Execute
rclone sync \
    --transfers ${transfers} \
    --fast-list \
    --copy-links \
    --min-age ${min_age} \
    --log-level ${log_level} \
    --log-file ${log_file} \
    --b2-hard-delete \
    --exclude "Thumbs.db" \
    --exclude "desktop.ini" \
    --exclude "AlbumArt*" \
    --exclude ".recycle/**" \
    --exclude ".windows/**" \
    --dry-run \
    ${src} ${dest}
success=$?

if [[ $success != 0 ]]; then
    subject="rclone backup: An error occurred. Please check the logs. (error code:${success})"
    body="Refer to https://rclone.org/docs/#exit-code for more information"
else
    subject="rclone backup: Backup succeeded"
    body=""
fi

### send the email using the email_attachments.sh script
/mnt/tank/Sysadmin/scripts/email_attachments.sh ${email} ${email} "${subject}" "${body}" /tmp/rclonelog.txt
###

### Tidy Up ###
## Delete the rclone log in preparation of a new log
rm /tmp/rclonelog.txt
### End ###

If I run the above script (using --dry-run) it works, and I get emailed the following:

Code:

2019/06/06 12:43:22 NOTICE: .windows: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Hannah school handwriting Book.docx: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Test.docx: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Untitled.png: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Hannahs House.png: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: jcob.png: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: adams house.png: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: our house.png: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: testing.txt: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: python/fun Q&A.py: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/BBC - CBeebies Home.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Bing.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Bob the Builder Official Site  Online Games, Videos, Activities, Printables.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Disney Junior  New Official Disney Junior UK Website  Disney UK.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Northwood Primary School/Year 3/Half-term project on Volcanos.pub: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/NORAD Santa.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Rabbits CCTV.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Northwood Primary School/Year 5/Grenada Hannah Year 5.pptx: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: LEGO Creations/Models/kjhgggtr5ewqasdxzcvb.lxf: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Links/BBC - CBeebies - Home - Play fun games and early years activities.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Links/Suggested Sites.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Links/Disney Junior  New Official Disney Junior UK Website  Disney UK.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Links/YouTube.url: Not copying as --dry-run
2019/06/06 12:43:22 NOTICE: Favorites/Links/Fun Educational Games for Kids  e-Learning Resources for Teachers  EducationCity UK.url: Not copying as --dry-run
2019/06/06 12:43:22 INFO  : Encrypted drive 'secret:FamilyHannah': Waiting for checks to finish
2019/06/06 12:43:22 INFO  : Encrypted drive 'secret:FamilyHannah': Waiting for transfers to finish
2019/06/06 12:43:22 INFO  : Waiting for deletions to finish
2019/06/06 12:43:22 INFO  :
Transferred:                0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:           24 / 24, 100%
Elapsed time:        2.4s

However, if I remove the "--dry-run \" line, i get the following error on screen:

Code:

./rclonebackup.sh: line 85: /mnt/tank/FamilyHannah: Is a directory

And am emailed:

Code:

sage:
  rclone sync source:path dest:path [flags]

Flags:
  -h, --help   help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command sync needs 2 arguments minimum

However, if I extract the code and just run it as a CLI command, i.e.:

Code:

rclone sync --transfers 16 --fast-list --copy-links --min-age 15m --log-level INFO --log-file /tmp/rclonelog.txt --b2-hard-delete  --exclude "Thumbs.db" --exclude "desktop.ini" --exclude "AlbumArt*" --exclude ".recycle/**" --exclude ".windows/**"  /mnt/tank/FamilyHannah secret:FamilyHannah

It works and the log file says:

Code:

2019/06/06 12:53:20 INFO  : Encrypted drive 'secret:FamilyHannah': Waiting for checks to finish
2019/06/06 12:53:20 INFO  : Encrypted drive 'secret:FamilyHannah': Waiting for transfers to finish
2019/06/06 12:53:22 INFO  : .windows: Copied (new)
2019/06/06 12:53:22 INFO  : Test.docx: Copied (new)
2019/06/06 12:53:22 INFO  : Untitled.png: Copied (new)
2019/06/06 12:53:22 INFO  : Hannah school handwriting Book.docx: Copied (new)
2019/06/06 12:53:22 INFO  : Hannahs House.png: Copied (new)
2019/06/06 12:53:22 INFO  : jcob.png: Copied (new)
2019/06/06 12:53:22 INFO  : python/fun Q&A.py: Copied (new)
2019/06/06 12:53:22 INFO  : our house.png: Copied (new)
2019/06/06 12:53:22 INFO  : testing.txt: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Rabbits CCTV.url: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Links/Fun Educational Games for Kids  e-Learning Resources for Teachers  EducationCity UK.url: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Links/YouTube.url: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/NORAD Santa.url: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Disney Junior  New Official Disney Junior UK Website  Disney UK.url: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/BBC - CBeebies Home.url: Copied (new)
2019/06/06 12:53:23 INFO  : LEGO Creations/Models/kjhgggtr5ewqasdxzcvb.lxf: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Links/BBC - CBeebies - Home - Play fun games and early years activities.url: Copied (new)
2019/06/06 12:53:23 INFO  : adams house.png: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Links/Suggested Sites.url: Copied (new)
2019/06/06 12:53:23 INFO  : Favorites/Links/Disney Junior  New Official Disney Junior UK Website  Disney UK.url: Copied (new)
2019/06/06 12:53:24 INFO  : Favorites/Bob the Builder Official Site  Online Games, Videos, Activities, Printables.url: Copied (new)
2019/06/06 12:53:24 INFO  : Favorites/Bing.url: Copied (new)
2019/06/06 12:53:24 INFO  : Northwood Primary School/Year 3/Half-term project on Volcanos.pub: Copied (new)
2019/06/06 12:53:29 INFO  : Northwood Primary School/Year 5/Grenada Hannah Year 5.pptx: Copied (new)
2019/06/06 12:53:29 INFO  : Waiting for deletions to finish
2019/06/06 12:53:29 INFO  :
Transferred:        4.419M / 4.419 MBytes, 100%, 375.146 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:           24 / 24, 100%
Elapsed time:         12s

Also, I'm not sure why the folder ".windows" is not being excluded.

A confused coder....

Jonathan

 

[Hazimil]

Polite bump, if anyone can assist?

Jonathan

 

[Hazimil]

Fixed it!

Problem was that if I hash-out "#" the line "--dry-run \", because of the escape code at the end, it also hashed-out the ${src} ${dest} line... Just deleting "--dry-run " instead fixes it.

Jonathan