SOLVED Cron/RSync job to move data to dated folder

[mcalr3]

Hi folks,

I have been looking into this for a small while and I am quite stuck (I am quite new to linux in general).

I have a bunch of 5 network cameras saving their images to a volume called /mnt/Data/Camera2/Cam1, /mnt/Data/Camera2/Cam2, etc....

The cameras themselves do not support putting the images in dated directories so I need the ability for freenas to move the files into an "archive directory" so that the users who have the share added, do not need to map another network drive in windows.

I would like the backups to go to /mnt/Data/Camera2/Archive/<date> on a weekly basis. So far I have come up with the following, but I dont know if it is correct:

Code:

#!/bin/bash
HOME_DIRS="/mnt/Data/Camera2/archive/Cam1/mnt/Data/Camera2/archive/Cam2 /mnt/Data/Camera2/archive/Cam3 /mnt/Data/Camera2/archive/Cam4 /mnt/Data/Camera2/archive/Cam5"
DATE_DIR=$(date +%Y-%m-%d)

for FOLDER in $HOME_DIRS; do
    mkdir -p "${FOLDER}/${DATE_DIR}"
done

mv /mnt/Data/Camera2/Cam1 /mnt/Data/Camera2/archive/Cam1/${DATE_DIR}

I have also tried:

Code:

rsync -rvuAth --log-file=/mnt/Data/Camera2/weeklymove.log --remove-source-files /mnt/Data/Camera2/Cam1 /mnt/Data/Camera2/archive/Cam1/"date -I"

Any help would be much appreciated!!

 

[DifferentStrokes]

Looks like you are moving in the right direction. I'd make a test folder with a few test files and run the script against that.

 

[mcalr3]

What would do the job best though, the rsync task or the mkdir and move?

 

[DifferentStrokes]

I guess that is up to you. If they both work and do what you want, just pick one.

 

[fracai]

mv will avoid the time required to copy the files.
you could also use the --link-dest argument with rsync to achieve the same thing.
Assuming that is that the dated directories are on the same dataset. If they're being moved to a different dataset, I'd probably stick with rsync.

Out of curiosity and off-topicy, what are the cameras that you use?

 

[mcalr3]

mv will avoid the time required to copy the files.
you could also use the --link-dest argument with rsync to achieve the same thing.
Assuming that is that the dated directories are on the same dataset. If they're being moved to a different dataset, I'd probably stick with rsync.

Out of curiosity and off-topicy, what are the cameras that you use?

The cameras I am using are Y-Cam Black edition. Quite old and by no means HD, but they do the job. I work at a golf club with a semi-low IT budget and I have inherited alot of the rubbish equipment they have had, the cameras are quite low on the list of the things to replace!

 

[mcalr3]

Hi again folks,

still having trouble writing this script. I have tested the individual elements in the shell and they work no problems but for my script i keep getting unexpected end of file.

Also, can someone show me how to tidy up the mv command?

Code:

#!/bin/bash

BACKUP_DIRS="/mnt/Data/Camera2/Archive/GCMF /mnt/Data/Camera2/Archive/GCMR /mnt/Data/Camera2/Archive/GCMS /mnt/Data/Camera2/Archive/CHF /mnt/Data/Camera2/Archive/CHR"
DATE_DIR=WB_$(date +%d-%m-%Y)
LOGDIR=/mnt/Data/Camera2/logs/WB_$(date +%d-%m-%Y)
LOGFILE=$LOGDIR/$(date +%d-%m-%Y).txt

/bin/mkdir -p $LOGDIR

/bin/echo "Weekly DDL camera backup:" > $LOGFILE
/bin/echo "" >> $LOGFILE
for FOLDER in $BACKUP_DIRS; do
    mkdir -p "${FOLDER}/${DATE_DIR}"
done >> $LOGFILE

mv -f -b /mnt/Data/Camera2/archive/GCMF/Snapshot_time /mnt/Data/Camera2/archive/GCMF/WB_$(date +%d-%m-%Y) >> $LOGFILE
mv -f -b /mnt/Data/Camera2/archive/GCMR/Snapshot_time /mnt/Data/Camera2/archive/GCMR/WB_$(date +%d-%m-%Y) >> $LOGFILE
mv -f -b /mnt/Data/Camera2/archive/GCMS/Snapshot_time /mnt/Data/Camera2/archive/GCMS/WB_$(date +%d-%m-%Y) >> $LOGFILE
mv -f -b /mnt/Data/Camera2/archive/CHF/Snapshot_time /mnt/Data/Camera2/archive/CHF/WB_$(date +%d-%m-%Y) >> $LOGFILE
mv -f -b /mnt/Data/Camera2/archive/CHR/Snapshot_time /mnt/Data/Camera2/archive/CHR/WB_$(date +%d-%m-%Y) >> $LOGFILE 

 

[Bidule0hm]

I would start by doing a rootDir variable with this inside "/mnt/Data/Camera2/archive" so you can shorten a lot of the other lines.

For the error I only see one thing: add doubles quotes around variables and integrated commands.

 

[mcalr3]

I am trying to do just a mkdir first (without doing the move) to check that it is doing it correctly, but I am getting no output from this even when doing a /bin/bash script.sh | tee -a log.txt

Code:

#!/bin/bash
ROOT_DIR='/mnt/Data/Camera2'
BACKUP_DIRS='${ROOT_DIR}/Archive/GCMF ${ROOT_DIR}/Archive/GCMR ${ROOT_DIR}/Camera2/Archive/GCMS ${ROOT_DIR}/Camera2/Archive/CHF ${ROOT_DIR}/Camera2/Archive/CHR'
#DATE=$(date +%d-%m-%Y)
DATE_DIR='WB_$(date +%d-%m-%Y)'
#LOG_DIR='$ROOT_DIR/Logs/WB_$(date +%d-%m-%Y)'
#LOG_FILE='$LOG_DIR/${DATE}.txt'
#
#/bin/mkdir -p $LOG_DIR
#
#/bin/echo "Weekly DDL camera backup:" > $LOG_FILE
#/bin/echo "" >> $LOG_FILE
 #
for FOLDER in $BACKUP_DIRS; do mkdir -p "${FOLDER}/${DATE_DIR}" #>> $LOG_FILE
#
#mv -f -b ${ROOT_DIR}/GCMF/Snapshot_time ${ROOT_DIR}/archive/GCMF/${DATE_DIR}
#mv -f -b ${ROOT_DIR}/GCMR/Snapshot_time ${ROOT_DIR}/archive/GCMR/${DATE_DIR}
#mv -f -b ${ROOT_DIR}/GCMS/Snapshot_time ${ROOT_DIR}/archive/GCMS/${DATE_DIR}
#mv -f -b ${ROOT_DIR}/CHF/Snapshot_time ${ROOT_DIR}/archive/CHF/${DATE_DIR}
#mv -f -b ${ROOT_DIR}/CHR/Snapshot_time ${ROOT_DIR}/archive/CHR/${DATE_DIR}
done

This is doing my head in!

 

[Bidule0hm]

Is the uppercase "a" in "archive" on purpose? because Unix systems are case sensitive so it matters and in some places it's an "A" and other it's an "a" so you might want to look at that.

You've included all the mv in the for but they're not in the old script, it's on purpose?

I believe you've forgotten some "Camera2".

Single quotes are bad, use double quotes.

 

[mcalr3]

OK, I am simplifying my scripts and creating 2, one that makes the dirs and one that moves the files.
I dont know how to get the shell or cron to know which directory it is working in, but I have this and when I am running it from the directory /mnt/Data/Camera2 it works, when I try to run it from elsewhere it doesnt work.

How should I change the working directory in a shell script, and how does cron see the directory structure when setting up in FreeNAS? Does it just go by the user's home directory?

 

[Bidule0hm]

That's because you've used relative paths instead of absolute paths in your script. So either you do a cd /the/path/you/want/ (ugly fix) or either you use absolute paths (the right method) :)

 

[mcalr3]

Managed to get some help over on stack overflow and tweaked it and now it is working with this script:

Code:

#!/bin/bash

ROOT_DIR="/mnt/Data/Camera2"
BACKUP_DIR="${ROOT_DIR}/Archive"
CAMERAS="GCMF GCMR GCMS CHF CHR"
DATE=$(date +%d-%m-%Y)
DATE_DIR="WB_$(date +%d-%m-%Y)"
LOG_DIR="${ROOT_DIR}/Logs/WB_$(date +%d-%m-%Y)"
LOG_FILE="${LOG_DIR}/${DATE}.txt"

/bin/mkdir -p ${LOG_DIR}

/bin/echo "Weekly DDL camera backup:" > ${LOG_FILE}
/bin/echo "" >> ${LOG_FILE}

for FOLDER in ${CAMERAS}
do
   mkdir ${BACKUP_DIR}/${FOLDER}/${DATE_DIR} >> $LOG_FILE

   mv -f -b ${ROOT_DIR}/${FOLDER}/Snapshot_time ${BACKUP_DIR}/${FOLDER}/${DATE_DIR}

done

 

[Bidule0hm]

That's looks good and clean, good job ;)