Script to delete old backup file?(Acronis, OO DiskImage, etc)

[paleoN]

Here's the script.

No sha-bang? Though it ought to work anyway.

Here's your commands and the output

You missed one. I was asking for crontab, well the relevant line(s) really, to see if anything looked off.

 

[cyberjock]

Didn't see the crontab... here it is


# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD: src/etc/crontab,v 1.33.2.1 2009/08/03 08:13:06 kensmith Exp $
#
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
#
#minute hour mday month wday who command
#
*/5 * * * * root /usr/libexec/atrun
#
# Save some entropy so that /dev/random can re-seed on boot.
*/11 * * * * operator /usr/libexec/save-entropy
#
# Rotate log files every hour, if necessary.
0 * * * * root newsyslog
#
# Perform daily/weekly/monthly maintenance.
1 3 * * * root periodic daily
15 4 * * 6 root periodic weekly
30 5 1 * * root periodic monthly
#
# Adjust the time zone if the CMOS clock keeps local time, as opposed to
# UTC time. See adjkerntz(8) for details.
1,31 0-5 * * * root adjkerntz -a

*/5 * * * * root /usr/local/bin/python /usr/local/bin/graph.py
0 * * * * root /bin/sh /root/save_rrds.sh

0 * * * * root /usr/local/bin/python /usr/local/bin/mfistatus.py > /dev/null 2>&1
*/5 * * * * root /usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1
* * * * * root /usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1

15 3 * * * root /usr/local/bin/python /usr/local/www/freenasUI/tools/cachetool.py expire >/dev/null 2>&1
30 3 * * * root /usr/local/bin/python /usr/local/www/freenasUI/tools/cachetool.py fill >/dev/null 2>&1
0 3 * * * root find /tmp/ -iname "sessionid*" -ctime +1d -delete
00 23 * * * root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/tank/.config-backup/.bkpconfig.sh > /dev/null 2> /dev/null
00 23 * * * root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/tank/.config-backup/.autorecycle.sh > /dev/null 2> /dev/null
05 03 * * * root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/tank/.SMARTdata/eSMART.sh > /dev/null 2> /dev/null
20 14 * * * root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/tank/.config-backup/backup-cleanup.sh
00 00 01,14,15 * * root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 14 tank

 

[HolyK]

Hi cyberjock:

First ... you should put "#!/bin/bash" at the beginning of your script
OR
you can use /bin/bash <scriptname> in CRON, so like this:
20 14 * * * root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /bin/bash /mnt/tank/.config-backup/backup-cleanup.sh

Then it should work fine :]

Code:

#!/bin/bash
ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list
ls -ltd1 /mnt/tank/Backups/HTPC/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list
ls -ltd1 /mnt/tank/Backups/Laptop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list

- No need to "cd <path>" since commands above will give you absolute path to files.
- Also lists are created in /tmp/ just to avoid creation of files all over the system ^^
- And also you don't need to "rm list" since it is (should be?) automatically overwritten by next command :]

 

[cyberjock]

Ok, correct me if I'm wrong, but doesn't the # mean comment for .sh files? So why would not having it matter?

Gonna test this new script. I'll report back in a minute.

 

[cyberjock]

And.. the cronjob just ran and I got an email.. which is even more puzzling...

/mnt/tank/.config-backup/backup-cleanup.sh: not found

But it is there. In fact, all I did was open the old file and pasted the contents of Holykiller's script. Remember that the old file had the errors I posted before, so clearly it is finding it.

In the FreeNAS GUI, should the command be "sh /mnt/tank/.config-backup/backup-cleanup.sh" or just "/mnt/tank/.config-backup/backup-cleanup.sh"


Well, out of desperation and "what could possibly go wrong" I tried doing "sh /mnt/tank/.config-backup/backup-cleanup.sh" for the cronjob instead of without the sh and I got an email:

/mnt/tank/.config-backup/backup-cleanup.sh: cannot open list
: No such file or directory
/mnt/tank/.config-backup/backup-cleanup.sh: cannot open list
: No such file or directory
ls: /mnt/tank/Backups/Laptop/*: No such file or directory
/mnt/tank/.config-backup/backup-cleanup.sh: cannot open list: No such file or directory

The laptop error is correct, the folder is empty of all files. The laptop is encrypted with Truecrypt and I haven't found a simple automated method for backing up my laptop.

I'm gonna try a few experiments. I'm kind of convinced that I can't read and/or write to my zpool through a cronjob.

 

[paleoN]

Ok, correct me if I'm wrong, but doesn't the # mean comment for .sh files? So why would not having it matter?

Actually, it isn't a comment. A sha-bang is special. Also, it's now a [post=53550]Bash script![/post]

Code:

ls -ald /mnt/tank/.config-backup

ls -al /mnt/tank/.config-backup/backup-cleanup.sh

 

[cyberjock]

So I tried touch /mnt/tank/testfile. It worked.
Then I tried cp /mnt/tank/.config-backup/backup-cleanup.sh /mnt/tank/.config-backup/testfile. That worked.

So clearly I have permissions to the zpool, my folder and files.

So I started breaking down the commands and I can do ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 for the cronjob command. It works.

But as soon as I add the > /tmp/list then it breaks. In fact, I get an email with the list of files. But I shouldn't be getting an email at all since the output should have gone to /tmp/list. So something must be wrong with the implementation of >. No file is created in /tmp at all(vice an empty file).

Last edit: Attention to details.. I also noticed that the "tail +3" isn't actually doing anything at all. I have 5 files(3 that should be deleted). When I run the command:

ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 from the command line I get:

/mnt/tank/Backups/Desktop/DESKTOP-02-15-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-02-01-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-01-25-2013.omg

Which seems to be correct.

But when the crojob sent me an email with the exact same command I got:

/mnt/tank/Backups/Desktop/DESKTOP-03-01-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-02-22-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-02-15-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-02-01-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-01-25-2013.omg

This would have deleted all of my backups. Clearly not preferred ;)

 

[paleoN]

So clearly I have permissions to the zpool, my folder and files.

Which permissions?

But as soon as I add the > /tmp/list then it breaks. In fact, I get an email with the list of files. But I shouldn't be getting an email at all since the output should have gone to /tmp/list. So something must be wrong with the implementation of >.

That appears to be a bug if you are entering it directly. Check what's in crontab.

 

[cyberjock]

Which permissions?

I verified I can read and write to the zpool. Although I didn't verify precisely to the /mnt/tank/Backups/Desktop(or Laptop or HTPC) folder.

That appears to be a bug if you are entering it directly. Check what's in crontab.

Crontab... is messed up. I did chose a bunch of different times while experimenting so ignore them. The command that was in the GUI was ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3

Code:

14,15,16,18,20,23,34	19	*	*	*	root	PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" ls -ltd1 /mnt/tank/Backups/Desktop/* 

So actually even the grep isn't executing, but I didn't know that because the only files in the folder are .omg files.

Edit:

So now I went back to having the cronjob go to my .sh file. With the #!/bin/bash at the beginning I get an email

/mnt/tank/.config-backup/backup-cleanup.sh: not found

I then changed the script to a single line: ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3

I got an email back saying:

tail:
: No such file or directory
grep: write error: Broken pipe

Now I'm very lost...

 

[paleoN]

I verified I can read and write to the zpool.

Who and where? It does seem to try and execute the script. Now that I think about it I'm not sure what cron's default working directory is anyway.

I then changed the script to a single line: ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3

I thought these were files, no -d.
Edit: Nevermind, that works.

Now I'm very lost...

I don't suppose these file names have spaces or anything?

 

[ProtoSD]

Now that I think about it I'm not sure what cron's default working directory is anyway.

throw a pwd > some-log-file.txt at the top of the script

I don't suppose these file names have spaces or anything?

I think the version of the command I posted was supposed to account for that, but I'm just jumping in here again and don't remember what I posted.

 

[paleoN]

throw a pwd > some-log-file.txt at the top of the script

/root in this case, duh. :o

I think the version of the command I posted was supposed to account for that, but I'm just jumping in here again and don't remember what I posted.

It doesn't, but for me it simply fails to delete the files. The script still works otherwise.

 

[cyberjock]

The working directory is /root.

paleoN, when you say the script works for you, do you mean the script works for you when run manually, via cronjob, or both? If I run the script manually it worked before. I don't think it actually ever worked as a cronjob. I think I got it working manually, created the cronjob and didn't ever check up on it expecting it would work. I haven't tried to run it tonight, but I'll test it with a test folder and files just to be sure it still works.

 

[paleoN]

paleoN, when you say the script works for you, do you mean the script works for you when run manually, via cronjob, or both?

Any/both, though I removed the rm list line and yes the cd is gratuitous.

Code:

#!/bin/sh
cd /mnt/tank/Backups/Desktop
ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 > /mnt/tank/list && xargs rm < /mnt/tank/list

Do the other .config-backup scripts run appropriately?

 

[cyberjock]

First, there is an error in your script paleoN. I questioned it in my head for a second or two before pasting and didn't look it over.

Your script:

Code:

#!/bin/bash
ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < list
ls -ltd1 /mnt/tank/Backups/HTPC/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < list
ls -ltd1 /mnt/tank/Backups/Laptop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < list

Notice that at the end of the line you have list instead of /tmp/list. Whoops!

So.. with that in mind I fixed my script, made a test folder and files and adjusted the script to go to the test folder and files.

But, with the #!/bin/bash I still get the email "/mnt/tank/.config-backup/backup-cleanup.sh: not found"

Without the #!/bin/bash I get no email and SUCCESS! It deleted my test files appropriately.

But now I'm left with the question why the other script didn't work before. It should have...

So now I've fixed the script to point to the backup folders, adjusted the cronjob time so it ran 1 min in the future and... it didn't work. Back to "cannot open /tmp/list: No such file or folder"

Strangely, I checked and the /tmp/list file is there and does contain the correct filenames!

Code:

/mnt/tank/Backups/Desktop/DESKTOP-02-15-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-02-01-2013.omg
/mnt/tank/Backups/Desktop/DESKTOP-01-25-2013.omg

Using #!/bin/bash is out of the question for my scripts though. I even tried touching a file after the #!/bin/bash and it doesn't touch the file. Anytime I add #!/bin/bash I get a guaranteed email saying "/mnt/tank/.config-backup/backup-cleanup.sh: not found". As soon as I get rid of it the script runs fine.

I looked at my other scripts that execute fine(all of my other scripts are executing fine as scheduled) and they have #!/bin/sh. I tried changing my script to have that at the beginning.. another "/mnt/tank/.config-backup/backup-cleanup.sh: not found" error.

So I changed the script to simply say "touch /mnt/tank/1.txt" to see what happens. When it ran I ended up with a file named "1.txt?" If I go to that location from Windows Explorer the filename is 1RKJB3~L. I get the same thing if I go to /tmp. I actually did see a /tmp/list? file once or twice but disregarded the question mark as a typo. Weirdness!

So any guesses what the heck that means? I'm about to do a reboot just because I'm desperate. I'm starting to wonder if my FreeNAS installation is somehow corrupted. I'm having no other issues at all with the system besides this one. I'm running FreeNAS-8.3.0-RELEASE-p1-x64 (r12825) and I don't believe I ever used any non-release versions or upgraded to this version.

Edit: Actually, I'm gonna wait on a reboot. Maybe someone can recommend a log file for me to look at that would be wiped on reboot.

 

[cyberjock]

Ok.. I'll try to keep my thoughts organized. This stuff with the script not working just isn't making any sense to me.

1. I checked out the /var/log/cron and there's nothing very valuable there.
2. I read /var/log/messages and there's lots of entries like this:

Code:

Mar  5 19:14:46 server notifier: [:  grep ".*\.omg": bad number
Mar  5 19:14:46 server notifier: [: 0: unexpected operator
Mar  5 19:14:46 server notifier: Stopping cron.
Mar  5 19:14:46 server notifier: Starting cron.
Mar  5 19:14:55 server notifier: [:  grep ".*\.omg": bad number
Mar  5 19:14:55 server notifier: [: 0: unexpected operator
Mar  5 19:14:55 server notifier: Stopping cron.
Mar  5 19:14:55 server notifier: Starting cron.
Mar  5 19:15:17 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:15:17 server notifier: [:  tail +3 : unexpected operator
Mar  5 19:15:17 server notifier: Stopping cron.
Mar  5 19:15:17 server notifier: Starting cron.
Mar  5 19:17:17 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:17:17 server notifier: [:  tail +3 > /tmp/list: unexpected operator
Mar  5 19:17:17 server notifier: Stopping cron.
Mar  5 19:17:17 server notifier: Starting cron.
Mar  5 19:17:24 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:17:24 server notifier: [:  tail +3 > /tmp/list: unexpected operator
Mar  5 19:17:24 server notifier: Stopping cron.
Mar  5 19:17:24 server notifier: Starting cron.
Mar  5 19:19:12 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:19:12 server notifier: [:  tail +3 > /tmp/list: unexpected operator
Mar  5 19:19:12 server notifier: Stopping cron.
Mar  5 19:19:12 server notifier: Starting cron.
Mar  5 19:21:59 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:21:59 server notifier: [:  tail +3 > /tmp/list: unexpected operator
Mar  5 19:21:59 server notifier: Stopping cron.
Mar  5 19:21:59 server notifier: Starting cron.
Mar  5 19:33:55 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:33:55 server notifier: [:  tail +3: unexpected operator
Mar  5 19:33:55 server notifier: Stopping cron.
Mar  5 19:33:55 server notifier: Starting cron.
Mar  5 19:40:30 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:40:30 server notifier: [:  tail +3 > /tmp/list: unexpected operator
Mar  5 19:40:30 server notifier: Stopping cron.
Mar  5 19:40:30 server notifier: Starting cron.
Mar  5 19:43:46 server notifier: [:  grep ".*\.omg" : bad number
Mar  5 19:43:46 server notifier: [:  tail +3 > /tmp/list: unexpected operator

Kinda weird but since I didn't keep a log of every test I did I'm not sure how helpful this actually is.

3. I checked my other scripts. All of my scripts are in /mnt/tank/.config-backup.
-I have a script that is just "cp /data/freenas-v1.db /mnt/tank/.config-backup/`date +%Y%m%d`.db
-I have a script to auto-delete everything in the recycle bin after it has been there for 7 days. Sure enough, nothing has been in the recycle bin longer than 7 days.

So it has something to do with this script and how it works. I'm so confused with complex command line stuff that I can't even make a guess at what is wrong.

So I'm going to install FreeNAS 8.3.0-p1 to a new USB stick, import the zpool and attempt to run the script as a cronjob. If that doesn't work then there is definitely a bug or something I'm doing wrong. If it works then its something with my configuration and I'll just have to suck it up and start over. Wish me luck!

 

[cyberjock]

So I got a new USB stick setup. I realized that I can't easily resetup my email, so I did a test with the following script knowing if it didn't work I won't get an email:

Code:

#!/bin/sh
cd /mnt/tank/Backups/Desktop
ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list

and sure enough.. it didn't work. I then imported my config file, waited for the reboots, then did the cronjob again and got an email. Once again.. the backup-config.sh file not found. Of course, I was expecting a file not found error since I had #!/bin/sh. Anytime I have those it always gives a file not found error.

Back to using a test folder.

 

[cyberjock]

I feel like throwing up. I've found the issue.. at last. I feel so incredibly stupid I'm ashamed to admit it.

I was editing and saving the file in notepad++. But the "EOL conversion" wasn't UNIX(as my working scripts were). It was in Windows mode. As soon as I changed it and saved it the damn thing just worked.

So, here's my final script for anyone that uses this thing in the future:

Code:

#!/bin/sh
ls -ltd1 /mnt/tank/Backups/Desktop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list
ls -ltd1 /mnt/tank/Backups/HTPC/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list
ls -ltd1 /mnt/tank/Backups/Laptop/* | grep ".*\.omg" | tail +3 > /tmp/list && xargs rm < /tmp/list

My guess is that it worked originally but I edited it in Notepad and screwed up the carriage returns, which broke it back in January. One thing is for sure. I will never EVER make this mistake again! LOL.

Thanks to everyone who tried to help me while I wasted everyone's time :)

 

[joeschmuck]

I feel like throwing up. I've found the issue.. at last. I feel so incredibly stupid I'm ashamed to admit it.

I was editing and saving the file in notepad++. But the "EOL conversion" wasn't UNIX(as my working scripts were). It was in Windows mode. As soon as I changed it and saved it the damn thing just worked.

My guess is that it worked originally but I edited it in Notepad and screwed up the carriage returns, which broke it back in January. One thing is for sure. I will never EVER make this mistake again! LOL.

Thanks to everyone who tried to help me while I wasted everyone's time :)

You know that this is a very good learning experience for anyone who reads this thread and I doubt anyone would have caught this except you. I like ee to create and edit scripts and I use Putty to SSH so I do use Windoze Notepad to create my scripts initially and then I cut and paste into ee, it has never let me down yet.

 

[HolyK]

2 cyberjock: Hehe, mystery resolved :D Gratz!

edit: missing /tmp/ also added in my post to avoid confusion for the further reader ^^ , thanks :]