Script to delete old backup file?(Acronis, OO DiskImage, etc)

[cyberjock]

So I have automated backups setup for my machines, each with its own folder.

If you look in /mnt/tank/Backups/HTPC I have 4 files:

HTPC-01-01-2013.omg
HTPC-01-09-2013.omg
HTPC-01-20-2013.omg
HTPC-01-26-2013.omg

I have this situation for each of my machines and it's taking up quite a bit of space. I've searched Google and I can't find any script smart enough to recognize the 2 newest files and delete the older ones. I'd rather not have a script that deletes files older than a certain value because that could backfire if my machine is off for an extended period of time.

I tried googling and that was almost useless. I was thinking if I could make a script that slowly goes backwards in time from today until it finds 2 files, then auto-delete all files older than that threshhold that would work. But alas I'm not script-friendly enough to get it to work. Anyone know how I can do this? I was thinking something using find -atime +X would be useful..

 

[joeschmuck]

As for Acronis, not sure which version you might have available but your backup plan can be customized to perform auto-deletes (cleanup). I have mine set for 62 days but you can also set it up to retain x number of backups.

I'd think a script would be easy to create if that is your preference.

Here is something to wet your whistle since I'm headed to bed for the evening... It lists all the files in date order, newest on top...

Code:

find . -type f -exec stat -f "%Sm %N" -t %Y%m%d%H%M%S {} \;

or this version would work better for your purpose but the first one demonstrates the dates better...

Code:

find . -type f -exec stat -f {} \;

 

[cyberjock]

OO Diskimage doesn't offer that feature and I abandoned Acronis over a year ago when nobody could get Acronis to properly backup over a network connection.

I will definitely experiment with that command and see what I can come up with :)

Edit: That command seems to only output one file. For me, today's backup. So something is wrong with that command but I'm going to read more on the find command and see if I can figure it out.

 

[joeschmuck]

Since you can list the files, you could put that list into a file and then chop off the first two, then pass the remaining names to delete the files. This actually should be easy. Myself I'd set up a few variables to pass via arguments like the path to check and how many files to retain. This would make it universal. You could add a third argument that if present could be a mask for the file types, or something like that just in case there were extra files there you didn't expect.

 

[cyberjock]

Since you can list the files, you could put that list into a file and then chop off the first two, then pass the remaining names to delete the files. This actually should be easy. Myself I'd set up a few variables to pass via arguments like the path to check and how many files to retain. This would make it universal. You could add a third argument that if present could be a mask for the file types, or something like that just in case there were extra files there you didn't expect.

That was exactly what I was thinking. I just lack the know-how to implement it :)

Actually, I can do ls -l -U > somefile, chop off 4 lines and get my list. There's only 10 ways to skin a cat ;)

 

[cyberjock]

I feel stupid. After trying to figure out why your command only lists one file.. that's because I deleted the others so there was only 1 file. /sigh

So I got:

set extensiontofind=omg
find /mnt/tank/test -type f -name "*.$extensiontofind" -exec stat -f "%N" -t %Y%m%d%h%m%s {} \; > /tmp/output.txt

Makes me a file with the names. I have to replace the variable with the actual omg because substituting the variable doesn't work.. yet. Haven't figured out why exactly. Been trying to figure out the following paragraph.

But I can't figure out how to delete the first 2 lines. I've read about grep, cut, colrm, and a whole slew of other commands but I can't seem to find one that does exactly what I want. I was sure grep would let you, but reading the manpage doesn't appear to be so.

 

[ProtoSD]

That was exactly what I was thinking. I just lack the know-how to implement it :)

.... There's only 10 ways to skin a cat ;)

That's the beauty of Unix.

This taking a break from the forums doesn't seem to be working out :p

Give this a try AFTER creating a copy of an existing directory to test it with (you could also just do "touch test-1.omg" "touch test-2.omg etc. etc. and create some empty test files):

Code:

find . -type f -name "*.omg" -exec stat -f "%Sm %N" -t %Y%m%d%H%M%S {} \; | sort -rzn | awk 'NR > 2 { sub("^[0-9]*(.[0-9]*)? ", ""); print }' | xargs -0 rm -f

 

[cyberjock]

WOW. That's a long command. It doesn't seem to do anything though. I created a test folder and pasted that whole thing into a .sh file and nada.

Edit:

# find /mnt/tank/test/1 -type f -name "*.omg" -exec stat -f "%Sm %N" -t %Y%m%d%H%M%S {} \; | sort -rzn

Causes the following output:

: No such file or directory

 

[ProtoSD]

Try doing it in steps copying the commands up to each pipe symbol "|"

Is the file extension "Oh" MG or Zero MG?

Somewhere you have a typo unless I pasted it wrong.

 

[ProtoSD]

Also, maybe try adding -maxdepth 1 before the -name argument.

 

[HolyK]

Huh ... why not to do it simple like this ?

Code:

ls -lt1 | grep ".*\.omg" | tail +3 > rmlist

Output is list of the .omg filenames without two newest files.

And now just call xargs rm < list to delete the files.

Or do it in one command ^^

Code:

ls -lt1 | grep ".*\.omg" | tail +3 > list && xargs rm < list

This will always keep 2 newest .omg files. If you will have only one or two files present, then tail +3 will return back empty output, so the list file will be blank and nothing will be removed


Or is there any catch which i am missing ? :]

 

[cyberjock]

Huh ... why not to do it simple like this ?

Code:

ls -lt1 | grep ".*\.omg" | tail +3 > rmlist

Output is list of the .omg filenames without two newest files.

And now just call xargs rm < list to delete the files.

Or do it in one command ^^

Code:

ls -lt1 | grep ".*\.omg" | tail +3 > list && xargs rm < list

This will always keep 2 newest .omg files. If you will have only one or two files present, then tail +3 will return back empty output, so the list file will be blank and nothing will be removed


Or is there any catch which i am missing ? :]

Nice!!! It worked perfectly.

I think I need to get a grasp on when | and > and < are used. They do different things but I'm not sure of their exact limitations yet.

 

[cyberjock]

Try doing it in steps copying the commands up to each pipe symbol "|"

Is the file extension "Oh" MG or Zero MG?

Somewhere you have a typo unless I pasted it wrong.

It is an "Oh" MG. For you electrical folks.. Ohm-G. :D

 

[joeschmuck]

Huh ... why not to do it simple like this ?

Code:

ls -lt1 | grep ".*\.omg" | tail +3 > rmlist

Output is list of the .omg filenames without two newest files.

And now just call xargs rm < list to delete the files.

Or do it in one command ^^

Code:

ls -lt1 | grep ".*\.omg" | tail +3 > list && xargs rm < list

This will always keep 2 newest .omg files. If you will have only one or two files present, then tail +3 will return back empty output, so the list file will be blank and nothing will be removed


Or is there any catch which i am missing ? :]

Nice work, I"m sure cyberjock will appreciate the help.

 

[HolyK]

Oh, I just made a mess in my panties....it worked perfectly.
I think I need to get a grasp on when | and > and < are used. They do different things but I'm not sure of their exact limitations yet.

If i will make it simple, it can be described like this:

"|" == pipe ... "piping" is used for let say for "merging" several commands into one, so for command ls -lt1 | grep ".*\.omg" its like
"Give me a list of the files (ls -lt1) and before you throw me the results on screen, filter it via lines which contains *.omg (grep ".*\.omg)"

">" is like forwarding the output into specified files ... so command mentioned above will normally throw results on screen, but with "> list" it will save a new file containing it isntead
"<" instead of typing part of the command into command line, you will load the *needed thing* from specified file

I hope this will help a bit ... for more description just google it ^^

 

[joeschmuck]

@CyberJock

Now I'd create a script which would take 3 arguments, here is an example of my thoughts, let's say the script is named deletebackups.sh so I'd call it with deletebackups [path] [file extension] [number of most recent files to save]

Code:

deletebackups /mnt/pool/backups omg 2

In my code I would check for:
$1 is a valid path
$2 check against a list of allowed extensions to prevent a goof from taking out files you don't want removed automatically
$3 if this value is not set then set it to "2" so you can save the last two most recent files.

I would follow up with using exit codes for checks that don't work out as well.

These are just suggestions, you have the key piece of code already and could stop there, but I normally make the code a bit fancier so I might be able to use it elsewhere easily and others could use it without having to change much coding. For me this also allows me to learn a bit more about scripting.

 

[cyberjock]

So I thought I'd take a look at my script and see how it was doing. It works great when I manually run it from the command line as root. But when I do a cronjob I get an email that says:

Subject: Cron <root@freenas> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/tank/.config-backup/backup-cleanup.sh
cd: can't cd to /mnt/tank/Backups/Desktop
/mnt/tank/.config-backup/backup-cleanup.sh: cannot create list: Read-only file system
rm: list
: No such file or directory

: not found
cd: can't cd to /mnt/tank/Backups/HTPC
/mnt/tank/.config-backup/backup-cleanup.sh: cannot create list: Read-only file system
rm: list
: No such file or directory

: not found
cd: can't cd to /mnt/tank/Backups/Laptop
/mnt/tank/.config-backup/backup-cleanup.sh: cannot create list: Read-only file system
rm: list
: No such file or directory

: not found

If I log into my freenas server and type the cd command they do work. I'm 100% sure there is no typo or capitalization error.

Any ideas?

 

[paleoN]

Any ideas?

You looking for guesses or is the script posted somewhere and I just missed it?

Code:

ls -ald /mnt/tank/Backups
ls -ald /mnt/tank/Backups/Desktop
ls -ald /mnt/tank/Backups/HTPC
ls -ald /mnt/tank/Backups/Laptop

cat /etc/crontab

May also be illuminating.

 

[cyberjock]

Here's the script. I could have sworn I included it...

Code:

cd /mnt/tank/Backups/Desktop
ls -lt1 | grep ".*\.omg" | tail +3 > list && xargs rm < list
rm list

cd /mnt/tank/Backups/HTPC
ls -lt1 | grep ".*\.omg" | tail +3 > list && xargs rm < list
rm list

cd /mnt/tank/Backups/Laptop
ls -lt1 | grep ".*\.omg" | tail +3 > list && xargs rm < list
rm list

 

[cyberjock]

Here's your commands and the output:


ls -ald /mnt/tank/Backups
drwxrwxrwx 9 root wheel 10 Feb 19 09:29 /mnt/tank/Backups/

ls -ald /mnt/tank/Backups/Desktop
drwxrwxrwx 2 root HTPC 7 Mar 1 16:28 /mnt/tank/Backups/Desktop/

ls -ald /mnt/tank/Backups/HTPC
drwxrwxrwx 3 root HTPC 3 Feb 26 16:56 /mnt/tank/Backups/HTPC/

ls -ald /mnt/tank/Backups/Laptop
drwxrwxrwx 2 root HTPC 2 Jan 15 12:49 /mnt/tank/Backups/Laptop/