Can't delete SNAPSHOTS

[BlazeStar]

Using FreeNAS-9.2.1.8-RELEASE-x64

I can't delete snapshots.

In the GUI, into STORAGE > ZFS Snapshots :
I select snapshots, click Destroy, confirm.
Nothing happens, they just stay there, whatever I do it has no effect.

I want to delete them all and start fresh.

Is there a command I can put in console so I can clear them ALL at once ?

Thanks!

 

[dlavigne]

Anything related in /var/log/messages? How many snapshots?

 

[BlazeStar]

Absolutely nothing appears in /var/log/messages

I need to get rid of exactly 2048 snapshots

 

[dlavigne]

That is going to take a LOOOOOOOOOOONG time, so it probably is doing it's thing. ZFS needs to do its checks to make sure it is sane to delete a snapshot before it actually deletes it.

 

[BlazeStar]

Yeah but I only checked ONE snapshot.

And it won't go away.

And first time I tried to deleted was about 6 hours ago.

And the snapshot is only 144K


Also, even if I get to delete this specific snapshot, through the GUI it'll take forever to get rid of 2048 (click them all, etc.)

So is there a command for the shell that I can use to wipe ou ALL snapshots ?

 

[cyberjock]

There's a way to script it and do it from the CLI yourself. You'd have to google around. But I think you should be focusing on the bigger thing.... "why didn't that snapshot delete?"

There could be one of a bunch of reasons for that, and that should be your primary goal right now. Of course, we've got no error messages or anything from you so far, so for all we know it's user error. *something* should be showing up *somewhere*. In the footer, or something.

 

[BlazeStar]

well of course I've been "tailing" the message.log a lot.

the footer is the console message so I guess it's the same thing... or is it not??

Just to make sure, I just tried deleting a snapshot again (ANY snapshot) and I look at the console messages (the footer) and NOTHING happens.

So I don't know where else to look for?


All those snapshots were taken during several weeks when the only thing that was on my FreeNAS were the datasets and the shares configuration.

I was doing all kind of testing before I migrate to FreeNAS, including replication, etc.
I flashed my RAID card... I tested various permission models... I messed around a lot.

Now I'm satisfied with everything and I'm ready to dump all my datas. The setup is perfect.

But I want to start clean and get rid of all those old "empty" snapshots.

The problem is that, before posting, I did look around to find a trace or an explanation why the snapshots wouldn't delete.

The only way I thought to get rid of all those snapshots would be to destroy the volume and start a new one, but I'd like to avoid that as much as possible...


By the way, you seem to be surprised by the quantities of snapshots.

I take a snapshot every 4 hours of my /mnt/Data

I have 10 datasets in my /mnt/Data and it appears there's a separate snapshot for every dataset

So everyday, my Periodic Snapshot Task generates 10 X 6 snapshots = 60. So after 30 days, I have over 2,000 snapshots.

Should I revisit that ?

 

[cyberjock]

Normally what we see people do is have hourly snapshots for 2 days, then daily snapshots for a month, then weekly snapshots for a a year (or something like that). You basically have multi-tiered snapshots that limit the number of snapshots you have. In the example above you'd have 48 + 30 + 52 = 130 snapshots.

2000 is definitely far more than most people have, and if you were a TrueNAS customer we'd make something like the recommendation I just mentioned. ;)

 

[diedrichg]

Normally what we see people do is have hourly snapshots for 2 days, then daily snapshots for a month, then weekly snapshots for a a year (or something like that). You basically have multi-tiered snapshots that limit the number of snapshots you have. In the example above you'd have 48 + 30 + 52 = 130 snapshots.

2000 is definitely far more than most people have, and if you were a TrueNAS customer we'd make something like the recommendation I just mentioned. ;)

I hadn't thought of doing it that way. I'll need to modify my schedule asap!

 

[BlazeStar]

Normally what we see people do is have hourly snapshots for 2 days, then daily snapshots for a month, then weekly snapshots for a a year (or something like that). You basically have multi-tiered snapshots that limit the number of snapshots you have. In the example above you'd have 48 + 30 + 52 = 130 snapshots.

2000 is definitely far more than most people have, and if you were a TrueNAS customer we'd make something like the recommendation I just mentioned. ;)

Wow that's just AWESOME

That's actually exactly how it worked before on my classic CentOS + SAMBA machine.

I had a script called RIBS which would do RSYNC Incremental Backups.

And we'd have hourly's, daily's, weekly's, monthly's and yearly's

Everything would be "cleaned" except the yearly's...

The last hourly of the day would become the latest daily, the last daily of them month would become the monthly, etc...

It was just programmed that way (the script) so I didn't think about replicating that with the Snapshot Tasks

This really needs to go in your awesome guide!



But in the end, how can I get rid of those damn snapshots ?

As described above, they're just laying there, with nothing going on in the console when I try to deleted them.

I'd really like to avoid googling it around... I did try, I ended up on some articles or threads about ZFS, but none of which referred to FreeNAS so I just don't want to mess something around, and the FreeNAS GUI won't help me on this one.

 

[rs225]

You could try running the command on command line with -n so it doesn't run. See if it gives an explanation. If not, then try it without -n and see if it deletes it. I don't know if that will goof up the GUI, somebody else can chime in on that.

zfs destroy -vn pool/dataset@snapshot1

 

[fracai]

I wrote an alternative to managing the snapshots yourself called ZFS Rollup. Set it to run in cron and it'll prune snapshots according to the schedule you set. I have snapshots set to every five minutes. My script keeps them all for two hours (24 snapshots), one per hour for the last day, one per day for the last week, one per week after that. You can customize this however you want and restrict the pruning to individual datasets in case you want different retention times for each.

 

[BlazeStar]

So here's what I did :

Code:

zfs list -t snapshot

To get a list of the snapshots

Then I selected one and ran this:

Code:

zfs destroy -vn Data/Virtualization/Machines@auto-20141107.1513-4w
would destroy Data/Virtualization/Machines@auto-20141107.1513-4w
would reclaim 80K

not sure what it meant so I did:

Code:

zfs list -t snapshot

again to see if the snapshot was still there, and it was.

So I ran:

Code:

zfs destroy Data/Virtualization/Machines@auto-20141107.1513-4w
cannot destroy snapshot Data/Virtualization/Machines@auto-20141107.1513-4w: dataset is busy

So I googled around for a bit and ended up on this:
https://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6r4f/index.html

You can use the zfs release command to release a hold on a snapshot or set of snapshots. For example:
# zfs release -r keep tank/home@now
If the snapshot is released, the snapshot can be destroyed by using the zfs destroy command. For example:
# zfs destroy -r tank/home@now

The way I understand is that they suggest to release the snapshot and then destroying it.

I just wanted to check if it was safe, before I proceed releasing ALL snapshots and deleting them.

Thanks!

 

[BlazeStar]

awww yisss I figured it out

there was a hold tag on the snapshots

Code:

 zfs holds Data@auto-20141107.1513-4w 
NAME                        TAG           TIMESTAMP
Data@auto-20141107.1513-4w  freenas:repl  Fri Nov  7 15:16 2014

So I released the flag with

Code:

[root@NAS] ~# zfs release -r freenas:repl Data@auto-20141107.1513-4w

Then I could destroy the snapshots

Code:

[root@NAS] ~# zfs destroy -r Data@auto-20141107.1513-4w

I had to do it manually for all the automatic snapshots taken but at least with -r I only had to go for the volume and not all datasets.

Thanks all for your help!

 

[BlazeStar]

So concerning this

Normally what we see people do is have hourly snapshots for 2 days, then daily snapshots for a month, then weekly snapshots for a a year (or something like that). You basically have multi-tiered snapshots that limit the number of snapshots you have. In the example above you'd have 48 + 30 + 52 = 130 snapshots.

and this :

That's actually exactly how it worked before on my classic CentOS + SAMBA machine.
I had a script called RIBS which would do RSYNC Incremental Backups.
And we'd have hourly's, daily's, weekly's, monthly's and yearly's
Everything would be "cleaned" except the yearly's...

Up to now I've set up :

• Hourly snapshots with a lifetime of 1 day
• Daily snapshots with a lifetime of 1 week
• Weekly snapshots with a lifetime of 1 month

That's done.

However I'd also like to set up :

• Monthly snapshots with a lifetime of 12 months
• Yearly snapshots with a lifetime of, like, forever!!

So with the GUI:
http://cl.ly/image/1D0V170B1k2C
The longest interval I can select is 1 week!

Any way I can achieve that ?

 

[fracai]

I can post an example of this with my script if you're interested.

 

[BlazeStar]

I can post an example of this with my script if you're interested.

Yes of course I do!

And your script looks very interesting.

However I'm still curious on why the longest interval I can select would be 1 week.

 

[fracai]

I'm not sure if it's still the case, but for a while there was a bug where multiple snapshot schedules could cause problems with replication. That would mean the only use case for a weekly snapshot schedule would be where your data changed so infrequently that rolling back a week is acceptable. Honestly though, it's probably just not a use case that has been given much thought.

So, to use my script you'd want a snapshot something like:

Code:

rollup.py -r -i hourly,daily,weekly,monthly,yearly:0 tank

The "-r" means look at all child datasets, not just the top level. If you have multiple datasets and want to apply different rules to each, just drop the "-r" and issue the command once for each dataset (you can specify multiple datasets at a time if they all fall into the same interval set). The "-i hourly,..." defines the interval rules to consider. The "yearly:0" means the script will keep as many as accumulate. Normally, the number following the colon is how many snapshots to keep, but I use "0" to keep everything. Looking at the script I just realized that the "0" behavior isn't documented; I should fix that.

Anyway, with this command run every hour, and a single snapshot task set to once an hour with expiration set to however long you want the yearly snapshots to last (in your case, however high you can set the expiration), you should end up with the hourly, daily, weekly, monthly, yearly intervals that you're looking for.

Until you feel comfortable with the script running unmonitored, you can run the command on your own and add the "-t" flag to see what snapshots would be deleted (the "zfs destroy <snapshot>" command won't be run) and "-v" to see which snapshots are being kept and why (you'll see columns indicating which intervals the snapshots satisfy).

I've been running the script for around a year in its present form and haven't suffered any data loss. I've had some feedback from other people that are using replication and it sounds like it's been working well for them. The only issue I'll see is that sometimes when my machine is under very high load, I'll get emails indicating the script couldn't find any snapshots to destroy. I assume this is a timeout in one of the zfs commands. I should add the ability to silence those errors as the script will catch any extra snapshots during the next iteration.