Understanding Snapshots beside hourly'S

[tiberiusQ]

Dear all,

I do currently use hourly snapshots incl. replications and I think I understand how it works BUT If I wanna do eg. Daily snapshots I start to getting confused.

For example, I setup daily snapshots, start 08:00 and end 18:00 so the snapshot will occur at 08:00, does this mean the snapshot will be open till 18:00 so all changes during that period will be part of this snapshot OR does it just contain the diffs. between the first and the second day ?

I hope that somebody can clarify this - Thx & Greets

 

[sretalla]

The snapshot always "contains" the entire target filesystem as it was at the point of the snapshot.

The snapshot itself isn't a copy of the data, but rather pointers to all the blocks on the disks where the filesystem had allocated data.

Subsequent snapshots only need to point out the blocks that had changed since the last one (and reference the last snapshot also to know where to start from).

Until you destroy a snapshot, the blocks that it uses can't be available for re-use, so after you delete files, you may need to destroy snapshots to really get the space back.

I'm not sure I get exactly what you set up when you say daily snapshots from 08:00 to 18:00... perhaps you set the interval to 1 day... this would mean that some time within the time window you gave, when the last snapshot is 1 day old, a new one will be taken... probably this will always happen at 08:00.

 

[tiberiusQ]

08:00 to 18:00 is the start and end time of the periodic snapshot but this seems to make no sense because the snapshot will be created instanteanous at 08:00....Thats why I was asking if then the snapshot will be kept open till 18:00 to capture all diffs in that time OR the snapshot will just record/capture the diffs between 08:00 first snapshot and 08:00 second daily snapshot ?

 

[sretalla]

The time window is intended to work where you have a snapshot interval shorter than the window... like every 1 hour.

If you set the interval longer than the window, it can only happen once in the window.

 

[tiberiusQ]

Based on that fundamental confusion I still run only hourly snapshots during working hours with an retention time of 1 year, which is crazy.....SO it is time to understand the snapshot logic beside the hourlies to be able to setup a more ressource friendly snapshot plan....

 

[sretalla]

Hourly snapshots for a year is indeed a bad plan.

Hourly for a few weeks is normally enough, then either use backups or replication to keep them longer.

You could also look for the pruning script from @fracai https://www.ixsystems.com/community...napshots-similar-to-apples-timemachine.10304/ which may help you to keep it under control without needing to shorten the retention on the hourly job.

Remember that snapshots are not your backup... they are just a tool that can help you to have a backup executed well. (and maybe to do some quick file recovery in the case of accidental deletions)

 

[tiberiusQ]

I try to ask differtly- If I setup a daily snapshot start and end 08:00 - 18:00 so the snapshot will be created every day at 08:00. between which period the snapshot will grow in size and why ?

 

[sretalla]

Snapshots don't change in size. They are a frozen point in time.

Perhaps you could try to explain what you want to happen and we can guide you on how to get to that.

 

[tiberiusQ]

From the documentation: For example, a snapshot where no files have changed takes 0 MB of storage, but as changes are made to files, the snapshot size changes to reflect the size of the changes.

So in my szenario is it between 8 and 18:00 or between 8 and 8 from day to day ?

 

[zenon1823]

SIDEBAR NOTE:

So in my szenario is it between 8 and 18:00 or between 8 and 8 from day to day ?

18:00 is 6PM not 8PM.

I too found the snapshot configuration logic confusing when I first went to set it up for daily and longer frequency. I'm going to approach the answer slightly differently. When setting up daily snapshots, the window is somewhat meaningless (to reiterate what sretalla said) the window has meaning when snapshots have an interval less then 1 day. ie if you have hourly snapshots they are taken hourly withing that window.

For example I wanted snapshots to run every day at 1AM and be retained for 2 weeks. I initially thought I could put a frequency of 1 day, from 1AM -2AM, but this caused the actual snapshot execution time to wander slightly every day, or even miss a day if it wandered to much.

So the configuration I use to have a reliable snapshot every day at 1AM is to use a 12hr frequency. this ensures a snapshot is waiting to happen as soon as 1AM hits every day. see below:

 

[sretalla]

From the documentation: For example, a snapshot where no files have changed takes 0 MB of storage, but as changes are made to files, the snapshot size changes to reflect the size of the changes.

No. Don't think of it like that.

After the snapshot, the contents of the disk can change, which can change the size of the snapshot in the sense that the snapshot requires that some blocks on the disk are reserved in order for the snapshot contents to remain the same (which they must).

The "contents" of the snapshot will not change and the size of those "contents" will also not change. The space that you can't use for new files on the disk because you're keeping that snapshot can change based on your changes to files that were in that snapshot. None of that relates in any way to the time window you specified.

 

[tiberiusQ]

Either I think too complicated or daily snapshots are the most complicated things ever....
Ok. the implemenation start and end is bad and confusing if you do not setup hourly's, right ?
The snapshot will not be kept open between if you setup eg. daylies...so if you do eg. a reboot during that time it has no impact to the snapshot, right ?
The snapshot size will just reflect the size since the last snapshot, but do not contain them, right ?
If this is true, and all I wanna do is to realize a trivial daily fuckin ;-) snapshot I just have to set it up and ignore start and end including working hours because the snapshots does not contain the start and end diffs, furthermore it is just what it is, a point in time snapshot of that dataset?

 

[sretalla]

Ok. the implemenation start and end is bad and confusing if you do not setup hourly's, right ?

I can agree that it can be confusing since they allow the interval to be bigger than 24 hours, but there are logical reasons for that which might apply in some cases where people understand what they are doing well.

The snapshot will not be kept open between if you setup eg. daylies...so if you do eg. a reboot during that time it has no impact to the snapshot, right ?

Snapshots can't be kept open, they are a point in time only.

The snapshot size will just reflect the size since the last snapshot, but do not contain them, right ?

I think you mostly understand... it's more the space required by the snapshot to make sure you can see all the files that were there when the snapshot was taken.

If this is true, and all I wanna do is to realize a trivial daily ****in ;-) snapshot I just have to set it up and ignore start and end including working hours because the snapshots does not contain the start and end diffs, furthermore it is just what it is, a point in time snapshot of that dataset?

If what you want is to have hourly snapshots that are kept for a while and then some of those (1 per day) are kept for some weeks, then some of those (weekly) are kept for months... look at the pruning script I referenced from @fracai and just keep your hourly snapshots running and let the pruning keep them under control.

Remember snapshots are not a backup. If you have a pool problem, a snapshot can't bring your data back.

 

[tiberiusQ]

This makes me curious, why you recommend me those scripts, why not simply setup eg. different snapshot tasks(hourly, daily, weekly, etc.) and replicate them to a different zfs pool or different freenas What I already do, which works great, it is a backup! As I said, the only thing I want to optimise is the snapshots but I am stil be a little confused about all beside the hourlies. I think I have to test that....for my self.

 

[sretalla]

You can do it as you suggest. The script just makes it a little simpler to manage as you only need to set hourly snapshots with a really long retention and the script will handle deleting snapshots that you don't require along the way. You can also manage your backups like that if you're replicating to a backup server (maybe with a different pruning schedule).

 

[tiberiusQ]

Nice threat but my main question is still not solved, It's pretty intresting because nowhere you will find informations beside hourly snapshots incl. the official documentation.....If I solve it I will share In a simple way, how it should be !

 

[sretalla]

So you want hourly snapshots (you said you have them). You keep those for a year (not sure if that's a good thing, but OK if you have the storage for it).

You want daily snapshots (meaning 1 snapshot per day), so you need to say when to take that snapshot (set the time window to at least one minute long at the time when you want the snapshot to be taken and set the interval to anything shorter than or equal to 24 hours).

Alternatively to get daily snapshots, you can mark one of your hourly snapshots each day to be on hold (https://www.thegeekdiary.com/understanding-holding-a-zfs-snapshot-feature/). This concept is already mentioned in the script thread as an alternative to the script.

As a further alternative, you can use the script I mentioned to remove the hourly snapshots (except for 1 per day, 1 per week, 1 per month, etc. as they get older).

What part is still not clear?

 

[tiberiusQ]

Yes, I do currently only use hourly snapshots because I think to understand them most and I do replicate them to my other freenas boxes and it works great and yes I still do have the storage and ram for it but I think it is time to understand the details or behaviour of snapshots beside the hourlies to be able to setup a more resource friendly snapshot plan- eg.

hourly (mo-fr, 08:00-18:00) keep for 2 weeks
daily (mo-fr, 08:00 - ?) keep for 2 month
weekly (fr, 08:00 - ?) keep for 4 month
monthly (fr. 08:00 - ?) keep for 1 year

 

[sretalla]

08:00 - ?)

08:00 to 08:01 would be enough.

You might want to consider staggering them to 08:00, 08:01, 08:02, etc... just so you give the system time to process each of them on the days when they will all run.

 

[Fredda]

hourly (mo-fr, 08:00-18:00) keep for 2 weeks
daily (mo-fr, 08:00 - ?) keep for 2 month
weekly (fr, 08:00 - ?) keep for 4 month
monthly (fr. 08:00 - ?) keep for 1 year

Just a small remark, apart from the hourly snapshot I would put snapshots into the night. Of course this depends on your usage scenario.
For my it's:
hourly: "Oh, my code was just working fine 3 hours ago, why the heck is it not working now?"
all others: "Oh, I accidently deleted the wrong directory when cleaning up"