A few questions about snapshots and replication

[hansch]

Snapshot don't use much space, that is correct. At least if you consider the following, for instance you copy a large file like 10GB, the data it occupies will be roughly 10GB. When you take a snapshot, the snapshot will not contain the file and take up 10GB or so. No, the snapshot will only contain the address of all the blocks pointing to the file. It could be a few 100 bytes or a few Kbyte.

When you replicate a snapshot, the "zfs send" will go through all the blocks defined in the snapshot and the block themselves are going to be copied over to the where "zfs receive" is pointing at.

If you have gazillions of snapshots, your volume could be filled up.

I suspect in your case you may have replicated one dataset and snapshots to 2 different locations. Both location containing the same data, unless you started replication but toward the end never finished and you might end up with a resumable replication.

Yes, that is the theory. That they don't use much space.
This system has been replicating for months without issue, so I don't think it contains two copies.

4 weeks of snapshots, 12 snapshots a day, is about 336 snapshots. Hardly a gazillion.

The most important questions for me are not yet answered. I am not yet ready to just delete the snapshots as I proposed above because this is a production system.

 

[Heracles]

Hey Hansch,

When you configured your replication, did you asked for it to delete expired snapshots ? If you did not, then they will accumulate...

 

[hansch]

I have not seen that option. I assume that if snapshots live 2 weeks, they are deleted after 2 weeks.

I will log in and see if they are still there.

Edit: I did and it seems snapshots from last August are still there...

How can I limit the age of snapshots on the replication system? The max age is 4 weeks on the original system.

Can I delete the oldest snapshots without data loss (of the last 4 weeks)?

If I delete a newer one, say, of yesterday, and keep the one of today and the day before yesterday, will I still have the data of today and 2 days ago?

What the heck, I deleted every snapshot before 1.1.2020. 1300 of them or so, it is still running.

 

[zamana]

I'm facing this problem too.

Due to some long replication tasks that didn't complete, I have a bunch of snapshots older than the 1 week limit that I did set.

What's the best and most secure way to get rid of them?

Thanks.
Regards.

 

[Apollo]

I'm facing this problem too.

Due to some long replication tasks that didn't complete, I have a bunch of snapshots older than the 1 week limit that I did set.

What's the best and most secure way to get rid of them?

Thanks.
Regards.

First you need to figure out why the replication didn't complete.
Normally, when a replication is starting, ZFS place a hold on the deletion of the snapshot on the source so that even if you have a lifetime of 5 minutes and you are transferring TB of data, the replication will still be underway. Only when the replication has completed will the hold be removed. If lifetime as expired, then ZFS will destroy the snapshot, if not it will be preserved until its times comes.
With one snapshot in Hold, every newer snapshots will be placed on Hold I think, until the original hold is resolved.
The most secure way to deal with your case is to finalize the replication and exerything should return to normal on its own.
If you were to destroy the snapshot that failed to be replicated, you will have to destroy it on the replicated side as well otherwise you won't be able to replicate back to it in most case.

You can delete the snapshot manually through GUI or via CLI but you want to be careful as you can easily mess things up.

 

[zamana]

Ok. Thanks for the explanation.

The replication is running right now and I need to wait until it's finished. Typically it takes almost 3 days to complete (when running from scratch, that's the case right now).

Thanks.
Regards.

 

[Apollo]

Ok. Thanks for the explanation.

The replication is running right now and I need to wait until it's finished. Typically it takes almost 3 days to complete (when running from scratch, that's the case right now).

Thanks.
Regards.

So you are starting a brand new replication on a volume that doesn't contain data?
If so, and for subsequent replication, Freenas 11.3 supports resumeable replication. I have used it via CLI once when it was introduced with ZFS and it is a nice feature to make use of. It is a bit tricky via CLI, but with Freenas 11.3, when enabled in the replication task it should take care of it for you.

 

[zamana]

So you are starting a brand new replication on a volume that doesn't contain data?

Yes.

If so, and for subsequent replication, Freenas 11.3 supports resumeable replication. I have used it via CLI once when it was introduced with ZFS and it is a nice feature to make use of. It is a bit tricky via CLI, but with Freenas 11.3, when enabled in the replication task it should take care of it for you.

Nice to know.

Thanks!