Replication TN13 what happens to deleted datasets on replica system?

[Johnny Fartpants]

Hi All,

Historically when using replication if you either on purpose or mistakenly deleted a dataset on your primary systems when replication next ran it would delete the dataset on the secondary system. I was never a big fan of this and would have liked to have to actually delete the dataset on the secondary myself just incase it was a mistake. I know you can tell the primary to never delete snapshots but then you end up will millions so not ideal either.

Anyway it appears the behaviour is different and rather strange in TN13. If indeed you delete a dataset on the primary system the next time replication runs its appears to delete all but the most recent snapshot leaving the dataset in-place. At first I thought this was brilliant and exactly the behaviour I wanted however although the dataset is indeed there you can't access it. You can't cd to it on the server, you can't mount it but if you do zfs list its there and consuming disk space. I've even tried to clone the dataset and promote but still can get access to the data. So how would you actually get this data back if deleting the dataset on the primary was a mistake?

Not sure if this is a bug or I'm missing something but I would really appreciate your thoughts if any of you have any experience with this.

Thanks

 

[Johnny Fartpants]

Update: It appears that if you delete a dataset that is being replicated then all but the most recent snapshot is deleted from the replica system. The dataset remains visible in the UI but you can't actually access it via CLI etc. To access the data you need to navigate to the last remaining snapshot, clone it and then promote (if needed) in order to access the data.

Just incase anyone needs this for the future.

 

[Doug183]

Your post just answered my primary set of questions but brings up a a few more.

1) In the documentation, it says that on the primary system, you can not delete a snapshot that has been replicated. Does that mean "you shouldn't" or that it's "prohibited" in software?

To access the data you need to navigate to the last remaining snapshot, clone it and then promote (if needed) in order to access the data.

2) You are referring to doing this on the "replicated system" correct?

 

[Johnny Fartpants]

1) In the documentation, it says that on the primary system, you can not delete a snapshot that has been replicated. Does that mean "you shouldn't" or that it's "prohibited" in software?

Where does it say that please?

2) You are referring to doing this on the "replicated system" correct?

Yes

 

[Doug183]

Ok, maybe I am confusing clones and replication. Its says it here in this document about clones. 2nd paragraph.

When a snapshot is cloned, an implicit dependency is created between the clone and snapshot. Even though the clone is created somewhere else in the dataset hierarchy, the original snapshot cannot be destroyed as long as the clone exists.

https://docs.oracle.com/cd/E19253-01/819-5461/gbcxz/index.html

 

[Johnny Fartpants]

Yes this statement is correct hence why you need to promote the clone to make it its own separate entity.

 

[Doug183]

I am going to ask a question based on a workflow/problem I am having.

Setup:
I am a small company with a lot of very large digital media files (4K resolution Quicktime files, so a single file can be from ~50GB up to ~750GB). I have built a big ZFS server which currently has 5 independent vdevs, and each vdev is set up with 10x18TB Hard Drives in RaidZ2. (Yes, that's 50 drives.) I have been told this isn't the most efficient use of drive space or for speed, however, it does permit the most data secure method.

Data on 4 out of the 5 vdevs, rarely changes. What might change infrequently (a few times over 6 months) is a file name alternation or perhaps a folder name is changed. On a rare occasions, a file is deleted.

Goal
I need to back these drives up to an additional set of drives that will sit on a shelf. (FYI, we are just not ready to pay for another live server somewhere, but that might be coming in the future.). Right now it is a stop gate emergency recovery set up in addition to the LTO-8 backup system I already employ.


Question/Issue:
I understand the first backup will take time. On subsequent disaster backups of one vdev, I want to avoid a secondary long backup times. Currently on my original drives, if a folder structure changes or files are renamed, and then I use rsync to perform the backup, rsync will delete and copy data again to the disaster backup vdev. I think replication avoids this, correct? I think with replication of a snapshot, the unchanged blocks aren't copies, just the file structure will be copied, this avoiding the rsync issue, correct?

 

[winnielinnie]

I have built a big ZFS server which currently has 5 independent vdevs, and each vdev is set up with 10x18TB Hard Drives in RaidZ2.

Data on 4 out of the 5 vdevs, rarely changes.

On subsequent disaster backups of one vdev, I want to avoid a secondary long backup times.

You're using the term "vdev" but it sounds like you're describing pools.

In your first sentence, are you saying that you have five separate pools, each comprised of only one vdev; or rather, you have one massive pool comprised of five vdevs?

Currently on my original drives, if a folder structure changes or files are renamed, and then I use rsync to perform the backup, rsync will delete and copy data again to the disaster backup vdev. I think replication avoids this, correct? I think with replication of a snapshot, the unchanged blocks aren't copies, just the file structure will be copied, this avoiding the rsync issue, correct?

Correct. Even though rsync has a "fuzzy" parameter, I find that it doesn't work that well, and I believe it can only handle a filename change inside the same directory.

 

[Doug183]

In your first sentence, are you saying that you have five separate pools, each comprised of only one vdev

Yes, each zpool is compromised of one vdev. (FYI, there is also one dataset on each zpool).


So back to my initial question/concern then.

Your initial post in this thread indicates that you can delete previous Snapshots of the original zpool even though they have been used to replicate, correct? This is as opposed to what the Oracle article states - although that article refers to clones not "replicating". So I guess I am not clear if a clone and replicate are the same process or different?

I am concerned that over time, my original zpool does not free up space as out of date snapshots can't be deleted.

 

[winnielinnie]

Your initial post in this thread indicates that you can delete previous Snapshots of the original zpool even though they have been used to replicate

That wasn't me.

Although this is a "yes and no" type of situation.

For superficial pedantic reasons, I'll use "dataset" in place of "pool", since you cannot create a snapshot of a pool. (You can create a snapshot, recursive or otherwise, of the root dataset.)

"Yes" you can delete/prune previous snapshots on the source, as long as the destination shares a base snapshot with the source. However, the answer is also "no" if the source does not share a common base snapshot with the destination. Your only choice will be to do a full replication all over again.

I am concerned that over time, my original zpool does not free up space as out of date snapshots can't be deleted.

Snapshots take up relatively little to no space as long as they're not pointing to large amounts of "deleted" records.

You can still "prune" old snapshots on the source, but you have to make sure that the destination shares a common base snapshot with the source. (The "Replication Tasks" in the GUI provides options to try to prevent this.)



Another lesser known feature of ZFS are "bookmarks". The TrueNAS GUI does not leverage this feature, so you'll have to resort to the command line.

In short, "bookmarks" work like this.

1. You use the "zfs bookmark" command to create a bookmark from a snapshot
2. If you destroy the snapshot, the bookmark remains
3. This bookmark (on the source) can be used for incremental replications if the snapshot is missing
4. This requires that the matching snapshot (of which the bookmark was created) to exist on the destination

Bookmarks will always take up zero space.

I'd only resort to using bookmarks for replications as a last resort. You could either create them manually or with a script. Even if you create many, many, many bookmarks, it won't take up any space on the source pool. But there's a chance you can fallback to them in such a situation where you're missing a base snapshot on the source pool, but you don't want to have to do a full (non-incremental) replication all over again.

 

[Doug183]

Sorry, I missed there were two different amazing people posting.

Got it - snapshots of a dataset only.

"Yes" you can delete/prune previous snapshots on the source, as long as the destination shares a base snapshot with the source. However, the answer is also "no" if the source does not share a common base snapshot with the destination. Your only choice will be to do a full replication all over again.

Got it - original snapshot can not be deleted from the Originating dataset.


So, to be 100% clear, the issue when using the replicating process over time, is that if some files are truly deleted from the originating dataset, that space can never be freed up again. (Without a complete "do-over".). And by implication, that space can not be freed up on the replicated dataset as well.

I could use-rsync to accomplish of freeing up space, but if there are some name changes or folder structure changes, I will pay a time penalty with re-copying data. Also probably a fragmentation penalty over time. Does that sound correct?

 

[Johnny Fartpants]

So, to be 100% clear, the issue when using the replicating process over time, is that if some files are truly deleted from the originating dataset, that space can never be freed up again. (Without a complete "do-over".). And by implication, that space can not be freed up on the replicated dataset as well.

No. Replication works by replicating snapshots of a dataset or datasets for that matter. Snapshots are often scheduled and automatically purged when they reach their expiration date. This purging process happens on both the primary systems and secondary (if you configure this in replication). So when you delete data it will eventually be freed up depending on when the relevant snapshots expire. For example if you had a snapshot scheduled that took a snapshot every day and expired after 2 weeks if you deleted data then approximately within 2 weeks the space would be freed and the deleted data would be unrecoverable. You can do lots of things with replication and actually make it so the backup systems snapshots never expire but then you will have to manually prune to avoid running out of space. It depends on what you want. Personally I keep my primary systems in perfect sync with my secondary systems as this is very low maintenance and gives me the desired outcome.

 

[Doug183]

Johnny,

For Replication, is the Pause, resume feature implemented? I saw you post something in 2018 and saw... some maybe implementation in 10.3, but not a lot of other posts.

 

[Johnny Fartpants]

I saw you post something in 2018

Help me out I can't remember, Im old :)