How to Avoid Accidental Deletion of Dataset with Replication

[Johnny Fartpants]

I've mentioned this at least once before in another post and either got the feeling nobody else felt like this was an issue or they didn't quite appreciate the implication so thought I'll make a specific post about it so you guys can either put my mind at rest or we can conclude this is an issue and look to raise it.

So in the scenario where somebody has two FreeNAS systems with A replicating to B (lets say every hour for arguments sake). If the administrator of the box accidentally foolishly whatever you want to call it deletes a dataset then during the next replication window that same dataset is also deleted from system B meaning that data is lost forever.

Now the morale of this story is be careful when deleting datasets but what if someone managed to gain unauthorised access to the system and maliciously started deleting datasets knowing the impact it would have then not only have you lost your primary data but also your backup data.

Personally I would like some sort of option I could check in replication to say never delete datasets on the replica and that I will control that myself from the replica systems on a manual process thus avoiding this potentially catastrophic outcome.

I hope I've explained this well enough and like I said would be really interested to hear your viewpoints as its something for a while now that has played on my mind.

 

[Chris Moore]

This is an advantage of using rsync. As it is working on files, you can have snapshots as a second level of backup. Then it is just a matter of rolling back the snapshots. .

 

[Mlovelace]

You could submit a feature request to purge associated replication tasks with dataset deletions, much like the option that pops up to purge associated shares with dataset deletions.

 

[Johnny Fartpants]

This is an advantage of using rsync. As it is working on files, you can have snapshots as a second level of backup. Then it is just a matter of rolling back the snapshots. .

Thanks Chris. Its a shame though don't you think as replication is so good. Just one little fix and its perfect.

 

[Johnny Fartpants]

You could submit a feature request to purge associated replication tasks with dataset deletions, much like the option that pops up to purge associated shares with dataset deletions.

Thanks I will. Where do you submit feature requests?

 

[cobrakiller58]

Isn't this something that should be considered when planning your back-up strategy? I have a copy offsite that is offline so the deletion of a dataset would not affect that particular copy.

 

[Mlovelace]

Thanks I will. Where do you submit feature requests?

https://redmine.ixsystems.com/projects/freenas/issues?set_filter=1&tracker_id=2

 

[Johnny Fartpants]

Isn't this something that should be considered when planning your back-up strategy? I have a copy offsite that is offline so the deletion of a dataset would not affect that particular copy.

So do you periodically offline and online that backup machine? I have backups running all the time on multiple servers so would be a really pain to keep off-lining and on-lining them. Also is kind of defeats the object/benefit of async replication. Would be much easier to have a feature that just says don't delete datasets on replica system.

 

[cobrakiller58]

My data is safe I'm not too concerned if it takes 2 extra steps, does anyone use tape libraries here? how often are they deleted?

 

[Mlovelace]

My data is safe I'm not too concerned if it takes 2 extra steps, does anyone use tape libraries here? how often are they deleted?

We have a tape library here for archiving, iirc it's an IBM TS4500 series. One of those full rack size tape robots, it holds 100s of petabytes worth of tape. Sometimes the old tech is still viable depending on the task.

 

[cobrakiller58]

We have a tape library here for archiving, iirc it's an IBM TS4500 series. One of those full rack size tape robots, it holds 100s of petabytes worth of tape. Sometimes the old tech is still viable depending on the task.

The storage capacity of them is hard to compete with as far as I have seen, so in this scenario you would be able to recover the dataset from the tape back-up right?

@Johnny Fartpants I've sure seen enough posts on here about accidentally deleted datasets to agree that a little extra protection would be a good idea.

 

[Johnny Fartpants]

The storage capacity of them is hard to compete with as far as I have seen, so in this scenario you would be able to recover the dataset from the tape back-up right?

@Johnny Fartpants I've sure seen enough posts on here about accidentally deleted datasets to agree that a little extra protection would be a good idea.

Thanks for the support. It just seems to me that snapshots and replication can be part of a good backup strategy but we could do with fixing this little gotcha which Im not sure that many people are aware of.

zpool snapshots when they arrive could help with this but I would rather something like I've suggested above.

 

[MatthewSteinhoff]

On one hand, I kinda feel like this isn't a problem because if you have that kind of access to FreeNAS, there are plenty of ways to maliciously destroy the data.

On the other hand, as a guy with two decades of IT experience, I know how easy it is for a SysAdmin to shoot himself in the foot and can easily see someone (ie: me) deleting the wrong dataset and losing both the original and replicated copy.

The most elegant solution I can think of would be able to set a retention time on the replication target such that, when the original goes away, the replicated copy sticks around for a specified unit of time along with an email notification...

"Dataset RarelyUsedButImporantStuff was deleted from PrimaryHost but will be retained on ReplicatedHost for 14 days and is scheduled to be deleted on Month Day, Year. To free that space immediately, log into ReplicatedHost and click the 'Delete Now' button under 'Replication Tasks'."

But, yeah, thanks for bringing this to our attention, @Johnny Fartpants. Totally worth a feature request.

Cheers,
Matt

 

[Johnny Fartpants]

https://redmine.ixsystems.com/issues/73270

 

[guermantes]

Personally I would like some sort of option I could check in replication to say never delete datasets on the replica and that I will control that myself from the replica systems on a manual process thus avoiding this potentially catastrophic outcome.

I'm confused. Isn't this why one keeps several snapshots for a long time, and in combination with leaving unticked the checkbox 'Delete stale snapshots on remote system:' your dataset will remain in the replica?

 

[Johnny Fartpants]

Nope. No matter how many snapshots you have or what boxes you have or haven’t checked if you delete a dataset on the primary the next time replication runs your dataset and all it’s snapshots will go from the replica. Feel free to try on a test system I know I have tried every possibility.

This is why it needs addressing because I bet there are lots of people out there like you that don’t actually realise this.

 

[pro lamer]

don’t actually realise

I guess the confusion comes when not remembering that dataset is not a snapshot...

Edit: or rather that snapshots are inside datasets. Even a recursive snapshot means a snapshot in every dataset and not a supersnapshot spanning across all the datasets...

Sent from my phone

 

[linchesterBR]

On one hand, I kinda feel like this isn't a problem because if you have that kind of access to FreeNAS, there are plenty of ways to maliciously destroy the data.

On the other hand, as a guy with two decades of IT experience, I know how easy it is for a SysAdmin to shoot himself in the foot and can easily see someone (ie: me) deleting the wrong dataset and losing both the original and replicated copy.

The most elegant solution I can think of would be able to set a retention time on the replication target such that, when the original goes away, the replicated copy sticks around for a specified unit of time along with an email notification...

"Dataset RarelyUsedButImporantStuff was deleted from PrimaryHost but will be retained on ReplicatedHost for 14 days and is scheduled to be deleted on Month Day, Year. To free that space immediately, log into ReplicatedHost and click the 'Delete Now' button under 'Replication Tasks'."

But, yeah, thanks for bringing this to our attention, @Johnny Fartpants. Totally worth a feature request.

Cheers,
Matt

That's exactly the feature I'm looking for on FreeNAS. This is the only way to avoid replicate a "snapshot deletion" and, at same time, not filling the Backup Server indefinitely.

Where can we to request this feature?

For now, is there some script we can write on Backup Server to delete old snapshots without depends on "Delete Stale Snapshots on Remote System"?

Thanks in advance.

 

[Johnny Fartpants]

I have filled a feature request which you can follow above if you like.

Regarding your script idea on the backup then this wouldn't solve the issue as it doesn't matter if you check "Delete Stale Snapshots on Remote System" or not if you delete a dataset on your primary at next replication it will be deleted from your backup system.

 

[linchesterBR]

I have filled a feature request which you can follow above if you like.

Regarding your script idea on the backup then this wouldn't solve the issue as it doesn't matter if you check "Delete Stale Snapshots on Remote System" or not if you delete a dataset on your primary at next replication it will be deleted from your backup system.

In addition to your request, I believe that the suggestion of @MatthewSteinhoff regards retention is also very important to avoid full fill the server.