Rolling back a snapshot deleted a months worth of snapshots

[bjb]

I have a dataset that I use for storing my media (photos, videos, etc). I have two automatic snapshot tasks configured.
1) Snapshot that occurs daily and expires after 1 month
2) Snapshot that occurs every 4 weeks and expires after 1 year

Today, I did something funny to a couple of files and wanted to revert the changes. So I rolled back to the most recent snapshot (it was this morning). After running the operation to roll back, I am now back to a snapshot that was taken manually, and it was taken a month ago. As a result, I have lost some really important photos.

Naturally, this is the first time I've performed the operation through the FreeNAS UI, and something has gone terribly wrong.

Before performing the snapshot, I could filter my list of snapshots by typing MediaStorage into the search bar, and it gave me a filtered list of 50 or so snapshots.
I then selected the most recent snapshot from the list and selected "Rollback". After performing that operation, all of my automated snapshots have disappeared from my list and I've lost Media that was recently (in the last couple of weeks) added to the storage pool.

Then, after some short time, I see this automatic snapshot show up, which doesn't make sense. I don't know when or why it was taken:

So I have a few questions:
1) Can anyone help me understand what happened?
2) Where can I find the logs of the commands I executed? I would like to see what time the commands were run, and what the commands actually were, etc.
3) Where did the auto snapshot shown above come from? It would appear to have been taken today (March 22nd), though I'm not sure at what time. But again, I'm missing files that were added even last week. So it seems like that snapshot was automatically taken after the Rollback?
4) I'm hoping that someone can make sense of this and that I didn't actually lose all of my data. Is it possible to recover or undo any of these operations? I know, in general, you can't undo a Rollback. But maybe, just maybe, something funny has happened here and all of those snapshots weren't really removed.

Oh, one other detail that might be relevant. I expose this storage pool to an Ubuntu server over nfs, which had an active nfs mount at the time of the Rollback. Hopefully I didn't break something by doing that.

Thanks in advance for any help

 

[fracai]

1) I suspect that when you filtered the snapshots, you filtered out all the recent automatic snapshots. The "most recent" one at that point was the manual one from a month ago. In order to rollback, all of the more recent snapshots must be destroyed.
2) I'm not sure. You may find these by running "zpool history"
3) The 1553 is in 24 hour time, so 3:53 PM. I think the auto snapshot tool runs every minute and works by creating a new snapshot if one doesn't exist for the desired interval. In this case you destroyed all the auto snapshots for the past month, so a new one was created immediately.
4) Unfortunately, I don't think you can recover anything. Rolling back destroyed your recovery points.

Were there any warnings when you clicked to rollback? I'd think there should be a warning that any intermediate snapshots will be destroyed. Maybe even listing what those are.

Rolling back shouldn't have broken anything, but it could have caused some inconsistencies for anything looking at the data.

 

[garm]

I’m sorry for your loss and although I don’t think there is any recovery other than restoring from backup, I hope you find a way to restore your data.

On a future note, snapshots are way to powerful to play with without understanding them completely. And having said that, you should know that rolling back a snapshot restores the entire filesystem to that point in time. If you mess up a few files you should instead simply browse the snapshot and recover your file. https://www.ixsystems.com/documentation/freenas/11.2/storage.html#browsing-a-snapshot-collection

And I stress the point of backup once more. ZFS parity and snapshots are not backup, you can still loose data due to a million reasons and having a solid backup strategy is crucial in reliable long term data storage

 

[Mr. Slumber]

Sorry for your troubles! Maybe in the future the link below can help you if you just want to restore some data of a snapshot and not roll back the entire snapshot. So just add the auxiliary parameters, mount the snapshot you want and copy all/some parts back. It's awesome :)

How to access ZFS snapshots over SMB

 

[Heracles]

Hi,

Indeed, to rollback a snapshot is a destructive action : it destroy every changes in the filesystem since that snapshot and it destroys every snapshot taken after the rolled back snapshot.

The non-destructive way for using snapshot and do what you needed here was not to rollback the desired snapshot, but to clone it.

What you should have do is :
--Clone the snapshot containing the data you wished to restore
--Extract these data using any file level access you wish
--Copy these files to the main instance of your data
--Destroy the clone

That procedure is non destructive : all snapshot keep existing and the main instance of your data is untouched but for the few files you wished to restore.

FreeNAS is very powerful so using it means you can shoot yourself in the foot with a bigger gun...

Sorry for you data lost and hope lesson is learned for both yourself and other people who may read this...

 

[bjb]

All,
Thanks for the replies, and the tips for the right way to recover files. I unfortunately learned about that while trying to see if I could recover from the error I had made. Luckily, I was able to find all the files that I lost, so no real harm done in the end. On the flip side, I've taken my lumps and learned a couple of painful lessons. I'll make a replica set on another server, and I've learned the right way to recover files.

What's left now is for me to find out how this happened. I know rolling back a data set is destructive. In this particular case, it wouldn't have mattered since no changes have been made for a couple of weeks, and I have a daily automatic backup in place. For the life of me I can't figure out how all of the automatic snapshots were lost.

In the UI, the only snapshot that I am even allowed to try to roll back to is the most recent:

See here, no option to roll back:

But here, there is the option to roll back:

When I performed the operation, there were 30 or so snapshots, most of them were the daily automatic ones. After rolling back, all automatic snapshots were gone, and I was left with about 7 manual ones. The only way that should have been able to happen is if I:
1) manually deleted every automatic snapshot
2) rolled back to the manual snapshot that I ended up with in the end.

So, in spite of me doing it wrong in the first place, I still don't understand how I made this happen. Can anyone shine any light, or give any ideas? This has naturally shaken my confidence and I want to understand it.

@fracai, ^^ is to your point specifically. How would I have rolled back to the old snapshot and inadvertently deleted the ones I wanted, when the web interface doesn't allow for it?

 

[fracai]

That is interesting. Maybe there's a bug that displayed the rollback option when it shouldn't have been available?
Or a bug that selected the wrong snapshot and supplied the flag to destroy intermediate snapshots? That sounds somewhat too much of a combination of bugs, but I suppose it's possible.

Can you post the output of running "zpool history" at a command line?
Maybe run: "zpool history > zpool-history.txt" and attach the result?
Review it first to be sure there's nothing sensitive in the listing.

 

[bjb]

@fracai THANK YOU for the tip :) I should have seen that in the man pages.

The log file does indeed show the command that rolls back to the snapshot where I ended up:
zfs rollback -r myHomeNAS/MediaStoragePool@manual-20181214-priortodedup

I did get a pop up box in the UI asking me to confirm, but that message was expected since I was doing a roll-back and I knew that it would be destructive. But I am very confident that I didn't select the manual-20181214-priortodedup snapshot as the one to roll back to, I selected the snapshot that was at the bottom of the list (most recent). Additionally, I've tried on a dataset on that same machine that has the same auto-snapshot schedule, and it does not even give me the option to roll back to any snapshot other than the latest. I also created a temporary dataset, made a few snapshots, and did a roll-back just to see. Again, the UI would never allow me to select Rollback on any snapshot other than the most recent.

I really can't prove it, and I'm not trying to place blame or anything. But it does seem like there was an error where I picked a recent auto-snapshot to roll back to, but somehow the system selected an older one. I tend to assume that there was some sort of user error on my part, but I can't get the UI to allow me to do the destructive behavior that has happened. So, hands in the air I guess?