File system reflects data from months ago even after snapshot rollback

[Sendai Iicx]

Earlier today a client called saying there's data missing on their server.

As I was on-site elsewhere already, I told them I'd remote (SSH) in and restore the file system to the latest₁ snapshot from the previous day.

Having done so I called them back to check if everything was fine, it wasn't as nothing had changed.

I initially thought that there was a problem with CIFS showing a clone (confirmed there are none on the system) or an earlier snapshot, but looking at it further I discovered that what they're seeing on their machines via CIFS is the same as what see on the file system via ls. It showed a file system from the 28th of August. I've verified that all the files and folders that had been created, moved, and deleted since then are in the hourly snapshots. As I mentioned earlier, I performed a rollback to snapshot that contains the files and folders as they should be, but that rollback did not change what was visible on the filesystem.

I'm currently doing a long test on the drives to ensure there's nothing wrong physically, and after that I'll restore the data in the latest known good snapshot to the file system manually₂, but I'd just like to know if anyone has seen something similar?

1. Verified to contain the files that were worked on at the end of the day via

Code:

zfs diff dataset@auto-date.1600-3w dataset@auto-date.1800-3w

and forced it to delete the first two snapshots of the day so as to allow rolling back to an earlier snapshot.

2. Likely through

Code:

rsync -a --delete --progress --size-only /source/ /destination

testing it with –dry-run first of course.

 

[dlavigne]

Did you figure this out?