Recover data from deleted dataset.

[pawel30w]

You helped me to build my NAS back in 2020, now I need your help restoring data. I am writing this post with a heart rate of 200 bpm and totally distraught by my own stupidity...

tl;dr: Platform: TrueNAS Scale. I used zfs destroy on the dataset tank/Photos_and_Videos in TrueNAS Scale inside the Shell. I received a message along the lines of "Dataset cannot be destroyed because it is busy". However, upon accessing the dataset, it was empty and all 2TB of files were missing. The Dataset is still visible though. Can I recover my files? I don't have any snapshots or backups as this happened during my preparations to add a backup pool.

Story of me being so uncareful... I have a 10TB RAIDZ2 tank pool where, recently, I copied large amounts of data back and forth, including a 2TB collection of photos and videos stored in the dataset tank/Photos_and_Videos. Because I've been copying and modifying large zipped files for the last few days, my snapshots significantly increased in size. I tried removing the unnecessary snapshots to reduce the size in order to fit it into my new backup pool. However, I was unable to delete these snapshots via the GUI. I resorted to using the Shell and executed zfs destroy on all snapshots individually. For some reason, at the end, I did zfs destroy on tank/Photos_and_Videos, mistakenly thinking it was the last top-level snapshot. This was the OMG moment. The shell immediately came back with a message something like "Dataset cannot be destroyed because it is busy". I checked the dataset via SMB and could access it, but the files were missing. Upon accessing the dataset via Shell, it was also empty. Yet, dataset itself still is showing in the GUI with the same amount of "Used/Available Space". After doing some reading, I immediately powered off my NAS and haven't turned it back on since.

1. Considering I received the message "Dataset cannot be destroyed because it is busy" is there a chance that the dataset is still intact despite the files seemingly being hidden in some way? The message came back instantaneously so I hope it didn't really remove anything because while removing snapshots earlier it took a bit longer before the delete confirmation.

The best solution I found online was to attempt a "rollback ZFS transaction" as described in a this post. I understood that every minute the NAS is running, there's the risk of data being overwritten, so I haven't turned it back on until I'm sure of what to do next.

I'm considering a professional service. However, I'd appreciate recommendations on free options if they exist.

How can I restore the data?

 

[winnielinnie]

You probably had this child dataset inadvertently unmounted in the process of (trying) to destroy it.

If you reboot, TrueNAS may in fact properly mount everything automatically, starting with the root path of /mnt/tank/

Regardless, you can see the real properties of the dataset like this:

Code:

zfs list -o space tank/Photos_and_Videos

This is the best place to start.

 

[pawel30w]

It worked... just as you said, TrueNAS mounted everything automatically. I didn't even had to check the properties.

Many thanks @winnielinnie ! It's 2 am here in UK and I prepared myself for a night of reading about zfs.

I guess, I am one of not many lucky people that got away with playing with fire (read zfs destroy).

I am happy for this thread to be locked.

 

[winnielinnie]

For future reference, try to remain within the GUI, and go slowly. If using the command-line, especially, don't use zfs commands with root/admin privilege, unless you absolutely intend to do something destructive.

For the meantime, you should definitely create snapshots of your important datasets right now, and give them a meaningful name, such as "important-2023-08-14_07-30". (The "datestamp" is created automatically if you use the GUI.)

I'd recommend you do this as a "recursive" snapshot for your root dataset, which will create snapshots of this name for all your datasets in the entire pool.

Consider also the "zfs hold" feature, as another safeguard against accidentally destructions.

 

[artlessknave]

this is also why backups are highly recommended. a snapshot would not help if the snapshot dataset is nuked. you have to replicate those snapshots somewhere else for it to be a backup.

you can, and should, schedule snapshots though. they can help with things like ransomware and file deletion.

 

[pawel30w]

For future reference, try to remain within the GUI, and go slowly. If using the command-line, especially, don't use zfs commands with root/admin privilege, unless you absolutely intend to do something destructive.

For the meantime, you should definitely create snapshots of your important datasets right now, and give them a meaningful name, such as "important-2023-08-14_07-30". (The "datestamp" is created automatically if you use the GUI.)

I'd recommend you do this as a "recursive" snapshot for your root dataset, which will create snapshots of this name for all your datasets in the entire pool.

Consider also the "zfs hold" feature, as another safeguard against accidentally destructions.

Thanks, this is a good advice and this is what I had before I decided to wipe my backup drives <facepalm>

All was set up in-line with this forums guidelines back in 2020 i.e. Daily/Weekly/Monthly snapshots. Never used "zfs hold" but I may start to.

this is also why backups are highly recommended. a snapshot would not help if the snapshot dataset is nuked. you have to replicate those snapshots somewhere else for it to be a backup.

you can, and should, schedule snapshots though. they can with things like ransomware and file deletion.

Thanks, I have a backuppool where I replicate all important data from tank. The reason for all of this was increasing the size of backuppool because it is only as large as I need for backing up important datasets (I don't replicate dataset with all Linux ISOs)

 

[pawel30w]

Just received a quotation from one of the Data recovery companies I contacted yesterday...

• RAID Rebuild and Deleted Data Scan: £895 + VAT (Upfront)
• Recovery: £1,725 + VAT (on Success only)