Looking for last ditch data recovery tips (ZFS)

[kwyjibo]

Firstly - Mea Culpa - I broke it, I was stupid and I am now trying to recover anything I can.

I have ZFS pool that I migrated to another microserver (Freenas 9.3 to Freenas 9.3) all was fine. Then comes the stupid bit. I was playing around with virtualisation (yes that dirty word) I had passthrough set up on the other box and it was working well to the point that I imported the pool into the virtual environment to see how it would perform. Which was great until I started playing around with passing through the GPU to another VM and kernel panic'd the box. When I got back online the zfs pool instantly reported issues. My response was to kill the VM environment and move the disks back to physical. Since then I have been unable to import the pool.

Quick overview of current status

4 x 2TB Disks single RaidZ1 pool (bfv1) -- All disks appear to be fine (no errors no smartctl issues)

zpool import tells me that the pool was last used on another system so use -f

zpool import -f fails tells me I need to recreate from backup (no backup .. yes I know)

zpool import -fF - fails - I/O error - advises Destroy and re-create pool

zpool import -fFX - runs for almost exactly 24hrs then causes a reboot - Have tried this under Freenas / FreeBSD & Opensolaris exact same behaviour

zpool import -fFXV actually works but doing anything with the pool fails ie zpool clear doesn't work

zdb -e - Runs and prints a bunch of details but then hangs with "Can't open pool, Input/Output error"

zdb -u(or uuu) -e - Fails with can't open pool Input/Output error

zdb -uuu -t 20876738 -e pool --- Works and prints the Uberblock (that being the last transaction ID I can see from zdb dump)

Labels look fine on all disks - Strangely all disks are paritioned the same way yet 2 disks have labels under the root ie ada1 and ada3 the other 2 are under ada0p1 and ada2p1 - Assume this is just because the pool is using /dev/gptid paths for the other 2 disks.

I am out of ideas and have tried every suggestion returned by google for ZFS pool recovery in the last 10 years! If anyone has any final suggestions please let me know. Worst case I know I can run photorec to pull the family photos off which is all I really care about.

Thanks in advance to anyone with ideas.

 

[cyberjock]

Actually, you'll find photorec isn't going to get you anything. At least, I have yet to see a program like photorec work. The problem is that ZFS writes the data in a method that pretty much requires the ZFS metadata, which photorec can't use. You are welcome to try it though, but now would be a *really* good time to look on all your disks laying around to see if any of the data exists anywhere else, because you've got few options.

Nice name.. back to the Simpsons. ;) I might be the only one that hasn't watched the simpsons in 10+ years but remembers that one part of that episode. LOL.

If you are desperate you can call iXsystems and do a 3 hour block of time for zfs recovery. That will likely end up in my lap. It's not cheap (you could have likely built new iron to store FreeNAS on for the same price) but if you want me to give it a shot, feel free to call.

 

[kwyjibo]

Thanks for the info no other disks apart from dead ones pulled from the raid unfortunately, I actually tested photorec early in my recovery attempts and successfully pulled files so I tucked that away as a last resort. I am not very excited about sorting and renaming thousands of files. I do have 90% of the photos backed up to cloud storage so I am not totally lost.

Kills me knowing the data is there just out of reach.. Live and learn I guess.

 

[joeschmuck]

Once you have recovered your photos, I recommend placing a copy on DVD-R media for safe keeping. This is a practice I live by.

 

[Naz]

I am also trying to recover photos (jpegs and raws) from a nonfunctioning raidz1 zpool. When I use PhotoRec on the drive, the recovered files cannot be read. Do I need to tweak the settings of PhotoRec to help the recovery process?