HELP ZFS Pool data recovery

[Dawson]

Okay, so pool imported! None of my shares work. So I went and deleted the current ones, I went to recreate the share, and I can only get as far as /mnt

Also when I go to one of the zvols and click edit permissions it does nothing. every other button works.

 

[Dawson]

 

[HoneyBadger]

Awesome!

Export your pool from the UI, then try a re-import from the UI.

 

[Dawson]

Awesome!

Export your pool from the UI, then try a re-import from the UI.

Should I wait for this to finish? I clicked Tank>status

Could be an automatic scrub? I haven't started anything

 

[HoneyBadger]

Should I wait for this to finish? I clicked Tank>status

View attachment 67561



Could be an automatic scrub? I haven't started anything

It'll be able to scan but not complete the scrub, as you've still got the one UNAVAIL device - I assume because you haven't installed/connected the "out of date" device with the false failure.

Let the scan phase complete (to correct any metadata issues) export the pool, power off, connect the "out of date" device, power up, and import from the UI. That should reset it to the proper mount point and allow for the share creation to proceed as normal.

Once that's done, scrub the pool, let it complete, and update that backup.

 

[Etorix]

Scrubs automatically resume when interrupted, so it shouldn't matter.
But I would wait for the scrub to finish before replacing the missing disk or attempting to use shares or zvols.

This looks like you're well in theway of recovery, thanks to the magic skills of @ElricTheNecromancer @HoneyBadger.

 

[Dawson]

It'll be able to scan but not complete the scrub, as you've still got the one UNAVAIL device - I assume because you haven't installed/connected the "out of date" device with the false failure.

Let the scan phase complete (to correct any metadata issues) export the pool, power off, connect the "out of date" device, power up, and import from the UI. That should reset it to the proper mount point and allow for the share creation to proceed as normal.

Once that's done, scrub the pool, let it complete, and update that backup.

Will do, I will let you know how it goes. So far 0 errors and it's 75% done. You have no idea how thankful I am for you helping me out. I greatly appreciate it.

 

[winnielinnie]

HoooooooooooOOOOOOOooooooooooooooOOOOOOOooooooooooooooooooooooooooooooooooooooooooly moly!!!!


No to jinx anything, nor do I wish to sound like a party-pooper, but even if this doesn't work out in the end, the fact that you even made it this far is something to behold! (Plus many lessons learned. Not just in these daunting recovery scenarios, but also what not to do in the future when you play around with virtualization and disks. ) Major, major, major props to @HoneyBadger!


This is incredible!

 

[Dawson]

Looks like it worked! My data is intact as far as I can tell. It is still finishing a scrub, but all looks good. It does appear that the data is a little old. As old as the outdated drive was. But I'm not too broken up about this.

 

[jgreco]

And the great thing is, I didn't have to get too heavily involved. Hahah. But persistence sometimes pays off with these things. Please, please, please do go back and read the stuff I wrote about proper virtualization strategy. Parts of that are intended to protect your devices against this sort of accidental mistreatment. I can tell you from long experience that the ability to recover from a charlie foxtrot like this is statistically unusual, you just happened to luck out in the "didn't do catastrophic levels of damage" department.

 

[winnielinnie]

It does appear that the data is a little old. As old as the outdated drive was.

My memory is fuzzy. About how old is that? A week or so?

 

[Dawson]

Overall thank you so much @HoneyBadger you are actually a wizard and also thank you to everyone else who helped out especially @jgreco

 

[Dawson]

And the great thing is, I didn't have to get too heavily involved. Hahah. But persistence sometimes pays off with these things. Please, please, please do go back and read the stuff I wrote about proper virtualization strategy. Parts of that are intended to protect your devices against this sort of accidental mistreatment. I can tell you from long experience that the ability to recover from a charlie foxtrot like this is statistically unusual, you just happened to luck out in the "didn't do catastrophic levels of damage" department.

Already did, I have parts on the way to do it right this time. Thanks for the advice. I am going to setup backups as well. If you guys see me in this forum again....it will be because the world ended

 

[Dawson]

My memory is fuzzy. About how old is that? A week or so?

4ish days

 

[winnielinnie]

If you guys see me in this forum again....it will be because the world ended

Ah, c'mon. We're pretty cool people. You're allowed to stop by and say "hi" once in a while.

 

[HoneyBadger]

Overall thank you so much @HoneyBadger you are actually a wizard and also thank you to everyone else who helped out especially @jgreco

 

[Dawson]

View attachment 67563

That apple represents my data

The scrub should get that outdated drive up to date I assume? or do I need to resilver it?

 

[winnielinnie]

So that's what happens when ZFS heals literal bit rot, eh...?

 

[HoneyBadger]

That apple represents my data

The scrub should get that outdated drive up to date I assume? or do I need to resilver it?

Not quite perfect, I'm afraid. A little miffed that it decided to revert back that far even without the "Drive B" in the system when the import happened.

Scrubbing should get it back "up to date" with the latest changes.

 

[Dawson]

So scrub finished. It still shows unhealthy, and it has 142 errors. How do I fix?