SOLVED ZFS pool not found after hardware swap and reboot

[donsev]

Hello,

Yesterday I was replacing a failing drive (turned the failing drive off, added new hdd, clicked replace, waited for resilvering). Afterwards my VMs and Jails stopped working, so I rebooted. Come to find out FreeNAS cannot see my ZFS pool.

I ran zpool import -fF ZFS, this came back with the error:

Code:

cannot import 'ZFS': I/O error
    Destroy and re-create the pool from a backup source.

System details and other info:

• Intel i3-4330 CPU at 3.50GHz (4 cores)
• 32 GB RAM
• FreeNAS-11.2-U7
• 2x Raidz2, for a total of 8 2tb hdds
• The hard drive failure was in one Raid, the other showed up as fine
• I messed up. There was a hard drive failure in August (?), but I didn't have the time to replace the hdd so I just unplugged it and let the raid run degraded. A second hard drive failure came up about two weeks ago, so yesterday I decided to replace it.
  • Turn off newly failed hdd, unplug it, add in new hdd, click "replace" from the webgui
  • Also plugged in old failed hdd. Then I stupidly turned it on, thinking it should "turn on" within the webgui so I can confirm the right hdd and then turn it off and replace it.
  • Seemed to be working fine (freenas gui was reporting resilvering and replacing progress), but plex and VMs weren't working.
  • Waited 24 hours, rebooted server. And here we are
• Last scrub of the pool was Dec 31, 2019

I ran the freenas-debug -A command, not sure what would be useful but the output is attached below. I can't find a history of messages, just a current diagnoses, but I'm not sure if I'm looking at the right places.

I do have a backup on old external drives, unfortunately they are roughly a year old. Really hoping we can save the data somehow, best case scenario reattach the pool and I get spanked and immediately create a backup. A tiny bit terrified right now.

Edit: Oh ffs. Apparently I unplugged a HDD somehow, so FreeNAS didn't attach the pool. Plugged the HDD back in, and all my files and stuff are back. Now to back everything up.

 

[artlessknave]

note that rebooting while zfs is doing an operation can make for a very bad day by irrevocably damaging the pool.
backups are ireplaceably important with zfs because, as you found out, as far as zfs is often concerned the fix to problems really is:

Destroy and re-create the pool from a backup source.