Cannot import pool (visible in drop down list in GUI)

[kgrass]

Hi.
I'm desperately in need for your help Guys to recover/import my Pool after FreeNas USB stick failure.
2 weeks ago my FreeNAS system 9.10.2-U3 didn't boot up with Panic error (Stopped at kdb_enter+0x3e: movq $0,kdb_why). I've got 2 HDD 2TB (mirror0).
My first idea was maybe it's a USB stick problem but I could copy it to another USB drive and run it with the same error.
Then I install FreeNAS again on another USB drive and tried to import volume in GUI (i can see pool and it's ID fine) so I pressed import and then after a minute of watching lots of pages scrolling really fast computer restarted and loaded FreeNAS again but no pool in GUI - Volumes. Same situation in Shell after command: zpool import -f "poolname".

Next idea was to upgrade to latest system (FreeNAS 11 Stable). I can see pool on Import volume page but this time it goes straight in to an error...
https://drive.google.com/open?id=0B0GnK4u_WozcSFZDdzhuOFplMGM

Please help me to recover my data; I still have the original USB - I've read that taking file containing info about pool's my be helpful...

Specification:
Platform AMD Phenom(tm) II X4 945 (64bit) Processor 3000MHz
Memory 8162MB dual channel 2x4096MB/800MHz DDR3 1600
Motherboard: Asrock N68-S3 UCC
2x SATA HDD WD 2TB Raid0 (mirror0)

Code:

root@freenas:~ # zpool status
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  ada0p2	ONLINE	   0	 0	 0

--------------------------------------------------------------------------------------------

root@freenas:~ # zpool import
   pool: NAS
	 id: 16553933547003700147
  state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
		the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

		NAS											 ONLINE
		  mirror-0									  ONLINE
			gptid/caf812f3-9774-11e5-9112-0025224caa76  ONLINE
			gptid/cbd00290-9774-11e5-9112-0025224caa76  ONLINE
root@freenas:~ #

 

[Dice]

Do you have a backup of your config file?

 

[Vito Reiter]

I hope that's ECC RAM. Also, your last bet is to use

Code:

zpool import -f 16553933547003700147

and hope it works, this isn't safe nor recommended but it's the only way you'll get it imported. Good luck :) Redundancy is not a backup.

Edit: It's rejecting because it's been last accessed by another system.

 

[danb35]

Next idea was to upgrade to latest system (FreeNAS 11 Stable). I can see pool on Import volume page but this time it goes straight in to an error...
https://drive.google.com/open?id=0B0GnK4u_WozcSFZDdzhuOFplMGM

This forum supports uploading images and other files directly to posts, and even including those images in your posts. Please take advantage of this capability when posting images in the future; it makes your posts much easier to read, and helps for archival use as well.

As to your problem, kernel panics when trying to import a pool usually indicate the pool has become rather badly corrupted, and it may well not be recoverable. What happens if you do zpool import -f -F -n NAS?

 

[Vito Reiter]

Well, that error just looks like a hostname mismatch (if you're lucky..). Don't know if you didn't name the old box 'freenas' or verse vica, but using -f should force import the volume. Otherwise, it might be gone :(

 

[danb35]

zpool import -f 16553933547003700147

The pool has a name; the numeric identifier isn't necessary. And the -f flag will only allow the system to import a pool that was last used on a different system (i.e., it will bypass the issue you mention with your edit). The -F flag will attempt to fix the pool and force an import, but is also potentially destructive. The -n flag in connection with -F will tell it to report if the pool will be recoverable, without actually changing anything.

Failing that, the next option would be to try to import the pool read-only. I forget the syntax for that, but a few minutes with Google should find it. Edit: looks like it's zpool import -o readonly=on NAS.

 

[kgrass]

Do you have a backup of your config file?

Unfortunately no config file...

I hope that's ECC RAM. Also, your last bet is to use

Code:

zpool import -f 16553933547003700147

and hope it works, this isn't safe nor recommended but it's the only way you'll get it imported. Good luck :) Redundancy is not a backup.

Edit: It's rejecting because it's been last accessed by another system.

non-ECC

As to your problem, kernel panics when trying to import a pool usually indicate the pool has become rather badly corrupted, and it may well not be recoverable. What happens if you do zpool import -f -F -n NAS?

This command definitely do something as well as regular zpool import -f but ending up with restart and back on again but nothing in GUI->Storage->Volumes.
My knowledge of freeBSD may be to little at this stage...

 

[Vito Reiter]

non-ECC

I wouldn't do any fixes or scrubs with non-ECC RAM, as that can make it worse.. Try the command a few different ways to force import it or make sure that the new hostname is the same as the old and see what happens.

 

[danb35]

This command definitely do something as well as regular zpool import -f but ending up with restart and back on again but nothing in GUI->Storage->Volumes.

So -f -F -n results in a reboot? What about zpool import -o readonly=on NAS?

 

[danb35]

I wouldn't do any fixes or scrubs with non-ECC RAM, as that can make it worse.. Try the command a few different ways to force import it or make sure that the new hostname is the same as the old and see what happens.

Well, you can't do a scrub on an offline pool, so no worries there. And even if you could, non-ECC RAM is only going to change things for the worse if it can simultaneously come up with bad data and a matching checksum for that bad data. I think it's safe to say the chances against that are astronomical.

The hostname has nothing to do with it. Even if the hostname is the trigger for the "The pool was last accessed by another system" message (which I don't believe it is), that isn't going to cause a kernel panic, and -f bypasses that, completely and cleanly.

If using the -f -F -n flags still results in panic and reboot, the only real options are (1) try to import the pool read-only, as I described above (which is sub-optimal for any number of reasons), or (2) try some zdb voodoo (but there be dragons, as far as I'm concerned).

 

[Ericloewe]

If -n fails, it's time to start making arrangements for the backups to be made available...

(1) try to import the pool read-only

And if that fails, consider the pool gone.

 

[kgrass]

So -f -F -n results in a reboot? What about zpool import -o readonly=on NAS?

How can I describe you what has happened when I am typing that command pressing enter and the screen is flooded with pages of text scrolling that it is impossible to read a single thing. After 20sec stops and it looks like an scheduled restart with no errors that I can see. Maybe i'll try to record it. Works only on ver. 9.10.2 , FreeNas 11 generate panic error immediately for zpool import in any configuration

How is that possible that one day all works fine and the other (i think it could have been an update to 9.10.2-U3) system goes down and data too? Recovering after system failure should not cause such a damage (that's fundamental idea of this system - isn't it?)

 

[StephenFry]

Weird idea: what if it's something hardware related? Did you try connecting the drives to a completely different system and importing them there?

I have noticed that the FN/FreeBSD systems I have built over the years are much more finicky than regular PCs when it comes to 'simple' things like controller and cables.
I've been able to solve many incomprehensible errors by re-seating connectors that were not obviously loose and by replacing controllers for seemingly no good reason.

 

[kgrass]

So -f -F -n results in a reboot? What about zpool import -o readonly=on NAS?

I setup a second PC - bit lower spec but need to eliminate hardware problem...
I've recorded what is happening after I press import button in GUI:

1st movie is FreeNas 11 (stop almost immediately)

2nd movie is FreeNAS 9.10.2

Generally nothing change so I don't thing it's a hardware.
Maybe one of you can see anything?
How can I check if pool is imported from shell level?
there is nothing in GUI...

Maybe I can upload an image of the original USB flash drive which stops with panic error while booting?

 

[kgrass]

FINAL EDIT

After days of trying different things I solved it...
Answer... I went back to FreeNas 9.3 to see this:

Now i can back up my data in 20 different ways....:)

Thanks to All of you Guys for ideas but I think that few programmers from FreeNas should look into their new versions to improve whole process of importing volumes.

Good Night.

 

[Ericloewe]

And you can actually access the data?

 

[kgrass]

And you can actually access the data?

Yes. It's all there and working. Will start to backup 1st thing after I sleep a bit :)

 

[DrKK]

This really is odd. The fundamental difference between 9.3 and 9.10 and beyond is the underlying FreeBSD version (9 vs 10+).

What do you think, @Ericloewe ? I feel like I also heard something similar a few weeks ago, dude with unrecommended hardware (like this guy) couldn't get 9.10 or beyond to work, and was OK when he backed off the 9.3

 

[Ericloewe]

I feel like I also heard something similar a few weeks ago, dude with unrecommended hardware (like this guy) couldn't get 9.10 or beyond to work, and was OK when he backed off the 9.3

I don't remember any quite like this. There are the occasional outright incompatibilities that keep FreeNAS from booting, but causing a pool to no longer import, even though it gets as far as seeing that there is a pool? That's a new one to me.

The top of the stack trace isn't very readable in the video, but it seems to be panic'ing at zfs_blkptr_verify, which has me worried that there might be some weird corruption on the pool that somehow isn't picked up by earlier versions of ZFS.

 

[DrKK]

worried that there might be some weird corruption on the pool that somehow isn't picked up by earlier versions of ZFS.

That's inherently very unsatisfying, if we don't get to the bottom of it. We really ought to understand this, if we can.