zpool import causes panic

Status
Not open for further replies.
Joined
Jul 29, 2012
Messages
10
I was running the FreeNAS 11 Nightly from 20180416. I came home and found the box had panic'd, so I reset and let the machine reboot. It panic'd again when it tried to import the main data set. It successfully imports the boot volume.

Things I have tried:

1. built new boot usb.
2. zpool import -f
3. zpool import -fF -o readonly
4. moved disks to another machine

zpool import command itself reports back that all 5 disks of the pool are online.

I am looking for suggestions. Thanks in advance.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
It would be useful to see the exact panic output.

Also, hardware specs and the version of FreeNAS you were using before the nightlies. Also, why are you running the nightlies?
 
Joined
Jul 29, 2012
Messages
10
I have been running the nightlies for fun. No specific reason. I liked testing the new features.

FreeNAS version at crash - FreeNAS-11-MASTER-201804160410-78cda93

Hardware:
CPU- intel i5 2500-K (Socket LGA1155)
MB - Intel Desktop Board DZ77SL-50K
RAM - 32GB - 4 x 8GB
4 x 3TB disks+ 1 SSD for Zil
32GB SanDisk Ultra Fit (Boot)

Working on reducing the DB panic info pic size to post.

The final line is:

Stopped at kdb_enter+0x3b: movq $0,kdb_why
 
Joined
Jul 29, 2012
Messages
10
Screenshot from panic uploaded
 

Attachments

  • screenshot001.pdf
    44.2 KB · Views: 570
Joined
Jul 29, 2012
Messages
10
I ran smartctl -a on each physical disk and all passed. The panic still happens on import.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
  • Plug in the install USB, boot into it and then select "Shell"
  • Once in shell, can you run zpool import, zpool status and post the output here?
  • Do not try to import just yet or you will have to reboot and get back into shell everytime you need to run a command
 
Joined
Jul 29, 2012
Messages
10
zpool import
Code:
   pool: vd3																	
	 id: 6662080228811196520													
  state: ONLINE																 
 status: The pool was last accessed by another system.						 
 action: The pool can be imported using its name or numeric identifier and	 
		the '-f' flag.														 
   see: http://illumos.org/msg/ZFS-8000-EY									 
 config:																		
																				
		vd3											 ONLINE				 
		  mirror-0									  ONLINE				 
			gptid/c31ed619-7891-11e4-afe7-002511e20540  ONLINE				 
			gptid/c43e301f-7891-11e4-afe7-002511e20540  ONLINE				 
		  mirror-1									  ONLINE				 
			gptid/f613211b-af2b-11e4-b0b6-002511e20540  ONLINE				 
			gptid/c5aa4fcb-7891-11e4-afe7-002511e20540  ONLINE				 
		logs																	
		  gptid/df7c20c6-e5c9-11e6-bcb1-e840f2d1ca87	ONLINE	

 
Joined
Jul 29, 2012
Messages
10
zpool status
Code:
  pool: freenas-boot															
 state: ONLINE																 
  scan: none requested														 
config:																		 
																			   
	   NAME		STATE	 READ WRITE CKSUM								 
	   freenas-boot  ONLINE	   0	 0	 0								
		 da0p2	 ONLINE	   0	 0	 0								 
																			   
errors: No known data errors
 
Joined
Jul 29, 2012
Messages
10
It causes a kernel panic. See the initial posts for screen shots of the panic.

Same results occur with zpool import -f vd3 also.

Disks seem to be error free.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
It causes a kernel panic. See the initial posts for screen shots of the panic.

Same results occur with zpool import -f vd3 also.

Disks seem to be error free.

Here is what I did to stop kernel panic and actually try to recover:

DISCLAIMER: I would recommend taking pictures/screenshots at each and every steps listed here. If you dont understand something or you think you did something wrong, stop and post output/screenshot here instead of ignoring or trying the next step. We can't help if you missed one step and then wonder why it didn't work as intended.


  • Reboot FreeNAS and when GRUB loads up, press "E" to edit
IMG_1039.jpg

  • Scroll down to where the single user mode section starts. You want to add following two lines. Make sure you scroll down to the right section and make sure to follow the format of existing lines.
Code:
set kFreeBSD.vfs.zfs.recover=1
set kFreeBSD.vfs.zfs.debug=1


After second line, press Enter and once you are on new line, press Tab and it should exit the edit mode.
IMG_1040.jpg

  • Once you are done editing, you can press Ctrl + X and it will reboot the system.
Once the GRUB menu loads up again, be sure to select the single user mode that we edited previously.
IMG_1042.jpg

Once it loads, you should get the following prompt:
"This will get you to the following screen, go ahead and press enter to start /bin/sh"

Type following command:

sh /etc/rc.initdiskless

Now you are ready to start trying to import your pools.

First pass is just trying a straight import:

# zpool import -R /mnt poolname

If it says recovery is possible but you might lose some data, take a look and see how much data has to be discarded:
Error5.jpg

If you get the recovery option but it complains that The pool was last accessed by another system and you have to use -f to force import, then try:

# zpool import -F -f poolname

If you get a panic then repeat the above steps, but try:

# zpool import -o readonly=on -R /mnt poolname

If this works your pool imported read only. Backup your data and life goes on.

As a final desperation move, or after you've managed to import the pool readonly and copy everything off you can try:

# zpool import -R /mnt -FX poolname

The above command has a good chance of destroying your pool, but it also has a chance of importing it.


Work Cited:
These steps were posted by @jpaetzel and @technopop on THIS thread. I just re-worded and re-arranged some of the things to make it easier to follow. It helped me with my issue so I figured it might help with yours.
 
Joined
Jul 29, 2012
Messages
10
I had the issue again. I was able to fix it, but modified the steps as 11.2 has a now version of GRUB.

I installed an 11.0 ISO to a different thumb drive and booted. I pressed E and ddded the ZFS parameters (
set kFreeBSD.vfs.zfs.recover=1
set kFreeBSD.vfs.zfs.debug=1
)
and press Ctrl-x to boot.
At the install menu and selected the Shell option.

Then I used the zpool import commands to import the volume, specifically the zpool import -F -f poolname.

This imported the volume. Then I rebooted using the original thumb drive and the volume has imported.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
11.2 does not use GRUB. 11.2 uses the FreeBSD bootloader, which has gained support for all necessary features since 9.3.
 
Joined
Jul 29, 2012
Messages
10
any suggestion how to do a similar process after upgrading the zpool, i.e. I can't do the previous process as the disks can't be imported as read/write.
 
Joined
Jul 29, 2012
Messages
10
with 11.3, I pressed space to stop the boot and selected the loader options. At the OK prompt I typed:
set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot

The booted the vanilla system and then I was able to run zpool import -f -F <volumename> to import the volume.
 
Last edited:
Status
Not open for further replies.
Top