Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

ZFS Has Failed you

Status
Not open for further replies.

jpaetzel

FreeNAS Core Team
iXsystems
Joined
May 27, 2011
Messages
226
Thanks
89
#1
This post is howto for what to do when ZFS has failed you. Generally you'll be interested in this when FreeNAS is panicing at boot. You've discovered that the reason the system is panicing is during the ZFS pool import.

Step 1. Don't panic.

Step 2.

Reboot FreeNAS, and select the Escape to Loader option in the boot menu.

freenas_bm.png

In the attached screenshot it's option 6.

enter the following commands:

set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot -s

This will get you to the following screen, go ahead and press enter to start /bin/sh

freenas_su.png

# sh /etc/rc.initdiskless

Now you are ready to start trying to import your pools.

First pass is just trying a straight import:

# zpool import -R /mnt poolname

If this works, great. Backup your data and life goes on.

If you get a panic then repeat the above steps, but try:

# zpool import -o readonly=on -R /mnt poolname

If this works your pool imported read only. Backup your data and life goes on.

As a final desperation move, or after you've managed to import the pool readonly and copy everything off you can try:

# zpool import -R /mnt -FX poolname

The above command has a good chance of grenading your pool, but it also has a chance of importing it.


ZFS is very robust, but non ECC RAM and desktop drives can cause problems that it can't fix, especially in larger pool configurations.
 

Sir.Robin

FreeNAS Experienced
Joined
Apr 14, 2012
Messages
552
Thanks
41
#2
Always good to have a plan-B the day Mr. Panic knocks on your door. Thanx :)
 
Joined
Oct 14, 2013
Messages
1
Thanks
0
#3
Hi there,

Thanks for the post, I have experienced exactly this problem! I followed your excellent guide and managed to mount the zpool in read-only mount mode. But... I don't know how to back it up from there. My Freenas machine is exactly that: a NAS. I only ever accessed the data through CIFS. Which I configured from the GUI. So, after mounting my zpool in read-only mode, I typed 'exit' and the NAS booted to Freenas. But once I was there, I couldn't see the zpool I just mounted, so I cannot share it to the network.... Help? :) Thanks in advance!

-update-
When importing the zpool in read-only mode from the shell thingy in the Freenas GUI, something weird happens: I can see the volume in the 'reporting' graphs, but I cannot see the zpool listed under my 'volumes' in the Storage area:


If I get this fixed, I'll be able to back the files up :)

-update 2-
Using SSH file transfer (SCP) to copy the files to my Windows machine(s). Slow but at least I'm securing my data! :D
 
Joined
May 11, 2015
Messages
1
Thanks
0
#5
+1

I have the same problem. in v9.3 I no longer see Escape to Loader. I did get to a command line but without the kernel loaded I couldn't do much. Is there an updated list of these steps for v9+?

Thanks!
 
Joined
Sep 14, 2015
Messages
30
Thanks
3
#6
V9.3 procedure that just worked for me.

When GRUB loads up, hit the letter "e" on the keyboard to edit.
IMG_1039.jpg

Go down to the SINGLE USER MODE section and insert two lines. Type in the following into the two lines.
IMG_1040.jpg

set kFreeBSD.vfs.zfs.recover=1
set kFreeBSD.vfs.zfs.debug=1

IMG_1041.jpg

Look on the bottom of the screen for instructions. You want to boot by hitting CTRL-X or F10.

You'll be into the grub sub menu to see Normal boot up, single user mode and Verbose boot up. Arrow down to Pick single user.
IMG_1042.jpg

Continue with the instructions from the original post at

"This will get you to the following screen, go ahead and press enter to start /bin/sh"

etc...
 
Last edited:
Joined
Feb 21, 2017
Messages
1
Thanks
0
#7
This post just saved my pool! Thankyou

I ran into an issue where the system would freeze on boot at importing pools. Before i found this post i tried reinstalling and updating to FreeNAS 11 but when i tried to import the pool the system would just freeze. In the end i was able to use the readonly command from this thread with the -f flag to mount the pool and i am currently in the process of transferring my files via FTP.
This did not require single user mode due to the fact that i had already reinstalled the os and was able to access the console.
 
Status
Not open for further replies.
Top