Trying to mount a ZFS volume after system crash results in kernel panic

[ArtVandelay]

I setup FreeNAS on a new system that I built and created a volume (ZFS mirror) with 3 1TB drives. I created several ZFS datasets and transferred a large amount of data from my old NAS box to the new system. Initially everything was working fine. About a month later, while transferring a large file to my desktop over the network, the FreeNAS box froze and became unresponsive. The only thing I could do was to power cycle the machine. Now, whenever I try to mount the volume, I get a kernel panic.

Example:

If I boot FreeNAS in single user mode and run zpool import, I get the output in screen_01.jpg.

But when I try zpool import -f vol1, the system crashes. See screen_02.jpg.

I've also tried to mount the volume using FreeBSD, mfsbsd, OpenSolaris, and Ubuntu, but I always get the same result. The output of zpool import makes me think that the volume is OK, so it's frustrating that I'm unable to mount it. Is there something that I'm missing? I'm fairly new to ZFS, so any help would be greatly appreciated.

PS - Luckily, I hadn't yet reused the drives from my old NAS box =), so I'm only missing about a months worth of data (which I would love to get back).

 

[ProtoSD]

Sorry to hear you're having problems. You might try the rescue CD mentioned in this link with ZFS version 28:

http://protosd.blogspot.com/2011/12/protosds-unofficial-freenas-8-faq.html#q34

I know you're not getting corrupt metadata messages, but v28 might be able to get your pool back online and then you can export it if that works and try FreeNAS again. Using the rescue CD doesn't upgrade your pool to v28, v28 just has improved import funtionality.

 

[ArtVandelay]

Hi protosd,

Thanks for the reply. I tried to import the volume with mfsbsd. They system still crashes, but with a different error (see attached screenshots). Any other ideas?

 

[ProtoSD]

Well you can give this a try (still using mfsbsd):

Try testing a forced import as if data loss was allowed:

# zpool import -nfF pool0

That will tell you to what point ZFS transactions would need to be "rewound" to import your pool. If rewinding to that time and losing data written after that time is ok, then repeat the command again without the "n" option, so only -fF.

The “import” option will report the exact number of “seconds” worth of data that cannot be restored. Depending on the bandwidth and utilization of your system, this could be very little data or several MB worth of transaction(s).

EDIT: There is also a poorly documented -X option which needs to be combined with the -F option which will do an exhaustive scrub like attempt to recover your pool. It can take quite awhile, HOURS if you have a lot of data and it is pretty much a last resort.

 

[ArtVandelay]

I tried both

# zpool import -nfF vol1

and

# zpool import -nfFX vol1

Both resulted in the same crash/message as before - "panic: solaris assert: ss == NULL..."

Thanks again for the quick replies.

 

[William Grzybowski]

Ok, then the next step is trying to do so with linux (using zfsonlinux.org), or OpenSolaris/Solari***press

 

[ArtVandelay]

I tried to import the volume using the Oracle Solaris 11 11/11 live cd and got the same kernel panic as in my last post.

 

[ArtVandelay]

This same problem happened to me again a few weeks ago, but this time I was able to regain access to my data. I wanted to update this post with the solution I found incase there was anyone else facing the same issue.

When attempting to import the pool, the system would crash with the following error:

Code:

panic: solaris assert: ss == NULL, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space
_map.c, line: 109

The first thing that I did this time was to image one of the drives in the ZFS mirror to a spare drive using dd. That way I wouldn't have to worry about any further damage to the original.

After extensive Googling, I found this post on the FreeNAS forum, which lead me to a post on sigtar.com. Using the info on that page, I was able to import the pool in read-only mode and copy all of my data onto a new drive. As a bonus, the same process worked on the pool I was having problems with in my original post!! (The drives had been sitting on the shelf, waiting for a fix for a year and a half.)