Unable to import zpool

[Kiko]

I have 4 drives in RAIDZ2. After one reboot I've got Kernel 12 and FreeNAS 8.3 won't import zpool.
I tried upgraded FreeNAS and different hardware (same disks).
Finally I installed Solaris 11 and zpool import output is:
tank Degraded
raidz2-0 Degraded
c7t1d0 Degraded - corrupted data
c7td20 Online
c7td30 Online
c7td40 Online

device details:
c7td10 Degraded scrub/resilver needed
status: ZFS detected errors on this device. The device is missing some data that is recoverable.

I tried zpool import -F tank and the message is : pool may be in use from other system
I tried zpool -fF tank and system crashes and restarts.

I would really appreciate any help.

This seems like very odd error, I was not able to find anything similar on the Internet.
Of course, data is important and there is no backup.

 

[dlavigne]

Which version of FreeNAS was it originally? What error did you get when you upgraded and how did you upgrade?

 

[Kiko]

My error is not from upgrade I just tried to import zpool on the upgraded FreeNAS after I tried several times with original 8.3 version.
It was Kernel 12, all the same with both versions.

 

[cyberjock]

What's your system specs?

 

[Kiko]

My system is
MSI FM2 A85 mainboard
8 GB no-ECC RAM
4x2TB WDBlack Caviar
Additional Intel network card

 

[cyberjock]

Try running a RAM test first. Something damaged your pool. Depending on what went wrong you might not see your data again...

 

[Kiko]

I am currently runninig Memtest86+ test.
I am still not sure how to determine if I will see my data again :(

 

[cyberjock]

I don't either. We need to figure out how devastating the corruption is. I will say that if your system fails the RAM test you have no chance of seeing your data again.

 

[paleoN]

I am still not sure how to determine if I will see my data again :(

Those are the wrong import commands to run. After the RAM test reboot into single-user mode, option 2, and only run:

Code:

sh /etc/rc.initdiskless

zpool import -f -R /mnt tank

If that panics reboot again into single-user mode and run:

Code:

sh /etc/rc.initdiskless

zpool import -f -R /mnt -o rdonly=on tank

I will say that if your system fails the RAM test you have no chance of seeing your data again.

Rather premature for such declarations.

 

[cyberjock]

Rather premature for such declarations.

I disagree. Every person that has had a pool that was unmountable and later found to have bad RAM never got their data back from the original pool. They had to use backups, or if no backups, they kissed it all goodbye.

 

[Kiko]

I quit Memtest after 19 hours without errors.

 

[Kiko]

I tried with those commands above:
zpool import -f -R /mnt tank
zpool import -f -R /mnt -o rdonly=on tank

and I got Fatal trap 12 on both.

 

[paleoN]

and I got Fatal trap 12 on both.

OK, let's back up and get all information from a FreeBSD/FreeNAS OS:

Code:

uname -a

zpool import
 
camcontrol devlist
 
gpart show
 
glabel status

You can follow that up with zdb -l dev for each ZFS partition. Use e.g. pastebin.com as the output will be quite lengthy.

 

[Kiko]

OK, let's back up and get all information from a FreeBSD/FreeNAS OS:

Code:

uname -a
 
zpool import
 
camcontrol devlist
 
gpart show
 
glabel status

You can follow that up with zdb -l dev for each ZFS partition. Use e.g. pastebin.com as the output will be quite lengthy.

Please help.
You are right about output, it is indeed very lenghty and I am not very familiar with console.
Problem is that I cant find a way to export or copy that output.
The only way I can boot FreeNas is Single user mode and than I couldn't find a way to communicate with outer world.
I tried to redirect output to a file with zdb -l /dev/ada3p1 >file.txt, message is file.txt:Read-only file system.

 

[warri]

Try to use a location in /tmp, this should be writable to.

 

[mikeyr]

Kiko,
I had something very similar.
I have a 8.2.0-release-p1-x64 FreeNAS server that has been running for 2 years.
I have 2 x 2TB drives in a mirror configuration.
It recently notified me that there were some data errors in about 5 files, although I could read the files and they didn't seem corrupted so I think the mirror was doing its job. (The pool was not degraded at this time, and everything was "ONLINE".)
I manually ran a scrub to clear out the errors from the command shell. The drives spun up and I heard activity for about 2 seconds and then suddenly the system crashed with a Fatal Trap 12.
I rebooted the system and during the bootup sequence I could see that it mentioned that now the pool was degraded and something about the geom/gpt was corrupted on one of the disks in the mirror. When the system tried to import the pool it crashed with another Fatal Trap 12.
I thought it might be my server dying, so I pulled the disks and put them into my backup server; and loaded FreeNAS 8.2.0-release-p1 onto it.
When I set it up and tried to import the volume the new machine crashed with Fatal Trap 12.
I was now extremely concerned that I had lost my data, despite having a mirror configuration. (Due to a set of circumstances that I won't get into here my backup was already in bad shape; in fact this error happened on my main server while trying to recreate my backup that had its own problem... so I was really in a pickle!)

There are a number of "recovery" type options on the zpool import command that have been mentioned in a lot of posts, but I wanted to secure whatever I had of my data before experimenting-- I figured most of the actual data was there, just somehow the headers or something were destroyed. I wanted to duplicate my drives so I could work on them without fear of destroying whatever data I had.

First I ran memtest86+ on my backup server and it made multiple passes with no errors.

So I downloaded FreeBSD 8.3 and installed it to get a FreeBSD machine working on my backup server that has more tools/flexibility than the FreeNAS system does. (I chose 8.3 because it was close to the 8.2 that my FreeNAS box was running, but 8.3 added an option to import a pool read only which is something i wanted while trying to recover my drives that wouldn't import)

FreeBSD 8.3 was able to see my mirrored drives, and had the same status the the pool was degraded and one drive corrupted somehow. I did not import the pool though, since that was what had crashed both my main server and my backup server.
I installed ddrescue and copied my mirrored drives onto fresh drives. They copied with zero errors, so I don't think the original drives were having any hardware issues.

Once I had the copies I put the originals of my mirror in a safe place and installed just the copies of the mirror on the FreeBSD 8.3 machine. I was then able to import the pool (readonly) and it didn't crash. I replicated the filesystem onto yet another fresh disk (so now I have all my data on a non-mirrored disk) I checked and on that fresh disk I can import the pool and see the data.

Now I'm trying to figure out if the mirror can be repaired. I removed my backup disk so I only have the mirrored array on the FreeBSD machine, I booted it up, and I tried to import the mirrored pool (same as before), only this time I didn't use the -o readonly=on flag. When I imported without the readonly flag the machine crashed again!

I've tried 3 times in a row now, and trying to import the mirror causes a Panic crash.
If I disconnect the disks, reboot the system, and "export" the pool so that FreeBSD "forgets" about the pool; then shutdown, reconnect the mirrored disks, reboot, and then import the pool -o readonly=on there is no problem.

So something about the set of errors on the mirrored disks does not allow ZFS to import the pool normally (causes a crash even!); but it can import the pool "readonly"

I'm loosing a bit of faith in ZFS here if it seems to be able to read the pool "readonly", but when I try to import the pool normally the system crashes-- this isn't what a well behaved filesystem should do, no matter what the problem on the disk is, it shouldn't crash/Panic!

I guess I'll need to move up to FreeBSD 9 and see if the problems are fixed in that version.

I welcome any comments, questions, etc. from those more expert than I. I felt I should post this since I've learned so much from reading what everyone else does. Maybe this will strike a chord with someone or give them another avenue to try when their data appears lost.

 

[cyberjock]

Have you run a RAM test on your original server? It could be you were getting errors from bad RAM...

 

[mikeyr]

Hi cyberjock, I've read a *lot* of your posts on here, thanks for all the good work and great suggestions you make to help out us noobs.

I'll eventually be getting back to my original server and doing a RAM test and some kind of burn-in to see if I can recreate the original crash that started all this before I trust it too much again. It's very tempting just to get a new and/or better machine since this has *not* been fun.

Right now I'm trying to decide if I have time/energy to try to figure out why this mirror pool can only import "readonly" -- or if I should just destroy the pool on them and rebuild from my newly made data disk. It would seem like a useful public service to figure out why the pool won't import... but I've got a family life and a job to keep up with; and while it is amusing I've already spent many many hours that I didn't really have getting to this point.

But it bothers the heck out of me that the mirror is definitely readable; but it crashes the system if not imported "readonly" This just shouldn't happen. I think it's kind of a confidence in ZFS thing for me now.

I'm going to be out of town for the next week, but maybe next weekend I'll try setting up a FreeBSD 9 or FreeNAS 9 machine and see what it thinks of this "toxic" pool I've got... maybe there's some bug that's been fixed.

 

[cyberjock]

The reason I asked about the RAM test is because it sure sounds like you have a pool that's been damaged somehow. Since both disks are good the possible causes range from "unlikely" to "very unlikely". Finding the cause of your problems would be a good start.

ZFS is ultra-safe, when properly used. It is designed to handle corruption from many sources, but not from RAM. Hence the ECC RAM requirement. But you aren't using ECC RAM. Also we recommend 8GB of RAM minimum with ZFS because many people have had problems with less. And again, you fall into that category.

My guess is your pool is damaged somehow and the only solution is going to be to destroy and recreate the pool. If it gets to that I'd strongly recommend you get hardware that is more in-line with the recommended hardware.

If you choose to continue diagnosing your problem you should make a new thread as your issue isn't the same as the OP aside from being unable to import the pool.