zpool FAULTED - Please help!

Status
Not open for further replies.

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
I'm afraid I've gone to far and I should have asked for help a while ago. I am new to *nix and ZFS and I'm over my head. I was attempting to replace a failing drive and now I have a FAULTED zpool. I've been reading and trying things for a week now and I am afraid I've gone too far. I have 5 Seagate 2T EARS drives in a raid1z configuration along with a smaller cache drive. I replaced the failing with a used drive of the same size and model but I didn't realize I needed to zero it out first. I pulled the replacement drive out and zeroed it out in another machine and tried the replacement steps again now the zpool is FAULTED. I've since tried a million other things and I am still in a faulted state Have I lost everything or is there still hope? Can someone guide me through this? I've got years of source code, family pictures and everything else I can't replace on there. Oh, and of course I don't have any backups!

Thanks,
Tom
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
Hi Tom,

No promises I can help you all the way through this, but to get started can you post the output of these 3 commands:

Code:
zpool status -v

camcontrol devlist

gpart show


You didn't necessarily need to zero your replacement disk, but that doesn't matter at this point.

What have you done with the drive that was failing? Don't wipe it or do anything with it if you haven't already, hang on to it.

-- Proto
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
Proto - Thank you for the prompt response! I have the failing drive, and I haven't done anything with it except to put it back in to see if that would help. Here is a taste of what you are dealing with - I can run the commands you asked for but how do I get the results to a text file?

Thanks,
Tom
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
I can run the commands you asked for but how do I get the results to a text file?

Thanks,
Tom

Hmmm, this could prove to be a little challenging... ;)

Here's a link to the docs on setting up SSH, but it's a little more than you need to setup. You can disregard the SFTP stuff, and creating a dataset (which you won't be able to do anyway), everything about permission, creating a home folder, and connecting with WinSCP you and ignore. I think those docs need some adjusting.

Basically you need to:

Create a user from the GUI
Add it to the "wheel" group
Enable the SSH service
Install Putty http://www.putty.org/

Login to your FreeNAS system using Putty and the user you added above.

After you do that you'll be able to copy & paste output.

In FreeNAS 8.2 you'll only need to click on the "Shell" icon from the GUI to copy & paste, but that's jumping waay ahead.
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
I didn't think about using Putty, I already had it setup. Here are the results of what you asked for. I hope this helps.

Thanks,
Tom


[tom@freenas] /> zpool status -v
no pools available
[tom@freenas] />

[tom@freenas] /> camcontrol devlist
camcontrol: couldn't open /dev/xpt0: Permission denied
[tom@freenas] />

[tom@freenas] />
[tom@freenas] /> gpart show
=> 34 1465149101 da0 GPT (699G)
34 94 - free - (47K)
128 1465149007 1 freebsd-zfs (699G)

=> 34 3907029101 da1 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 da3 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 da4 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 da5 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 63 8125866 da6 MBR (3.9G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 4220937 - free - (2.0G)

=> 0 1930257 da6s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)

[tom@freenas] />
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
I logged in as root and got a little more information.

[root@freenas] ~# camcontrol devlist
<ATA ST3750640AS E> at scbus0 target 0 lun 0 (da0,pass0)
<ATA WDC WD20EARS-00M AB50> at scbus0 target 1 lun 0 (da1,pass1)
<ATA WDC WD20EARS-00M AB51> at scbus0 target 2 lun 0 (da2,pass2)
<ATA WDC WD20EARS-00M AB50> at scbus0 target 3 lun 0 (da3,pass3)
<ATA WDC WD20EARS-00M AB50> at scbus0 target 4 lun 0 (da4,pass4)
<ATA WDC WD20EARS-00M AB50> at scbus0 target 5 lun 0 (da5,pass5)
<HL-DT-ST DVDRRW GSA-H30L S856> at scbus4 target 0 lun 0 (cd0,pass6)
<3SYSTEM USB Flash Disk 1.00> at scbus5 target 0 lun 0 (da6,pass7)
[root@freenas] ~#


[root@freenas] ~# zpool import
pool: FreeNAS
id: 14911761039937691724
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
see: http://www.sun.com/msg/ZFS-8000-72
config:

FreeNAS FAULTED corrupted data
raidz1 DEGRADED
da1p2 ONLINE
replacing UNAVAIL insufficient replicas
2603211288499742476 UNAVAIL cannot open
da2p2 UNAVAIL cannot open
da3p2 ONLINE
da4p2 ONLINE
da5p2 ONLINE
[root@freenas] ~#
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
Hmmm,

Well I can't give you much hope at the moment. The last time I saw someone do something very similar to what you did and they got the corrupted metadata error, they weren't able to recover anything. They even tried sending their disks in for data recovery.

Here's what I would try, get a live CD with ZFS v28 (I'll post a couple of links). I haven't personally tried either of these, but ZFS v28 has some newer features that *might* make a difference.

I'm a little torn about whether to suggest putting the old disk back in or just sticking with the new one. I guess I'm leaning toward putting the old one back in. Since I personally haven't used either of these links below, I can't say one is easier or better than the other. They both have ZFS v28. I might go with the PCBSD Live DVD or possibly the Live USB. After you get one of these booted, post back the results of a "zpool import". Then wait and I'll tell you what to try next.

http://mfsbsd.vx.sk/

http://pcbsd.org/index.php?option=com_zoo&view=item&Itemid=98
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
Proto,
Thanks for hanging in there with me - I do appreciate it!!!

After almost a week of flailing in the wind on my own and trying a million different things before I asked for help, I am afraid I went too far. At this point, I've all but given up on a successful recovery so think of this as a learning experience for both of us. One of the million things I did was download the image at http://mfsbsd.vx.sk/ and had no luck but I don't remember if I had the new or old disk in. Here is the results of "zpool import" using the cd downloaded from there with the original failing disk. I don't know how to enable/configure ssh to allow me to putty in so this is just a brief rundown.

state: FAULTED
status: the pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.

FreeNAS
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
oopps... sorry about that.

FreeNAS FAULTED corrupted data
raidz1 DEGRADED
da1p2 ONLINE
replacing-1 DEGRADED
da2p2 ONLINE
135596100085734653217 UNAVAIL cannot open
da3p2 ONLINE
da4p2 ONLINE
da5p2 ONLINE
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
Tom,

If you're just getting the "corrupted data" and not "metadata", I think there's a small margin of hope. The other problem I haven't seen a solution for is getting the "replacing" status to go away (or continue). I'm glad you were already familiar with the mfsBSD Live CD, but you're right, asking for help sooner could have made a difference.

What was going on with the disk that was failing?
Did you try a scrub before you tried to replace it (not required, just curious)?


Below is a section from my blog, see how it goes using the mfsBSD CD:

zpool import -nfF tank

This will tell you to what point ZFS transactions would need to be "rewound" to import your pool. If rewinding to that time and losing data written after that time is ok, then repeat the command again without the "n" option which only shows you the actions that "would be" performed, so do "zpool import -fF tank" only if you're satisfied to actually do those actions.

The “import” option will report the exact number of “seconds” worth of data that cannot be restored. Depending on the bandwidth and utilization of your system, this could be very little data or several MB worth of transaction(s).

There is also a poorly documented -X option (v28 only?) which needs to be combined with the -F option which will do an exhaustive scrub like attempt to recover your pool. It can take quite awhile, HOURS if you have a lot of data and it is pretty much a last resort. It's used like this: "zpool import -fFX tank"
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
If you can't copy & paste from mfsBSD, you can always try taking a pic of your screen with your cellphone or other cam.
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
Well all too often I see people begging for help when it is obvious they didn't even bother to figure it out on there own. I guess I didn't want to be one of them. My previous post might be confusing because I accidently submitted it before I was finished. I do indeed have a "metadata" error.

state: FAULTED
status: the pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.

With that noted, here is what I tried next following your previous comments.

zpool import -nfF FreeNAS
- quickly returns to the command prompt with no text displayed

zpool import -fF FreeNAS
Cannot import 'FreeNAS': I/O error
Destroy and re-create the pool from a backup source.

zpool import -fFX FreeNAS
- It's doing something, wish me luck!!!


I really appreciate your help, thanks for giving me at least some hope! What is your blog? I should start reading it.

Thanks,
Tom
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
Tom,

I'll keep my fingers crossed, I know how much it can suck to loose important data, especially if you've done a backup and didn't realize it didn't work until you needed to recover something!

Hmm, my blog, there is so much that needs to be updated and added! You can find it at:

protosd.blogspot.com


*Fingers crossed*
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
It still seems to be chugging away, thanks for all of your help. I'll keep you posted!!!

Thanks,
Tom
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
Well it ran all night and this morning I had an error: I/O Error Disk unreadable. Do you have any other ideas?

Thanks,
Tom
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
Hi Tom,

The fact that it ran all night is good, I wouldn't give up yet. Where did you see the I/O error? Could you tell if the message was from the import command or from the OS? Did it give any other details like a device or anything?

Two ideas, one swap the bad disk with the one you were going to replace it with and try the command again, if that fails, try cloning the bad disk on Linux using "ddrescue" to the replacement disk. See how far ddrescue gets cloning (it can take AWHILE, like all night or day & night, or even a week a *really bad* disks). My thinking is if you can duplicate the failing disk as accurately as possible without it having any I/O errors and then swapping it and trying the import again, you *might* have a chance. There are 2 versions of ddrescue, the GNU version and the ? version. I *think* the default that Ubuntu installs is the correct one. It copies data bit by bit skipping failures until after it reads everything it can, then goes back and tries to get the failed spots. It really does a great job if you give it time.

If you were able to let ZFS do the replace, it would recontsruct the data on that disk anyway, but since you can't even mount/import the pool I think trying to get that to work is the key.
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
I accidently left something out. zpool import -fFX FreeNAS ran all night with the new drive and I got an error - I forget what it said, so I ran it again on the OLD drive and when I got home from work I got the I/O error. So the I/O error actually was with the old drive. I am going to attempt ddrescue, I'll keep you posted.

Thanks,
Tom
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
Well Proto, it's not looking good. I struggled with cloning the drive. The different ubuntu distros I tried always went to "sleep" after ten minutes or so and I couldn't wake them up. After a couple days of trying I finally was able to clone the drive



FreeNAS
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
That's strange about Ubuntu sleeping, it shouldn't do that if something's happening like cloning. Did you use a Live version that booted off USB?

So does that mean you've already put the clone in and tried doing an import again with all of those options that made it chug along overnight?

A clone of the failing drive without the IO errors should have a better chance of completing the import.
 

tefoster

Dabbler
Joined
Dec 15, 2011
Messages
26
The clone process finished without errors, the clone did have lost information, but the process did not fail. After putting the cloned drive in place and running zpool import -fFX for about 12 hours now I get the error "The pool cannot be imported due to damaged devices or data". Here is what zpool import currently returns.

FreeNAS FAULTED corrupted data
raidz1-0 DEGRADED
da1p2 ONLINE
replacing-1 DEGRADED
da2p2 ONLINE
13559610085734653217 UNAVAIL cannot open
da3p2 ONLINE
da4p2 ONLINE
da5p2 ONLINE

I've all but given up but if there is still hope I am willing to keep trying, do you have any ideas? It seems to me that if I can in some way get it to forget about replacing the drive it could work. All of the 5 drives are online.

Thanks,
Tom
 
Status
Not open for further replies.
Top