Need a Freenas expert to help get us out of this mess

[joelsawyer]

Sorry, yes I did do -D.

There IS four drives, protosd. I made sure everything was plugged in properly and did it again with the same results as above. Not sure what the problem is. Any ideas?

 

[ProtoSD]

Try doing another 'gpart show' and see if all four listed, for some reason ZFS only sees 3 or thinks there were only 3 in your pool.

Did have 4 disks from the very beginning?

I don't remember, did you say what the situation was leading up to the failure?

 

[freeflow]

No there are four disks. One is OK three are saying degraded.

 

[joelsawyer]

On the first day of our disaster, an IT contact told me to buy three new drives and try to rebuild each of the three harddrives showing errors. (I was initially getting sector errors on three drives, but Kroll only found two of the drives to be bad)

So that's what I did. Last time I left this system, there were three new harddrives and one original. Could this be an issue?

 

[Durkatlon]

May I just say that you are (or at least appear to be) remarkably stoic during this whole ordeal. I'd be climbing the curtains and screaming in agony by now.

 

[joelsawyer]

Thanks for noticing. I feel like I've lost a few years off the end of my life, but trying to keep a positive outlook on this.

 

[ProtoSD]

On the first day of our disaster, an IT contact told me to buy three new drives and try to rebuild each of the three harddrives showing errors.

Well that explains why all three say 'replacing'.

Sorry if these seem like stupid questiona, but I'm just trying to verify/get a clear picture.

Were you trying to replace all three drives at the same time?
Did any of them finish?
What happened to the drives you were doing the replacements with (new ones)?

I'm assuming the replacement(s) didn't finish, you took out the new replacement disks, sent them along with the old disks for them to restore stuff to?

 

[joelsawyer]

not a stupid question at all.

I replaced one drive at a time. Put it in, booted, clicked replace in the gui, waited for it to replicate, then shutdown and went onto the next drive. All three finished.
The "new" ones we kept. It was just the four "original" drives that were sent to Kroll. I assumed (and maybe incorrectly) that since the three replaced drives didn't resolve the issue, that they were useless. I hoped that restoring the broken drive, and just sticking them back in would work.

Did I make it worse by doing that?

 

[ProtoSD]

Hmmm, I don't think you made it worse, it *might* actually be better!

So you still have the 3 drives that finished replicating and haven't done anything with them since?

If you can figure out which one of the 4 you got back was the remaining disk that goes with the three that finished replicating, and add the clone you made of it to the other 3 (which you should clone to the other 3 spare disks), and put all 4 of those together and try and import them, you might have another chance!

I hope that made sense? Does everyone else following this thread follow what I'm suggesting? What do you think?

EDIT: The more I think about this the more encouraging it is. The 3 drives that finished should be 'clean', not in a 'replacing' state like the original ones. The 4th disk, the one in the zpool status that doesn't say it's degraded, might be the match that gives you a complete pool that is importable!

 

[joelsawyer]

uhhh... remember when we decided to clone the original four drives? I took those three "new" drive plus one more recently purchased one and used those to be the clones.

 

[ProtoSD]

Ohhhh Sheeeet!!!! That SUCKS! I was really thinking you had a good chance! Damn....

 

[survive]

Hi joelsawyer,

It might be worth it to contact these guys:

http://www.joyent.com

It's run by a bunch of ex-sun guys who should be able to point you in the direction of a data recovery company who has experience with damaged ZFS volumes.

-Will

 

[joelsawyer]

FYI, I'm in contact with Steve over at myharddrivedied.com. He says he has experience with RAID-Z (though he called it Z-RAID) and FreeNAS recovery.

Hopefully he can help.

 

[mr_clark]

joelsawyer...whatever happened with this. I'm new to FreeNAS but reading your story was like a mini soap opera. Did you manage to recover your data?

 

[joelsawyer]

Sadly, no data recovery. We've resigned ourselves to the fact that we're not going to get our data back, despite Kroll's insistence that there's still a chance to recover usable files from the raid. At a cost, of course. They're wanting an additional $14,000 to try a build a tool to recover the files.

At this point, it's not worth it. The issue now is try to recoup some of the initial $6,000 we spent with Kroll that resulted in no data recovered. We're trying to work with our insurance company to honor a liability claim. This is a whole opera in itself. The insurance adjuster assigned to our claim is about as clueless about the workings of hard drives as a rock. Not to mention the whole concept of a RAID array and why specifically we can't recover our data.

This opera, I'm afraid, has a sad ending.

 

[survive]

Hi joelsawyer,

Before you give up, contact somebody at Joyent or try your local Oracle office for a referral to a SUn specialist. You can't be the first person to run into a problem like this...you just need to talk to somebody who has some insight into this particular world.

-Will