Help recover my data please. Freaking out.

[3dmdlr]

well thanks all for the support today. i definitely need to step away and clear my head. i'll be gone all day tomorrow, plenty of time to relax and start fresh on saturday i guess. btw where do i go to get an ix ticket? thanks again!

 

[mjws00]

I'd just call them. 1-855-GREP-4-iX (toll-free) https://support.ixsystems.com/

@cyberjock may have a smarter way. It doesn't mean you are out of the woods, but they should be able to at least confirm the severity. Honestly, if you can get a nice solid platform running, getting a few commands in to see what is up is easy for them remotely.

Good luck.

 

[Noctris]

Well that didn't go as expected.

I honestly think you are still a long way off from needing 4 figure recovery. But i'll also admit that i'm the type of it guy that will only call support when i have exhausted my own bag of tricks, which grew to a rather large collection after 16 years in the field of it. I don't feel i'm at that point yet. I've pruposely killed several pools and attempted (done) recovery before starting to use zfs in production in my smb and i'v seen/done ugglier stuff then this. But I do recognize however that, if entering some commands is already hard, doing back and forths on a bulletin board might not be the best way to go.

I do however strongly support mjws00's view that you MAY have been extremly lucky and at least one of both disks is ok. His approach of first getting a stable system with a test disk is indeed a slightly better version of my proposal ( i can live on the edge a bit more since i do have backups :s

Ultimately it is up to you to decide where to go. If you don't feel comfortable with executing commands, swapping disks etc, ix support maybe the best way to go ( and i don't enjoy bringing bad news either, just in case so). If you do want to do an attempt for yourself first, let us know.

Meanwhile: try not to get killed by your wife :)

Sent from my portable microwave

 

[3dmdlr]

Thanks again. Having slept (a little anyway) It's horrible having a dream about losing data and how to save it and, uh, yeah really weird hahah. I am a little calmer today though. I plan to call ix today just to see what my options are. I am also going to create a new box using an old shuttle I was using for the HTPC this weekend. Once I check that off the list I will then be back on here to get help with the next step. And just breathe a lot :) So stay tuned gurus, I am deeply thankful thus far and hope to see you guys help me thru to the end.

 

[Robert Trevellyan]

When I read this thread and I see the output of zpool import, I find myself thinking, "I bet I could get that data back lickety-split." Of course, I might be deluding myself, since I've never actually done recovery on a ZFS pool. Anyway, I mention this because, unlike some of the other participants, I'm at least on the same continent as you. I might need help from the forum too, but at least I'm a techie so I know what it means to use the CLI.

Depending on how things progress, you might want to consider the option of carefully removing the two hard drives, placing them in anti-static bags, wrapping them in a two-inch thickness of bubble wrap, and Fedexing them to me.

Send me a PM if this ends up looking like a good option.

 

[cyberjock]

When I read this thread and I see the output of zpool import, I find myself thinking, "I bet I could get that data back lickety-split." Of course, I might be deluding myself, since I've never actually done recovery on a ZFS pool. Anyway, I mention this because, unlike some of the other participants, I'm at least on the same continent as you. I might need help from the forum too, but at least I'm a techie so I know what it means to use the CLI.

As someone who has done ZFS recovery, I can tell you that the zpool output not showing a problem often means things are much worse because ZFS isn't aware of the corruption, or is crashing before it can actually log the problem. I'd rather work with a pool claiming to be corrupt than one that is corrupt but doesn't know it. This is along the same lines that are often a sign of failing RAM. :/

Depending on how things progress, you might want to consider the option of carefully removing the two hard drives, placing them in anti-static bags, wrapping them in a two-inch thickness of bubble wrap, and Fedexing them to me.

Send me a PM if this ends up looking like a good option.

The problem I have with offers like this is that it is easy to run commands from the CLI that are permanent and unrecoverable. If the data is as important as he claims, he should be looking for someone that specializes in ZFS recovery.

Unfortunately, in my years here I have told quite a few people that their data appeared to be recoverable until they started doing stuff they didn't understand. At which point it went from recoverable to unrecoverable. And by unrecoverable I mean either that the data has an extremely low chance of being recovered because metadata is totally missing or that they'd need 5-figure recovery services (which is pretty much the "starting price" if you go to a company that actually specializes in data recovery). For most of us mere-mortals, needing even $10k to pay for recovery is as useful as an option as claiming there are no options. :P

 

[Robert Trevellyan]

As someone who has done ZFS recovery, I can tell you that the zpool output not showing a problem often means things are much worse because ZFS isn't aware of the corruption, or is crashing before it can actually log the problem. I'd rather work with a pool claiming to be corrupt than one that is corrupt but doesn't know it. This is along the same lines that are often a sign of failing RAM. :/

Point taken.

Unfortunately, in my years here I have told quite a few people that their data appeared to be recoverable until they started doing stuff they didn't understand. At which point it went from recoverable to unrecoverable.

Yes, and what I like most about @3dmdlr is in the very first: "I am scared to just start trying things for fear of losing otherwise recoverable data."

 

[cyberjock]

Point taken.

Yes, and what I like most about @3dmdlr is in the very first: "I am scared to just start trying things for fear of losing otherwise recoverable data."

Me too. Being that fearful is good. There's dozens of threads where users have sabotaged themselves because they show up and ask questions about getting their data back. Then then find a bunch of websites that involve commands that don't even apply to FreeBSD, and they start running them in various combinations trying to invoke the power of the great ZFS gods to save them. This is generally where things go totally bonkers and stuff ends up permanently broken.

When I used to do ZFS recovery as a side-job, my conditions for recovery was based on what they had tried, what the output of zpool history showed (if it could be obtained) and the output of "zpool import".

If I didn't think there was even a remote chance because of the stuff that had been tried, I generally would refuse to do the work, or I'd do the work with it in writing that I felt the recovery chances were extremely slim. I didn't want the person needing recovery to have unrealistically high hopes of what would happen nor did I want to be blamed if the data wasn't recovered. ;)

If it was determined that your RAM was bad, I wouldn't even try to do recovery. I had done several of those and I knew better than to try to play that game. They were always damaged so severely that it was pretty amazing and beautiful (in a scientific way only) how it would get to where it was.

Data recovery is one of those things that is high stakes. There's a lot on the line and it's something that either works brilliantly or not at all when dealing with zpools. Very few people get something between 1% and 99% of their files back. It's generally 0% or something like 99.999% of their files.

 

[Robert Trevellyan]

Data recovery is one of those things that is high stakes. There's a lot on the line and it's something that either works brilliantly or not at all when dealing with zpools. Very few people get something between 1% and 99% of their files back. It's generally 0% or something like 99.999% of their files.

Yes, I've only done a few but it's been 100% file recovery each time, as far as I could tell. But only NTFS and HFS+, never ZFS.

 

[cyberjock]

Yes, I've only done a few but it's been 100% file recovery each time, as far as I could tell. But only NTFS and HFS+, never ZFS.

Oh, ZFS is a very different beast. ZFS depends on everything being uncorrupted (or corrupted but repairable with redudancy) to function. As soon as you break that basic philosophy, things go ugly real fast.

That's why we are so adamant about avoiding RAIDZ1, using non-ECC RAM, or anything that could, in theory, compromise your zpool. Once done, undoing it is difficult, if not impossible. It doesn't take much for things to blow up in your face in some ugly way.

So long as the zpool never gets corrupted beyond the ability to repair ZFS, things are great. But it's like a cliff, once jumped off there is only one way to go.. and it is painful and often fatal.

This is also why I do scrubs twice a month, and recommend them that frequently when feasible. I want some assurance that everything I think is there is actually there, because someday I'll have to rely on redundancy to replace a disk. I want confidence that I will be able to rebuild using the redundancy that I've verified that I have via scrubs.

 

[Robert Trevellyan]

I understand.

This is also why I do scrubs twice a month

Same here.

 

[Bidule0hm]

It's just me or is this a scam?

 

[Robert Trevellyan]

Looks pretty scammy to me. Brand new member, no previous postings. Where's the evidence for such a confident diagnosis?

 

[pirateghost]

This is 100% a degraded drive, there is not a software solution for this. I can resolve this problem for you.

No. You can't. Because your account is about to be suspended for spam.

 

[gpsguy]

Perhaps it's cyberjock in disguise. [emoji57]


Sent from my phone

 

[cyberjock]

It is not me. Had 3 mods talk to him in PM already. I won't say his offer isn't genuine. But the service he's offering (imaging the disk) is not what is needed at this point.

I'd be very leery about offering to send your disks to someone that made an account today, has no reputation but is offering services. *I* would never take that kind of offer. I know the mods here, and if they offered it, I'd probably take it. But that's because they have an established reputation of knowing and understanding ZFS. It appears that the user simply has an expensive imaging system and optional recovery software, neither of which seem appropriate for the problem at hand.

If he's wanting to get in the business of ZFS recovery (regulars know there is a demand, albeit small) this is definitely not the way to go about it. The hardware and software he has mentioned in PM makes no mention of supporting ZFS and almost certainly doesn't support ZFS. Any company designing tools for ZFS would most certainly mention ZFS support because that is a market that they would be the only choice in town. ;)

Establish yourself as a knowledgeable person in all things FreeNAS and/or ZFS. But an imaging service is not what is needed here.

 

[Noctris]

Kinda curious if you got this fixed vs your wife smacked you into oblivion using a large trout?

Sent from my portable microwave

 

[3dmdlr]

Noctris
Thank you for asking. Lol, no, no smacks yet. The short of it is, I know enough to be dangerous, obviously. So rather continue to potentially destroy any hopes of recovery I have turned off, unplugged and given up any attempts at self recovery. I have started the process with a firm, not sure if I can plug them or not, that seems legit. Fingers crossed they are, reviews look good and they are semi local (5 hr drive) if I need to pay them a visit. The hit sounds like it will be anywhere from $800-$5900! Ouch, shame on me I guess. They sort of got you by the shorts. Post recovery, if all goes well, I am now working on the new crash plan. I am researching a decent cloud service, 3.0 USB desktop HDD and a new NAS, maybe the FreeNAS Mini. I want to be able to pull a failing drive and stick in a spare e z p z. Paired with some decent syncing software I am hoping it will be enough. I can never let this happen again. My hear t couldn't take it.

 

[Noctris]

If you don't feel comfortable, handing it over into capable hands maybe the best decision.

Just make REALLY sure the guys you trust upon know their stuff on zfs. This is not your usual file system to recover ( for example: to my knowledge, no actual recovery tools exist , and i am still thinking this will not become a disk recovery rather then a "put disks in different machine and run zpool import -f" but ok..). I think a mention is ok since you are not advertising them or working for them so a happy customer can say he's happy ;-)


Eitherway: best of luck !

 

[Darren Myers]

best of luck, and godspeed! I saw this thread and read through it but...could offer no support nor input. However if youre looking for cloud i have 27Tb of data with Crashplan Pro, yes 27Tb...unlimited storage. Now mind you most of this is archives as i set it to archive forEVER, only about 10Tb of it is current on my server, they have great support, everyone i have spoken to speaks very good english and dont sound foreign (to me it matters...) they offer a "seed" drive where you back it up to there external, ship it to them, they plug it into there datacenter and your shiz is right there rather then waiting for a month or 3 while it gets uploaded, and they offer a recovery drive where if something (like this) happens they send you a drive with your data.

They also offer 448 bit encryption, http://www.code42.com/business/

P.S i do not work for them, i like them and there support very much, and have been very helpful. I pay i think its 16$ a month for unlimited cloud backup