Data recovery scenario

[auralsun]

I have a somewhat tragic FreeNAS scenario and am trying to determine the best option for data recovery.

One of my clients had a FreeNAS server in his office, set up by another developer. Set-up looks pretty straightforward: two 1.5TB hard drives running ZFS on RAID1. The server was working fine until last week when one of the hard drives failed.

When I visited the office, I found that the flash drive on which FreeNAS resides was missing, and to make matters worse the client does not have a backup of his FreeNAS configuration. We're not sure what version of FreeNAS the server was using, and the theory is that the flash drive was snatched by the business owner's nephew and is now lost.

So, I now have a 1.5TB FreeNAS data drive on ZFS in an external enclosure. My plan was to bring it home, plug it into my home FreeNAS server, and find some way to recover the data. But apparently the FreeNAS GUI cannot import external ZFS volumes (only UFS), and importing ZFS volumes using FreeBSD 9.2 seems to be a tricky proposition.

Please let me know what you guys would suggest? Would it be easier to mount the external hard drive to FreeNAS or using some other method? To clarify, we are not trying to recover the client's FreeNAS server entirely -- just the data on the hard drive. Are there any how-to guides / documentation geared toward this type of scenario in particular?

My home FreeNAS server is running FreeNAS 9.2, by the way. Any tips would be greatly appreciated!!

 

[BigDave]

@cyberjock is our resident ZFS recovery Guru in these parts, but he's been too busy lately
and has not been around a whole lot. In case he responds, I would gather up ALL the info
on the original setup (and post it here) that you can, i.e.

• importance/value of the data involved
• weather or not the volume was encrypted
• anything else that might be of value

 

[Robert Trevellyan]

I now have a 1.5TB FreeNAS data drive on ZFS in an external enclosure.

Please clarify exactly what you mean by this.

Please let me know what you guys would suggest?

I think your plan to install the drive in your existing FreeNAS is viable, assuming the pool was a ZFS mirror. If it was a stripe, the pool is already lost. If it was hardware RAID1, the challenge will be greater.

I believe you won't be able to auto-import a degraded volume from another system, but you should be able to force it to import. Don't attempt this until directed to do so by an experienced member.

For now, attach the drive, type zpool import at the command line, and post the output between CODE tags.

 

[Mirfster]

Hmm, interesting. If I were presented with this situation, I would start by making a complete clone of the drives first.

Thinking I would:

1. Use CloneZilla to make the complete backup/clone of the drives to a network share.
   • Sure you could use other tools as well or even P2V it directly to VM, but using a tool like this will allow you to easily make clones to physical or virtual systems (if needed)
2. Create a Virtual Machine (ESXi in my case) and create hard drives that are the same size (or larger) as well as one for the FreeNas install.
   
3. Mount the CloneZilla ISO on the VM, boot up to it and apply the drive images to the Virtual Hard Drives I created
4. Shutdown the VM and take a snapshot (just to be able to revert if needed)
   • This way I can easily revert and perhaps try other versions of FreeNas if needed or even other Tools/OSes (worst case)
5. Install FreeNas (whichever version you want to try) to the VM
6. Do all work or recovery attempts in the VM so you are not risking the physical drives

Edit: *** This is just me "spitballing" and has not been done by me personally. FreeNas may not even like the fact that the drives may not have the same serial numbers so it is unknown how well this would work out.

Edit #2: I just looked at the drives in my VM instance of FreeNas (they are VMDKs) and they have no serial numbers... S0, that may be a "good" sign?

If you are successful, then I would recommend that you tell the client they need to:

1. Get rid of the USB and use an internal SSD (or two mirrored)
2. Upgrade so at the very least they are able to run RAIDz2
3. Backup, Backup, Backup... Configs and Data
4. Lock the Server in a closet and not leave it where it is so easily accessible

 

[auralsun]

BigDave:

The volume is not encrypted. It contains various business data that is not essential to daily operations but which they would like to recover.


Robert Trevellyan:

I grabbed the drive from the FreeNAS box in my client's office and brought it home, thinking it would be easier to recover the data using my tools (a FreeNAS server, an external enclosure, various unused but functional boxes which I could boot to FreeNAS using a flash drive) than using the Windows laptop I had on me at the time.

The drive is using ZFS file system and was previously set up in FreeNAS with software RAID1. I'm missing various details about the FreeNAS server as it wasn't properly documented: FreeNAS version, shares, permissions, and so on. Luckily the goal is to simply recover the data on the drive -- not to recover FreeNAS entirely. Client already has an alternative storage / backup solution in place and simply hasn't done anything with FreeNAS in ages because they frankly don't understand how it works -- the guy who set it up initially jumped ship.

To clarify: my FreeNAS box can accommodate six SATA devices, and all SATA ports are already taken. I would prefer not to remove one and degrade my server's current RAIDZ2 set-up in order to recover my client's data, if possible. Can I import the volume if the drive is attached via an external enclosure?

If I need to connect the hard drive via SATA, I have another unused box I can throw the hard drive in, with a fresh FreeNAS install on a flash drive. If you could, let me know which you think will make more sense?

 

[Robert Trevellyan]

The drive is using ZFS file system and was previously set up in FreeNAS with software RAID1

No, it wasn't. I'm not being picky, you're in a tricky situation and it's important to be very clear. In this case, your terminology needs to be correct. If there were two drives in the original box, and there was redundancy in the pool, then most likely it was a ZFS mirror. Please confirm.

I would start by making a complete clone of the drives first.

Oh, yes indeed, thanks @Mirfster. In fact, best practice is to make a clone, then use a clone of the clone for recovery (i.e. your next step is to purchase two 1.5TB or larger drives). That way, you can make another copy of the clone if you screw up, without touching the original drive again. I would choose GNU ddrescue over CloneZilla, because it can recover from an interrupted copy, but that's up to you.

Once you have a clone, you could attempt to import the volume using a USB enclosure, but eSATA would be better.

 

[auralsun]

I think we can mark this problem as resolved. I cloned the drive per your suggestions, using CloneZilla as I'm already familiar with it. I attached the cloned drive to FreeNAS via USB enclosure, and the FreeNAS web GUI was thankfully able to auto-import the volume in its degraded state. Data has now been recovered.

Many thanks to everyone who contributed advice! :)

Robert Trevellyan:

Sorry -- I was confused because the name of the volume is "RAID1". When I got the drive imported to FreeNAS and ran "zpool status", the volume shows up as "mirror-0" which I assume means that it is a ZFS mirror.

 

[Mirfster]

Glad things worked out.

 

[Yatti420]

Good ol joys of ZFS :)