I finally got a call from a rep at KrollOnTrack today. He confirmed that 2 drives mechanically failed and that they did a sector copy of the recovered data onto two brand new drives.
Quite possibly had to remove the platters from the mechanically dead disks and install them in donor drives using a clean room. In which case I wonder why they could only recover 98% of the data, unless the platters were also damaged?
I also voiced my frustration that I was originally told we would be getting FILES back, not a bunch of drives still in an array. His response was that if this were a "normal" RAID configuration (like RAID5 or RAID10) that it wouldn't be an issue, but that because it's RAID-Z, it's something they haven't run across before so would have to reverse-engineer a tool to extract and reassemble the actual files.
That's a really, really poor excuse from Kroll - I find it hard to believe Kroll haven't "run across" ZFS before, considering it's likely to be fairly widely used in enterprises, ie. their bread & butter business (or so I'd imagine).
As for cloning the disks... I've never done it before, but I'm confident that with the right guidance I can make it happen. We already have three 2TB disk at the office that I purchased on the day we discovered the failure. I attempted to replace three of the drives I thought had died, hoping to rebuild the array... but obviously that failed. So buying one more 2TB drive isn't a stretch. What software would you recommend I use to clone the drives? I'm assuming this is something I'll need to run from my main computer and not the FreeNAS box.
You should be able to use your FreeNAS system, you just need console access. What you need to do is identify the disks - I would suggest connecting one "original" disk (source) and one "new" (destination) at a time, boot the system then use "dmesg" to check which is which (or run "smartctl -a /dev/daX" to obtain the serial number for each disk).
Obviously, correctly identifying the source and destination disks is of paramount importance - if you're not sure, do nothing and ask here!
So, once you have identified the disks, you just need to use the dd command to transfer the data, eg:
Code:
dd if=/dev/daX of=/dev/daY bs=1M
where /dev/daX is the source and /dev/daY is the destination. The cloning of each disk will take several hours, possibly up to 12 hours for 2TB disks, but at least you'll have your original disks as backups should it all go wrong.