Defective USB boot drive

[Job76]

Hi,
I am a novice to FreeBSD, FreeNAS and new to this forum.
I have encountered a serious problem with my FreeNAS installation and after searching for hours for an answer, I am hoping for your help!
I would, if possible, like to know how to:
1: restore and recover my USB boot memory
2: rescue the data on the hard drives, and attach them to a new installation, if the first option is not available.

I have accidentally created this problem myself, by powering off the server with the front button, by holding it in… Something on my network stopped working and with the server in another room, without any attached monitor, I did not know if it was down or not. I pressed the power button and waited, in hope that it would power down correctly but nothing happened, so I tried again and waited, and still nothing.
So I pressed the power button for more that 3-5 second and it turned off…
The problem was a fault network switch and not the server, and after I had replaced the switch, the server refused to boot.
I have four 3TB HDDs connected in a ZFS raid as data storage (one drive as spare), and booted FreeNAS from a USB memory.
This USB memory no longer boots as the server says it’s not a boot disc.
I am not even sure of the file system on the USB since I just went with the default install option.
I have tried to boot FreeBSD Live CD, and run a “fsck -t usf /dev/da1”
(I presumed da1 was my USB stick since da1 was listed when I inserted it)
It says something like “following disk sectors could not be read; 128, 129.. etc. up to 143.
ioctl inappropriate ioctl for device”

Best case would be if it was possible to repair the faulty USB, but I am ok with a new installation, but I would really like to keep the data on the hard drives.
This installation is new, and I think I updated it to 9.10, but I am unable to verify the version number since it does not start. I do not have any critical data on these drives, only movies.
I spent hours on configuring Plex and searching for ways to automate the update process…
I spent days transferring my legally purchased DVD movies in to MKV format and adding them to Plex.
I guess that I will not actually lose any critical data on a complete re-install, but I prefer not to do this, since that would mean that I have to transfer all DVD’s again, and I do not think that I actually will do that process again. It was way too time consuming.

I know that this is my own fault, and that I do not have a backup. This server is newly purchased, with 16GB ECC RAM, etc. It was purchased for this purpose only.
I was planning to purchase a UPS, but this happened first, and a UPS would not have mattered anyway. Backup and the backup process is something that I had not taken the time to do yet… My reasoning behind that was that it was too much data to backup anyway, and with the original DVD’s still in my bookcase, and with one drive to spare, I should not lose any data.
I never considered that the actual boot drive/database needed to be backed up, or copied... or that a USB crash would render all HDD data useless.
I really wish I had… I estimate that I had it up and running for something like four weeks before this happened…
I am thankful for any help, guides, links or suggestions that you might give me!

 

[dlavigne]

What version of FreeNAS are you running? You should be able to install that same version to another USB stick. If you saved a copy of your config, you can restore it, otherwise you will need to recreate your configuration. The data on your storage disks is separate so should be fine.

 

[Job76]

I think it was 9.10, and I have installed 9.10 to a separate USB, but I do not know how to re-attach the raid without destroying the data. I have not found any guide on how to reattach the raid to a new installation.
I do not have a backup of the config. Its possible that this is stored somewhere on the "old" usb, and that its still accessible, even if the usb will not boot, but I do not know where to look for it, and I do not know how to retrieve it!

 

[Mirfster]

but I do not know how to re-attach the raid without destroying the data. I have not found any guide on how to reattach the raid to a new installation.

FreeNAS 9.10 Manual; Section "8.1.6. Import Volume"

 

[Mirfster]

Also, once you are "Up and Running", you may want to take a look at these:
@DrKK 's: How-to: First Configuration for Small FreeNAS Deployments
@Bidule0hm 's: Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config

 

[Job76]

Thank you! It would seem that I missed that part in the manual! But it seems that it’s not only the USB boot that is defective, but the entire pool...?
Import doesn’t work, and if I run a shell with zpool import, it still does not work!

So I followed http://illumos.org/msg/ZFS-8000-EY and ran zpool import -f, but that does not work…

I guess that the last thing left is to reformat everything and restart from the beginning!?

 

[Mirfster]

Per your screenshot and the comment below:

I have four 3TB HDDs connected in a ZFS raid as data storage (one drive as spare)

All this points to the fact that you were running a RaidZ1 configuration; which only allows for 1 drive redundancy. While you did mention one was a "Spare", per the screenshot it looks like all 4 drives were part of the vDev. So, I would surmise that you lost two drives in the RaidZ1; which equates to total loss of the Pool.

FYI, RaidZ1 is rarely (if ever) recommended; especially when using drives over 1 TB in size.

 

[joeschmuck]

I concur, it appears you lost two drives which had data on them and thus a loss of all your data because you were running a RAIDZ1. You should review the Hard Drive Troubleshooting Guide and run a SMART Long Test on all your drives. I suspect that you do not have email notifications set and/or do not run routine SMART tests on your drives. Not knowing your system status is a real killer.

I doubt the fact that you powered off your machine was the reason your pool died, it was likely the fact that you had two failed drives that took your system down.

Also, in the future, do not physically connect up a spare hard drive until you need it. If you setup FreeNAS properly then you will have time to connect the drive and start the repair process. BTW, the repair process is not automatic not matter how you slice it.