Freenas wont boot becuase of error mounting volume.

[willhorner]

The error I get is

panic solaris assert 0==dmu_buf_hold_array.....
there is much more i can providce if relevant.
I installed a fresh copy of freenas to another usb stick, booted, and tried to auto import the volume and it gets the same error. Normally I'd just rebuild, but this drive has family photos I really would like to save. Can anyone help me get this array mounted long enough to get them off?

 

[cyberjock]

Did you have a UPS installed to this server? Stuff I'm reading indicates that there was file system corruption due to partial writes being performed during a power loss causing ZFS corruption preventing mounting.

Do yourself a favor and do not try to use the command line options that include force mounting(typically -f and -F). That's the last step and if you do that then any other attempts to recover your data are pointless.

Edit: I don't have immediate recommendations, can you explain the "story" behind how you got to this point. Either you have a hardware failure somewhere or there's a story to how you got to this point...

Also post your system specs and how you had it setup(ZFS type.. etc)

Also did you ever figure out what your problem was in your previous thread?

 

[willhorner]

First the easy part. No i never resolved the performance issues. i just lived with it.:)

Second
I wish there was a story here. Yes i have a UPS attached. But it didn't come into play here. Yesterday afternoon in the middle of watching a movie, the system simply rebooted. no error no power outage on the outlet, no warning. when it came back up, it halted mid boot with that error.

Config is freenas 8.3 booting of a usb stick
intel i5 2400
32 gigs ram

5 x 3tb drive array zfs single parity drive
7 x 1.5tb drive array zfs single parity drive
8 x 1.5tb drive array zfs single parity drive



as i said , i've installed to a clean usb stick, and removed all drives except the first array (5x3tb) and get the same error in the middle of the auto import. it sees the array, its tried to start the import and then dies, leaving me at a prompt that i don't recognize.

 

[cyberjock]

First the easy part. No i never resolved the performance issues. i just lived with it.:)

This was probably a very big mistake. If something is wrong it is important that you find the exact issue so you can at least understand if it is something that could cost you your data lateror if its just an inconvenience. In your case, now that things are much uglier, I'm thinking it might have been something that has cost you your data.

Second
I wish there was a story here. Yes i have a UPS attached. But it didn't come into play here. Yesterday afternoon in the middle of watching a movie, the system simply rebooted. no error no power outage on the outlet, no warning. when it came back up, it halted mid boot with that error.

Yeah, I have absolutely no recommendation at this point. I'm gonna have to think about this overnight and see if I happen to come up with any good ideas.

Config is freenas 8.3 booting of a usb stick
intel i5 2400
32 gigs ram

5 x 3tb drive array zfs single parity drive
7 x 1.5tb drive array zfs single parity drive
8 x 1.5tb drive array zfs single parity drive

Yeah, hardware looks good and is seriously overpowered for the issues you were having before now. There was/is definitely something wrong and it should have been found a long time ago. That's why I sent you the PM back then. There was(in my opinion) something very wrong with your setup.. hardware and/or software.

I would do a RAM test and make sure your RAM is good. Not that I have any clues that its bad, but its easy to make the memtestx86+ cd and let it run overnight. Typically 3 passes without a failure means the RAM is good.

 

[willhorner]

I suppose you could be correct. But other than not knowing you had sent me a pm, i tried everyone's suggestions. and it was completely unaltered install of freenas config wise (by that i mean yes i had mounted drives and set up a jail, but i had done no tuning or changing of any of the default environment variables) it has run fine if slowly since then. Im definitely grateful for any help you can offer, and for the time already spent. I ran mem tests overnight last night and they came up ok (that was my first assumption, my second was that the system may have overheated, but i can find no evidence of that either). i just cant imagine how it could have been fine 1 minute, and literally 5 mins later it be so corrupted that there's no way to mount the array. Is making a freebsd bootstick worth trying to see if i can get it working with its toolset? or is there anything do able from a command prompt? (sorry, im sure if there was you'd have suggested it already).

 

[willhorner]

a bad pic of the error

 

[cyberjock]

There really are no tools to correct heavy ZFS corruption. That's why my guide has a slide that says:

ZFS has very few “recovery tools” unlike many other file systems. For this reason, backups are very important. If the zpool becomes unmountable and cannot be repaired there are no easy software tools or reasonably priced recovery specialists you can use to recover your data. This is because ZFS is enterprise-class software, and no enterprise would waste their time with recovery tools or data recovery specialists. They would simply recover from a known good backup or mirror server.

In my slideshow I also have:

• Having a UPS is virtually mandatory for all servers.
• This is because a file system is a living document and is constantly changing. Even when you aren’t using it there are occasional reads and writes as maintenance and system tasks are performed.
• Any partial write performed on the file system due to a sudden improper shutdown(i.e. a loss of power) could render the file system corrupt. This is an inherent vulnerability with all file systems and is the primary reason why a UPS should be used to protect file servers in particular.
• You do not need to use a very expensive UPS. The UPS should be sized and setup to provide at least 8-10 minutes of power so that the server can shutdown itself in the event of an extended loss of power.

Things can(and have) gone from perfect to horribly crappy in milliseconds. That one bad reboot can ruin it for everything. Both of those slides I quoted above are the results of blood, sweat and tears from forum users that have lost everything because of inadequately planning, building, and maintaining their server. Lucky for you, you did have an UPS so those risks weren't likely.

Strictly speaking on statistics, your chances of recovering data is less than 5%. In fact, I've only seen 1 person recover their data after a zpool was made unmountable. It took him the better part of a week of trying various commands and waiting for them to finish. I believe he also had to buy another SATA controller to rule out the controller as the cause.

 

[willhorner]

I think i found the culprit. the power cable on the back of the system had worked its way loose. when i moved the system to check something it powered off and back on. So i imagine that was the problem, something must have brushed into the cable. although i cant imagine what.

- - - Updated - - -

I'm willing to spend a week trying if you can point me to a list of the commands to try.

- - - Updated - - -

does freenas not store a secondary copy of its allocation table equivalent somewhere on the disk?

 

[cyberjock]

It does not. Corruption is considered "impossible" with proper design and implementation. That's why I make all the recommendations I do regarding ECC RAM, UPSes, not tweaking stuff, backups, etc. If you do everything right you can have a very reliable setup. I won't lie though.. right now I have 10TB of stuff with no backup and I have several friends that just don't backup their ZFS protected stuff at all because they consider ZFS to be "uncorruptible" regarding their implementation.

I'm sending you a PM.

 

[paleoN]

as i said , i've installed to a clean usb stick, and removed all drives except the first array (5x3tb) and get the same error in the middle of the auto import.

Do the other two arrays import?

does freenas not store a secondary copy of its allocation table equivalent somewhere on the disk?

ZFS doesn't have a FAT.

Try installing a mfsBSD 9 to a different USB stick and attempt to import from there.

Edit: Come to think of it, you can just use one of the nightlies. Attempt the import from the CLI.