Help Unable to boot FreeNAS Trap 19 Error (Solved)

[Jerzy Sobski]

Need help Desparately. Running Freenas 9.10.

Today when I went to reboot Freenas It stopped in middle of boot with several errors. See attached screenshot. What can I do to get this booted and up running. I have about 39 terrabytes of data that needs to be accessed.

 

[melloa]

Please post your setup configuration as requested in the forum rules, so people can help you troubleshoot.

The pic states you might have a parity error on your RAM ... so, if you want, start by isolate your RAM problem by booting with the minimum number of sticks you can get your box running.

 

[Jerzy Sobski]

Info on System:
Server: Proliant ML10
Bios - P88 (07/02/2013)
32,768 MB Memory Configured
Processor Intel Xeon CPU E3-1220 V2 3.10GHz, 8MB L3 Cache

I have 2 - 10 Bay Towers.
- 1st Tower is complete filled with Seagate Constellation ES.3 Drives st4000nm0033 (4TB per)
-2nd Tower only 5 Bays are filled with Seagate Constellation ES.3 Drives st4000nm0033 (4TB per)

All 15 drives are under Vol0 with subset data sets.

I believe one of the drives had failed and I was about place drive but need to verify which drive it was so I was going to do a reboot and watch as bootup went through each drive identifying the #. Once I knew I had right drive I was planning on doing a replace with a spare drive I had sitting around. I never got to that point when the crash happened. Tried a second time and it occured a second time. Thats when I felt I needed to reach out for help.
--------------------------------------

Update I was able to boot with Freenas with both Storage Towers turned Off. Now that I have been able to bootup completely with drive off, how do I get the volume back online so that I can do the replace of the defective drive?

 

[SweetAndLow]

Start by adding the drives back in little by little and see which drive is causing your problems. What was your pool layout? Can you verify your freenas install and make sure it's not corrupt.

Sent from my Nexus 5X using Tapatalk

 

[Jerzy Sobski]

All drives were Part of Vol0 pool. Within the pool I had datasets approximately 7 media sets (Media, Work, Misc,, ect......) The Level that it degraded at was at the VOL0 Level. Attached are screen shots captured just before the the original post was put up.

How would I go about getting each drive online. To get Freenas to boot up completely, I turned the power off to both 10-bay towers. With them all ofline, what is the best way to start bring them back online?
Will Freenas be able to see the towers if they are turned on while Freenas is running?
With entire 15 drives part of the dataset, will it even be able to see individual drives as they come one?

 

[SweetAndLow]

Well it looks like that 2TB drive has failed in your raid z1 vdev. There is a very high chance you have data corruption and will lose the pool. What did you do before rebooting, if anything?

Sent from my Nexus 5X using Tapatalk

 

[SweetAndLow]

You should do a memory test or try booting with less sticks to see if you can find the bad memory model.

Sent from my Nexus 5X using Tapatalk

 

[Jerzy Sobski]

Prior to a reboot, All I did was a Status check on drives which is one of the screen shots uploaded. What is wierd is that 2TB should have been a 4TB drive. ( Is there a anyway to recover the ZFS pool if there was a data corruption)

As for the Memory test, how do I go about doing the test?

Wouldnt Freenas have crashed on bootup after Drives had been disconnected(Turned off) from the servier? (I have Freenas Running at present without the drives.)
It only seemed to crash while drives were connected (Turned on) during server bootup. When Towers with drives were turned off, it booted fine without any errors.

Earlier you mentioned bring drives on individually is that possible when all drives are part of that pool? Could it be possible that I remove just the drive that went bad and boot with missing drive in the pool since it is using ZFS.

 

[SweetAndLow]

You have a hardware problem. You need to figure out what is bad. Removing things is how you do that. Pick something and try it. Removing the bad disk is a good start.

Sent from my Nexus 5X using Tapatalk

 

[Jerzy Sobski]

SweetAndLow, First Thanks for all the Help. (thank you, Thank You)
I was finally able to identify the problem and have ZFS being resilvered with new 4TB drive that was used to replace. Just an FYI what I discovered and how I discovered it.

When I got hope I decided my first step was to map out each drive and the slot it was in and compared serial numbers to those that I had done a screen capture. What I discovered was:

In Tower 1: (This was the tower that was the original tower put on line with 10 4 TB drives.)
- The last drive in that tower was not showing up in the serial #'s. (This ended up being the drive that was causing all the problems)

In Tower 2: This tower was only to have only the top 5 drives online which were all 4 TB. While mapping it out I discovered that in the bottom row of 5 drives a 2TB drive had been pushed into bay completely. This should not have been in and Freenas for some reason (possiblely a bug) picked it up in place of the drive that was not showing up in tower 1 as part of the pool. Note: I have entire bottom 5 rows of this tower filled with drives but they are not inserted completely in primarly to muffle out the fan noise. I recalled that on 1/15 I was working under the desk where the towers are located moving stuff and its very possible that it may have been pushed in accidetnly as I was moving things around. (Not sure how the 2TB drive would have affected the drive on Tower 1.)


I started with removing the 2TB drive and the last drive in Tower 1. This solved the boot issue and the server booted up with entire pool online but degraded.

I then tried a 2nd reboot with the last Drive from Tower 1 reinserted. When I tried booting up again the error and crash occured.
I then replaced the last drive in tower 1 with a new drive and tried a boot once again. This solved the issue. Server booted up and loaded up the degraded pool. I then used the replace command to resilver pool with new drive.

All is well now and looks like I will recover all my data once the resilvering is complete.