ZFS import crashes freenas

[kleinejan]

Hi, after upgrading to 9.2.1 every thing worked like a charm. Yesterday my nas suddenly crashed.
The console displayed:

Code:

KDB: enter: panic

Rebooting did not help, so I made a new USB stick.
Now, when I try to import the ZFS volume (GUI of CLI), my console gets tons of errors and reboots. Errors are to fast to read. ;)

Really no clue where to start, look next. The forum did not give me the answer yet, have sifted a lot of pages. :-|

System:
Non ECC Ram 16Gb (working on it)

ZFS pool:

Code:

[root@freenas] ~# zpool import
  pool: grinder
    id: 13947999836474807747
  state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
 
    grinder                                        ONLINE
      raidz2-0                                      ONLINE
        gptid/f28d9c40-7309-11e3-bac0-60a44cb1b1b3  ONLINE
        gptid/f30ff67e-7309-11e3-bac0-60a44cb1b1b3  ONLINE
        gptid/f379853f-7309-11e3-bac0-60a44cb1b1b3  ONLINE
        gptid/e523a987-8153-11e3-8c42-60a44cb1b1b3  ONLINE
      raidz2-1                                      ONLINE
        gptid/f44f00eb-7309-11e3-bac0-60a44cb1b1b3  ONLINE
        gptid/f4a243ce-7309-11e3-bac0-60a44cb1b1b3  ONLINE
        gptid/f4fe1a7f-7309-11e3-bac0-60a44cb1b1b3  ONLINE
        gptid/f56d471b-7309-11e3-bac0-60a44cb1b1b3  ONLINE

Gpart status

Code:

[root@freenas] ~# gpart status
  Name  Status  Components
ada0p1      OK  ada0
ada0p2      OK  ada0
ada1p1      OK  ada1
ada1p2      OK  ada1
ada2p1      OK  ada2
ada2p2      OK  ada2
ada3p1      OK  ada3
ada3p2      OK  ada3
ada4p1      OK  ada4
ada4p2      OK  ada4
ada5p1      OK  ada5
ada5p2      OK  ada5
ada6p1      OK  ada6
ada6p2      OK  ada6
ada7p1      OK  ada7
ada7p2      OK  ada7
da0s1      OK  da0
da0s2      OK  da0
da0s3      OK  da0
da0s4      OK  da0
da0s1a      OK  da0s1
da0s2a      OK  da0s2

 

[cyberjock]

Have you done a RAM test? You're already in a bad way if its rebooting when you try to import it. I wouldn't try to do anything from the CLI until you determine what actually went wrong. Kernel panics are usually the result of hardware problems. Might be useful to post your hardware specs too.

 

[kleinejan]

Try to figure out how to to the RAM test, getting back on that one.
dmesg for hardware specs:
http://pastebin.com/L2jp2nU0

I did try to swap ECC memory into the box last friday. Then I realised my CPU nor my Motherboard supports it. So the box did not boot.
Swapped everything back, Bob was my uncle, of we go. After 3 day's it got the crash.
That makes it so weird to determine what the problem could be.

 

[cyberjock]

RAM test: www.memtest.org

Boot from the disk and let it run for 3 passes. Could take some hours, so start it and go to bed.

dmesg isn't overly useful. It doesn't always give model numbers. For example, no motherboard model number is provided, no NIC model number is provided, no SATA controller model number is provided, etc. 10 for effort though!

 

[kleinejan]

For trying the ECC Ram or dmesg ;)
Asus P8H77-I,s1155
Realtek® 8111F, 1 x Gigabit LAN Controller(s)

PCI-e SiI3132 SATA controller
Memtest will be executed after I get some sleep.

 

[kleinejan]

Did run the memtest and got right away one error. After 3 runs I collected 2 error. Assuming this is the problem, I ordered a brand new ATOM ECC compatible mainboard (AsRock C2750D4I). Which is tested by several users in the forum. Just to get rid of my not 'server grade' stuff.
And now I know why Cyberjock is aways hammering on "just get ECC Ram","Don't buy cheap hardware". If I just spend € 200,- more, my nas was already server grade with the correct RAM.
The first Freenas I build: Supermicro miniITX, 4Gb ram. And that worked like a charm. All my later upgrades i5/i7, more RAM did not take in consideration the really important hardware specs freenas really depends on.
I just can say, So now I know :)
The board I got from the shop was a 'monday morning model' and needs to be swapped tomorrow. Then I finally know where my data is ;)

 

[HoneyBadger]

Ouch. I hope your pool is OK and it didn't get completely damaged by the bad RAM, but you should hope for the best and prepare yourself for the worst.

 

[cyberjock]

Did run the memtest and got right away one error. After 3 runs I collected 2 error. Assuming this is the problem, I ordered a brand new ATOM ECC compatible mainboard (AsRock C2750D4I). Which is tested by several users in the forum. Just to get rid of my not 'server grade' stuff.
And now I know why Cyberjock is aways hammering on "just get ECC Ram","Don't buy cheap hardware". If I just spend € 200,- more, my nas was already server grade with the correct RAM.
The first Freenas I build: Supermicro miniITX, 4Gb ram. And that worked like a charm. All my later upgrades i5/i7, more RAM did not take in consideration the really important hardware specs freenas really depends on.
I just can say, So now I know :)
The board I got from the shop was a 'monday morning model' and needs to be swapped tomorrow. Then I finally know where my data is ;)

Yep... I guess I'll add you to the victims list..

Ouch. I hope your pool is OK and it didn't get completely damaged by the bad RAM, but you should hope for the best and prepare yourself for the worst.

Probably unrepairable being that it crashes the system on import.

 

[HoneyBadger]

Probably unrepairable being that it crashes the system on import.

Last ditch effort he can try an import (once he gets healthy hardware!) with zfs_recover and some other flags as outlined in this recovery guide. It's been linked a couple times out of this forum and some folks have had success being able to import their zpool as read-only and pull data from it.

But yeah, if this doesn't work ...

 

[cyberjock]

Oh.. Oh.. that picture is amazing!

 

[kleinejan]

LOL

 

[kleinejan]

HoneyBadger thanks for the flowers! Zpool died lonely.
So remember kids, listen to Cyberjock, he's not a bully, he's just right about ECC RAM, good hardware, ECC RAM, ECC RAM, oh and good hardware as well ECC RAM.

I upgraded my UNAS-8 with the ASUS C2750-d4i, 16GB ECC RAM. Did I mention the practical use of ECC RAM? :)
Please read the guide BEFORE you buy stuff/crap.

 

[cyberjock]

Aww. Sorry to hear that. :(

 

[norbs]

Had exact thing happen to me recently. I found my FreeNAS 9.2.1.7 vm crashed after trying to access some shares with no luck. I ran it in the same configuration on ESXi for almost 4 years and just after last night's scrub this happened. I also was not using ECC as the OP. Running the "zbd -e -bcsvL <zpool>" command on my pool right now but after 12 hours it still has another 30 hours to go.

I actually ended up just caving and buying a very similar setup to what I have but with ECC. Only a few minor differences that I can understand...

Before | After
non-ecc --> ecc
i7-3770 ivy bridge --> e3-1231v3 haswell
no remote management --> IPMI
no extra cost --> $870!!

Any other big benefits of going to a xeon beyond ECC support?

Here is what I ordered today:
1x Supermicro X10SLL+-F-O LGA1150
1x Intel Xeon E3-1231 v3
4x Samsung DDR3-1600 8GB/1Gx72 ECC CL11 Server Memory

I already have the raid controller which i've been running in esxi passthrough (v-td) mode for about a year now.

Hope this saves me some future headaches and the IPMI management sounds awesome.