KDB Panic - No fix?

kjemison

Dabbler
Joined
Jul 9, 2015
Messages
32
So, I have a Freenas 11.2 install ... was with Raid z2 ... Core i3 8 GB RAM... was running just fine and then today I get an error:

Beginning ZFS volume imports

Importing 14519832302454782612

txg 6063307 import pool version 5000; software version 5000/5; uts 11.0-STABLE 1100512 amd64
ZFS volume imports complete
panic: Solaris(panic): blkptr at 0xfffff8011945c180 DVA 0 has invalid VDEV 512
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0467a8b0c0
vpanic() at vpanic+0x186/frame 0xfffffe0467a8b140
panic() at panic+0x43/frame 0xfffffe0467a8b1a0
vcmn_err() at vcmn_err+0xc2/frame 0xfffffe0467a8b2e0
zfs_panic_recover() at zfs_panic_recover+0x5a/frame 0xfffffe0467a8b340
zfs_blkptr_verify() at zfs_blkptr_verify+0x2d3/frame 0xfffffe0467a8b380
zio_read() at zio_read+0x2c/frame 0xfffffe0467a8b3c0
arc_read() at arc_read+0x71b/frame 0xfffffe0467a8b460
dbuf_read() at dbuf_read+0x7d1/frame 0xfffffe0467a8b510
dmu_buf_hold_by_dnode() at dmu_buf_hold_by_dnode+0x3d/frame 0xfffffe0467a8b550
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x4d/frame 0xfffffe0467a8b5b0
fzap_length() at fzap_length+0x96/frame 0xfffffe0467a8b620
zap_length_uint64() at zap_length_uint64+0xd7/frame 0xfffffe0467a8b680
ddt_zap_lookup() at ddt_zap_lookup+0x3d/frame 0xfffffe0467a8b7e0
ddt_lookup() at ddt_lookup+0x2a2/frame 0xfffffe0467a8b9b0
zio_ddt_write() at zio_ddt_write+0x75/frame 0xfffffe0467a8bad0
zio_execute() at zio_execute+0xac/frame 0xfffffe0467a8bb20
taskqueue_run_locked() at taskqueue_run_locked+0x127/frame 0xfffffe0467a8bb80
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xfffffe0467a8bbb0
fork_exit() at fork_exit+0x85/frame 0xfffffe0467a8bbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0467a8bbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 0 tid 101534 ]
Stopped at kdb_enter+0x3b: movq $0,kdb_why
db>

A quick Google took me to a ixsystems.com site and a ixsystems tech stated the following:

"This looks like a pool corruption triggered this panic. Do you experience this only with FreeNAS 11, or it happens for 9.10 too?
If it is indeed a pool corruption, you may try to set vfs.zfs.recover sysctl/tunable and try to import the pool to copy-out the data. There are no mechanisms in ZFS to fix corruptions like that."

My question is this... was I mislead into thinking that ZFS was much stronger and better than any other o/s ? I thought corruption was not possible with ZFS ? ... with CoW and all... And, as stated in the last sentence.. "There are no mechanisms in ZFS to fix corruptions like that." ... is this true?

Just curious on others thoughts about this issue .. Makes me nervous knowing that I have about 20 of these FreeNAS systems in production environments... should I be worried?

Appreciate everyone's time concerning this matter.
Kell
 
D

dlavigne

Guest
Couple of things to try. First, that message may just indicate that the boot device died. Try installing the same FreeNAS version to a new stick and see if it boots successfully. If it does, you can restore your config.

If that fails and it really is an issue with the storage pool, will the system take more RAM? 8GB is the minimum and bumping it up might allow the pool to import.
 

kjemison

Dabbler
Joined
Jul 9, 2015
Messages
32
I have reloaded to a new hard drive the o/s ... I tried to import the pool first before uploading the saved config file. I will reload again and upload the config file then, import the pool and see what happens. I will also see if I can get 2 (8GB) memory sticks to bump to 16 GB.

I will let you know tomorrow how it goes.
Thank you
Kell
 

kjemison

Dabbler
Joined
Jul 9, 2015
Messages
32
After trying multiple repairs / suggestions, I finally had to just wipe out the data sets and the pool. I tried adding more RAM, changing to a different version, updating etc... each time I would try and import the pool the entire system would crash... Lesson learned... FreeBSD is a good operating system with ZFS but not as "bullet proof" as I am always reading about. The positive thing from all of this is that I came away learning something.. which is never a bad thing!
Have a nice weekend!
Kell
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
Nothing is perfect and completely bullet proof. That is why we always say that RaidZ is not a substitute for a proper backup strategy.

As you never gave any description of your hardware, it is not possible to comment on your specific situation. Glad you got it working.
 

kjemison

Dabbler
Joined
Jul 9, 2015
Messages
32
From the first post: "I have a Freenas 11.2 install ... was with Raid z2 ... Core i3 8 GB RAM "

the components are based on a Intel chipset motherboard.. the drives are WD Red 2 TB

As I stated above, I did learn from this experience so... all is not lost. I now have the groundwork laid out to setup offsite backup to another FreeNAS box using Rsync... I have tested it and it works very well!!

As always, I do appreciate the quick responses from the group and admins!

Have a great weekend!!

Sincerely,
Kell
 
Top