FreeNAS (FreeBSD) kernel crash - how to interpret logs

[DarekM]

Hi there,

Recently I experienced crash under heavy load. Not that it didn't pass that much before before but still a scrub, a backup write and a copy over NFS. Scrub started just before crash, say a minute or two while accepting writes at 1.5Gbps. Whole thing has SD boot, L2ARC, SLOG on mirror and 6x SATA in Z2.
After autoreboot scrub continued and found nothing wrong. Drives are encrypted by geli.

I've got files in /data/crash, simple explanation is "double fault" and in addition dump of everything.

How to find what actually has failed?

 

[DarekM]

Code:

Fatal double fault
rip 0xffffffff80440d8a rsp 0xfffffe137347df90 rbp 0xfffffe137347e450
rax 0xfffff8013e42c800 rdx 0x340191bd rbx 0xfffff8013bf62000
rcx 0x3199c4cc rsi 0xfffff8013e42cd70 rdi 0xfffff80b316ea660
r8 0xfffff806bfdaca70 r9 0xfffff80d5cfd1a70 r10 0
r11 0x2 r12 0xfffff8013e42cc48 r13 0xfffff8013b77e000
r14 0xfffff8013e42c800 r15 0xfffff8013e42cd70 rflags 0x10286
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0xc42049a070 gsbase 0xffffffff821c4980 kgsbase 0
cpuid = 3; apic id = 34
panic: double fault
cpuid = 3
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe1331560d80
vpanic() at vpanic+0x186/frame 0xfffffe1331560e00
panic() at panic+0x43/frame 0xfffffe1331560e60
dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe1331560f30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe1331560f30
--- trap 0x17, rip = 0xffffffff80440d8a, rsp = 0xfffffe137347df90, rbp = 0xfffffe137347e450 ---
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x2a/frame 0xfffffe137347e450
vdev_queue_io_done() at vdev_queue_io_done+0xc8/frame 0xfffffe137347e4a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347e4d0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e520
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347e580
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e5d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347e620
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347e650
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e6a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347e700
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e750
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347e7a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347e7d0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e820
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347e880
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e8d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347e920
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347e950
zio_execute() at zio_execute+0xac/frame 0xfffffe137347e9a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347ea00
zio_execute() at zio_execute+0xac/frame 0xfffffe137347ea50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347eaa0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347ead0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347eb20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347eb80
zio_execute() at zio_execute+0xac/frame 0xfffffe137347ebd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347ec20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347ec50
zio_execute() at zio_execute+0xac/frame 0xfffffe137347eca0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347ed00
zio_execute() at zio_execute+0xac/frame 0xfffffe137347ed50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347eda0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347edd0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347ee20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347ee80
zio_execute() at zio_execute+0xac/frame 0xfffffe137347eed0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347ef20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347ef50
zio_execute() at zio_execute+0xac/frame 0xfffffe137347efa0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f000
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f050
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f0a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f0d0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f120
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f180
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f1d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f220
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f250
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f2a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f300
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f350
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f3a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f3d0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f420
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f480
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f4d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f520
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f550
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f5a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f600
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f650
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f6a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f6d0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f720
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f780
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f7d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f820
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f850
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f8a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347f900
zio_execute() at zio_execute+0xac/frame 0xfffffe137347f950
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347f9a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347f9d0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fa20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347fa80
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fad0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347fb20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347fb50
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fba0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347fc00
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fc50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347fca0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347fcd0
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fd20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347fd80
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fdd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347fe20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347fe50
zio_execute() at zio_execute+0xac/frame 0xfffffe137347fea0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe137347ff00
zio_execute() at zio_execute+0xac/frame 0xfffffe137347ff50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe137347ffa0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe137347ffd0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480020
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480080
zio_execute() at zio_execute+0xac/frame 0xfffffe13734800d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480120
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480150
zio_execute() at zio_execute+0xac/frame 0xfffffe13734801a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480200
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480250
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe13734802a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe13734802d0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480320
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480380
zio_execute() at zio_execute+0xac/frame 0xfffffe13734803d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480420
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480450
zio_execute() at zio_execute+0xac/frame 0xfffffe13734804a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480500
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480550
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe13734805a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe13734805d0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480620
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480680
zio_execute() at zio_execute+0xac/frame 0xfffffe13734806d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480720
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480750
zio_execute() at zio_execute+0xac/frame 0xfffffe13734807a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480800
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480850
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe13734808a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe13734808d0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480920
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480980
zio_execute() at zio_execute+0xac/frame 0xfffffe13734809d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480a20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480a50
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480aa0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480b00
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480b50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480ba0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480bd0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480c20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480c80
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480cd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480d20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480d50
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480da0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480e00
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480e50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373480ea0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373480ed0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480f20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373480f80
zio_execute() at zio_execute+0xac/frame 0xfffffe1373480fd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373481020
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373481050
zio_execute() at zio_execute+0xac/frame 0xfffffe13734810a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481100
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481150
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe13734811a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe13734811d0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481220
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481280
zio_execute() at zio_execute+0xac/frame 0xfffffe13734812d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373481320
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373481350
zio_execute() at zio_execute+0xac/frame 0xfffffe13734813a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481400
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481450
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe13734814a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe13734814d0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481520
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481580
zio_execute() at zio_execute+0xac/frame 0xfffffe13734815d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373481620
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373481650
zio_execute() at zio_execute+0xac/frame 0xfffffe13734816a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481700
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481750
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe13734817a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe13734817d0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481820
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481880
zio_execute() at zio_execute+0xac/frame 0xfffffe13734818d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373481920
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373481950
zio_execute() at zio_execute+0xac/frame 0xfffffe13734819a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe1373481a00
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481a50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe1373481aa0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe1373481ad0
zio_execute() at zio_execute+0xac/frame 0xfffffe1373481b20
taskqueue_run_locked() at taskqueue_run_locked+0x147/frame 0xfffffe1373481b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe1373481bb0
fork_exit() at fork_exit+0x85/frame 0xfffffe1373481bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1373481bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

 

[dlavigne]

Best bet is to create a report at bugs.freenas.org that includes an attached debug so a dev can assist in determining the cause.