FreeNAS crashing, stack backtrace errors with NIC

[freeaq2004]

FreeNAS appliance has intermittently crashed twice over the past few days and I'm unsure what's causing it. In checking /var/log/messages, I noticed that there are several "stack backtrace" messages related to the network interfaces after the system has recovered from the kernel panic.

Code:

Dec 26 13:57:42 scratchfs-01 lock order reversal:
Dec 26 13:57:42 scratchfs-01 1st 0xfffff8000df4c858 ix0:rx(7) (ix0:rx(7)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763
Dec 26 13:57:42 scratchfs-01 2nd 0xfffff8002779b608 if_lagg rmlock (if_lagg rmlock) @ /freenas-11-releng/freenas/_BE/os/sys/net/if_lagg.c:1603
Dec 26 13:57:42 scratchfs-01 stack backtrace:
Dec 26 13:57:42 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70
Dec 26 13:57:42 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23
Dec 26 13:57:42 scratchfs-01 #2 0xffffffff80a85129 at _rm_rlock_debug+0x119
Dec 26 13:57:42 scratchfs-01 #3 0xffffffff80ba39c2 at lagg_input+0x42
Dec 26 13:57:42 scratchfs-01 #4 0xffffffff80b9e30c at ether_nh_input+0x1dc
Dec 26 13:57:42 scratchfs-01 #5 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 26 13:57:42 scratchfs-01 #6 0xffffffff80b9d84f at ether_input+0x8f
Dec 26 13:57:42 scratchfs-01 #7 0xffffffff80bfb517 at tcp_lro_flush+0x1b7
Dec 26 13:57:42 scratchfs-01 #8 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b
Dec 26 13:57:42 scratchfs-01 #9 0xffffffff806ee967 at ixgbe_rxeof+0x787
Dec 26 13:57:42 scratchfs-01 #10 0xffffffff806e48e7 at ixgbe_msix_que+0x97
Dec 26 13:57:42 scratchfs-01 #11 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99
Dec 26 13:57:42 scratchfs-01 #12 0xffffffff80a4ec86 at ithread_loop+0xb6
Dec 26 13:57:42 scratchfs-01 #13 0xffffffff80a4bcf4 at fork_exit+0x84
Dec 26 13:57:42 scratchfs-01 #14 0xffffffff80f250fe at fork_trampoline+0xe

Dec 26 13:57:42 scratchfs-01 lock order reversal:
Dec 26 13:57:42 scratchfs-01 1st 0xfffff8000df4c858 ix0:rx(7) (ix0:rx(7)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763
Dec 26 13:57:42 scratchfs-01 2nd 0xfffffe0000dabe08 pcbinfohash (pcbinfohash) @ /freenas-11-releng/freenas/_BE/os/sys/netinet/in_pcb.c:1959
Dec 26 13:57:42 scratchfs-01 stack backtrace:
Dec 26 13:57:42 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70
Dec 26 13:57:42 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23
Dec 26 13:57:42 scratchfs-01 #2 0xffffffff80a86162 at __rw_rlock+0x82
Dec 26 13:57:42 scratchfs-01 #3 0xffffffff80be571b at in_pcblookup_hash+0x3b
Dec 26 13:57:42 scratchfs-01 #4 0xffffffff80bf5da1 at tcp_input+0x921
Dec 26 13:57:42 scratchfs-01 #5 0xffffffff80beb367 at ip_input+0x137
Dec 26 13:57:42 scratchfs-01 #6 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 26 13:57:42 scratchfs-01 #7 0xffffffff80b9d4ae at ether_demux+0x16e
Dec 26 13:57:42 scratchfs-01 #8 0xffffffff80b9e4cc at ether_nh_input+0x39c
Dec 26 13:57:42 scratchfs-01 #9 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 26 13:57:42 scratchfs-01 #10 0xffffffff80b9d84f at ether_input+0x8f
Dec 26 13:57:42 scratchfs-01 #11 0xffffffff80bfb517 at tcp_lro_flush+0x1b7
Dec 26 13:57:42 scratchfs-01 #12 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b
Dec 26 13:57:42 scratchfs-01 #13 0xffffffff806ee967 at ixgbe_rxeof+0x787
Dec 26 13:57:42 scratchfs-01 #14 0xffffffff806e48e7 at ixgbe_msix_que+0x97
Dec 26 13:57:42 scratchfs-01 #15 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99
Dec 26 13:57:42 scratchfs-01 #16 0xffffffff80a4ec86 at ithread_loop+0xb6
Dec 26 13:57:42 scratchfs-01 #17 0xffffffff80a4bcf4 at fork_exit+0x84

Dec 26 13:57:42 scratchfs-01 lock order reversal:
Dec 26 13:57:42 scratchfs-01 1st 0xfffff8000df4c858 ix0:rx(7) (ix0:rx(7)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763
Dec 26 13:57:42 scratchfs-01 2nd 0xfffff80143fb7338 tcpinp (tcpinp) @ /freenas-11-releng/freenas/_BE/os/sys/netinet/in_pcb.c:1966
Dec 26 13:57:42 scratchfs-01 stack backtrace:
Dec 26 13:57:42 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70
Dec 26 13:57:42 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23
Dec 26 13:57:42 scratchfs-01 #2 0xffffffff80a8571e at _rw_wlock_cookie+0x5e
Dec 26 13:57:42 scratchfs-01 #3 0xffffffff80be57e0 at in_pcblookup_hash+0x100
Dec 26 13:57:42 scratchfs-01 #4 0xffffffff80bf5da1 at tcp_input+0x921
Dec 26 13:57:42 scratchfs-01 #5 0xffffffff80beb367 at ip_input+0x137
Dec 26 13:57:42 scratchfs-01 #6 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 26 13:57:42 scratchfs-01 #7 0xffffffff80b9d4ae at ether_demux+0x16e
Dec 26 13:57:42 scratchfs-01 #8 0xffffffff80b9e4cc at ether_nh_input+0x39c
Dec 26 13:57:42 scratchfs-01 #9 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 26 13:57:42 scratchfs-01 #10 0xffffffff80b9d84f at ether_input+0x8f
Dec 26 13:57:42 scratchfs-01 #11 0xffffffff80bfb517 at tcp_lro_flush+0x1b7
Dec 26 13:57:42 scratchfs-01 #12 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b
Dec 26 13:57:42 scratchfs-01 #13 0xffffffff806ee967 at ixgbe_rxeof+0x787
Dec 26 13:57:42 scratchfs-01 #14 0xffffffff806e48e7 at ixgbe_msix_que+0x97
Dec 26 13:57:42 scratchfs-01 #15 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99
Dec 26 13:57:42 scratchfs-01 #16 0xffffffff80a4ec86 at ithread_loop+0xb6
Dec 26 13:57:42 scratchfs-01 #17 0xffffffff80a4bcf4 at fork_exit+0x84

Currently running FreeNAS-11.0-U4 (54848d13b) on an iXSystems built appliance server NFSv3 to about 50 clients. Smartctl checked out OK on all hard disks and ZFS scrub sees no errors with zpool.

 

[joeschmuck]

Exactly which iXsystems appliance?
Had the device been working fine before these messages started showing up?
Have you made ANY changes to the system in the past 30 days?
Had the system experienced a power outage and dropped unexpectedly?
What is your boot device?

1) Backup your configuration information.

@Ericloewe Is this another issue with the ASRack motherboard?

 

[Ericloewe]

Ericloewe Is this another issue with the ASRack motherboard?

I don't think so, mostly because the workload seems excessive for a C2750.

 

[freeaq2004]

It's the 36 JBOD model (similar to iX-4236P) with Intel C612 Chipset, FreeNAS installed on 32GB SATADOM. Appliance has been working fine after the last FreeNAS major update from 9 to 11. Previously it was crashing after high NFS loads which I then enabled autotune (along with OS upgrade) and it has been stable until 12/24. No power outage and no system/OS changes in the past 30 days either.

 

[Ericloewe]

Do you have the output of the panic?

 

[freeaq2004]

I do not or at least do not know where to find it. Does FreeNAS store the crash logs to a local area during a kernel panic or is there is a switch I need to enable this feature? TIA!

 

[freeaq2004]

Unfortunately the server just experienced another crash 2 days ago;

Code:

Dec 31 00:42:57 scratchfs-01 lock order reversal:
Dec 31 00:42:57 scratchfs-01 1st 0xfffff8000df4c5f8 ix0:rx(5) (ix0:rx(5)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763
Dec 31 00:42:57 scratchfs-01 2nd 0xfffff8016626e168 tcpinp (tcpinp) @ /freenas-11-releng/freenas/_BE/os/sys/netinet/in_pcb.c:1966
Dec 31 00:42:57 scratchfs-01 stack backtrace:
Dec 31 00:42:57 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70
Dec 31 00:42:57 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23
Dec 31 00:42:57 scratchfs-01 #2 0xffffffff80a8571e at _rw_wlock_cookie+0x5e
Dec 31 00:42:57 scratchfs-01 #3 0xffffffff80be57e0 at in_pcblookup_hash+0x100
Dec 31 00:42:57 scratchfs-01 #4 0xffffffff80bf5da1 at tcp_input+0x921
Dec 31 00:42:57 scratchfs-01 #5 0xffffffff80beb367 at ip_input+0x137
Dec 31 00:42:57 scratchfs-01 #6 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 31 00:42:57 scratchfs-01 #7 0xffffffff80b9d4ae at ether_demux+0x16e
Dec 31 00:42:57 scratchfs-01 #8 0xffffffff80b9e4cc at ether_nh_input+0x39c
Dec 31 00:42:57 scratchfs-01 #9 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f
Dec 31 00:42:57 scratchfs-01 #10 0xffffffff80b9d84f at ether_input+0x8f
Dec 31 00:42:57 scratchfs-01 #11 0xffffffff80bfb517 at tcp_lro_flush+0x1b7
Dec 31 00:42:57 scratchfs-01 #12 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b
Dec 31 00:42:57 scratchfs-01 #13 0xffffffff806ee967 at ixgbe_rxeof+0x787
Dec 31 00:42:57 scratchfs-01 #14 0xffffffff806e48e7 at ixgbe_msix_que+0x97
Dec 31 00:42:57 scratchfs-01 #15 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99
Dec 31 00:42:57 scratchfs-01 #16 0xffffffff80a4ec86 at ithread_loop+0xb6
Dec 31 00:42:57 scratchfs-01 #17 0xffffffff80a4bcf4 at fork_exit+0x84

What's also disconcerting is that it takes almost 40 minutes from kernel panic to boot up;

Code:

Dec 30 23:46:14 scratchfs-01 mountd[2478]: mount request succeeded from 10.10.10.97 for /mnt/export/
Dec 30 23:46:14 scratchfs-01 mountd[2478]: mount request succeeded from 10.10.10.107 for /mnt/export/
Dec 31 00:21:06 scratchfs-01 syslog-ng[1798]: syslog-ng starting up; version='3.7.3'
Dec 31 00:21:06 scratchfs-01 Copyright (c) 1992-2017 The FreeBSD Project.
Dec 31 00:21:06 scratchfs-01 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
[\CODE]

 

[woyteck]

Interesting. I have observed similar thing just yesterday (lots of them and 3 crashes):

Code:

lock order reversal:
 1st 0xfffff80117003758 ix1:rx(6) (ix1:rx(6)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1660
 2nd 0xfffff8015537ea78 tcpinp (tcpinp) @ /freenas-11-releng/freenas/_BE/os/sys/netinet/in_pcb.c:1966
stack backtrace:
#0 0xffffffff80b01140 at witness_debugger+0x70
#1 0xffffffff80b00fc2 at witness_checkorder+0xe02
#2 0xffffffff80a9ae59 at _rw_wlock_cookie+0x69
#3 0xffffffff80c00720 at in_pcblookup_hash+0x100
#4 0xffffffff80c10e0e at tcp_input+0x7de
#5 0xffffffff80c067a1 at ip_input+0x3a1
#6 0xffffffff80bd3a4f at netisr_dispatch_src+0x9f
#7 0xffffffff80bb7b7e at ether_demux+0x16e
#8 0xffffffff80bb8dc1 at ether_nh_input+0x3c1
#9 0xffffffff80bd3a4f at netisr_dispatch_src+0x9f
#10 0xffffffff80bb808f at ether_input+0x8f
#11 0xffffffff80bc583c at vlan_input+0x2bc
#12 0xffffffff80bb7b64 at ether_demux+0x154
#13 0xffffffff80bb8dc1 at ether_nh_input+0x3c1
#14 0xffffffff80bd3a4f at netisr_dispatch_src+0x9f
#15 0xffffffff80bb808f at ether_input+0x8f
#16 0xffffffff80c16958 at tcp_lro_flush+0x208
#17 0xffffffff80c16acb at tcp_lro_flush_all+0x10b