FreeNAS 9.10.2-U2 Rebooting Every Night

Status
Not open for further replies.
R

Rich Wagner

Cadet
Joined
Mar 10, 2017
Messages
6
Good day...

I have an issue with my FreeNAS machine rebooting for no apparent reason every night. I use this to provide 3 Hyper-V servers iSCSI storage. Each day I come into the office and find all of the Hyper-V virtual machines offline due to the storage rebooting by itself. I took a snippet of the log files below and you can see when it reboots but I cant see a reason for it. Can anyone see anything in the log file that I'm missing?

Code:
Apr 20 02:12:08 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
Apr 20 02:12:08 freenas (0:6:7/6): Tag: 0x5ce66d00, type 0
Apr 20 02:12:08 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): tasks terminated
Apr 20 02:12:08 freenas (0:6:7/6): ctl_process_done: 709 seconds
Apr 20 02:12:08 freenas (0:6:7/6): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
Apr 20 02:12:08 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): tasks terminated
Apr 20 02:12:08 freenas (0:6:7/6): Tag: 0x5de66d00, type 0
Apr 20 02:12:08 freenas (0:6:7/6): ctl_process_done: 681 seconds
Apr 20 02:12:08 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
Apr 20 02:12:08 freenas (0:12:0/2): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): tasks terminated
Apr 20 02:12:08 freenas (0:12:0/2): Tag: 0x582c1000, type 0
Apr 20 02:12:08 freenas (0:12:0/2): ctl_process_done: 681 seconds
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:08 freenas (0:12:0/2): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:08 freenas (0:12:0/2): Tag: 0x592c1000, type 0
Apr 20 02:12:08 freenas (0:12:0/2): ctl_process_done: 653 seconds
Apr 20 02:12:08 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:08 freenas (0:12:0/2): Tag: 0x5a2c1000, type 0
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:08 freenas (0:12:0/2): ctl_process_done: 625 seconds
Apr 20 02:12:08 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:08 freenas (0:12:0/2): Tag: 0x5b2c1000, type 0
Apr 20 02:12:08 freenas (0:12:0/2): ctl_process_done: 597 seconds
Apr 20 02:12:08 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:08 freenas (0:12:0/2): Tag: 0x5c2c1000, type 0
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:08 freenas (0:12:0/2): ctl_process_done: 569 seconds
Apr 20 02:12:08 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:08 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x5d2c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 541 seconds
Apr 20 02:12:09 freenas (0:8:4/4): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:8:4/4): Tag: 0x90100000, type 0
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:8:4/4): ctl_process_done: 525 seconds
Apr 20 02:12:09 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:cmchyperv2.cmcn2.local): connection error; dropping connection
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:cmchyperv2.cmcn2.local): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x5e2c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 513 seconds
Apr 20 02:12:09 freenas (0:8:4/4): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:8:4/4): Tag: 0x91100000, type 0
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:8:4/4): ctl_process_done: 497 seconds
Apr 20 02:12:09 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:cmchyperv2.cmcn2.local): connection error; dropping connection
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:cmchyperv2.cmcn2.local): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x5f2c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 485 seconds
Apr 20 02:12:09 freenas (0:8:4/4): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:8:4/4): Tag: 0x92100000, type 0
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:8:4/4): ctl_process_done: 469 seconds
Apr 20 02:12:09 freenas (0:12:0/2): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:cmchyperv2.cmcn2.local): connection error; dropping connection
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:cmchyperv2.cmcn2.local): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x612c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 220 seconds
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:12:0/2): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x622c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 193 seconds
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x632c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 165 seconds
Apr 20 02:12:09 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x642c1000, type 0
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 137 seconds
Apr 20 02:12:09 freenas (0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:12:0/2): Tag: 0x652c1000, type 0
Apr 20 02:12:09 freenas (0:12:0/2): ctl_process_done: 109 seconds
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
Apr 20 02:12:09 freenas (0:6:7/6): READ(10). CDB: 28 00 0c 6e bc 00 00 02 00 00
Apr 20 02:12:09 freenas WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x4ee66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas (0:6:7/6): READ(10). CDB: 28 00 0c 6e be 00 00 02 00 00
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x4fe66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas (0:6:7/6): READ(10). CDB: 28 00 0c 6e c0 00 00 02 00 00
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x50e66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas (0:6:7/6): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 04 34 00 00 02 28 00 00
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x4be66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): tasks terminated
Apr 20 02:15:16 freenas WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): connection error; dropping connection
Apr 20 02:15:16 freenas WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): waiting for CTL to terminate 62 tasks
Apr 20 02:15:17 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
Apr 20 02:15:17 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 12 tasks
Apr 20 02:15:22 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): no ping reply (NOP-Out) after 5 seconds; dropping connection
Apr 20 02:15:22 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 1 tasks
Apr 20 02:16:06 freenas WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): connection error; dropping connection
Apr 20 02:16:06 freenas WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): waiting for CTL to terminate 1 tasks
Apr 20 02:16:07 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
Apr 20 02:16:12 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): no ping reply (NOP-Out) after 5 seconds; dropping connection
Apr 20 02:16:12 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 1 tasks
Apr 20 02:16:35 freenas WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
Apr 20 02:22:20 freenas syslog-ng[1865]: syslog-ng starting up; version='3.6.4'
Apr 20 02:22:20 freenas Copyright (c) 1992-2016 The FreeBSD Project.
Apr 20 02:22:20 freenas Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Apr 20 02:22:20 freenas	 The Regents of the University of California. All rights reserved.
Apr 20 02:22:20 freenas FreeBSD is a registered trademark of The FreeBSD Foundation.
 
m0nkey_

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
Looks like one of your drives is throwing errors. How are the drives attached?

Code:
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x4ee66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas (0:6:7/6): READ(10). CDB: 28 00 0c 6e be 00 00 02 00 00
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x4fe66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas (0:6:7/6): READ(10). CDB: 28 00 0c 6e c0 00 00 02 00 00
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x50e66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
Apr 20 02:12:09 freenas (0:6:7/6): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 04 34 00 00 02 28 00 00
Apr 20 02:12:09 freenas (0:6:7/6): Tag: 0x4be66d00, type 1
Apr 20 02:12:09 freenas (0:6:7/6): ctl_process_done: 922 seconds
 
R

Rich Wagner

Cadet
Joined
Mar 10, 2017
Messages
6
Good morning. It's an HP storage server with 12bays each with a 1TB drive. Connection is via gigabit switch
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Read the forum rules and follow them. Post all hardware specs and other important information. In this case that would be pool layout, zpool status, smart data for disks.

Sent from my Nexus 5X using Tapatalk
 
R

Rich Wagner

Cadet
Joined
Mar 10, 2017
Messages
6
The hardware specs are

HP StorageWorks P4500 G2 Network Storage Server
12 - 1TB drives
8 GB RAM
DUAL 36GB USB Boot Drives
1GB NIC

ZPOOL Status
Code:
[root@freenas] ~# zpool status
  pool: data
state: ONLINE
  scan: scrub repaired 0 in 3h12m with 0 errors on Sun Apr 16 03:12:46 2017
config:

		NAME										  STATE	 READ WRITE CKSUM
		data										  ONLINE	   0	 0	 0
		  gptid/c3a4f63a-051d-11e7-924c-68b599b069a4  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Wed Apr 19 03:45:48 2017
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			da2p2   ONLINE	   0	 0	 0
			da1p2   ONLINE	   0	 0	 0

errors: No known data errors


ZPOOL LIST
Code:
[root@freenas] ~# zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
data			10T   751G  9.27T		 -	 4%	 7%  1.00x  ONLINE  /mnt
freenas-boot  29.2G   658M  28.6G		 -	  -	 2%  1.00x  ONLINE  -
 
R

Rich Wagner

Cadet
Joined
Mar 10, 2017
Messages
6
My latest dump file
Code:
[root@freenas] ~# gzip -dc /data/crash/textdump.tar.last.gz | tail -n 50
(0:12:0/2): INQUIRY. CDB: 12 00 00 00 24 00
WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
(0:12:0/2): Tag: 0x652c1000, type 0
(0:12:0/2): ctl_process_done: 109 seconds
WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): connection error; dropping connection
(0:6:7/6): READ(10). CDB: 28 00 0c 6e bc 00 00 02 00 00
WARNING: 192.168.137.4 (iqn.1991-05.com.microsoft:cmchyperv1lab): tasks terminated
(0:6:7/6): Tag: 0x4ee66d00, type 1
(0:6:7/6): ctl_process_done: 922 seconds
(0:6:7/6): READ(10). CDB: 28 00 0c 6e be 00 00 02 00 00
(0:6:7/6): Tag: 0x4fe66d00, type 1
(0:6:7/6): ctl_process_done: 922 seconds
(0:6:7/6): READ(10). CDB: 28 00 0c 6e c0 00 00 02 00 00
(0:6:7/6): Tag: 0x50e66d00, type 1
(0:6:7/6): ctl_process_done: 922 seconds
(0:6:7/6): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 04 34 00 00 02 28 00 00
(0:6:7/6): Tag: 0x4be66d00, type 1
(0:6:7/6): ctl_process_done: 922 seconds
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): tasks terminated
WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): connection error; dropping connection
WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): waiting for CTL to terminate 62 tasks
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 12 tasks
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 1 tasks
WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): connection error; dropping connection
WARNING: 192.168.137.10 (iqn.1991-05.com.microsoft:cmchyperv3.cmcn2.local): waiting for CTL to terminate 1 tasks
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 1 tasks
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): connection error; dropping connection
WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): no ping reply (NOP-Out) after 5 seconds; dropping connection

WARNING: 192.168.137.9 (iqn.1991-05.com.microsoft:win-u8a8go18le7): waiting for CTL to terminate 1 tasks

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 12
fault virtual address   = 0x1e8
fault code			  = supervisor write data, page not present
instruction pointer	 = 0x20:0xffffffff8061ae79
stack pointer		   = 0x28:0xfffffe023a2ae7d0
frame pointer		   = 0x28:0xfffffe023a2ae860
code segment			= base 0x0, limit 0xfffff, type 0x1b
						= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags		= interrupt enabled, resume, IOPL = 0
current process		 = 57668 (ctld)
version.txt06000025113076060130  7605 ustarrootwheelFreeBSD 10.3-STABLE #0 41eb257(9.10.2-STABLE): Mon Mar  6 17:03:14 UTC 2017
	root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64
 
Vito Reiter

Vito Reiter

Wise in the Ways of Science
Joined
Jan 18, 2017
Messages
232
Well, looks like you've tried to scrub the boot volume already. You just have a one drive volume? If that's failing then it could definitely be causing an issue. I definitely see that you wrote 12 - 1TB drives but that's not how it looks in that file, haha. Do a smartctl on the drives and see if anything comes up suspicious or just link the results here for me or someone else to pick apart. Also, if your RAM is non-ECC then a scrub may not be the best choice given that it could be a RAM error.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Hardware RAID.
Get rid of it, read the hardware recommendations guide, then come back and we can try to fix the fallout from doing exactly what you should not do.
 
Vito Reiter

Vito Reiter

Wise in the Ways of Science
Joined
Jan 18, 2017
Messages
232
Ericloewe said:
Hardware RAID.
Get rid of it, read the hardware recommendations guide, then come back and we can try to fix the fallout from doing exactly what you should not do.
Click to expand...

Good catch, that's why I'm all confused. You should definitely try to move away from that, if you have a third source to backup your data put it there and switch that RAID card to JBOD or IT mode, that should give ZFS direct access to your disks. Obviously, you have a lot of stuff running on it, VM's and such, when I was in this position I built a whole new NAS box to transfer to and started from scratch but there are definitely better ways to do it. You'll be lucky if the boot volume is causing this issue and not the RAID card, because that's also another single point of failure.
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Raid card! Remove it and try again

Sent from my Nexus 5X using Tapatalk
 
B

Bart Van Kerckhove

Cadet
Joined
Apr 21, 2017
Messages
1
Try to boot your freenas in debugging mode, we're seeing similar issues since moving iscsi target to ctl (9.3 -> 9.10 upgrade).
When booting in normal mode, a kernel crash will result in an automatic reboot in 38 seconds.
( cf. debug.ddb.scripting.scripts - watchdog 38)

Our kernel consistently crashes with:

panic: solaris assert: MUTEX_HELD(&vq->vq_lock), file: /freenas-9-10-releng/_BE/os/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, line: 488

After setting initiator queue sizes to 32, this seems to be more stable.

Hope this helps, and I hope you have the same issue - I'm having a hard time tracking this bug down to any known issue.
 
Status
Not open for further replies.

Similar threads

Steven Sedory
  • Locked
New CTL Warnings - connection error; dropping connection
Replies
7
Views
2K
bestboy
B
jjkester
  • Locked
FreeNAS 9.10 spontaneously reboots
Replies
3
Views
1K
jjkester
jjkester
Fidelity
SOLVED TrueNAS hangs every saturday
Replies
4
Views
1K
jgreco
jgreco
TheShellshock67
  • Locked
task terminated errors
Replies
6
Views
3K
TheShellshock67
TheShellshock67
H
  • Locked
time going out of sync
Replies
5
Views
2K
hotdog
H
Top