Hi, have an odd problem where my FreeNAS box keeps shutting down (gracefully).
It seems to be related to load - it can idle for days, but if used hard it will happen in a couple of hours.
It's complaining about a drive that's being used as a cache / ZIL drive.
If I restart, it all works fine... for a while.
It looks to me like there was some kind of problem with the ZIL drive (maybe it crashed, maybe there was a read error, maybe the code that does caching just broke) and a graceful shutdown was initiated?
I'd really appreciate some thoughts!
Specifically, I've got the following excerpt from the log at the time when the fault happened. There was nothing in the log in the 14 hours before this. I've also tried to include any relevant other information.
Debug File:
https://www.dropbox.com/s/4q5fqheag1tg7dx/debug-freenas-20151226003325..tgz?dl=0
PC Info:
It seems to be related to load - it can idle for days, but if used hard it will happen in a couple of hours.
It's complaining about a drive that's being used as a cache / ZIL drive.
If I restart, it all works fine... for a while.
It looks to me like there was some kind of problem with the ZIL drive (maybe it crashed, maybe there was a read error, maybe the code that does caching just broke) and a graceful shutdown was initiated?
I'd really appreciate some thoughts!
Specifically, I've got the following excerpt from the log at the time when the fault happened. There was nothing in the log in the 14 hours before this. I've also tried to include any relevant other information.
Code:
Dec 24 18:44:14 freenas zfsd: DevfsEvent::ReadLabel: /dev/ada4p2: Unable to extract pool GUID from vdev label data. Dec 24 18:44:15 freenas afpd[2966]: AFP Server shutting down Dec 24 18:44:15 freenas cnid_metad[2967]: shutting down on SIGTERM Dec 24 18:44:15 freenas netatalk[2951]: Netatalk AFP server exiting Dec 24 18:44:15 freenas smbd[14822]: dnssd_clientstub DNSServiceProcessResult called with DNSServiceRef with no ProcessReply function Dec 24 18:44:17 freenas winbindd[14826]: STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=1) Dec 24 18:44:17 freenas winbindd[14979]: STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0) Dec 24 18:44:17 freenas winbindd[14827]: STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0) Dec 24 18:44:17 freenas winbindd[14980]: STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0) Dec 24 18:44:18 freenas nmbd[14818]: [2015/12/24 18:44:18.599678, 0] ../source3/nmbd/nmbd.c:57(terminate) Dec 24 18:44:18 freenas nmbd[14818]: Got SIGTERM: going down... Dec 24 18:44:18 freenas ntpd[2349]: ntpd exiting on signal 15 Dec 24 18:44:19 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:19 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:19 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:20 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:20 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:20 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:21 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:21 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:21 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:22 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused Dec 24 18:44:22 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused Dec 24 18:44:22 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused Dec 24 18:44:25 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:25 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:25 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:26 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:26 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:26 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:27 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:27 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:27 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:28 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused Dec 24 18:44:28 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused Dec 24 18:44:28 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused Dec 24 18:44:31 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:31 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:31 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:34 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused Dec 24 18:44:34 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused Dec 24 18:44:34 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused Dec 24 18:44:37 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:37 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:37 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:38 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:38 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:38 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:40 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused Dec 24 18:44:40 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused Dec 24 18:44:40 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused Dec 24 18:44:43 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:43 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:43 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:44 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:44 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:44 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:45 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:45 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:45 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:46 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused Dec 24 18:44:46 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused Dec 24 18:44:46 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused Dec 24 18:44:49 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:49 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:49 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:50 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:50 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:50 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:51 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:51 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:51 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:52 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused Dec 24 18:44:52 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused Dec 24 18:44:52 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused Dec 24 18:44:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Dec 24 18:44:57 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:57 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:57 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Dec 24 18:44:58 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:58 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Dec 24 18:44:58 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Debug File:
https://www.dropbox.com/s/4q5fqheag1tg7dx/debug-freenas-20151226003325..tgz?dl=0
PC Info:
Code:
Make / Model: HP Microserver N54L. CPU: AMD Turion II Neo N54L - 2.2 GHz. Memory: 16GB DDR3 ECC, 2x 8GB DDR1600. Hard Drives: 14.3TB RAID-Z1, 1x OCZ Vertex 2 120GB SSD (Cache), 5x Seagate 7200.15 4TB, 1x Sandisk Cruzer Contour 8GB. Power Supply: XFX Pro 450w Core Edition. Network Card: Intel PRO/1000 PT PCI-E Gigabit Network Card. Operating System: FreeNAS 9 64-bit (FreeNAS-9.3-STABLE-201512121950)