I've been noticing my FreeNAS system shutting itself down once or twice a day over the last week and I'm struggling to find an explaination. I've looked through the logs after it happens and not found anything that seems obvious. I've pasted the log below in case anyone can decipher it.
The log seems to just show the system shutting down, although when I do a clean shutdown I don't get anything about "Lost x pages of memory".I̶'̶v̶e̶ ̶a̶l̶s̶o̶ ̶n̶o̶t̶i̶c̶e̶d̶ ̶t̶h̶a̶t̶ ̶w̶h̶e̶n̶ ̶I̶ ̶r̶e̶s̶t̶a̶r̶t̶ ̶t̶h̶e̶ ̶s̶e̶r̶v̶e̶r̶ ̶a̶f̶t̶e̶r̶w̶a̶r̶d̶s̶ ̶S̶S̶H̶ ̶i̶s̶n̶'̶t̶ ̶r̶u̶n̶n̶i̶n̶g̶ ̶a̶n̶d̶ ̶I̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶r̶e̶s̶t̶a̶r̶t̶ ̶i̶t̶ ̶t̶h̶r̶o̶u̶g̶h̶ ̶t̶h̶e̶ ̶w̶e̶b̶u̶i̶.̶ (fixed)
A̶l̶s̶o̶ ̶j̶a̶i̶l̶s̶ ̶w̶o̶n̶'̶t̶ ̶s̶t̶a̶r̶t̶ ̶a̶f̶t̶e̶r̶ ̶r̶e̶b̶o̶o̶t̶i̶n̶g̶,̶ ̶w̶h̶e̶n̶ ̶I̶ ̶t̶y̶p̶e̶ ̶"̶w̶a̶r̶d̶e̶n̶ ̶s̶t̶a̶r̶t̶ ̶p̶l̶e̶x̶_̶1̶"̶ ̶i̶t̶ ̶s̶a̶y̶s̶ ̶i̶t̶'̶s̶ ̶a̶l̶r̶e̶a̶d̶y̶ ̶r̶u̶n̶n̶i̶n̶g̶ ̶b̶u̶t̶ ̶t̶h̶e̶ ̶w̶e̶b̶u̶i̶ ̶s̶a̶y̶s̶ ̶i̶t̶'̶s̶ ̶o̶f̶f̶ ̶a̶n̶d̶ ̶I̶ ̶c̶a̶n̶'̶t̶ ̶c̶o̶n̶n̶e̶c̶t̶ ̶t̶o̶ ̶i̶t̶.̶ (fixed)
This system has been reliable for the 363 days of uptime it had prior to this.. Any advice would be greatly appreciated :)
The system: http://au.pcpartpicker.com/p/F4cT99
/var/log/messages:
The log seems to just show the system shutting down, although when I do a clean shutdown I don't get anything about "Lost x pages of memory".I̶'̶v̶e̶ ̶a̶l̶s̶o̶ ̶n̶o̶t̶i̶c̶e̶d̶ ̶t̶h̶a̶t̶ ̶w̶h̶e̶n̶ ̶I̶ ̶r̶e̶s̶t̶a̶r̶t̶ ̶t̶h̶e̶ ̶s̶e̶r̶v̶e̶r̶ ̶a̶f̶t̶e̶r̶w̶a̶r̶d̶s̶ ̶S̶S̶H̶ ̶i̶s̶n̶'̶t̶ ̶r̶u̶n̶n̶i̶n̶g̶ ̶a̶n̶d̶ ̶I̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶r̶e̶s̶t̶a̶r̶t̶ ̶i̶t̶ ̶t̶h̶r̶o̶u̶g̶h̶ ̶t̶h̶e̶ ̶w̶e̶b̶u̶i̶.̶ (fixed)
A̶l̶s̶o̶ ̶j̶a̶i̶l̶s̶ ̶w̶o̶n̶'̶t̶ ̶s̶t̶a̶r̶t̶ ̶a̶f̶t̶e̶r̶ ̶r̶e̶b̶o̶o̶t̶i̶n̶g̶,̶ ̶w̶h̶e̶n̶ ̶I̶ ̶t̶y̶p̶e̶ ̶"̶w̶a̶r̶d̶e̶n̶ ̶s̶t̶a̶r̶t̶ ̶p̶l̶e̶x̶_̶1̶"̶ ̶i̶t̶ ̶s̶a̶y̶s̶ ̶i̶t̶'̶s̶ ̶a̶l̶r̶e̶a̶d̶y̶ ̶r̶u̶n̶n̶i̶n̶g̶ ̶b̶u̶t̶ ̶t̶h̶e̶ ̶w̶e̶b̶u̶i̶ ̶s̶a̶y̶s̶ ̶i̶t̶'̶s̶ ̶o̶f̶f̶ ̶a̶n̶d̶ ̶I̶ ̶c̶a̶n̶'̶t̶ ̶c̶o̶n̶n̶e̶c̶t̶ ̶t̶o̶ ̶i̶t̶.̶ (fixed)
This system has been reliable for the 363 days of uptime it had prior to this.. Any advice would be greatly appreciated :)
The system: http://au.pcpartpicker.com/p/F4cT99
/var/log/messages:
Code:
Apr 29 19:11:54 freenas kernel: delete: cannot locate 192.168.1.99 Apr 29 19:11:54 freenas kernel: epair0a: link state changed to DOWN Apr 29 19:11:54 freenas kernel: epair0b: link state changed to DOWN Apr 29 19:11:55 freenas upsmon[2487]: UPS [ups]: connect failed: Connection failure: Can't assign requested address Apr 29 19:12:05 freenas last message repeated 2 times Apr 29 19:12:06 freenas kernel: . Apr 29 19:12:06 freenas kernel: ifa_del_loopback_route: deletion failed Apr 29 19:12:06 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:12:06 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items). Lost 3 pages of memory. Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcpreass) was not empty (252 items). Lost 3 pages of memory. Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcptw) was not empty (50 items). Lost 1 pages of memory. Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcpcb) was not empty (16 items). Lost 4 pages of memory. Apr 29 19:12:06 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items). Lost 2 pages of memory. Apr 29 19:12:06 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required Apr 29 19:12:06 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required Apr 29 19:12:07 freenas kernel: jexec 2 /usr/pbi/transmission-amd64/control stop Apr 29 19:12:15 freenas kernel: epair1a: link state changed to DOWN Apr 29 19:12:15 freenas kernel: epair1b: link state changed to DOWN Apr 29 19:12:19 freenas kernel: tun0: link state changed to DOWN Apr 29 19:12:22 freenas kernel: . Apr 29 19:12:23 freenas kernel: Done Apr 29 19:12:23 freenas kernel: jexec 3 /usr/pbi/btsync-amd64/control stop Apr 29 19:12:32 freenas kernel: delete: cannot locate 192.168.1.6 Apr 29 19:12:32 freenas kernel: epair2a: link state changed to DOWN Apr 29 19:12:32 freenas kernel: epair2b: link state changed to DOWN Apr 29 19:12:45 freenas last message repeated 2 times Apr 29 19:12:50 freenas kernel: . Apr 29 19:12:50 freenas kernel: ifa_del_loopback_route: deletion failed Apr 29 19:12:50 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (90 items). Lost 9 pages of memory. Apr 29 19:12:50 freenas kernel: Freed UMA keg (udpcb) was not empty (672 items). Lost 4 pages of memory. Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcpreass) was not empty (84 items). Lost 1 pages of memory. Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcptw) was not empty (300 items). Lost 6 pages of memory. Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (80 items). Lost 8 pages of memory. Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcpcb) was not empty (32 items). Lost 8 pages of memory. Apr 29 19:12:50 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items). Lost 2 pages of memory. Apr 29 19:12:50 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required Apr 29 19:12:50 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required Apr 29 19:12:51 freenas kernel: Stoppingthejail.... Apr 29 19:12:51 freenas kernel: delete: cannot locate 192.168.1.22 Apr 29 19:12:51 freenas kernel: epair3a: link state changed to DOWN Apr 29 19:12:51 freenas kernel: epair3b: link state changed to DOWN Apr 29 19:12:56 freenas kernel: . Apr 29 19:12:56 freenas kernel: ifa_del_loopback_route: deletion failed Apr 29 19:12:56 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (50 items). Lost 5 pages of memory. Apr 29 19:12:56 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items). Lost 3 pages of memory. Apr 29 19:12:56 freenas kernel: Freed UMA keg (tcpreass) was not empty (252 items). Lost 3 pages of memory. Apr 29 19:12:56 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:12:56 freenas kernel: Freed UMA keg (tcpcb) was not empty (20 items). Lost 5 pages of memory. Apr 29 19:12:56 freenas kernel: Freed UMA keg (ripcb) was not empty (10 items). Lost 1 pages of memory. Apr 29 19:12:56 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required Apr 29 19:12:56 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required Apr 29 19:12:57 freenas kernel: Stoppingthejail.... Apr 29 19:12:57 freenas kernel: delete: cannot locate 192.168.1.23 Apr 29 19:12:57 freenas kernel: epair4a: link state changed to DOWN Apr 29 19:12:57 freenas kernel: epair4b: link state changed to DOWN Apr 29 19:13:01 freenas kernel: . Apr 29 19:13:01 freenas kernel: ifa_del_loopback_route: deletion failed Apr 29 19:13:01 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:13:01 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items). Lost 3 pages of memory. Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcpreass) was not empty (252 items). Lost 3 pages of memory. Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcptw) was not empty (50 items). Lost 1 pages of memory. Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcpcb) was not empty (20 items). Lost 5 pages of memory. Apr 29 19:13:01 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items). Lost 2 pages of memory. Apr 29 19:13:01 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required Apr 29 19:13:01 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required Apr 29 19:13:02 freenas kernel: Stoppingthejail.... Apr 29 19:13:02 freenas kernel: delete: cannot locate 192.168.1.98 Apr 29 19:13:02 freenas kernel: epair5a: link state changed to DOWN Apr 29 19:13:02 freenas kernel: epair5b: link state changed to DOWN Apr 29 19:13:07 freenas kernel: . Apr 29 19:13:07 freenas kernel: ifa_del_loopback_route: deletion failed Apr 29 19:13:07 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:13:07 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items). Lost 3 pages of memory. Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcpreass) was not empty (336 items). Lost 4 pages of memory. Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcptw) was not empty (100 items). Lost 2 pages of memory. Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcpcb) was not empty (20 items). Lost 5 pages of memory. Apr 29 19:13:07 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items). Lost 2 pages of memory. Apr 29 19:13:07 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required Apr 29 19:13:07 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required Apr 29 19:13:08 freenas kernel: jexec 7 /usr/pbi/minidlna-amd64/control stop Apr 29 19:13:08 freenas kernel: delete: cannot locate 192.168.1.26 Apr 29 19:13:08 freenas kernel: epair6a: link state changed to DOWN Apr 29 19:13:08 freenas kernel: epair6b: link state changed to DOWN Apr 29 19:13:15 freenas kernel: . Apr 29 19:13:15 freenas kernel: ifa_del_loopback_route: deletion failed Apr 29 19:13:15 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items). Lost 4 pages of memory. Apr 29 19:13:15 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items). Lost 3 pages of memory. Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcpreass) was not empty (336 items). Lost 4 pages of memory. Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcptw) was not empty (250 items). Lost 5 pages of memory. Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (70 items). Lost 7 pages of memory. Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcpcb) was not empty (32 items). Lost 8 pages of memory. Apr 29 19:13:15 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items). Lost 2 pages of memory. Apr 29 19:13:15 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required Apr 29 19:13:15 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required Apr 29 19:13:16 freenas kernel: jexec 8 /usr/pbi/mediabrowser-amd64/control stop Apr 29 19:13:18 freenas rc.shutdown: 90 second watchdog timeout expired. Shutdown terminated. Apr 29 19:13:18 freenas init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode Apr 29 19:13:18 freenas syslogd: exiting on signal 15
Last edited: