Hello FreeNAS community,
last saturday, we updated two of our storage systems to FreeNAS 9.3-RELEASE-p31 and since then, one of these systems resets itself every day exactly at 6am UTC (local timezone is CET).
Unfortunately we were not yet able to discover what is causing these resets.
This morning, I was logged in and tried to observe what is going on when the system reset itself.
Here are the last lines of /var/log/cron before the reset:
There are absolutely NO messages in /var/log/messages shortly before the reset, also dmesg does not show anything suspicious.
While I was waiting for the reset this morning, I saw that the RRD graphs in the reporting section of the Web-UI showed values for the timeframe between 5am UTC and 6am UTC. Then, after the reset happened, this timeframe was empty in all RRD graphs, and the first new values were starting at 6:07am UTC when the system was completely up again after the reset.
These resets occur really every day, but only on one of our two storage systems. The hardware of these systems is different but both are running FreeNAS 9.3-RELEASE-p31.
Does anybody have any idea what could be causing these resets? At least any idea on how to dig further could be helpful...
Thank you very much in advance!
Best regards,
Christian
last saturday, we updated two of our storage systems to FreeNAS 9.3-RELEASE-p31 and since then, one of these systems resets itself every day exactly at 6am UTC (local timezone is CET).
Unfortunately we were not yet able to discover what is causing these resets.
This morning, I was logged in and tried to observe what is going on when the system reset itself.
Here are the last lines of /var/log/cron before the reset:
Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56205]: (root) CMD (/usr/local/bin/python /usr/local/bin/mfistatus.py > /dev/null 2>&1)
Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56206]: (root) CMD (/bin/sh /usr/local/sbin/save_rrds.sh > /dev/null 2>&1)
Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56207]: (operator) CMD (/usr/libexec/save-entropy > /dev/null 2>&1)
Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56208]: (root) CMD (/usr/libexec/atrun > /dev/null 2>&1)
Here are the top 3 process in top output before the reset:Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56206]: (root) CMD (/bin/sh /usr/local/sbin/save_rrds.sh > /dev/null 2>&1)
Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56207]: (operator) CMD (/usr/libexec/save-entropy > /dev/null 2>&1)
Feb 25 07:00:00 cloudstorage02 /usr/sbin/cron[56208]: (root) CMD (/usr/libexec/atrun > /dev/null 2>&1)
56248 root 1 103 0 36952K 11256K CPU6 6 0:41 98.58% bsdtar
4111 root 6 20 0 466M 105M usem 7 0:24 1.56% python2.7
4116 root 1 52 0 208M 35572K select 1 1:14 0.29% python2.7
4111 root 6 20 0 466M 105M usem 7 0:24 1.56% python2.7
4116 root 1 52 0 208M 35572K select 1 1:14 0.29% python2.7
There are absolutely NO messages in /var/log/messages shortly before the reset, also dmesg does not show anything suspicious.
While I was waiting for the reset this morning, I saw that the RRD graphs in the reporting section of the Web-UI showed values for the timeframe between 5am UTC and 6am UTC. Then, after the reset happened, this timeframe was empty in all RRD graphs, and the first new values were starting at 6:07am UTC when the system was completely up again after the reset.
These resets occur really every day, but only on one of our two storage systems. The hardware of these systems is different but both are running FreeNAS 9.3-RELEASE-p31.
Does anybody have any idea what could be causing these resets? At least any idea on how to dig further could be helpful...
Thank you very much in advance!
Best regards,
Christian