guermantes
Patron
- Joined
- Sep 27, 2017
- Messages
- 213
I have six nightly tasks running, four rsync push tasks between 1 AM-2 AM and two cron tasks at 6:45 AM (one copying log files for the rsync tasks and the other a community script sending SMART reports by email). I noticed a few days ago that I had not received any SMART emails since 10 Nov so I went to investigate.
I noticed that the destination for the cron task copying log files did not have any new files since 10 Nov. So that cron task did not seem to have been running either. However, at the source, the rsync log files had been updated for three of the rsync tasks but not for the fourth. The fourth one was dated 11 Nov. I looked at it and saw that it seemed to have stopped during execution. Heading over to the Freenas GUI I saw that the rsync task in question was indeed listed as "Running" with a yellow button. Apparently it had hung.
I went to the command line and killed the hung rsync process and its corresponding lock. Briefly the Freenas GUI button went red for "Failed" and then went to no color and "Pending". And a few minutes later my mailbox received 19 emails from the script sending smart reports (in my count it should have been 20 since it was 2 PM on 30 Nov, but I was happy enough with 19). The following night all rsync tasks and both cron jobs executed and finished without problem, and no problems since.
The fact that the cron tasks never ran from 11 Nov until I killed the hung rsync task puzzles me.
I noticed that the destination for the cron task copying log files did not have any new files since 10 Nov. So that cron task did not seem to have been running either. However, at the source, the rsync log files had been updated for three of the rsync tasks but not for the fourth. The fourth one was dated 11 Nov. I looked at it and saw that it seemed to have stopped during execution. Heading over to the Freenas GUI I saw that the rsync task in question was indeed listed as "Running" with a yellow button. Apparently it had hung.
I went to the command line and killed the hung rsync process and its corresponding lock. Briefly the Freenas GUI button went red for "Failed" and then went to no color and "Pending". And a few minutes later my mailbox received 19 emails from the script sending smart reports (in my count it should have been 20 since it was 2 PM on 30 Nov, but I was happy enough with 19). The following night all rsync tasks and both cron jobs executed and finished without problem, and no problems since.
The fact that the cron tasks never ran from 11 Nov until I killed the hung rsync task puzzles me.
- Is it expected that cron tasks never get to execute because an rsync task has has hung? I would expect those processes to be completely independent of each other.
- I never received alerts from Freenas that the two cron tasks could not be run successfully either. I have a vague memory that such alerts should be issued.
