Cron Emails every 5 Minutes

[TravisT]

/var filesystem full

I keep getting emails with the following body from my freenas server:

Traceback (most recent call last):
File "/usr/local/bin/graph.py", line 467, in <module>
GenerateInterfaceGraph()
File "/usr/local/bin/graph.py", line 122, in GenerateInterfaceGraph
'GPRINT:global_tot_tx:ca. %5.1lf%s Total\l'
rrdtool.error: Could not save png to '/tmp/graph/if-em0-1h.png'

The emails are from the root account with the name Cron. Not sure what's causing this, and it didn't start until I installed 8.0.1-Beta 4. I had one cron job configured via the GUI that I deleted to see if that was causing the problem, but I'm still getting emails every 5 minutes.

 

[TravisT]

Ok, so I found the cron job that is causing this error, but still no clue as to why I'm getting the error. I'm just going to turn off email reporting until I can figure it out.

 

[TravisT]

I believe I have this fixed. I noticed that my /var filesystem was full, and logs were unable to be written. Not sure why that happened, but I assume that it was messed up from upgrading from the 8.0 Release to one of the 8.0.1 releases. I increased the size of the system disk from 1GB to 2GB, but maybe the partitioning wasn't correct. I reinstalled from the iso image and restored the configurations from beta4 and everything seems to be working. I'm keeping an eye on the partition sizes to see if any of them are getting too full.

 

[TravisT]

Well, I thought I had this fixed, but got another email last night. This time it was just one though. Still not sure what is going on that is causing the emails. Email contents were:

Code:

tar: collectd/rrd/localhost/df/.df-mnt-cargo-depot.rrd: Cannot stat: No such file or directory
tar: collectd/rrd/localhost/df/.df-mnt-cargo-starlifter.rrd: Cannot stat: No such file or directory
tar: collectd/rrd/localhost/df/.df-mnt-stripe.rrd: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

Again if anyone can help me to figure out what is causing this, I'd greatly appreciate it!

 

[TravisT]

The emails have started again as of last night. I haven't looked into it yet, but this has to be some sort of bug. Is anyone else experiencing this problem?

 

[TravisT]

So I noticed on the footer console messages that an error was thrown because the /var filesystem was full. This is the same thing that happened last time I encountered the problem, but I thought it may have been a bad install. I re-installed from scratch and imported the configs and here I am again. Am I missing something here?

 

[ProtoSD]

Are you comfortable navigating around at the command line?

Try looking around in the /var/log directory and see if you notice any particular log file that is a lot bigger than the others. When you find that, look inside and see what kind of errrors/messages there are and that might give you a clue to what is happening.

 

[TravisT]

I'm comfortable enough to be dangerous :D

I checked the file and directory sizes, and my entire /var dir is 75 M. It looks like the problem may be my zfs snapshots. I was a little overzealous when setting them up and it looks like they are being stored in the /var/db/collectd/rdd/localhost/df path. I suspected this before reading you post and modified my snapshot tasks to be a little more reasonable.

Because the process takes so long to do from the GUI, can I just delete all snapshots from that directory and start over?

If I have diagnosed this correctly, then hopefully the size of snapshots will be taken into consideration in future FreeNAS releases....

Thanks for the pointer!

 

[ProtoSD]

Yes, you can delete them all. Just think of each snapshot like a new recycle/trash bin in Windows, they hold any deleted files since the time they were created, so if you deleted any files that you want to keep they'll get deleted with the snapshot. It's strange they're being kept on /var, they can eat up tons of space if you have big files that you've deleted.

 

[TravisT]

Having problems deleting the files. I can delete them with the -f switch, but they seem to come right back.

I've deleted *.rrd in the directory, and can see that the files are gone. After a couple of seconds they reappear.

Just to give you an idea, the file names are as follows:

df-mnt-cargo-globemaster-.zfs-snapshot-auto-20110624.rrd

cargo is my zfs volume name and globemaster is the dataset. Any idea what's going on?

 

[ProtoSD]

Sorry, when you asked if you could remove the snapshots, I assumed you meant with proper zpool command. I tried snapshots, but decided not to use them. There's a specific command for deleting them. I'll have to look it up, I don't remember it off the top of my head. How often are you creating them, it sounds like they're reproducing faster than rabbits!

I'll update when I find the command(s).

Show snapshots:

zfs list -t snapshot

Delete snapshot:

zfs destroy poolname/filesystem@name-of-snapshot

 

[TravisT]

Yea, I found it and tried it. I deleted all of the snapshots, but is seems the /var directory is still full. It looks like it's something with the collectd files which seem to be used for the reporting tab (maybe). The files are still in the /var/db/collectd/rrd/localhost/df directory. Deleting the snapshots didn't clear them.

Not sure what to do, but I'm really tired of getting all of these emails!

 

[ProtoSD]

I guess you could just delete your email address in the settings until you figure out what the problem is. Are there just a lot of files in that directory, or are they just getting large? Are you sure there's nothing in one of the /var/log directories growing?

 

[TravisT]

Yea, I could do that, but I'm afraid other things will end up being affected. Case in point, just restarted and I can't access it from http and I'm getting a ton of errors on the console.

Code:

kernel: pid 1344 (nmbd), uid 0 inumber 15627 on /var: filesystem full

Right now, /var is 75M in size. 68M of that is in the /var/db/collectd/rrd/locahost/df directory. After deleting the snapshots using This command:

Code:

zfs list -H -o name -t snapshot | xargs -n1 zfs destroy

I was able to delete all of the snapshots. When I went back to the /var directory that was full, I deleted all of the .rrd files and they seem to have deleted permanently. Time will tell, but I hope that fixed it.

 

[TravisT]

Never mind. After a reboot, all the files were back in the /var directory and I'm getting an error in by browser that /var/tmp is not writable.

I only have a config from a week or so ago when I upgraded to beta 4. The only way I know to fix this is to revert to that config, but I don't know how long it will fix it.

 

[sschen]

Hi there

Just wondering if you got to the bottom of this?

I just upgraded from the 32-bit FreeNAS 8.01 Beta 4 to the 64-bit FreeNAS 8.01 RC1, upgraded memory from 2GB to 8GB at the same time. Since the upgrade, I am getting the following cron email every 5 minutes. Is this the same problem as yours? I used the same config from the 32-bit 8.01B4

Traceback (most recent call last):
File "/usr/local/bin/graph.py", line 468, in <module>
GenerateCpuGraph()
File "/usr/local/bin/graph.py", line 189, in GenerateCpuGraph
'GPRINT:avg4:LAST:%5.2lf Last\l',
rrdtool.error: This RRD was created on another architecture

 

[TravisT]

The only way I was able to get them to stop was to re-install freenas from scratch, and not enable snapshots. It sucks because I'd really like to take advantage of that feature, but the snapshots were filling up the /usr directory and I was running out of disk space, which I believe caused the emails to start.

Are you using snapshots?

 

[Durkatlon]

I believe this has to do with the fact that each snapshot is considered a new dataset by FreeNAS (up until recently!), so the RRD statistics collection process creates a graphic for each of them. The data stored to create the graphics really starts adding up after a while so /var goes full very rapidly.

Although I haven't tried the new version yet outside VMWare toy installations, the new 8.0.1-RC1 is supposed to solve this problem by not including snapshots in the RRD graphs anymore. If you haven't already I would suggest you upgrade to RC1 and then if all seems well, reenable the autosnapshots.

 

[TravisT]

I'm away from home for a long while, so I will probably not upgrade unless there are substantial enhancements in the builds until I'm actually there to work through any issues that may arise. That is really good news though. I was happy to see that feature came along when it did - I'd love for it to be working correctly.

Even though I'm not planning to upgrade remotely, I've been checking the website for the release of the new build every couple of days. Really looking forward to 8.1...

 

[sschen]

The only way I was able to get them to stop was to re-install freenas from scratch, and not enable snapshots. It sucks because I'd really like to take advantage of that feature, but the snapshots were filling up the /usr directory and I was running out of disk space, which I believe caused the emails to start.

Are you using snapshots?

Hi there

Thanks for this tip "re-install from scratch".

I had originally intended to do that. I normally have the FreeNAS image installed on two separate USB drives, one is used in the machine itself, the other as spare/redundancy. Whenever a new release comes out, I will just use one of the USB drive to install the new image onto. When I did that this time, I had not realised the first option on the install image screen was asking if I wanted to upgrade the existing image and I answered "Yes" to that question. As I was using the 32-bit version previously, although the upgrade to 64-bit version was fine and I was able to see all my drives etc, it obviously didn't upgrade the WebGUI "Reporting" modules properly hence why the "Reporting" page on the WebGUI was empty. I re-do it again when I saw your post, this time selecting "No" forcing it to do a fresh install.

The Cron email has gone away, the "Reporting" page is fine now too :)