Out of space

[greenhobby]

I'm not sure this is a bug, but while scrubbing the main volume last night, my inbox was flooded with this:
Cron <root@clanker> /usr/local/bin/python /usr/local/bin/graph.py

Traceback (most recent call last):
File "/usr/local/bin/graph.py", line 480, in <module>
GenerateInterfaceGraph()
File "/usr/local/bin/graph.py", line 135, in GenerateInterfaceGraph
'GPRINT:global_tot_tx:ca. %5.1lf%s Total\l'

rrdtool.error: Could not save png to '/tmp/graph/if-bge0-1h.png'

It seems that running for 90 days somehow filled up the /tmp area? My guess is that the emails it sends out are cumulative in /tmp and will eventually fill it up. The system was up for 3 months with scrubbing happening every Wednesday and Thursday night. I rebooted and /tmp was erased.

thoughts?

 

[jgreco]

It is not unheard-of; figuring out exactly how large to make things like /tmp on any UNIX system is always fun.

/tmp is actually /var/tmp on FreeNAS, which is in turn part of a memory-based /var filesystem. It is exceedingly likely that something filled up space on /var. It would have been potentially useful to know exactly what, so if it happens again, log into the console and inspect with du.

Code:

# df /var
Filesystem 1K-blocks  Used  Avail Capacity  Mounted on
/dev/md2      152791 22719 117849    16%    /var
# du /var
1      /var/.snap
1      /var/empty
1      /var/tmp/vi.recover
1      /var/tmp/nginx/proxy_temp
1      /var/tmp/nginx/fastcgi_temp
1      /var/tmp/nginx/uwsgi_temp
1      /var/tmp/nginx/scgi_temp
3      /var/tmp/nginx
1      /var/tmp/firmware
5      /var/tmp/graph
13      /var/tmp
27      /var/backups
1      /var/mail
1      /var/spool/samba
1      /var/spool/lock
1      /var/spool/lpd
1      /var/spool/mqueue
1      /var/spool/output/lpd
1      /var/spool/output
1      /var/spool/opielocks
4      /var/spool
1      /var/preserve
1      /var/at/spool
1      /var/at/jobs
2      /var/at
1      /var/log/samba/cores/nmbd
1      /var/log/samba/cores/smbd
2      /var/log/samba/cores
21      /var/log/samba
1      /var/log/proftpd
197    /var/log
1      /var/msgs
1      /var/games
1      /var/crash
1      /var/named/etc/namedb/working
1      /var/named/etc/namedb/dynamic
1      /var/named/etc/namedb/slave
1      /var/named/etc/namedb/master
3      /var/named/etc/namedb
3      /var/named/etc
1      /var/named/var/dump
1      /var/named/var/log
1      /var/named/var/run/named
1      /var/named/var/run
1      /var/named/var/stats
3      /var/named/var
1      /var/named/dev
7      /var/named
1      /var/cron/tabs
1      /var/cron
1      /var/agentx
1      /var/audit
1      /var/account
1      /var/db/ports
1      /var/db/samba/printing
1466    /var/db/samba
1      /var/db/ipf
1      /var/db/freebsd-update
58      /var/db/fontconfig
1      /var/db/entropy
1      /var/db/portsnap
761    /var/db/collectd/rrd/localhost/cpu-0
153    /var/db/collectd/rrd/localhost/uptime
305    /var/db/collectd/rrd/localhost/swap
761    /var/db/collectd/rrd/localhost/memory
1065    /var/db/collectd/rrd/localhost/processes
437    /var/db/collectd/rrd/localhost/load
7115    /var/db/collectd/rrd/localhost/df
7405    /var/db/collectd/rrd/localhost/interface
761    /var/db/collectd/rrd/localhost/cpu-1
18759  /var/db/collectd/rrd/localhost
18760  /var/db/collectd/rrd
18760  /var/db/collectd
1      /var/db/dbus
1      /var/db/nut
380    /var/db/graphs/hourly
378    /var/db/graphs/daily
392    /var/db/graphs/weekly
390    /var/db/graphs/monthly
397    /var/db/graphs/yearly
1937    /var/db/graphs
22227  /var/db
1      /var/rwho
1      /var/run/ppp
2      /var/run/samba
1      /var/run/proftpd
1      /var/run/wpa_supplicant
1      /var/run/named
1      /var/run/dbus
28      /var/run
1      /var/yp
1      /var/heimdal
81      /var/etc/private
81      /var/etc
1      /var/nmbd
22595  /var

I would expect something to blow up in /var/log or /var/db/samba maybe. You should be able to spot differences. Then identify more specifically what it is that's filling things up.

 

[greenhobby]

Thanks jgreco. I will check it every month or so. Considering how well it runs, I have actually been negligent on checking things periodically.

YES, I agree, it would have been nice to look at it in this state, but it was hung and not responding in anyway (except sending out these emails every 5 minutes). Could log into console, or through the network. Maybe this is expected?

This is an awesome collection of bits! And checking once a month for /tmp usage is no biggie...

-gh