Received critical alert emails overnight for freenas-boot volume

[ggoldfingerd]

Received emails due to the freenas-boot volume running out of space. I am not sure what it was yesterday, but I received multiple emails which shows the size went from 90% to 96% overnight without any system updates. I am not sure why it is growing in size when nothing was updated or a backup was created. I have two 32GB USB thumb drives, one is a mirror.

I had a total of 6 boot environments, 5 backups and 1 active environment. I tried to delete two, but the WEB GUI returned errors, yet the disappeared from the GUI.

Code:

df -h
Filesystem                                                      Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201506292332                27G     27G    9.4M   100%    /
devfs                                                           1.0k    1.0k      0B   100%    /dev
tmpfs                                                            32M    5.3M     26M    16%    /etc
tmpfs                                                           4.0M    8.0k      4M     0%    /mnt
tmpfs                                                            21G    119M     21G     1%    /var
freenas-boot/grub                                                16M    6.8M    9.4M    42%    /boot/grub
matrix                                                           15T    192k     15T     0%    /mnt/matrix
matrix/jails                                                     15T    351k     15T     0%    /mnt/matrix/jails
matrix/jails/.warden-template-pluginjail--x64                    15T    576M     15T     0%    /mnt/matrix/jails/.warden-template-pluginjail--x64
matrix/jails/.warden-template-pluginjail--x64-20150627180151     15T    576M     15T     0%    /mnt/matrix/jails/.warden-template-pluginjail--x64-20150627180151
matrix/jails/.warden-template-pluginjail--x64-20150712123217     15T    576M     15T     0%    /mnt/matrix/jails/.warden-template-pluginjail--x64-20150712123217
matrix/jails/couchpotato_1                                       15T    920M     15T     0%    /mnt/matrix/jails/couchpotato_1
matrix/jails/plexmediaserver_1                                   15T    3.9G     15T     0%    /mnt/matrix/jails/plexmediaserver_1
matrix/jails/sabnzbd_1                                           15T    903M     15T     0%    /mnt/matrix/jails/sabnzbd_1
matrix/jails/sickrage_1                                          15T    938M     15T     0%    /mnt/matrix/jails/sickrage_1
matrix/jails/transmission_1                                      15T    706M     15T     0%    /mnt/matrix/jails/transmission_1
matrix/media                                                     21T    5.1T     15T    24%    /mnt/matrix/media
matrix/.system                                                   15T    208k     15T     0%    /var/db/system
matrix/.system/cores                                             15T    2.0M     15T     0%    /var/db/system/cores
matrix/.system/samba4                                            15T    759k     15T     0%    /var/db/system/samba4
matrix/.system/syslog-26bec1b654b54f3489f57aa43ee342f9           15T      1M     15T     0%    /var/db/system/syslog-26bec1b654b54f3489f57aa43ee342f9
matrix/.system/rrd-26bec1b654b54f3489f57aa43ee342f9              15T    192k     15T     0%    /var/db/system/rrd-26bec1b654b54f3489f57aa43ee342f9
matrix/.system/configs-26bec1b654b54f3489f57aa43ee342f9          15T    192k     15T     0%    /var/db/system/configs-26bec1b654b54f3489f57aa43ee342f9
devfs                                                           1.0k    1.0k      0B   100%    /mnt/matrix/jails/couchpotato_1/dev
procfs                                                          4.0k    4.0k      0B   100%    /mnt/matrix/jails/couchpotato_1/proc
/mnt/matrix/media/3D Movies                                      21T    5.1T     15T    24%    /mnt/matrix/jails/couchpotato_1/media/3D Movies
/mnt/matrix/media/Download                                       21T    5.1T     15T    24%    /mnt/matrix/jails/couchpotato_1/media/Download
/mnt/matrix/media/Movies                                         21T    5.1T     15T    24%    /mnt/matrix/jails/couchpotato_1/media/Movies
devfs                                                           1.0k    1.0k      0B   100%    /mnt/matrix/jails/plexmediaserver_1/dev
procfs                                                          4.0k    4.0k      0B   100%    /mnt/matrix/jails/plexmediaserver_1/proc
/mnt/matrix/media/3D Movies                                      21T    5.1T     15T    24%    /mnt/matrix/jails/plexmediaserver_1/media/3D Movies
/mnt/matrix/media/Movies                                         21T    5.1T     15T    24%    /mnt/matrix/jails/plexmediaserver_1/media/Movies
/mnt/matrix/media/TV                                             21T    5.1T     15T    24%    /mnt/matrix/jails/plexmediaserver_1/media/TV
devfs                                                           1.0k    1.0k      0B   100%    /mnt/matrix/jails/sabnzbd_1/dev
procfs                                                          4.0k    4.0k      0B   100%    /mnt/matrix/jails/sabnzbd_1/proc
/mnt/matrix/media/Download                                       21T    5.1T     15T    24%    /mnt/matrix/jails/sabnzbd_1/media/Complete
devfs                                                           1.0k    1.0k      0B   100%    /mnt/matrix/jails/sickrage_1/dev
procfs                                                          4.0k    4.0k      0B   100%    /mnt/matrix/jails/sickrage_1/proc
/mnt/matrix/media/Download                                       21T    5.1T     15T    24%    /mnt/matrix/jails/sickrage_1/media/Download
/mnt/matrix/media/TV                                             21T    5.1T     15T    24%    /mnt/matrix/jails/sickrage_1/media/TV
devfs                                                           1.0k    1.0k      0B   100%    /mnt/matrix/jails/transmission_1/dev
procfs                                                          4.0k    4.0k      0B   100%    /mnt/matrix/jails/transmission_1/proc
/mnt/matrix/media/Torrents                                       21T    5.1T     15T    24%    /mnt/matrix/jails/transmission_1/media/torrents

It looks like freenas-boot/ROOT/FreeNAS-9.3-STABLE-201506292332 is using 27GB.

What should I check or do next to fix this? I will be shutting my NAS down for today until I can address this. I don't want to loose anything.

 

[cyberjock]

I don't know what is wrong, but something is taking up 27TB of space on your boot device. So you need to use du to try to figure out where all the data is being stored. I'm somewhat expecting the cause to be user-error as that's the only reasonable conclusion I can come to. But what you'd have done to cause it is very much in the air. There's so many ways you could cause this that I won't even try to speculate.

But check out the manpage on du and see where all that storage space is being used.

 

[ggoldfingerd]

Thanks for the quick response and pointer. I will investigate this tonight and post back. It could definitely be my fault as I'm a newb to FreeNAS. Live and learn!

 

[ggoldfingerd]

So it looks like two torrent files were moved to a folder called Downloads in the root directory of FreeNAS. I deleted the folder and now I am down to 2% usage. I remember that I tried to copy one directory using SSH and must have typed something that caused it to create a folder and copy the contents. This was my fault. Thanks for informing me about du. I was able to find the issue using the threshold option.

 

[cyberjock]

You naughty boy! :P