Boot-pool full following automatic scrub [Solved]

[tgyk]

Hello all!

I have been using TrueNAS SCALE for a little bit now, and have run into a bit of an issue. My boot-pool zpool "fills up" and I get constant critical alerts notifying me of this. I put that into quotes because I am not sure what could be using that space. Ncdu reports a total usage of only 4.7Gb on a 64Gb SSD, df and zfs report 57GB used. I do not have snapshots enabled. I am fairly stumped and now turning to the community for help. Below are a few command outputs for zpool/zfs.

Code:

truenas# zfs list -r boot-pool
NAME                                                         USED  AVAIL     REFER  MOUNTPOINT
boot-pool                                                   57.1G  35.1M       96K  none
boot-pool/.system                                           7.70M  35.1M      128K  legacy
boot-pool/.system/configs-0efd0a67fe784978b852714f919da917    96K  35.1M       96K  legacy
boot-pool/.system/cores                                       96K  35.1M       96K  legacy
boot-pool/.system/glusterd                                    96K  35.1M       96K  legacy
boot-pool/.system/rrd-0efd0a67fe784978b852714f919da917      5.82M  35.1M     5.82M  legacy
boot-pool/.system/samba4                                     144K  35.1M      144K  legacy
boot-pool/.system/services                                    96K  35.1M       96K  legacy
boot-pool/.system/syslog-0efd0a67fe784978b852714f919da917   1.15M  35.1M     1.15M  legacy
boot-pool/.system/webui                                       96K  35.1M       96K  legacy
boot-pool/ROOT                                              57.1G  35.1M       96K  none
boot-pool/ROOT/20.12-ALPHA                                  57.1G  35.1M     56.9G  legacy
boot-pool/ROOT/Initial-Install                                 8K  35.1M     2.65G  legacy
boot-pool/grub                                              5.76M  35.1M     5.76M  legacy

Code:

truenas# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Data       14.5T  5.83T  8.67T        -         -     0%    40%  1.00x    ONLINE  /mnt
boot-pool    59G  57.1G  1.88G        -         -    69%    96%  1.00x    ONLINE  -

Code:

truenas# df -h /
Filesystem                  Size  Used Avail Use% Mounted on
boot-pool/ROOT/20.12-ALPHA   57G   57G   35M 100% /

Edit: Using CODE tags instead of QUOTE tags for readability

 

[artlessknave]

please use code tags instead of quote(?) so it can be read.

Code:

Filesystem                  Size  Used Avail Use% Mounted on
boot-pool/ROOT/20.12-ALPHA   57G   57G   35M 100% /  

Code:

boot-pool/ROOT                                              57.1G  35.1M       96K  none
boot-pool/ROOT/20.12-ALPHA                                  57.1G  35.1M     56.9G  legacy

it looks like you have 2 datasets that are using it all for some reason. i would guess you or some process created 50GB worth of files in / somewhere. the easiest way to fix it would be to reinstall fresh, but you should be able to use du in /, find where the usage is, delete it, make a new boot environment, and delete the other other boot environments.
(sudo) du -hd 1 /
(mine sits at /mnt forever cuz it's huge, so you might have to do the folders in / individually for this to finish in your lifetime if you have anything there)
(sudo) du -hd 1 -I mnt / <that's an uppercase I (eye) not an l (el)

 

[tgyk]

Ahh, sorry. Not used to these forums just yet.

du threw an error with the -I option

Code:

truenas# du -hd 1 -I mnt /
du: invalid option -- 'I'
Try 'du --help' for more information.

This is the output of du without it. I exclude /mnt/Backup because that is where my external HDD is mounted for backups, and exclude /mnt/Data because that is where my data pool is mounted, both about 10TB in size

Code:

truenas# du -hd 1 --exclude /mnt/Backup --exclude /mnt/Data /
2.6G    /usr
67M     /boot
8.0M    /etc
3.7M    /run
44K     /root
du: cannot read directory '/proc/2124/task/2124/net': Invalid argument
du: cannot read directory '/proc/2124/net': Invalid argument
du: cannot access '/proc/3287025/task/3287025/fd/4': No such file or directory
du: cannot access '/proc/3287025/task/3287025/fdinfo/4': No such file or directory
du: cannot access '/proc/3287025/fd/3': No such file or directory
du: cannot access '/proc/3287025/fdinfo/3': No such file or directory
0       /proc
512     /media
0       /sys
512     /mnt
39M     /tmp
6.0K    /srv
39M     /opt
1.0K    /home
317K    /conf
1.9G    /var
260K    /data
104K    /dev
4.7G    /

 

[artlessknave]

OH!!! my bad that's scale+linux not BSD. gonna take a bit to wrap my head around freenas on linux...
that...doesnt show it. hmm.
are you sure there are no snapshots somewhere? what about:
zfs list -t all -o space -r boot-pool
other than that, the only thing I could remend is delete those dataset, which I believe are boot environments?

 

[tgyk]

It seems there was a single snapshot made automatically on the date of install.

Code:

truenas# zfs list -t all -o space -r boot-pool
NAME                                                        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
boot-pool                                                   37.7M  57.1G        0B     96K             0B      57.1G
boot-pool/.system                                           37.7M  7.70M        0B    128K             0B      7.58M
boot-pool/.system/configs-0efd0a67fe784978b852714f919da917  37.7M    96K        0B     96K             0B         0B
boot-pool/.system/cores                                     37.7M    96K        0B     96K             0B         0B
boot-pool/.system/glusterd                                  37.7M    96K        0B     96K             0B         0B
boot-pool/.system/rrd-0efd0a67fe784978b852714f919da917      37.7M  5.82M        0B   5.82M             0B         0B
boot-pool/.system/samba4                                    37.7M   144K        0B    144K             0B         0B
boot-pool/.system/services                                  37.7M    96K        0B     96K             0B         0B
boot-pool/.system/syslog-0efd0a67fe784978b852714f919da917   37.7M  1.15M        0B   1.15M             0B         0B
boot-pool/.system/webui                                     37.7M    96K        0B     96K             0B         0B
boot-pool/ROOT                                              37.7M  57.1G        0B     96K             0B      57.1G
boot-pool/ROOT/20.12-ALPHA                                  37.7M  57.1G      204M   56.9G             0B         0B
boot-pool/ROOT/20.12-ALPHA@2021-01-09-00:04:30                  -   204M         -       -              -          -
boot-pool/ROOT/Initial-Install                              37.7M     8K        0B      8K             0B         0B
boot-pool/grub                                              37.7M  5.76M        0B   5.76M             0B         0B

Code:

truenas# zfs destroy boot-pool/ROOT/20.12-Alpha@2021-01-09-00:04:30
cannot open 'boot-pool/ROOT/20.12-Alpha': dataset does not exist
truenas# zfs destroy boot-pool/ROOT/20.12-ALPHA@2021-01-09-00:04:30
cannot destroy 'boot-pool/ROOT/20.12-ALPHA@2021-01-09-00:04:30': snapshot has dependent clones
use '-R' to destroy the following datasets:
boot-pool/ROOT/Initial-Install

I would delete the ALPHA dataset, but it is my current and only boot environment. I am really stumped.

 

[artlessknave]

only boot environment

um. make a new boot environment and delete the old ones? Im not sure at this point. im not sure why it sounds like you have data in a SCALE environment at all, since it is like (pre?)-alpha still.

 

[sretalla]

Why aren't you managing the boot pool from the system | boot menu in the GUI and removing no longer needed boot environments from there?

 

[tgyk]

um. make a new boot environment and delete the old ones? Im not sure at this point. im not sure why it sounds like you have data in a SCALE environment at all, since it is like (pre?)-alpha still.

The data stored on this instance is only Plex-used moviex/tv shows. It's all stored in an 8-disk array that can be easily exported/imported and is otherwise backed up to a 10TB external HDD should there be an issue with the integrity of SCALE. As far as making a new boot env, are there any guides that could walk me through the process? As you can see, this current environment is one which is still very young.

Why aren't you managing the boot pool from the system | boot menu in the GUI and removing no longer needed boot environments from there?

The only no-longer needed environment would be Initial-install and I see that it is not taking much room at all. At this point, I am trying to identify the phantom usage within the ALPHA dataset.

 

[artlessknave]

Plex-used moviex/tv shows.

otherwise backed up

ok fair enough, testing with real data is good for scale.
create a new boot environment is super easy....it's like 4 clicks. the essence of it is in the truenas manual, the only difference on scale is that the menu locations have been moved.
system settings->boot->actions->add->type a name->save
then just delete the other ones and see if the usage goes away. if it doesnt, then at least you know that whatever it is that's using space is captured by boot environments.

 

[tgyk]

I found my phantom usage. It was staring me in the face. An automated rsync backup script that did not check that a disk was mounted before running tried to backup ~5TB of data to 64GB of storage. Unmounted the disk, deleted the data from the Backup folder, and all is hunky-dory. Interesting point to learn from though.

Thanks everyone who pitched in some help!

 

[artlessknave]

ah. good.