Plex jail filling up my drive

[Maxburn]

90% full warning.

Does it help to see the full list?
root@parentstruenas[/mnt/parents/busync]# df -h Filesystem Size Used Avail Capacity Mounted on boot-pool/ROOT/13.0-U4 214G 1.3G 213G 1% / devfs 1.0K 1.0K 0B 100% /dev tmpfs 32M 10M 22M 32% /etc tmpfs 4.0M 8.0K 4.0M 0% /mnt tmpfs 2.6G 27M 2.6G 1% /var fdescfs 1.0K 1.0K 0B 100% /dev/fd parents 1.2T 128K 1.2T 0% /mnt/parents parents/busync 14T 13T 1.2T 91% /mnt/parents/busync parents/iocage 1.2T 8.7M 1.2T 0% /mnt/parents/iocage parents/iocage/images 1.2T 128K 1.2T 0% /mnt/parents/iocage/images parents/iocage/templates 1.2T 128K 1.2T 0% /mnt/parents/iocage/templates parents/iocage/jails 1.2T 128K 1.2T 0% /mnt/parents/iocage/jails parents/iocage/download 1.2T 128K 1.2T 0% /mnt/parents/iocage/download parents/iocage/releases 1.2T 128K 1.2T 0% /mnt/parents/iocage/releases parents/iocage/log 1.2T 134K 1.2T 0% /mnt/parents/iocage/log parents/iocage/download/13.1-RELEASE 1.2T 434M 1.2T 0% /mnt/parents/iocage/download/13.1-RELEASE parents/iocage/jails/pms 1.2T 421K 1.2T 0% /mnt/parents/iocage/jails/pms parents/iocage/download/13.0-RELEASE 1.2T 401M 1.2T 0% /mnt/parents/iocage/download/13.0-RELEASE parents/iocage/releases/13.0-RELEASE 1.2T 128K 1.2T 0% /mnt/parents/iocage/releases/13.0-RELEASE parents/iocage/releases/13.1-RELEASE 1.2T 128K 1.2T 0% /mnt/parents/iocage/releases/13.1-RELEASE parents/iocage/jails/pms/root 1.2T 3.2G 1.2T 0% /mnt/parents/iocage/jails/pms/root parents/iocage/releases/13.1-RELEASE/root 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/releases/13.1-RELEASE/root parents/iocage/releases/13.0-RELEASE/root 1.2T 1.6G 1.2T 0% /mnt/parents/iocage/releases/13.0-RELEASE/root parents/.system 1.2T 171K 1.2T 0% /var/db/system parents/.system/cores 1.0G 128K 1.0G 0% /var/db/system/cores parents/.system/samba4 1.2T 421K 1.2T 0% /var/db/system/samba4 parents/.system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996 1.2T 1.2M 1.2T 0% /var/db/system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996 parents/.system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996 1.2T 51M 1.2T 0% /var/db/system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996 parents/.system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996 1.2T 7.3M 1.2T 0% /var/db/system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996 parents/.system/webui 1.2T 128K 1.2T 0% /var/db/system/webui parents/.system/services 1.2T 128K 1.2T 0% /var/db/system/services fdescfs 1.0K 1.0K 0B 100% /var/run/samba/fd devfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/releases/13.1-RELEASE/root/dev devfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/releases/13.1-RELEASE/root/dev /mnt/parents/iocage/releases/13.1-RELEASE/root/bin 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/bin /mnt/parents/iocage/releases/13.1-RELEASE/root/boot 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/boot /mnt/parents/iocage/releases/13.1-RELEASE/root/lib 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/lib /mnt/parents/iocage/releases/13.1-RELEASE/root/libexec 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/libexec /mnt/parents/iocage/releases/13.1-RELEASE/root/rescue 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/rescue /mnt/parents/iocage/releases/13.1-RELEASE/root/sbin 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/sbin /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/bin 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/bin /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/include 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/include /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/lib 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/lib /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/libexec 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/libexec /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/sbin 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/sbin /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/share 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/share /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/libdata 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/libdata /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/lib32 1.2T 1.7G 1.2T 0% /mnt/parents/iocage/jails/pms/root/usr/lib32 /mnt/parents/busync 14T 13T 1.2T 91% /mnt/parents/iocage/jails/pms/root/mnt devfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/jails/pms/root/dev fdescfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/jails/pms/root/dev/fd root@parentstruenas[/mnt/parents/busync]#

root@parentstruenas[/mnt/parents/busync]# zfs list NAME USED AVAIL REFER MOUNTPOINT boot-pool 2.59G 213G 24K none boot-pool/ROOT 2.58G 213G 24K none boot-pool/ROOT/13.0-U4 2.58G 213G 1.29G / boot-pool/ROOT/Initial-Install 1K 213G 1.29G legacy boot-pool/ROOT/default 190K 213G 1.29G legacy parents 13.3T 1.08T 128K /mnt/parents parents/.system 61.2M 1.08T 170K legacy parents/.system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996 7.34M 1.08T 7.34M legacy parents/.system/cores 128K 1024M 128K legacy parents/.system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996 50.9M 1.08T 50.9M legacy parents/.system/samba4 1.30M 1.08T 421K legacy parents/.system/services 128K 1.08T 128K legacy parents/.system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996 1.16M 1.08T 1.16M legacy parents/.system/webui 128K 1.08T 128K legacy parents/busync 13.3T 1.08T 13.3T /mnt/parents/busync parents/iocage 8.62G 1.08T 8.73M /mnt/parents/iocage parents/iocage/download 835M 1.08T 128K /mnt/parents/iocage/download parents/iocage/download/13.0-RELEASE 401M 1.08T 401M /mnt/parents/iocage/download/13.0-RELEASE parents/iocage/download/13.1-RELEASE 434M 1.08T 434M /mnt/parents/iocage/download/13.1-RELEASE parents/iocage/images 128K 1.08T 128K /mnt/parents/iocage/images parents/iocage/jails 4.46G 1.08T 128K /mnt/parents/iocage/jails parents/iocage/jails/pms 4.46G 1.08T 421K /mnt/parents/iocage/jails/pms parents/iocage/jails/pms/root 4.46G 1.08T 3.22G /mnt/parents/iocage/jails/pms/root parents/iocage/log 133K 1.08T 133K /mnt/parents/iocage/log parents/iocage/releases 3.34G 1.08T 128K /mnt/parents/iocage/releases parents/iocage/releases/13.0-RELEASE 1.63G 1.08T 128K /mnt/parents/iocage/releases/13.0-RELEASE parents/iocage/releases/13.0-RELEASE/root 1.63G 1.08T 1.63G /mnt/parents/iocage/releases/13.0-RELEASE/root parents/iocage/releases/13.1-RELEASE 1.71G 1.08T 128K /mnt/parents/iocage/releases/13.1-RELEASE parents/iocage/releases/13.1-RELEASE/root 1.71G 1.08T 1.71G /mnt/parents/iocage/releases/13.1-RELEASE/root parents/iocage/templates 128K 1.08T 128K /mnt/parents/iocage/templates root@parentstruenas[/mnt/parents/busync]#

Extra data does not exist according to NCDU, and I'll mention it's physically impossible to send that much to the NAS over cable modem in one night.
--- /mnt/parents/busync 3.1 TiB [########################] /Backup to server 2.8 TiB [##################### ] /TV 2.2 TiB [################ ] /Movies 40.2 GiB [ ] /Music 6.0 KiB [ ] .DS_Store

 

[Maxburn]

I'm wondering if this is a problem with the jail mount being counted twice for disk usage?

disk is full, but cannot find big files or folders

Ubuntu server shows me that I use almost all disk: Usage of /: 95.5% of 118.12GB And I try to find big folders and files, run ncdu: ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help ...

askubuntu.com

When I stop the jail maybe it cleans out that folder like this guy is describing?

 

[Maxburn]

This is just completely out of control, I ran NCDU against the root instead of the sub folders and that's taking about forever as expected. Also throwing errors like below. BUT magically the high usage alert cleared and the dashboard is reporting "only" 68% full. Couple minutes later it's 62% full (NCDU still running). Now it's 57% full... which is probably about right.

That's the same odd behavior I saw in my original post here.

Warning: error scanning /mnt/parents/iocage/jails/pms/root/dev/fd/3 │
│ some directory sizes may not be correct


root@parentstruenas[~]# df -h Filesystem Size Used Avail Capacity Mounted on boot-pool/ROOT/13.0-U4 214G 1.3G 213G 1% / devfs 1.0K 1.0K 0B 100% /dev tmpfs 32M 10M 22M 32% /etc tmpfs 4.0M 8.0K 4.0M 0% /mnt tmpfs 2.6G 27M 2.6G 1% /var fdescfs 1.0K 1.0K 0B 100% /dev/fd parents 6.2T 128K 6.2T 0% /mnt/parents parents/busync 14T 8.2T 6.2T 57% /mnt/parents/busync parents/iocage 6.2T 8.7M 6.2T 0% /mnt/parents/iocage parents/iocage/images 6.2T 128K 6.2T 0% /mnt/parents/iocage/images parents/iocage/templates 6.2T 128K 6.2T 0% /mnt/parents/iocage/templates parents/iocage/jails 6.2T 128K 6.2T 0% /mnt/parents/iocage/jails parents/iocage/download 6.2T 128K 6.2T 0% /mnt/parents/iocage/download parents/iocage/releases 6.2T 128K 6.2T 0% /mnt/parents/iocage/releases parents/iocage/log 6.2T 133K 6.2T 0% /mnt/parents/iocage/log parents/iocage/download/13.1-RELEASE 6.2T 434M 6.2T 0% /mnt/parents/iocage/download/13.1-RELEASE parents/iocage/jails/pms 6.2T 421K 6.2T 0% /mnt/parents/iocage/jails/pms parents/iocage/download/13.0-RELEASE 6.2T 401M 6.2T 0% /mnt/parents/iocage/download/13.0-RELEASE parents/iocage/releases/13.0-RELEASE 6.2T 128K 6.2T 0% /mnt/parents/iocage/releases/13.0-RELEASE parents/iocage/releases/13.1-RELEASE 6.2T 128K 6.2T 0% /mnt/parents/iocage/releases/13.1-RELEASE parents/iocage/jails/pms/root 6.2T 3.2G 6.2T 0% /mnt/parents/iocage/jails/pms/root parents/iocage/releases/13.1-RELEASE/root 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/releases/13.1-RELEASE/root parents/iocage/releases/13.0-RELEASE/root 6.2T 1.6G 6.2T 0% /mnt/parents/iocage/releases/13.0-RELEASE/root parents/.system 6.2T 171K 6.2T 0% /var/db/system parents/.system/cores 1.0G 128K 1.0G 0% /var/db/system/cores parents/.system/samba4 6.2T 421K 6.2T 0% /var/db/system/samba4 parents/.system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996 6.2T 1.2M 6.2T 0% /var/db/system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996 parents/.system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996 6.2T 51M 6.2T 0% /var/db/system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996 parents/.system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996 6.2T 7.3M 6.2T 0% /var/db/system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996 parents/.system/webui 6.2T 128K 6.2T 0% /var/db/system/webui parents/.system/services 6.2T 128K 6.2T 0% /var/db/system/services fdescfs 1.0K 1.0K 0B 100% /var/run/samba/fd devfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/releases/13.1-RELEASE/root/dev devfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/releases/13.1-RELEASE/root/dev /mnt/parents/iocage/releases/13.1-RELEASE/root/bin 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/bin /mnt/parents/iocage/releases/13.1-RELEASE/root/boot 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/boot /mnt/parents/iocage/releases/13.1-RELEASE/root/lib 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/lib /mnt/parents/iocage/releases/13.1-RELEASE/root/libexec 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/libexec /mnt/parents/iocage/releases/13.1-RELEASE/root/rescue 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/rescue /mnt/parents/iocage/releases/13.1-RELEASE/root/sbin 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/sbin /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/bin 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/bin /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/include 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/include /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/lib 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/lib /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/libexec 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/libexec /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/sbin 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/sbin /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/share 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/share /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/libdata 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/libdata /mnt/parents/iocage/releases/13.1-RELEASE/root/usr/lib32 6.2T 1.7G 6.2T 0% /mnt/parents/iocage/jails/pms/root/usr/lib32 /mnt/parents/busync 14T 8.2T 6.2T 57% /mnt/parents/iocage/jails/pms/root/mnt devfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/jails/pms/root/dev fdescfs 1.0K 1.0K 0B 100% /mnt/parents/iocage/jails/pms/root/dev/fd root@parentstruenas[~]#

 

[Maxburn]

What this looks like when things are "normal"

root@parentstruenas[~]# zfs list NAME USED AVAIL REFER MOUNTPOINT boot-pool 2.59G 213G 24K none boot-pool/ROOT 2.58G 213G 24K none boot-pool/ROOT/13.0-U4 2.58G 213G 1.29G / boot-pool/ROOT/Initial-Install 1K 213G 1.29G legacy boot-pool/ROOT/default 190K 213G 1.29G legacy parents 8.15T 6.25T 128K /mnt/parents parents/.system 61.3M 6.25T 170K legacy parents/.system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996 7.34M 6.25T 7.34M legacy parents/.system/cores 128K 1024M 128K legacy parents/.system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996 51.0M 6.25T 51.0M legacy parents/.system/samba4 1.30M 6.25T 421K legacy parents/.system/services 128K 6.25T 128K legacy parents/.system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996 1.16M 6.25T 1.16M legacy parents/.system/webui 128K 6.25T 128K legacy parents/busync 8.14T 6.25T 8.14T /mnt/parents/busync parents/iocage 8.62G 6.25T 8.73M /mnt/parents/iocage parents/iocage/download 835M 6.25T 128K /mnt/parents/iocage/download parents/iocage/download/13.0-RELEASE 401M 6.25T 401M /mnt/parents/iocage/download/13.0-RELEASE parents/iocage/download/13.1-RELEASE 434M 6.25T 434M /mnt/parents/iocage/download/13.1-RELEASE parents/iocage/images 128K 6.25T 128K /mnt/parents/iocage/images parents/iocage/jails 4.46G 6.25T 128K /mnt/parents/iocage/jails parents/iocage/jails/pms 4.46G 6.25T 421K /mnt/parents/iocage/jails/pms parents/iocage/jails/pms/root 4.46G 6.25T 3.22G /mnt/parents/iocage/jails/pms/root parents/iocage/log 133K 6.25T 133K /mnt/parents/iocage/log parents/iocage/releases 3.34G 6.25T 128K /mnt/parents/iocage/releases parents/iocage/releases/13.0-RELEASE 1.63G 6.25T 128K /mnt/parents/iocage/releases/13.0-RELEASE parents/iocage/releases/13.0-RELEASE/root 1.63G 6.25T 1.63G /mnt/parents/iocage/releases/13.0-RELEASE/root parents/iocage/releases/13.1-RELEASE 1.71G 6.25T 128K /mnt/parents/iocage/releases/13.1-RELEASE parents/iocage/releases/13.1-RELEASE/root 1.71G 6.25T 1.71G /mnt/parents/iocage/releases/13.1-RELEASE/root parents/iocage/templates 128K 6.25T 128K /mnt/parents/iocage/templates root@parentstruenas[~]#

 

[sretalla]

Why can I not walk directories with zfs?

the zfs command works with zfs objects (datasets and zvols).

Although datasets effectively work like directories as far as all other tools and commands are concerned (a small number of caveats not worth going into here), it doesn't work the other way around...

directories can't act as zfs datasets, hence the zfs command won't work with them.

The zfs object that contains the data for a directory is the closest parent directory in the hierarchy which is a dataset.

 

[sretalla]

For sure df and du are not entirely zfs aware, so can produce strange/incorrect results when used in a ZFS environment, so that won't be helping by potentially adding some confusion.

When I stop the jail maybe it cleans out that folder like this guy is describing?

There is indeed a long-standing bug that jails will hold "freed" space on a mounted path until the jail is restarted (when the space is then available to the host/other consumers).

 

[Maxburn]

I'm going to note here that the plex jail mount is set to read only, so it shouldn't be possible that it is changing anything in that mount. I don't know if that could impact the hold on free space, it shouldn't be able to claim it in the first place...

What I find extremely strange is that running NCDU actually corrects the problem. Incorrect results I could understand but forcing the system to take another look and recalculating the disk total is just a really weird bug.

I guess my main takeaway at this point is my gut instinct to trust TrueNAS/FreeNAS as only a bulk storage device was correct. Forget the thing has jails and virtualization host capability. Next time I'm out there I will move plex to a local ESXi server in a VM there, with a share mount to TrueNAS. I've been doing that for years at my home and it's been flawless.

 

[Maxburn]

In the time between the last post and now this has happened again two more times. I now know two methods to reliably reclaim this falsely used space;

1. Restart the jail
2. SSH in and run NCDU against the drive

 

[Maxburn]

Forgot to restart the jail this past weekend and 100% full and I've got messages things are failing. Restart this jail and suddenly 6TB free.

 

[sretalla]

Are you transcoding in that jail? Where is your temp transcode location?

 

[Maxburn]

Are you transcoding in that jail? Where is your temp transcode location?

I don't think I'm transcoding, the TV I'm playing on should be able to do the media I have just fine. But this is a interesting Idea.

In plex the transcoding directory for temp files is empty, as in there's nothing in that setting entered at all.

But the only directory that's mounted in the jail is where my media files are and I don't think that is filling up. It shouldn't even be possible because the plex mount is read only. So if it is filling up a temp directory it's inside the jail.

I'm still baffled that I can run NCDU against the pool and get all my space back, that isn't making any sense to me. If I could find where all the space was being used that would answer what's causing it.

 

[sretalla]

Since you're already using ncdu, maybe a little inspiration from my resource (https://www.truenas.com/community/resources/pool-cleanup-tool-core-only.231/)...

ncdu -1xqo- / > /tmp/files.txt

Then later when full, try that again and compare the outputs.

 

[Maxburn]

Since you're already using ncdu, maybe a little inspiration from my resource (https://www.truenas.com/community/resources/pool-cleanup-tool-core-only.231/)...

ncdu -1xqo- / > /tmp/files.txt

Then later when full, try that again and compare the outputs.

Did you read my posts saying running ncdu against the drive fixes the problem and gets all my free space back? Does that make any sense to you?

At the moment the dashboard free space calculation is 59%, I think that's slightly bigger than it should be but not much.

ncdu -1xqo- / > /tmp/files.txt /usr/local/man/man1/rlog.1.gz 14333 files

 

[sretalla]

Did you read my posts saying running ncdu against the drive fixes the problem and gets all my free space back? Does that make any sense to you?

Yes, but I'm curious to see if you get a difference in the output in terms of a before/after (kind of hoping that the act of grabbing the directory listings comes before the "fix").

 

[Maxburn]

Yes, but I'm curious to see if you get a difference in the output in terms of a before/after (kind of hoping that the act of grabbing the directory listings comes before the "fix").

Thank you! I’ll report back in a couple of days and we can use that last post as a baseline.

 

[sretalla]

Just remember to change the filename for the output so you don't overwrite that first baseline one.

 

[hertzsae]

Do you have Video Preview Thumbnails turned on?

 

[Maxburn]

OK, 80% warning now.

ncdu -1xqo- / > /tmp/files80.txt /dev 2 files

Generate video preview files = never. Must be the default, I never touched that. Pretty much everything in this plex server is default.

 

[Maxburn]

I guess I'm going to have to do a diff on those files or something, where were you thinking of going with that? I know they are the same file size but I don't get the contents.

-rw-r--r-- 1 5701373 Aug 16 04:09 files.txt
-rw-r--r-- 1 5701373 Aug 17 18:18 files80.txt

 

[Patrick M. Hausen]

diff files.txt files80.txt