Swap usage is killing my Plex jail - after running for years until now

[2fst4u]

Version: TrueNAS-12.0-U7
CPU: Intel(R) Core(TM) i3-4170 CPU @ 3.70GHz
Memory: 16GB ECC
Storage: 1 volume with 1 vdev of 6 X 3TB drives in RaidZ2

I have a really weird issue that just started happening in the past couple of weeks. I run a Plex jail and every day I've been waking up to a notification that the Plex server is down.

When I look into the logs it shows I'm running out of swap space (which shouldn't be used in the first place) and that is causing the server to kill Plex.

The strange part is that this server has been working absolutely fine with this amount of ram for years without this problem. The only thing I can think of that happened around the same time as this issue was I upgraded to 12.0-U7 but I reverted to 12.0-U5 and the issue stayed.

When I check the logs I get the following:

And when I check swap usage in the morning it's around a few hundred MB out of a total of 4GB, with the main user of swap being SMB - but here's the thing, I don't use SMB and have disabled the service every time but it keeps re-enabling itself.

I did enable autotune at one point but this was over a month ago, with this issue only happening in the last two weeks. I deleted all the tunables that autotune created but no luck.


I'm at a loss and have no idea what's eating up all my memory to the point that swap is being used.

I don't think the answer is more ram. I realise I don't have that much and more is always better, but this problem is clearly the result of something manifesting recently and I want to fix the underlying issue rather than mask the symptoms.

What can I do from here to troubleshoot this?

 

[Samuel Tai]

How are you sharing media to Plex? Also, what Plex jobs do you have running overnight?

 

[2fst4u]

How are you sharing media to Plex? Also, what Plex jobs do you have running overnight?

Mount from the dataset into the jail.

All the regular overnight tasks, however these have run for years unwaveringly. I'm not ruling them out but I'd find it very strange if it was killing itself when other services appear to be running with swap.

 

[Samuel Tai]

I think you may have a Plex DB issue. See this thread https://www.truenas.com/community/threads/jails-dying.97328/ which has a similar swap blowup with Plex overnight. Unfortunately, you may have to delete and recreate your Plex plugin.

 

[2fst4u]

Interesting. That specific thread is quite inconclusive but it did help to go down a rabbit hole of thread links which led to this one:

Slow library performance due to increasingly growing database

I have a new alpha build with the fix and avoids the delays seen on NAS for users with massive extras. I can send it to users that have the high number of extras as showing when doing this query on the database select count(metadata_item_id),metadata_item_id from metadata_relations group by...

forums.plex.tv

Which seems like a fix is on the way and a workaround exists to disable extras in the meantime. That's great to finally have something to go by. I'll try that workaround for now and report on if it resolves it.

 

[2fst4u]

Unfortunately this doesn't appear to be the issue I'm having and I'm back at square 1. Any ideas on where to go from here?

The next thing I might do is shut down Plex overnight and see if swap starts to be used by something else.

Edit: I think I've narrowed it down to the Plex intro detection task ballooning out to consume all memory immediately. Disabling it seems to stop the issue. I'll take this over to the Plex forums to see if they can figure it out.

 

[georgelza]

I"m sitting with the same issuer, by the sounds, Plex get killed by TrueNas, well Plex gets killed eventually after some of my other jails were killed due to swop usage.
you mention plex into detection task, can you tell me more. what where how ? Plex forum actually pointed me at this thread.

G

 

[2fst4u]

Honestly I'm pretty sure this fixed itself after applying updates. Is your Plex jail updated to the latest version?

I think I was able to prevent Plex from being killed by disabling the scheduled tasks from within Plex, and I just enabled them after the problem was patched.

 

[georgelza]

Honestly I'm pretty sure this fixed itself after applying updates. Is your Plex jail updated to the latest version?

I think I was able to prevent Plex from being killed by disabling the scheduled tasks from within Plex, and I just enabled them after the problem was patched.

I"m one version behind atm.
but this ye it's probably fixed in the next version I've been told before, hehehe, still waiting for "that" next version :)

G