Freenas 9.2.x and Plex, running out of swap space

[hammong]

I've be on the Plex forum, and they have several people that are experiencing the same issue - but very little documentation about what's causing it and when.

My graphs look almost identical to danb35's -- and the problem in my case occurs ONLY when writing streams to the server via CIFS and ONLY when Plex is actually doing a background or foreground scan of the library. My Plex logs show a DEBUG message saying that it's doing expensive tag write for "abcefg" because something changed.

danb35 - see if your Plex logs show something like this, with corresponding date/times to when you're running out of swap.... My theory is that you DID have a background scan running (either initiated by a web browser, or the Plex scheduler) and it got stuck on the file you were ripping.

Greg


/mnt/ssd1/plexmediaserver_1/usr/pbi/plexmediaserver-amd64/plexdata/Plex Media Server/Logs# fgrep expensive *

Plex Media Server.log.5:Jun 04, 2014 13:36:28 [0x80bdd8c00] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:36:51 [0x808081400] DEBUG - Doing expensive tags write for 'Shrek 2' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:47:13 [0x80acd3400] DEBUG - Doing expensive tags write for 'Tarzan' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:47:13 [0x80acd3400] DEBUG - Doing expensive tags write for 'Tarzan' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:47:13 [0x80acd3400] DEBUG - Doing expensive tags write for 'Tarzan' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:47:13 [0x80acd3400] DEBUG - Doing expensive tags write for 'Tarzan' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:47:13 [0x80acd3400] DEBUG - Doing expensive tags write for 'Tarzan' because something changed.
Plex Media Server.log.5:Jun 04, 2014 13:47:13 [0x80acd3400] DEBUG - Doing expensive tags write for 'Tarzan' because something changed.


Looking at this particular log, my swap peaked to 27GB used (out of 48GB swap) beginning as 13:47:13 almost to the second. At that point "Tarzan" finished ripping, the file was closed, and the errors stopped - and my swap returned back down to 300 KB used instead of 27 GB.

 

[danb35]

Interesting. I was running a rip at the time (two or three, actually, from different computers), so this would track your theory. I also saw a lot of "expensive tags write" messages in the plex logs, though they didn't exactly correspond in time to when I noticed the issue. Seems easy enough to work around, though--rip to a working directory, and copy the files over once they're done.

 

[danb35]

I don't think that's what's doing it, at least for me. I had a couple of rips that were running to the movies directory, but once those were finished I started ripping instead to a different directory which Plex doesn't scan. I then restarted the Plex server and just let it run. Sure enough, when it started a scheduled scan, the memory usage started skyrocketing. I was watching the log with tail -f, and I didn't see anything about "expensive tags write", either. Noticed lots of this in the system log, too:

Code:

Jun  7 15:59:21 freenas2 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3152064, size: 4096

...which I suspect correspond pretty well with the system becoming unresponsive.

 

[danb35]

I realize I'm posting in kind of a "stream of consciousness" mode, but I'm wondering what can be done to protect FreeNAS from misbehaving plugins. Isn't it possible to set resource limits on jails? If so, how would that be done? This seems to be known, albeit poorly-understood, issue with the Plex Media Server, but it seems it should be possible to limit the damage. Certainly one option would be to run it in a VM, but that seems like it would be more resource-intensive than a jail. OTOH, that should completely protect the FreeNAS installation from Plex (or anything else) being a memory hog, as anything on the VM would not be able to exceed its assigned limits.

 

[Joshua Parker Ruehlig]

I realize I'm posting in kind of a "stream of consciousness" mode, but I'm wondering what can be done to protect FreeNAS from misbehaving plugins. Isn't it possible to set resource limits on jails? If so, how would that be done? This seems to be known, albeit poorly-understood, issue with the Plex Media Server, but it seems it should be possible to limit the damage. Certainly one option would be to run it in a VM, but that seems like it would be more resource-intensive than a jail. OTOH, that should completely protect the FreeNAS installation from Plex (or anything else) being a memory hog, as anything on the VM would not be able to exceed its assigned limits.

that's not possible, I think the jail implementation in freebsd 10 has some resource limitation.

EDIT

looks like its in freebsd 9 but I don't know how you would se that in freenas.

 

[Soundscape]

I'm having the same swap issues and they only started after adding plex (only plugin i've added). My system only has 8GB of ram and default 2GB swap per drive (6 drives zfs2)
Is there a way to prepare for pool loss and recovery or is backing up all my data my only option here? if so, i'm ditching plex altogether. can't risk losing my life which is now embedded on this diy nas. I do have my original data on a pile of dns-323's so I'd only lose some recent stuff...

Ok, so I'm gonna say this, and I know what your answer is going to be.

If you are having to rely on swap to keep the system stable you are at serious risk for losing your pool. Now you're going to tell me that upgrading to 64GB of RAM is too expensive or your hardware won't support it. But, that doesn't change the reality of it.

You stand a very real risk of losing your pools in your configuration. Fix it ASAP or don't say I didn't warn you. ;)

ZFS won't forgive you if you have to 'wait until your tax return' or something else. It is ruthless and doesn't care. It sucks, but ZFS expects you to be responsible with it and not do thing that outside of the realm of handling a server smartly.

 

[solarisguy]

@Soundscape, ditch Plex!

If an application can exhaust GBs of swap, most likely nothing would stop it. Even getting 128GB of RAM and 256GB of swap would only slow the process down, but not stop it.

A stopgap measure could be to have 16GB of RAM, more swap than your largest video (so likely 64GB), and restart Plex daily(?)?

 

[danb35]

Or install Plex in a VM once 9.2.1.6 is released. Then you can put a hard limit on its RAM usage.

 

[solarisguy]

@danb35, have you done that with 9.2.1.6-RC ? How much RAM do you have to allocate to such a VirtualBox? (Did I understand you correctly?)

 

[danb35]

I'm running that way on one of the beta releases, allocating 8 GB RAM to the vm. It works fine though a little slow. Need to see if I can allocate another core to the vm.

 

[Soundscape]

@Soundscape, ditch Plex!

If an application can exhaust GBs of swap, most likely nothing would stop it. Even getting 128GB of RAM and 256GB of swap would only slow the process down, but not stop it.

A stopgap measure could be to have 16GB of RAM, more swap than your largest video (so likely 64GB), and restart Plex daily(?)?

Yup looks like Plex WAS indeed the issue. So, I was about to ditch it and uninstall the plugin when there appeared an update to 0.9.9.12.504. I belive the previous running version was .479. Anyway I upgraded and since then everything has been solid. Sad that a plugin can bring down FreeBSD/FreeNas. Anyway, happy again. Was about to ditch the whole DIY NAS idea and ok with a Synology or QNAP.

 

[danb35]

I had disabled the Plex plugin since the VM solution was working (albeit slowly), but decided to try again with .504. I think it's improved, but still not fixed. The good news is that it isn't chewing up nearly as much swap--in fact, it's hardly consuming any swap. The bad news is that the python process still consumes 12-13 GB during a scan, reducing the ARC as low as a few hundred MB, and just generally bringing the system to its knees. I've ordered another 16 GB of RAM; hopefully that will help.

 

[danb35]

Just received and installed an additional 16 GB of RAM, bringing the system up to 32 GB total, and Plex has started its first scan. I guess I shouldn't be too surprised that Plex's python process now consumes ~ 27 GB of RAM (and rising) while the scan is running. ARC is significantly reduced, though not as much as before--it's down to around 3 GB (from 17 GB or so before the scan started), though as I type this it's dropped to 1.3 GB. Again, swap usage is minimal, under 1 GB (of 60). It's keeping the system busy enough that the nut process loses (and re-establishes, and loses again) communication with the UPS, the rrd logging isn't happening consistently, and CIFS and AFP sharing are slowed down considerably, though not completely stopped.

Plex is clearly very resource-hungry, but it seems to be a major design flaw in FreeNAS that a plug-in (really, anything in a jail) can cause that much degradation to the rest of the system.

 

[cyberjock]

Sorry, but I have to disagree. FreeNAS has no design flaw with this. This is where the expectation is that a responsible administrator of a server will properly manage and admin the resources as required for their system. Not everything should be run in a jail.

Problems with Plex are well known right now and several people have reported them. We don't own Plex and we run the same jail framework that FreeBSD runs.