Daily security run - swap_pager_getswapspace(16): failed

[kryptoman]

Hi, my daily emailed security run output is reporting dozens of get_swap_space failed messages e.g.
> swap_pager_getswapspace(16): failed
In amongst these messages I also see one instance of:
> pid 36951 (python), uid 972, was killed: out of swap space

Following these I see the regular, expected, login failure messages.

Any ideas on the cause/fix for this apparent swap space issue?

I have made a number of changes to my system over the last month - upgraded memory from 8GB to 16 to 24, added a ServeRAID card (LSI crossflashed for JBOD mode), replaced my 4 disk raidz1 with a 6 disk raidz2, upgraded to 9.2.1.2 and now 9.2.1.3. For a period I hadn't re-enabled email alerts after upgrading my disks, so not sure when this issue was triggered.
Memory chart in the memory reporting tab shows a max free swap of 15G and the tallest utilization spike I see is about 10GB.
Any ideas appreciated.

 

[kryptoman]

Ok, just got home from a trip and found GUI has died - just getting an nginx error page at the moment. Have done a little bit of digging and I see the content in the security log is just the messages found in the "dmesg" file and these are output by the system. The uid from the message above looks like it is the id of my jail user that is running the Plex processes.... so maybe it is Plex that is chewing up the swap space and causing this to happen?
swapinfo tells me this currently:

Code:

$ swapinfo
Device          1K-blocks    Used    Avail Capacity
/dev/da0p1.eli    2097152    62176  2034976    3%
/dev/da1p1.eli    2097152    58428  2038724    3%
/dev/da2p1.eli    2097152    61384  2035768    3%
/dev/da3p1.eli    2097152    59872  2037280    3%
/dev/da4p1.eli    2097152    60672  2036480    3%
/dev/da5p1.eli    2097152    60740  2036412    3%
/dev/ada0p1.eli  2097152    60680  2036472    3%
Total            14680064  423952 14256112    3%

The bit of reading I have done seems to indicate I can't increase my swap partition sizes but I can add swap files to increase my swap space. If there happens to be a memory leak somewhere though I don't imagine increasing my swap space will help, but if my configuration is just a heavy user of swap then maybe it will. The possibly unique thing about my system is that I am running two Plex jails - one to keep kids content separate and one where everything is accessible (since Plex doesn't support parental controls yet).

Can anyone shed any light on this issue?

(Incidently, what is the GUI process to restart, so I can avoid a reboot to get the GUI going again?)

 

[mpfusion]

I'm having the same issue here. I'm not using Plex, though. However, I do have a Transmission jail, among others.

The command to restart the GUI in this case is “/usr/local/etc/rc.d/django restart”. I didn't have any time to investigate this issue. Just letting you know that you're not alone with this.

 

[mpfusion]

This issue is solved for me in freenas 9.2.1.4, it was a memory leak (bug #4759).

 

[kryptoman]

Upgraded to 9.2.1.5, but there still seems to be a memory leak somewhere (and I suspect it is related to smbd):

My FreeNAS box can run days without issues, but whenever this memory problem occurs it seems it is when I am doing video encodes from a windows box using files on the NAS. In this case I had 2 windows boxes each running Handbrake sessions that were each reading separate .ISO files from my FreeNAS box and writing the re-encoded video files back to the FreeNAS box. I might of been moving some files around at this point in time also, and the FreeNAS web GUI was running on one of the Windows PC's, but nothing else was going on on the FreeNAS box.
Any thoughts?

 

[hummer5]

I am having the same problem and had to uninstall plex which i was using regularly and now unable to due to swap space failure. Just wondering if any of you found the solution to the problem.

Thanks

 

[DrKK]

Have you looked at "top" to see if you can tell which running process is chewing up all this memory? If you run "top" in the FreeNAS appliance (i.e., not the jails), the list you get will include all processes in jails.

 

[kryptoman]

Have you looked at "top" to see if you can tell which running process is chewing up all this memory? If you run "top" in the FreeNAS appliance (i.e., not the jails), the list you get will include all processes in jails.

Yes I ended up running a simple script that runs a single iteration of top every 20 seconds and displays the top 5 processes based on memory. It turned out to be Plex that was leaking memory. This seems to happen when I have a long running process (i.e Handbrake) creating a file across CIFS in a directory that Plex is scanning for content. It looked like while Handbrake was encoding a file and outputting to one of my Plex media directories, a scheduled Plex scan (configured to run every 4 hrs) of the directory occurs and this is when the memory leak is witnessed.

I haven't found time to look at this in any more detail, but I notice a new version of Plex has recently been released. I'll wait for this to make it to the FreeNAS plugins and test in a bit more detail. When I am sure of the conditions that cause the leak I will raise a bug with Plex.

 

[DrKK]

Well, two things I'd mention.

First of all, this is not a Plex forum, so I think in general we won't be able to help much with whatever problems they might have with a memory leak, etc.

Second of all, we have a post in the forum about building your own Plex, in your own jail, without using the PBI (push-button-install) plugin. Those of us with the self-administered Plex are on the new version literally the day it comes out, since no new PBI has to be fabricated by the iXS guys.

Something to think about. You should be able to find the post. I think Cyberjock made it.

 

[kryptoman]

Fi

Second of all, we have a post in the forum about building your own Plex, in your own jail, without using the PBI (push-button-install) plugin. Those of us with the self-administered Plex are on the new version literally the day it comes out, since no new PBI has to be fabricated by the iXS guys.

Finally got around to doing the manual update and found the latest Plex version seems to be significantly worse - the memory leak occurring far more frequently now and not just when I am doing other things on the system. Looks like it is related to Plex scans of my media directories so maybe it is something to do with some of my content if not everyone is experiencing this.
I have posted on the Plex forums where others have also raised the problem.

 

[balrog76]

@kryptoman any chance you could post link to yor plex forum thread? I have searched for it with out luck as I am experiencing the same issue :)
Merry Christmas