Freenas 9.2.x and Plex, running out of swap space

[hammong]

OK, so I've got this issue going on with Freenas 9.2.1.5 related to swap space. The server will be humming along just fine and periodically it begins running out of swap. Here's a short snippet from /var/log/messages:

...
May 15 19:00:27 fnas1 smbd[29911]: matchname: host name/name mismatch: 192.168.1.12 != (NULL)
May 15 19:00:27 fnas1 smbd[29911]: [2014/05/15 19:00:27.886593, 0] ../source3/lib/util_sock.c:1199(get_remote_hostname)
May 15 19:00:27 fnas1 smbd[29911]: matchname failed on 192.168.1.12
May 16 08:00:48 fnas1 kernel: swap_pager_getswapspace(16): failed
May 16 08:00:48 fnas1 kernel: swap_pager_getswapspace(16): failed
May 16 08:00:48 fnas1 kernel: swap_pager_getswapspace(8): failed
May 16 08:00:48 fnas1 kernel: swap_pager_getswapspace(16): failed
...

As you can see, it went from 19:00 until 08:00 the next day and no log entries, then it started running out of swap. After a few hundred of those swap_pager messages, it will usually start killing non-essential processes, starting with the Plex media server, and usually the Freenas HTTP GUI.

Anybody else seeing this? There aren't any Plex users up at 8:00am log time to cause any activity, and sometimes I do ANYDVD HD rips from my Windows workstation D: drive directly to a CIFS share on the Freenas box. It's usually done before the swap space messages begin.

Server hardware is ASUS P9X79 WS motherboard, Intel E5-2630V2 Hexa-core Xeon, 32GB ECC DDR3-1600 RAM, 6x WD RED 2TB in RAIDZ2, booting from USB 2.0 memory stick. Everything appears to work GREAT 99% of the time, except when the swap space runs out. System usually (lol) reports between 0MB and 300 MB swap utilized out of 14GB.

Getting frustrated. Tried watching some movies on ROKU at the beach house via Plex this weekend (remotely over Internet) and had to terminal service to my desktop PC back home, putty to the Freenas SSH prompt and do remote shutdown/reboot to get it back up and running. My father in law wasn't impressed. LOL.


Things I've tried --- break and rebuild disk pool as 3x2 disk mirrors thinking it was RAIDZ2 issue, no effect. Tried with and without a SSD ZIL and SSD L2ARC, and then both the ZIL + L2ARC. Performance was zippy, but didn't help the swap issue.

This has been going on a few weeks now and very frustrating.

Greg

 

[Yatti420]

Are you sure its related to plex? This causes hard crash?

Is this new hardware fully tested?

Sent from my SGH-I257M using Tapatalk 2

 

[hammong]

Not 100% sure it's Plex, but that's the ONLY plug-in I'm running that isn't vanilla Freenas. Plex is usually the first thing to "go" when it runs out of Swap, the server kills the Plex pid. Sometimes it recovers back to 0 swap usage, sometimes it doesn't. Usually it doesn't.

Greg

 

[cyberjock]

Well, if you are using swapspace at all you need more RAM. So you need to figure out if:

1. You have a memory leak problem.
2. You just don't have enough RAM for what you are expecting from the server.
3. Combination of the above.

If it's anything about 0 or 144kb(one place always says 144kb for me) then you have a problem.

Did you disable the swapspace in the WebGUI? Hint: You shouldn't have and if you did you can't undo it now.

 

[hammong]

I definitely have swap enabled and FreeNAS configured 2GB swap file per physical device just as it's supposed to when I created the ZFS volumes. Here's what I get when I run swapinfo --- clearly shows six 2GB swaps, and 0% used.

[root@fnas1] ~# swapinfo
Device 1K-blocks Used Avail Capacity
/dev/ada4p1.eli 2097152 0 2097152 0%
/dev/ada0p1.eli 2097152 0 2097152 0%
/dev/ada1p1.eli 2097152 0 2097152 0%
/dev/ada3p1.eli 2097152 0 2097152 0%
/dev/ada2p1.eli 2097152 0 2097152 0%
/dev/ada5p1.eli 2097152 0 2097152 0%
Total 12582912 0 12582912 0%


I've got 32GB of RAM installed in this box, with only 7.1 TB usable storage, NO deduplication, NO L2ARC to eat up indexing space, etc.

As soon as I get a little more disposable cash I'm going to swap the P9X79 WS motherboard for a Supermicro X9-series board so I can get IPMI and a few more SATA ports. I guess I can shelf this discussion until after the rebuild to see if the problem persists. There should be a number of different drivers/etc. in play at that point, so it might shed some light on the situation.

I was just hoping somebody else had the same problem and might point me to an obvious answer. Plex + Freenas out of the box shouldn't be hemorrhaging memory to the point where it needs 32GB + 12GB Swap to run a single 1080 transcode stream and/or copy a few gigs of data via CIFS from a single session.

Greg

 

[cyberjock]

Very true. It's possible you have a program/driver with a memory leak. But you'd have to play around in the CLI to find out where it is.

 

[hammong]

Just at status update... I was reading around the web on FreeBSD in general, and have seen many many recommendations for setting swap size to 2x RAM size. While I don't think I need THAT much swap, obviously I need more than I have - regardless of what is causing the out-of-swap situation. So, I backed up the RAIDZ2 yesterday via rsync, blew it away, and this time created the volume with 8GB swap per disk instead of the default 2GB per disk. Total swap size now is 48 GB with 32GB RAM installed, that's 1.5x RAM.

Restored the media, reloaded my Plex jail data, and set off to do my usual nightly routine, and lo and behold - the system has made it 24 hours now without any issues, and curiously enough swap utilization has shown 0% the entire time. Odd.

I will continue to monitor another week or two and report back. I am wondering if there is any truth to the old 1-2x RAM recommendation for swap. We'll see!

Thanks for the feedback so far --

Greg

 

[hammong]

Mark this as "fixed". It's been almost a week, and no swap or stability issues since making the default swap size change. I've been using Plex extensively over the last week, adding content via my usual methods nightly and moving at least 50-70 GB into the repository nightly. No hiccups!

Summary - Changed the swap per device from 2GB to 8GB on my 6-drive RAIDZ2 configuration to provide 48GB swap on my system with 32GB RAM. 2GB default was clearly not enough, at least not for FreeNAS/Plex/CIFS combination and the work load I'm doing nightly.

Greg

 

[cyberjock]

Ok, so I'm gonna say this, and I know what your answer is going to be.

If you are having to rely on swap to keep the system stable you are at serious risk for losing your pool. Now you're going to tell me that upgrading to 64GB of RAM is too expensive or your hardware won't support it. But, that doesn't change the reality of it.

You stand a very real risk of losing your pools in your configuration. Fix it ASAP or don't say I didn't warn you. ;)

ZFS won't forgive you if you have to 'wait until your tax return' or something else. It is ruthless and doesn't care. It sucks, but ZFS expects you to be responsible with it and not do thing that outside of the realm of handling a server smartly.

 

[solarisguy]

As cyberjock wrote, your increased swap is only masking some problem.

What do you see when you go to Reporting, Memory, Swap Utilization and scroll to check utilization for the past 7 days?

 

[hammong]

There is clearly some event that is using the swap. I'm beginning to think the issue is how L2ARC is allocated on the system as wired memory as a percentage of total RAM. When opening large streams via CIFS (>10-12 GB) is when I seem to be running out of swap. This might be an artifact of how I do my Handbrake sessions:

1. I rip Bluray to a local HDD on my workstation via AnyDVD during the day, usually 3-6 titles at a time. Total raw data runs 150-300 GB.
2. I queue up via Handbrake the conversions at night before I go to bed. This takes 3-8 hours, and reads the rips from my local HDD and writes the output MP4 to FreeNAS via CIFS.

I wonder if holding the file open for such a long time (a blu-ray rip can take 2 hours on my 3930K @ 4.2 GHz) is the culprit. Holding a 12GB+ file lock on a FreeNAS system that has most of it's memory locked up as L2ARC could be pushing swap utilization up.

I might swap RAM and CPU with my iSCSI system and see if the system runs out of physical RAM with 64 GB. My hypothesis is that it will do the same thing as it does with 32 GB, because MOST of the RAM is going to be held in L2ARC wired state, regardless of how much you have total.

There's a reason FreeBSD, System V, Linux all recommend 1-2x system RAM for swap. This could be it. I won't accept that it's a recipe for disaster, but it might be a recipe for changing how I do my rips and staging. I may need to rip from one local HDD to another local HDD, and then copy the completed Mp4 files to FreeNAS all at once instead of stream ripping from local to NAS.

Greg

 

[solarisguy]

32GiB -12GiB that is still 20GiB of RAM for 7.4GiB of storage = plenty. Unless your batch processes overlap.

RAM is ARC. L2ARC would be a pair of very fast SSDs. And with the usage you had described (everything is either written or read only once), L2ARC would offer no benefit whatsoever.

By the way, some serious operating systems recommend that the best use of swap is not to use it, as swap is many times slower than RAM. One day, compare bandwidth of your RAM with sustained transfer speed of your drives... 200 times faster? The rules, you wrote about, were sometimes a necessary evil due to programming methodology that is no longer recommended. Imagine 3s response time, it is noticeable but can live with that. Now 200*3s = 600s = 10 minutes...

Anyway, try to use top or Display System Processes, maybe you would be able find the culprit. Good luck!

 

[cyberjock]

Well:

1. I do keep telling people in the forums not to use L2ARCs with <64GB of RAM and I keep telling them it can backfire drastically if they don't listen(not to mention that the performance often decreases without that quantity of RAM. I'm not sure I've seen it spill into swap(as it shouldn't unless you've tuned things you shouldn't have....). There's also relationship between the ARC and L2ARC you almost certainly are ignoring or you'd know better than to try to use an L2ARC with <64GB of RAM. ;)
2. I've used handbrake many times virtually identical to what you are describing without any problem.
3. I'm going to ignore your comments about swap space being 1-2x system RAM because it doesn't apply to how FreeNAS is engineered and your considerations from FreeBSD are incorrect. If you do any searching around the forums about my previous comments I've made it very clear regularly that if you are using swap space you have basically screwed something up or some software program is running amok(I'll let you figure out which applies to you since I can't). I won't even discuss your comparison to System V or Linux because that's so dumb I don't even know how to argue against something that's apples to oranges to begin with.
4. Hey, if you don't want to admit it's a recipe for disaster keep doing what you are doing. But don't you dare try to tell us you weren't told when things go bad... and they *will* go bad. This isn't my first time at the rodeo, and some people want to come up with their own excuses for why I'm wrong. But I know better and I'd be very irresponsible to not warn you of impending doom from something like relying on swap space.

I might swap RAM and CPU with my iSCSI system and see if the system runs out of physical RAM with 64 GB. My hypothesis is that it will do the same thing as it does with 32 GB, because MOST of the RAM is going to be held in L2ARC wired state, regardless of how much you have total.

Umm, well yeah. That's how L2ARC and ARC work. If you need more information read my "#1" above...

 

[SweetAndLow]

Hammong:
You could have a file descriptor open for days and zfs wouldn't care. It is just a number in a table. Zfs also doesn't keep all the data from the write in memory, it writes it to disk when it wants. Writing it well before it needs swap space.

Your entire post is pretty much wrong. Your problem probably isn't with writing data. There is something else going on with the hardware or os that is broken.

 

[hammong]

My mistake I mentioned L2ARC when I meant ARC. I have no L2ARC on my system, although I have tried the system with one with no appreciable benefit in performance.

3. I'm going to ignore your comments about swap space being 1-2x system RAM because it doesn't apply to how FreeNAS is engineered and your considerations from FreeBSD are incorrect.

You can ignore them if you like. That's your prerogative.

4. Hey, if you don't want to admit it's a recipe for disaster keep doing what you are doing. But don't you dare try to tell us you weren't told when things go bad... and they *will* go bad. This isn't my first time at the rodeo, and some people want to come up with their own excuses for why I'm wrong. But I know better and I'd be very irresponsible to not warn you of impending doom from something like relying on swap space.

I'm a big boy, I've been doing this for 25 years on AIX, HP-UX, Linux, System V Unix. I run into some strange swap issue on a FreeBSD distro and bam, I get two people calling my assessment plain wrong and one of them calling it stupid. Nice.

For what it's worth, I replaced the ASUS P9X79 WS board with a bonafide server grade motherboard today, Supermicro X9SRL-F-O. If nothing else, it puts different NIC and chipset drivers into play, and I'll keep an eye on swap utilization. So far it's 0.0 max, 0.0 min on the reporting, and I'm converting some Blu-Rays now via HB and will see how it looks in the morning.

Greg

 

[cyberjock]

I'm a big boy, I've been doing this for 25 years on AIX, HP-UX, Linux, System V Unix. I run into some strange swap issue on a FreeBSD distro and bam, I get two people calling my assessment plain wrong and one of them calling it stupid. Nice.

Do you know how many people have made the same claim and lost data? Far more than you think.

Do you know what group of people tend to lost the most data? The ones that think they know better because "I've been working in IT for XY years!" and they think that experience makes them a pro at this. They're wrong.

Do you know what group the more experienced people laugh at the most? The guys that throw around the "I've been doing this for XY years and I know better". We laugh and watch. The drama unfolds and we get a laugh as they find out first hand how well that 25 years of experience on a bunch of unrelated OSes helps them.

 

[hammong]

Every time I post a message on the FreeNAS forum, it's like chum to the sharks. If you gain so much enjoyment out of watching people make mistakes and laughing at them - more power to you.

Shining star example for the whole forum to follow. Peace to you brother, I'm not going to bother helping or sharing anything more. You clearly have it all under control, personally, all by yourself.

Greg

 

[cyberjock]

Here.. give this thread a read...

How many of you have experienced catastrophic pool failure and why?

 

[danb35]

I've been noticing the same issue in my security email the last couple of days, and a few minutes ago I happened to be sitting at my computer with the FreeNAS WebGUI up and noticed the "swap_pager_getswapspace(16): failed" messages in the bottom pane.

System specs are an X9SCL-F-O motherboard, Xeon E3-1230V2, 16 GB RAM. Pool size is 10 TB. I'm running FreeNAS 9.2.1.5 with the Plex Media Server 0.9.9.7.429 as my only plugin. I've attached screen shots of the memory reporting page, the CPU reporting page (a bit delayed), and the top output. I noticed that top indicates python is using over 19 GB of memory, which sounded ridiculously high. When I turned off the Plex plugin, the memory usage decreased drastically.

At the time, nobody was watching or listening to any Plex media. The Plex server may have been reindexing the library on its own schedule (I hadn't manually started a rescan). I was ripping a DVD to a CIFS share on the FreeNAS server. As an interim measure, I've created an additional 16 GB swap file. I'll see what that does, but I don't think that's a reasonable long-term solution.

Edit: I've restarted the Plex plugin. Right now, python is in top as using about 12 GB of memory. User 972 is also the user idea listed for Plex Media Server and its other processes. So I think it's accurate to pin this memory usage on Plex.

 

[cyberjock]

Yep, you may want to go to the Plex forum and see if they know about or have fixed this problem. ;)