minidlnad was killed: out of swap space

[Zofinur]

So i bought myself a

• HP ProLiant MicroServer G7 N54L 4 GB-U SATA-Server

and installed

• Freenas 9.2
• on a 16GB USB2-stick
• added 4 * 1TB western digital SATA drives

then i

• did set up a (non-optimal) ZFS
• created userspecific directory
• a public directory
• eneabled (local) FTP and SSH access
• switched to a 8GB USB3-stick and Freenas 9.2.1.3 x64 by ex- and re-importing the settings

All was fine until now.
After enabling minidlna-plugin i recognized the media-directories was not available to my TV now amplifier. Just "some" of the Media was scanned.
Did several restarts and reboots but it never scanned the complete media.

Since i assumed this might be some memory-issue i upgraded to 16GB ECC memory (KV§1333D3E9SK2/16G) but had exactly the same issue.

So (finally) i started to take a look at the console output and saw
minidlnad was killed: out of swap space
(PFA screenshots of the swap utilization)

My first guess was "let´s increase the swap size" ... though this seems to be exactly not the thing to do after cunsultig dr. google. Everbody suggests to increase the RAM ... but 16GB should be sufficient, right?

My guess: Since i did install the system with 4GB and upgraded to 16GB later the system is not (really) aware of the increased RAM.

And ideas how to fix this?

 

[jgreco]

See that nice step function on your swap space graph? Something bad going on.

There is no guarantee that 16GB of RAM is going to be "sufficient" for a RAM-hungry NAS that has an 8GB minimum requirement PLUS whatever other stuff you load on top of it. I would say that 8GB of RAM is likely to be fine for the NAS itself and 16GB would be very comfy.

So what I think you need to do is to get in there and monitor what's going on. Something is probably growing, and it is probably minidlnad, and it probably gets to the point where the system has to kill processes. I don't actually recall the current algorithm FreeBSD uses for that, but it is likely that it is killing the requester or the piggiest, either of which would point to minidlnad being the likely culprit.

Code:

last pid: 43689;  load averages:  1.40,  1.19,  1.26   up 27+18:41:44  06:31:39
54 processes:  1 running, 53 sleeping
CPU:  0.0% user,  0.0% nice, 22.4% system,  3.1% interrupt, 74.4% idle
Mem: 4023M Active, 373M Inact, 19G Wired, 466M Buf, 8041M Free
ARC: 18G Total, 4879M MFU, 12G MRU, 18K Anon, 297M Header, 96M Other
Swap: 24G Total, 24G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3637 root          4  20    0  9904K  1564K rpcsvc  1 829:46  0.00% nfsd
 8433 root         12  20    0   146M 22740K uwait   2  47:51  0.00% collectd
 3671 root          1  20    0 83952K 35340K select  2  29:29  0.00% vmtoolsd
 4776 root          6  20    0  3867M  3705M usem    3  28:35  0.00% python2.7
 4301 root          1  20    0 22220K  3920K select  0   1:31  0.00% ntpd
 9017 root          1  20    0 34852K  4984K kqread  1   1:04  0.00% netatalk
 4592 root          1  20    0   275M 17668K select  3   0:57  0.00% smbd
 4589 root          1  20    0   208M 12568K select  0   0:50  0.00% nmbd
 3557 root          2  20    0   332M 23708K kqread  2   0:41  0.00% smbd
 4878 nobody        1  20    0  9908K  2296K select  1   0:10  0.00% mdnsd
 3532 root          1  20    0 12044K  1768K select  1   0:09  0.00% syslogd
93847 root          1  20    0 28212K  4696K nanslp  1   0:09  0.00% smartd
 4652 root          1  32   10 18592K  3240K wait    2   0:09  0.00% sh

So get on into your console, PuTTY and ssh is fantastic for this because you can cut and paste your output. Run "top" and you'll see a display something like above. The system shown is very healthy and not too busy. Look at the "Mem" line and you can see it has 32GB of RAM with lots of elbow room. The "Wired" is basically kernel stuff which is mostly ZFS. Your "Active" is probably what is growing; you need to use the process list to identify what is growing. You can tell when the system's in trouble by monitoring swap.

So I would bet a small sum on minidlnad being bad here. I don't know anything about it though so that's where my suggestions stop, sorry.

 

[cyberjock]

I'll just pipe up as comments about swap come up from time to time here and in IRC. In short, your system shouldn't ever be using swap. If it is, you need to upgrade your RAM. The way FreeNAS is designed swap is supposed to be nothing more than a spare tire. A healthy FreeNAS system should never use any swap space, ever. If it is then you need to up the system RAM. The only reason that swapspace is recommended so much even though its not supposed to be used is because you have 2 possible outcomes when you run out of RAM; programs are terminated or the system crashes. Neither of which are particularly good. Swap space can act like a spare tire that you can lean on while more RAM is ordered and en-route. Unfortunately though, the N54L maxes out at 16GB of RAM. So I'd find that process that is out of control and stop using it(see jgreco's explanation to figure this out) or go look at buying a system with more RAM. I'd bet money minidlna is to blame, but definitely verify that with #top. But using swap space religiously can be a dangerous game. It's not something you should be doing with FreeNAS. ;)

 

[jgreco]

I see no reason to think adding RAM would help. As I noted, there's a step function shown on the swap graph which suggests something is allocating, failing to free, then waiting a while before repeating.

Since the default data size is 32GB, the argument could be made that with a bit more than 32GB RAM plus swap, the system would be able to sustain the process, but there is no reason to expect that the process would gracefully handle hitting the limit if it isn't cleaning up after itself. So the process would just be crashing a different way.

 

[cyberjock]

I see no reason to think adding RAM would help. As I noted, there's a step function shown on the swap graph which suggests something is allocating, failing to free, then waiting a while before repeating.

Since the default data size is 32GB, the argument could be made that with a bit more than 32GB RAM plus swap, the system would be able to sustain the process, but there is no reason to expect that the process would gracefully handle hitting the limit if it isn't cleaning up after itself. So the process would just be crashing a different way.

That's very possible. I agree with your assessment, but I don't know enough about minidlna to have a good informed decision on what is going on inside the process(and you've said the same thing). The only thing I know about minidlna is that I hate it. Plex in a jail supports minidlna, so you might want to give that a try instead. I'd like to think that Plex's minidlna process is a little better written than the minidlnad process you are using. ;)

 

[jgreco]

Just to be picky of nits, Plex supports DLNA, not minidlna. minidlna is a very specific implementation of DLNA that was designed to be run on the ReadyNAS but is basically an open source project. No idea what Plex is doing though I expect the Plex guys are way clever enough to write their own DLNA implementation if they had a keg and an afternoon.

 

[cyberjock]

You're right. My bad. ;)

My phone has a built-in DLNA app, I've never used it because my stuff isn't compatible with my phone. :0

 

[joeschmuck]

MiniDLNA is very conservative with resources so I doubt it's MiniDLNA causing the problem. But I haven't been using the "Official" plugin for a while. Read this link about the MiniDLNA file I'm hosting until this comes out officially. Delete your current MiniDLNA installation and reinstall per my instructions. If you have any question then post them and I'll help you out. If this version doesn't fix the problem then maybe it's not MiniDLNA. And as suggested above, you could use Plex as it does have a DLNA server built in.

 

[ser_rhaegar]

In my short experience with MiniDLNA, it filled up my RAM and spawned hundreds of processes. Would work great for a bit until my RAM capped, then it tanked the system. This was using the version from the Debian repos on a Debian system (not FreeNAS). I ended up switching to Plex and never looked back.

I did not invest much time into figuring out the issue though.

 

[cyberjock]

Interesting! Thanks for the info ser_rhaegar! I'll keep that in mind. I've always hated DLNA because it's just hokey enough to not be the end-all that people think it is. If it had included codec support and transcoding it would have rocked the house!

 

[Zofinur]

Thanks all for your answers.
Problem has been solved by removing minidlna and installing Plex: No more swap-going-crazy since then.
This definietly occured while minidlna was indexing my media

Now i am digging into configuring Plex. In general it works but it might be some finetuning especially when it comes to subtitles ... but that´s a different story.

Sidenote: I´d guess that even the 4GB would be sufficient now, but i´ll just keep the 16 installed.

 

[jgreco]

Your guess would be wrong, sorry. The base system is stable in 8GB but really does need most of that.