FN11 -- swap_pager_getswapspace(#): failed.

[MrToddsFriends]

I read about a bug with SMB that can lead to increasing memory usage under certain conditions, but being far from a systems engineer I don't know if these are related or how I could investigate the issue further.

Are you able to identify a Samba process that is using a large amount of memory using a command like ps -auxc | grep smbd? The third column of the output of this command (%MEM) is the percentage of real memory used by this process. Example not showing a memory leak behavior (that is: normal memory usage):

Code:

root@blunzn:~ # ps -auxc | head -1
USER		 PID  %CPU %MEM	VSZ	RSS TT  STAT STARTED		 TIME COMMAND
root@blunzn:~ # ps -auxc | grep smbd
root		2780   0.0  0.4 174052 140896  -  Is   23Mar18	  1:31.34 smbd
root		2791   0.0  0.3 130604  98156  -  I	23Mar18	  0:19.56 smbd
root		2792   0.0  0.3 131628  99160  -  S	23Mar18	  0:03.21 smbd
root	   16390   0.0  0.5 198812 160296  -  S	06:56		0:40.95 smbd
root	   34657   0.0  0.5 192136 156296  -  S	08:13		0:24.21 smbd

 

[MrToddsFriends]

Well lets see how this goes, so far it has not stabilized, still goes up... can someone explain why the memory usage does that? Should it not load everything and be happy? Why does it ramp up, down and up again?

It's normal that all RAM that isn't used otherwise gradually gets used by ARC (depending on disk read activity). Ramping up and down might be caused by plugins.

 

[pomah]

This does seem a little high or? smbusr is the main user id I have.

 

[WolFree]

Same problem here as pomah.

The main smbd thread is comitting 23G+ at a 32GB system. Any workarounds? The service is crashing way to often because of that.

 

[MrToddsFriends]

Umbrella #28585 is an aggregation of several fixes that are already implemented for 11.1-U5. @pomah: I didn't see a relation of these Samba memory leak(s) with your problem after your first posting in this thread.

 

[WolFree]

Umbrella #28585 is an aggregation of several fixes that are already implemented for 11.1-U5. @pomah: I didn't see a relation of these Samba memory leak(s) with your problem after your first posting in this thread.

Tried the workaround I saw in this thread. Hope it will work. Hope the fixes will be released soon. :)

 

[pomah]

Umbrella #28585 is an aggregation of several fixes that are already implemented for 11.1-U5. @pomah: I didn't see a relation of these Samba memory leak(s) with your problem after your first posting in this thread.

I had no reason to suspect a memory leak, but after your post describing how to check I did it and saw that it looked odd. I will try the fixes as well

Edit1: I restarted the smb service:

Edit2: Now in the morning again:

Well I cant disable nix service since I am running solus on my laptop and libreelec on my media player... So any suggestions on what to do until u5 comes?

Edit3: So far so good, all ram populated and swap not used.

 

[Dariusz1989]

Is this global thread about this? I have the same issue! 0vm, 0 jail. Just hdd space for SAMBA, freenas 11.0u4 worked fine, freenas 11.1u4 and I'm getting those errors all around till crash today!

 

[WolFree]

at all who has the same problem with leaks - please try the following "workaround" from the Umbrellaticket:

Can you try the following configuration changes under Services->SMB:
1) Uncheck Unix Extensions
2) Add auxiliary parameter "wide links" = yes

 

[toadman]

There was also a memory leak found in snmp. May have been 11.1 for that one though. Both expected to be fixed in 11.1-U5.

 

[sam09]

Are you able to identify a Samba process that is using a large amount of memory using a command like ps -auxc | grep smbd? The third column of the output of this command (%MEM) is the percentage of real memory used by this process.

Okay, back home and I have checked my smbd, which is not leaking memory (%MEM is 0.1). I also checked with grep snmp which returned and empty set. I checked with top ordered by size and res, this gives bhyve on top with size 5145M and res changing around 900M, the rest are <300M size. Ordered by swap gives a handful of processes with < 4000K swap, from the top <syslog-ng>, <bash>, <nginx>, <sleep>, <smartd>, <sh>, nginx and syslog-ng. Restarting SMB did not help even with the fix there. I have tried to sort ps by %MEM using -aucxm, -avcx, aucxo %MEM to no avail. It does show bhyve quite high on the list with 5.4% memory usage, but nothing with >20% like in those memory leak examples. I'm pretty much at a loss here.

 

[sam09]

Actually, I found this older thread https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-4 and I think it might be discussing my problem. So far it's a bit out of my league but I'll see if I can fix this with the info in that thread.

 

[John Hixson]

at all who has the same problem with leaks - please try the following "workaround" from the Umbrellaticket:

1) Uncheck Unix Extensions
2) Add auxiliary parameter "wide links" = yes

This fix will work for now. There was a really bad memory leak in Samba and it has been fixed and is scheduled to go out in 11.1-U5.

 

[pomah]

Well this did nothing for me...

 

[MrToddsFriends]

Actually, I found this older thread https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-4 and I think it might be discussing my problem. So far it's a bit out of my league but I'll see if I can fix this with the info in that thread.

The discussion in the "Swap with 9.10" thread embraces at least two problems:

1) The "Inactive Memory problem", see for example screenshots in this posting and follow-ups by Stux on the same page. I never saw this problem on my system and can't say anything about it. (Maybe doesn't cause any swap usage, maybe just causes --- well --- inactive memory, that isn't used for ARC or for anything else that could be called useful.)

2) A problem (most probably caused by imperfect timing in memory management) resulting in marginal swap usage while plenty of memory is free, with the possible side effect of a painfully slow GUI (GUI stuff swapped out). In prior FreeNAS versions not having mirrored swap this could also cause system instability. See for example screenshots here and workaround here. (Typically responsible for at most a few hundred MBytes of swap, never saw a "swap exhaustion" due to this problem.)

AFAICT both of these problems don't have anything in common with the smbd / snmp memory leaks discussed here in this thread (at least since last friday). For the smbd / snmp memory leaks (fixes are said to be included in -U5) see also

https://forums.freenas.org/index.ph...11-1ux-and-system-crash-after-overflow.63172/
https://forums.freenas.org/index.php?threads/samba-using-up-most-of-the-ram.61039/
https://forums.freenas.org/index.ph...u4-catastrophic-software-failure-avoid.63474/

If you think that you have still another problem my recommendation would be to create a new thread to gain more attraction. It gets more difficult than necessary if too many things get mixed up in one and the same thread.

 

[sam09]

Thanks for clearing that up, MrToddsFriends. The combination of VMs and scrubs causing swap usage caught my attention there, but after your explanation I don't believe it is the same as my problem. I think I will leave my system settings as they are for the time being. My swap utilization trends look somewhat similar to what other people have posted in this thread and would (possibly?) be related to some memory leak going on. Here are a couple of screenshots. The system was started only a couple of days before the first graph begins and started building swap immediately.

The sharp drop in swap use is when I restarted the VM with 2Gb less memory. The jumps in swap usage at the end of weeks 13 and 17 coincided with a scrub which made me think they could be connected, but that's probably just normal for the system to use more memory during scrubs. I'll try to troubleshoot this a bit further myself and probably will create a new thread as per your suggestion, since I haven't been able to identify any memory leaks so far.

Edit: I think I will shut down the VM this weekend to see if the problem persists in absence of it. It's a pia, as for some reason the docker process in the VM stops working when the VM is stopped, so if I pause or shut it down I will have to recreate the whole thing from scratch. This swap problem has brought the VM to its knees though and made it virtually unusable (no pun intended), so I'm afraid it's unavoidable at this point anyways.

 

[pomah]

After running with more ram for a few more days this is what I get:

So no more issues with swap, but smbd is still using 1,7 gigs of ram, even after the suggested fixes above. at least for now it does not seem to be an issue.

 

[pomah]

Well after being away for 10 days and comming back I find my server like this:

And SMB service is off...

So sadly for me the changes suggested did not work...