Out of Swap Space error

[Alister]

I got a lot of

Mar 17 14:57:06 FREENAS kernel: pid 1883 (syslog-ng), uid 0, was killed: out of swap space
Mar 17 14:57:06 FREENAS kernel: pid 1883 (syslog-ng), uid 0, was killed: out of swap space
Mar 17 14:57:06 FREENAS kernel: pid 2580 (smartd), uid 0, was killed: out of swap space
Mar 17 14:57:06 FREENAS kernel: pid 2580 (smartd), uid 0, was killed: out of swap space
Mar 17 14:57:06 FREENAS kernel: pid 2762 (winbindd), uid 0, was killed: out of swap space


on my server

I have 32GiB RAM with a 32 GiB Swap

/dev/md99 33144832 0 33144832 0%

running FreeNAS-11.2-U2.1 with one windows 10 VM using 8GB of RAM and 6x4TB in RAIDZ2 as a SMB shared volume

I'm going to monitor the system's swap usage and see what happens but the number of users of the system is small (3 PCs 1user)

 

[Meyers]

Maybe start by seeing what processes are eating all your memory. Run top, type o, type in res, and hit enter.

 

[Alister]

last pid: 39361; load averages: 0.75, 0.67, 0.60 up 0+09:16:48 00:12:58
92 processes: 1 running, 90 sleeping, 1 waiting
CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
Mem: 1947M Active, 4922M Inact, 2199M Laundry, 21G Wired, 841M Free
ARC: 20G Total, 7473M MFU, 12G MRU, 536K Anon, 83M Header, 47M Other
19G Compressed, 19G Uncompressed, 1.03:1 Ratio
Swap: 32G Total, 32G Free

bhyve is using most

I'm going to have to monitor this and see - it's been running for ~20 days with no issue

 

[Meyers]

You didn't list any processes. How much is bhyve using?

 

[Alister]

3624 root 14 20 0 9279M 7827M kqread 0 18.6H 51.99% bhyve
237 root 21 20 0 219M 190M kqread 0 4:37 0.00% python3.6
75873 root 1 20 0 175M 154M select 0 0:11 0.00% smbd
74129 root 1 20 0 172M 152M select 0 0:20 0.01% smbd
74287 root 1 20 0 171M 151M select 2 0:19 0.00% smbd
3904 root 1 20 0 167M 147M select 0 0:00 0.00% smbd
2573 root 1 20 0 161M 145M select 3 0:01 0.00% smbd
2914 root 15 20 0 155M 125M umtxn 1 0:12 0.00% uwsgi-3.6
2620 root 1 20 0 118M 102M select 0 0:00 0.00% smbd
2614 root 1 20 0 118M 102M select 3 0:01 0.00% smbd
2760 root 1 20 0 115M 100M kqread 0 0:04 0.00% uwsgi-3.6
2631 root 1 20 0 78368K 62928K select 0 0:00 0.00% winbindd
3664 root 1 52 0 65232K 59364K ttyin 2 0:01 0.00% python3.6
343 root 2 20 0 64740K 54412K usem 1 0:01 0.00% python3.6
2939 root 11 20 0 85196K 48332K nanslp 0 2:53 0.00% collectd

so just over 9GiB which isn't unexpected


root@FREENAS:~ # swapinfo
Device 1K-blocks Used Avail Capacity
/dev/md99 33144832 0 33144832 0%

I don't know what went wrong as at present the system isn't using any swap. Something oblivious caused a massive spike in Memory (32GiB of swap being used), but that's not on the normal usage

 

[pro lamer]

IIRC when one uses a VM then limiting the ARC size is recommended...

Sent from my phone

 

[toadman]

You can use this script to help detect when you are having an issue and potentially debug from there if needed.

https://www.ixsystems.com/community...ny-used-swap-to-prevent-kernel-crashes.46206/