FreeNAS swap_pager_getswapspace: failed (No VMs, Jails)

[junior466]

Hey all,

Today I logged in to FreeNAS and was greeted with a ton of these messages in log:

freenas swap_pager_getswapspace: failed

I started doing some research and it appears that most people that have these issues are running either jails or Virtual machines. I am not running neither and simply have some shares mounted on a few virtual machines on a couple ESXi hosts.

Here's a screenshot of the memory section of the reporting tab:



My system specs:

Build FreeNAS-11.1-U6
Platform Intel(R) Xeon(R) CPU E5-2403 0 @ 1.80GHz
Memory 16GB

Any help would be greatly appreciated.

 

[jgreco]

It'd be helpful if you posted a lot more information about your system and setup. NFS? iSCSI? Pool config? Etc.?

In general, 16GB is a bit shy for block storage.

 

[junior466]

It'd be helpful if you posted a lot more information about your system and setup. NFS? iSCSI? Pool config? Etc.?

In general, 16GB is a bit shy for block storage.

I apologize for being so vague. I am running a mirror pool and the majority of the shares are SMB. I am not serving iSCSI for the ESXi hosts.

 

[jgreco]

Ah, I misread your initial post as "mounted for a few virtual machines."

I'll let someone else ponder SMB issues. It looks like something momentarily maxxed out your system's memory. It would be worthwhile to review the logs for clues as to whether something launched right at the time swap usage started to skyrocket.

 

[junior466]

Ah, I misread your initial post as "mounted for a few virtual machines."

I'll let someone else ponder SMB issues. It looks like something momentarily maxxed out your system's memory. It would be worthwhile to review the logs for clues as to whether something launched right at the time swap usage started to skyrocket.

@jgreco - It happened again and after further investigation, I was greeted with this in the log:

Code:

PID 86297 (smbd) , uid 1002, was killed: out of swap space

Is it safe to assume that I maxed out the smbd service and need to consider adding more memory? I have a few servers that heavily depend on the smbd service.

 

[jgreco]

@jgreco - It happened again and after further investigation, I was greeted with this in the log:

Code:

PID 86297 (smbd) , uid 1002, was killed: out of swap space

Is it safe to assume that I maxed out the smbd service and need to consider adding more memory? I have a few servers that heavily depend on the smbd service.

No, it means the system ran out of memory, swapped a bunch of stuff to swap, and then was still (maybe later, eventually, etc) out of memory. When that happens, FreeBSD will kill semi-arbitrary processes, not necessarily the one actually causing the problem.

Therefore it is not safe to assume that it was smbd causing the problem. It would be nice to know what the problem is, and you might want to consider setting up a little shell script cron to periodically snapshot the process list, so that you can look back. This would be the "best" path forward.

Adding more memory is likely to either reduce the frequency or eliminate the issue, depending on whether you are experiencing some sort of memory leak, insufficient amount of memory to begin with, etc. Without knowing the offender it is hard to provide any answer that is more accurate than just tossing darts at a dartboard.

 

[junior466]

No, it means the system ran out of memory, swapped a bunch of stuff to swap, and then was still (maybe later, eventually, etc) out of memory. When that happens, FreeBSD will kill semi-arbitrary processes, not necessarily the one actually causing the problem.

Therefore it is not safe to assume that it was smbd causing the problem. It would be nice to know what the problem is, and you might want to consider setting up a little shell script cron to periodically snapshot the process list, so that you can look back. This would be the "best" path forward.

Adding more memory is likely to either reduce the frequency or eliminate the issue, depending on whether you are experiencing some sort of memory leak, insufficient amount of memory to begin with, etc. Without knowing the offender it is hard to provide any answer that is more accurate than just tossing darts at a dartboard.

Thanks for your reply. That makes sense. I've noticed my Nextcloud server acting up recently so I suspect it may have something to do with it since it uses one of the shares through smbd.

I will keep an eye on it but unfortunately I don't have the skills necessary to setup a script as you suggested.

Again, thanks for the prompt reply.

 

[hervon]

I experienced multiple "swap_pager_getswapspace(32): failed" errors last night. Newly updated to RC2. Problem started @ 3h. Here is a link to memory usage. Jails : urbackup, MiniDLNA, Transmission. Specs are in my signature.

Edit : this has nothing to do with snapshots. No snapshot task occurred last night.

https://drive.google.com/file/d/1oj4qOZoBLewbmKPvYoW8Jl9frJaMtAaI/view?usp=sharing

 

[Silvan Burch]

hey guys Im looking through the forum since I have the same issue

there's are several threads on this issue, e.g.
https://forums.freenas.org/index.php?threads/swap_pager_getswapspace-7-failed.71505/#post-494870

Did anyone figure out yet why this happens? maybe nextcloud could be the issue? If I think back, Ive only seen this issue since I installed nextcloud as a jail
I used the script from here
https://github.com/danb35/freenas-iocage-nextcloud

 

[hervon]

I don't have nextcloud. Might be a jail/plugin issue nonetheless. Only occurred once so far for me. No idea what is the cause.

 

[junior466]

Just posting an update as this happened again yesterday. I went over the logs over and over and don't see anything that gives any clue as to what may have happened. I see a series of messages

freenas swap_pager_getswapspace(32): failed

followed by

freenas kernel: pid 50269 (smbd), uid 1003, was killed: out of swap space

I also noticed that it also kills the python service

(python3.6), uid 0, was killed: out of swap space