VM Shut down - Out of Swap Space

[JamesWorts]

Hi All,

My FreeNAS 11 shut down and had a few issues at 3am this morning, below is the attached log from the console.
Can anyone possibly help? I have looked at "Reporting" -> "Memory" -> "Swap Utilization" but it lists min and max as 0 all the way across, as if there isn't any swap space. That said, I had heard that its best to not use swap?

This is the first "crash" I've had in some time, the only thing that has changed recently is the latest FreeNAS update that I ran 3 days ago, I've just checked the reporting again and the Swap Utilization shows that the system has stopped using Swap since I performed the update on the 17th December....

The box had 8gb ram, I have just put another 8gb in, just in case this is the issue.

Runs one VM that hosts a website - This is one of things that went down.
Then it has 4 Jails:-
Transmission - Rarely used
Nextcloud - Rarely used
Plex - Rarely used but more than the 2 above
Team Speak3 Server very very rarely used.

6x SMB Shares
1x AFP TimeMachine

Thanks all,

Code:

Dec 23 03:03:46 freenas kernel: pid 21692 (bhyve), uid 0, was killed: out of swap space

Dec 23 03:03:46 freenas kernel: pid 21692 (bhyve), uid 0, was killed: out of swap space

Dec 23 03:03:46 freenas kernel: tap0: link state changed to DOWN

Dec 23 03:03:46 freenas kernel: tap0: link state changed to DOWN

Dec 23 03:03:46 freenas kernel: pid 6787 (mysqld), uid 88, was killed: out of swap space

Dec 23 03:03:46 freenas kernel: pid 6787 (mysqld), uid 88, was killed: out of swap space

Dec 23 03:03:46 freenas upsmon[2116]: Poll UPS [ups] failed - Write error: Broken pipe

Dec 23 03:03:46 freenas upsmon[2116]: Communications with UPS ups lost

Dec 23 03:05:20 freenas upsmon[2116]: Communications with UPS ups established

Dec 23 03:05:20 freenas collectd[59827]: aggregation plugin: Unable to read the current rate of "freenas.local/cpu-0/percent-system".

Dec 23 03:05:20 freenas collectd[59827]: utils_vl_lookup: The user object callback failed with status 2.

Dec 23 03:05:20 freenas daemon[2852]:	 2017/12/23 03:05:20 [WARN] Timed out (30s) running check '/usr/local/etc/consul-checks/freenas_health.sh'

Dec 23 03:05:20 freenas daemon[2852]:	 2017/12/23 03:05:20 [ERR] yamux: keepalive failed: session shutdown

Dec 23 03:05:20 freenas daemon[2852]:	 2017/12/23 03:05:20 [ERR] yamux: keepalive failed: session shutdown

Dec 23 03:05:20 freenas daemon[2852]:	 2017/12/23 03:05:20 [WARN] consul.kvs: Rejecting lock of consul-alerts/leader due to lock-delay until 2017-12-23 03:05:35.585505652 +0000 GMT m=+472834.185489132

Dec 23 03:06:18 freenas daemon[2852]:	 2017/12/23 03:06:18 [ERR] http: Request PUT /v1/kv/consul-alerts/leader?acquire=f6ad37e1-b597-d73a-af57-b59770a762de&dc=dc1&flags=3304740253564472344&token=%22%22, error: invalid session "f6ad37e1-b597-d73a-af57-b59770a762de" from=127.0.0.1:35595

 

[BigDave]

The box had 8gb ram, I have just put another 8gb in, just in case this is the issue.

Today's FreeNAS versions require a minimum of 8GB to run a simple file server!
When you start adding other things on top of that (as you have listed above)
you will have lots of issues when running only 8GB of memory.

 

[toadman]

Agreed, 8GB might not be enough. But the OP hasn't listed hardware, so hard to say what the problem is really.

I suppose one could add more RAM and/or add more swap space (I run swap on the pool in a file). While I have a lightly loaded system (fileserver only) mine still swaps, which seems whacky. I run a script once an hour that flushes it. But it rarely swaps over a few KB. Happened when I ran 8GB and happens when I now run 16GB.

While I haven't specifically tested it, my initial view was 11.1 is using more memory than 11.0, so that could be a factor as well.

 

[tvsjr]

Today's FreeNAS versions require a minimum of 8GB to run a simple file server!
When you start adding other things on top of that (as you have listed above)
you will have lots of issues when running only 8GB of memory.

Maybe it's time to update the docs/recommendations.
File server ONLY - 8GB minimum
Any jails/VMs/whatevers? - 16GB minimum

There seems to be an awful lot of "I'm having trouble running my 17 jails on my 8GB system" lately!

 

[BigDave]

Maybe it's time to update the docs/recommendations.
File server ONLY - 8GB minimum
Any jails/VMs/whatevers? - 16GB minimum

There seems to be an awful lot of "I'm having trouble running my 17 jails on my 8GB system" lately!

That's a good point about the documentation. You could post a bug report and the Doc engineers will take a look at the possibility of making changes.

 

[tvsjr]

That's a good point about the documentation. You could post a bug report and the Doc engineers will take a look at the possibility of making changes.

Well hell, that'll be a first for me.
Done - https://redmine.ixsystems.com/issues/27403

 

[BigDave]

Well hell, that'll be a first for me.
Done - https://redmine.ixsystems.com/issues/27403

Nicely authored, you are now a true FreeNAS Guru ;)

 

[JamesWorts]

Wow! Check you guys out! Seems like my post has resulted in a very worthy bug report! Lol.

Thanks for all your Comments, however.
The 8gb situation, it was working spot on with 8gb running 11.0 but within days of 11.1 it crashed and also the lack of swap is troubling me. It used to have swap space and used it effectively, now it appears to have none. Can we work on that? See why it has stopped using swap space since the update?

Thanks

 

[Bill McCormick]

Similar problem here. FreeNAS 11.1 w/ 16G RAM, 4+TB free. Newly created Ubuntu Server 16.04.3 with a 250 GB zvol, 4 CPU's, and 8GB RAM. Was loaded up and running for a few hours then died.

Jan 16 17:06:33 freenas kernel: pid 52657 (bhyve), uid 0, was killed: out of swap space
Jan 16 17:06:33 freenas kernel: pid 52657 (bhyve), uid 0, was killed: out of swap space
J

 

[toadman]

Doesn't 11.1 have a memory leak? (There is a bug on it.) That might explain why it runs out of memory while 11.0 doesn't.

 

[JamesWorts]

Sooo, I was right about there being a problem linked to 11.1 then. Nice!

 

[fzitelli]

I have similar problem, FreeNAS-11.1-U1, +20gb RAM Swap 4G...

pid 3417 (smbd), uid 0, was killed: out of swap space

any one have a solution?

 

[toadman]

The fact your wired memory is less than half the total seems weird. Seems like the memory leak I mentioned earlier, esp since the process involved is smb. If you are on 11.1-U1 I would file a bug, as I thought it was resolved on -U1.