Freenas 11.1 U6 unstable

[NDelamotte]

We install two freenas 11.1 U6 but they are unstable.
After 100 days running, in general, they begin to crash :

> pid 3560 (collectd), uid 0, was killed: out of swap space pid 229
> (python3.6), uid 0, was killed: out of swap space pid 3877 (uwsgi),
> uid 0, was killed: out of swap space pid 3352 (uwsgi), uid 0, was
> killed: out of swap space pid 3357 (python3.6), uid 0, was killed: out
> of swap space pid 4653 (python3.6), uid 0, was killed: out of swap
> space pid 3379 (consul), uid 0, was killed: out of swap space pid 2356
> (ntpd), uid 0, was killed: out of swap space pid 3372 (consul-alerts),
> uid 0, was killed: out of swap space pid 1718 (devd), uid 0, was
> killed: out of swap space pid 4353 (consul), uid 0, was killed: out of
> swap space pid 4352 (consul), uid 0, was killed: out of swap space pid
> 14227 (syslog-ng), uid 0, was killed: out of swap space pid 3085 (sh),
> uid 0, was killed: out of swap space pid 4397 (zfsd), uid 0, was
> killed: out of swap space pid 96225 (nginx), uid 80, was killed: out
> of swap space pid 1890 (mdnsd), uid 65534, was killed: out of swap
> space pid 3289 (mdnsd), uid 65534, was killed: out of swap space pid
> 1887 (mdnsd), uid 65534, was killed: out of swap space pid 1923
> (syslog-ng), uid 0, was killed: out of swap space pid 3246 (nginx),
> uid 0, was killed: out of swap space pid 2092 (ctld), uid 0, was
> killed: out of swap space pid 3286 (dbus-daemon), uid 201, was killed:
> out of swap space pid 4660 (getty), uid 0, was killed: out of swap
> space pid 4659 (getty), uid 0, was killed: out of swap space pid 4658
> (getty), uid 0, was killed: out of swap space pid 56808 (python3.6),
> uid 0, was killed: out of swap space pid 56811 (getty), uid 0, was
> killed: out of swap space pid 56812 (getty), uid 0, was killed: out of
> swap space pid 56813 (getty), uid 0, was killed: out of swap space pid
> 56814 (getty), uid 0, was killed: out of swap space pid 56815 (getty),
> uid 0, was killed: out of swap space pid 56816 (getty), uid 0, was
> killed: out of swap space pid 56817 (getty), uid 0, was killed: out of
> swap space pid 56818 (getty), uid 0, was killed: out of swap space pid
> 56819 (getty), uid 0, was killed: out of swap space pid 56972
> (python3.6), uid 0, was killed: out of swap space

last pid: 10356; load averages: 0.94, 0.47, 0.50 up 105+16:17:01 11:32:58
19 processes: 1 running, 18 sleeping
CPU: 0.2% user, 0.0% nice, 0.7% system, 0.0% interrupt, 99.1% idle
Mem: 9736K Active, 4K Inact, 61G Wired, 770M Free
ARC: 7804M Total, 3064M MFU, 4513M MRU, 24M Anon, 161M Header, 41M Other
5409M Compressed, 14G Uncompressed, 2.65:1 Ratio
Swap: 10G Total, 498M Used, 9741M Free, 4% Inuse, 1256K In, 1056K Out

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
58628 root 20 20 0 184M 11256K kqread 14 27:50 5.32% python3.6
99727 root 1 20 0 7948K 2452K CPU2 2 0:08 0.19% top
99723 root 1 20 0 13216K 2336K select 13 0:01 0.01% sshd
2052 root 1 -52 r0 3520K 3584K nanslp 14 27:19 0.00% watchdogd
57336 root 1 42 0 102M 3296K select 0 2:57 0.00% python3.6
4411 root 1 20 0 6496K 0K nanslp 14 1:05 0.00% <cron>
56821 root 1 52 0 72988K 2012K ttyin 3 0:04 0.00% python3.6
57329 nobody 1 20 0 7144K 1768K select 3 0:01 0.00% mdnsd
57330 nobody 1 20 0 7144K 1768K select 13 0:00 0.00% mdnsd
99725 root 1 20 0 7480K 0K pause 14 0:00 0.00% <csh>
2722 root 1 20 0 12904K 1984K select 11 0:00 0.00% sshd
56809 root 1 20 0 6364K 1820K ttyin 8 0:00 0.00% getty
4654 root 1 52 0 6364K 1820K ttyin 12 0:00 0.00% getty
56810 root 1 20 0 6364K 1820K ttyin 2 0:00 0.00% getty
56820 root 1 20 0 6364K 1820K ttyin 12 0:00 0.00% getty
4656 root 1 52 0 6364K 1820K ttyin 3 0:00 0.00% getty
4655 root 1 52 0 6364K 1820K ttyin 7 0:00 0.00% getty
4657 root 1 52 0 6364K 1820K ttyin 0 0:00 0.00% getty
227 root 1 20 0 6344K 720K piperd 7 0:00 0.00% daemon

After this event, the vm are still runing on proxmox (iscsi) but I know that I have few day before it crashes.
I am going to plan to update the freenas but I don't undesrstant why they crash so easyly.

I am not an expert and freenas and any help would be nice to undesrtand how I can make this plateforme more stable.

The platform is on a dl380 gen10 with 64 GB ram, 4110 silver xeon and a H240 hba card
Nicolas

 

[sretalla]

Looks like your memory is oversubscribed and you don't have enough swap space to expand into... perhaps a memory leak causing that to slowly arrive.

You can give yourself more time to play with by making sure you have more swap space... be prepared for bad performance when it's used.

I suspect it will not be a profitable use of anyone's time to troubleshoot an older version for memory leaks, but you can try to get the developers involved by logging a bug report (top of the screen here).

 

[mjt5282]

there is a finite amount of physical ram and some allocated swap space. The BSD kernel after running for a while starts to run out of free memory due to either : a process or processes on one of your jails leaking memory, or: a system process that has been running a long time on FreeNas has been leaking. Eventually system fills up and starts swapping, evening that can't keep the system running so the kernel starts killing processes, in your case, a python process. I would suggest planning an upgrade to the latest supported non-beta FN.