Freenas memory leak

[apulaski]

I've been using freenas for about 7 years and have had very little trouble with it in that time. Recently after an hour or so of operation my server began to crash. I couldn't load the web interface and upon running a command while ssh'd into it, it would simply freeze. I've come to realize that the reason the server is crashing is because it is running out of RAM. I've had 16 gb of memory in this server since the day I first turned it on and as of now it's been enough to run 2 pools on the freenas server itself as well as several jails (transmission, plex, ubiquiti unifi controller, homebridge, etc. When I load system processes shortly after boot, it shows about 4 gigs being used but over about an hour or so wired memory use slowly increases in size until the server runs out of memory and crashes. Looking at the system processes I see nothing growing out of the ordinary. Certainly nothing using gigabytes of RAM. I'm not really sure what would have changed to make this happen but to be sure it wasn't one of my jails I've shut them all down. It however has made no difference and memory continues to leak somewhere. Not really sure where to look from here so I'm hoping one of you kind souls can help. I've attached two screenshots of my system processes taken about a minute and a half apart. Any help would be much appreciated.

 

[Patrick M. Hausen]

The wired memory is the ZFS ARC and this is how it is supposed to be. Free memory is wasted memory. Your system should not crash from using all available RAM as cache.
I suggest you keep a top display running until it crashes again so we can see if it does start to swap - that would not be good.

HTH,
Patrick

 

[apulaski]

I've been watching the server for about an hour and a half. After the free memory hits about 100m the system has begun using swap. Do I need to reinstall and restore a backup config?

 

[Patrick M. Hausen]

Press "o" followed by "res" and ENTER in top, please. It is not your ARC that is eating all that wired memory ...

 

[apulaski]

Here is a screenshot of the results after about an hour runtime. I was able to get to 2 hours before crash last I tried.

 

[apulaski]

And 30 minutes later heres another screenshot of the same screen

 

[apulaski]

And here is a final screenshot at the time of system crash.

 

[Patrick M. Hausen]

Have you possibly enabled deduplication? If not, how large are your pools? And if we cannot get a clue on this forum, why don't you open a ticket with iXsystems and give them a debug archive?

 

[apulaski]

I have never used disk deduplication and have not enabled it. I have 2 pools with 4 disks in each pool, raidz2. One pool is 4x 3TB disks and the other is 4x 4TB disks. I'm not sure how to give them a debug archive. I'll look into it. Just for kicks I ran a zpool status and everything looks okay in that regard.

 

[Patrick M. Hausen]

System --> Advanced --> Save Debug

https://jira.ixsystems.com/