SOLVED Laundry grows out of control

[spoonman]

Hi,

I am new to freenas and have a question about the 'laundry' growing and using ever greater amounts of RAM.

My Freenas Server Specs;

1x Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz
128 GB RAM
FreeNAS-11.3-U2.1

Running;

1x Ubuntu 18.04 VM (2 vCPU and 16GB memory)
7x Jails (Jackett, Plex, QBtorrent, Sonarr, Radarr, Unifi Controller, Tautulli)

I noticed this morning that in the memory section of the dashboard, Services was using 98.1GB or RAM, when I drilled down it was the 'Laundry' that was using the space (Screenshots attached).

I had a look at Processes and didn't seem to be anything out of the ordinary there, also had a look at swap utilization and it wasn't swapping a whole lot either.

I restarted the Freenas server and the Services utilization went down to 5.4 GB.

What would cause Laundry to grow uncontrollably like this?

 

[sretalla]

Laundry

• Queue for managing dirty inactive pages, which must be cleaned ("laundered") before they can be reused
• Managed by a separate thread, the laundry thread, instead of the page daemon
• Laundry thread launders a small number of pages to balance the inactive and laundry queues
• Frequency of laundering depends on:
  • How many clean pages the page daemon is freeing; more frees contributes to a higher frequency of laundering
  • The size of the laundry queue relative to the inactive queue; if the laundry queue is growing, we will launder more frequently
• Pages are scanned by the laundry thread (starting from the head of the queue):
  • Pages which have been referenced are moved back to the active queue or the tail of the laundry queue
  • Dirty pages are laundered and then moved close to the head of the inactive queue

From: https://wiki.freebsd.org/Memory

It's probably only important when you run out of free memory as memory in the "laundry" state isn't used.

If you see your ZFS Cache memory going up, probably no need to be concerned.

 

[spoonman]

I was under the impression that the laundry was memory that was waiting to be laundered though and that it would decrease over time and free memory would increase?

In my case it was like the Laundry memory wasn't being laundered...

 

[SweetAndLow]

Turn off your VM and see if things change

 

[spoonman]

I tried that, it freed up a little bit of memory but didn't have an affect on the Laundry usage

 

[spoonman]

So I woke up to find that it looks like the laundry is growing again, turned off my VM and all my jails and that only reduced the services utilization by a few MB, but laundry usage is still high.

 

[SweetAndLow]

Here is an example of what mine looks like on a working system. I have no clue what you have going on. Do you have any tunables set?

 

[spoonman]

Funny you mention that. When I first installed Freenas I must have turned on the Auto Tune function and that created a whole lot of tunables. I compared to a friend who has the exact same server hardware running pretty much an identical setup to me (same VM type, settings and same Jails) and his Laundry sits around 21GB.

I have since turned off Auto Tune, deleted all of the tunable items it created and restarted the server. Will see if that makes any difference over time.

Attached is what the tunables looked like before I removed them all.

 

[spoonman]

Better picture of what it was like before I removed them all

 

[SweetAndLow]

Yeah remove every single tunable. And don't add them unless you do it manually and understand the affects.

 

[spoonman]

So it happened again, and it seems to happen at the same time every night

 

[SweetAndLow]

If it's the same time every night it's some application or service you are running.

 

[spoonman]

Well yes that is obvious, just don't know how to find what is causing it.

I am going to shutdown my jails over night to see if it stops happening, then can test each jail one at a time to isolate with Jail is doing it (if it is caused by a Jail)

 

[spoonman]

So I figured out that the Jail that is causing the issue is Plex, I turned off this option in Plex scheduled tasks (Perform Extensive Media Analysis during Maintenance) and that seems to have stopped the behaviour.

So the question is now, why would that be causing me an issue when I am sure other people are using that option.I guess that process is quite intensive cos it reads all my movies but not sure why it would cause laundry to use all the memory.

 

[spoonman]

So it end up this was caused by me mounting NFS shares into Freenas. I currently have my media files on a Synology NAS so I created NFS mounts in the Freenas shell and every time my Plex server would run it's nightly maintenance tasks and do an in-depth analysis of the media files it would cause the laundry to go crazy on Freenas. I switched to using SMB mounts and the issue went away.

I confirmed the behaviour on another freshly built freenas server by mounting an NFS share and copying a load of data from it.

 

[SweetAndLow]

So it end up this was caused by me mounting NFS shares into Freenas. I currently have my media files on a Synology NAS so I created NFS mounts in the Freenas shell and every time my Plex server would run it's nightly maintenance tasks and do an in-depth analysis of the media files it would cause the laundry to go crazy on Freenas. I switched to using SMB mounts and the issue went away.

I confirmed the behaviour on another freshly built freenas server by mounting an NFS share and copying a load of data from it.

You should not be using freenas for mounting remote filesystems. It's an untested and unsupported configuration.

If it works for you go for it, just know it's not expected to work.