12.0-U4 Memory leak?

[revengineer]

I am trying to chase down what appears like a memory leak and could use some advice to chase this down. My baseline is for comparison is FreeNAS 11.3-U5 where my installation with 3 jails and 3 Windows virtual machines showed ~30 GB memory usage for "services" in the dashboard. On TrueNAS 12.0-U4, the service memory usages starts out the same but after 9 days of operation services memory use has grown to ~55 GB.

I then did the following:
1. Shut down all 3 VMs -> service memory usage decreased to 35 GB
2. Shut down all 3 jails -> service memory usage decreased to 33 GB
2. Restarted all 3 VMs -> service memory usage increased to 69 GB

This smells like a memory leak but do not know how to chase this down. When I use "top" and sort by memory, there is nothing that sticks out. Any advice to get to the bottom of this is appreciated.

Thank you in advance.

 

[sretalla]

SMB would possibly be one source of "leaking" memory... it will cache quite a lot and is counted in the Services category.

 

[Alecmascot]

This thread is running for your reference :
https://www.truenas.com/community/threads/high-services-memory-consumption.93195/#post-645567

 

[revengineer]

SMB would possibly be one source of "leaking" memory... it will cache quite a lot and is counted in the Services category.

This would make sense from the perspective that after killing VMs and jails, the SMB server was the only thing left running. I also restarted the middlewared, and that what not it. I will try restarting the SMB server when memory creeps back up. Are there any commands other than "top" which would shed more insight onto this?

 

[revengineer]

This thread is running for your reference :
https://www.truenas.com/community/threads/high-services-memory-consumption.93195/#post-645567

Yes, this thread applies. Sadly there is no solution or instructions for debugging, so I am not sure how to file a bug report telling the developers that that they have an issue. Given this I have little hope that U5 will be any better in this regards.

 

[Alecmascot]

I am not sure how to file a bug report telling the developers that that they have an issue.

In the GIU: System | Support.....
Fill out a bug report and attach a Debug file.

 

[sretalla]

Are there any commands other than "top" which would shed more insight onto this?

htop

And/or pressing J in top to show the jail ID of each process (jls to get the ID mapping if it isn't already clear).

 

[revengineer]

htop

And/or pressing J in top to show the jail ID of each process (jls to get the ID mapping if it isn't already clear).

Thanks. I did not know about the jail ID but tried top and htop before. The problem I have is that the numbers do not seem to add up. The screen shot below shows the processes sorted by size. If I add up the numbers, I get to about 30 GB, and this is what I have immediate after boot and what I had in 11.3-U5 all the time. In contrast, at this time, actual wired memory usage is about 50 GB, and I cannot explain where the other 20 GB went. I just have no clue where to look for it.