Swap with 9.10

[Ericloewe]

I am on the freshly virtualized 11.1 and my system is swapping ~450mb practically idling with almost all arc wired and 30mb used. 10GB ram for VM.
I am going to disable uma and increase arc free target, will report how it goes.

That's likely because 11.1-U0 has a nasty memory leak that happens notably with sync writes and TRIM requests.

 

[toadman]

I am on the freshly virtualized 11.1 and my system is swapping ~450mb practically idling with almost all arc wired and 30mb used. 10GB ram for VM.
I am going to disable uma and increase arc free target, will report how it goes.

Yes, I'm gonna play with vfs.zfs.arc_free_target as well, on 11.0-U4. I took out the vfs.zfs.arc_max setting of 12GB. System defaulted to just under 15GB as you would expect.

Starting high on arc_free_target at 262,144, where I would expect to see over 1 GB of free memory. Will back down from there to see where the threshold seems to be. It was 28,xxx before I touched it.

 

[nick31]

So after 5 days with uma disabled and arc_free_target increased to 640mb, my vm system with 8gb ram (down from 10gb) did not use swap at all. Physical memory utilization stayed constant with 734-794mb free.
I am going to enable uma and see what changes.

 

[toadman]

Thx for the update.

My primary system ran for days on a free_target of 1GB under 11.0-U4. I backed that down two days ago to 512MB. Averaging 768MB free. UMA enabled. No swapping yet. I am going to keep backing the free_target down to 256MB in another couple days if all is well at 512MB.

Backup system (backup only 1x per day, serves 1 SMB share for config updates) running on 4GB total with a free_target of 256GB. Averaging 314MB free. Stable, no swapping under 11.1-U1.

 

[nick31]

So here is another update.
With 8GB RAM, uma enabled (default) and arc_free_target set to match vm.v_free_target at 43275, after 4 days my system used no swap at all.

 

[toadman]

Update. For two systems running 11.0-U4 and 11.1-U1 as a NAS only (no jails/VMs) with NFS and SMB I am seeing no swap on both of my systems for over 2 weeks with the following:

Code:

vm.v_free_target: 32768
vfs.zfs.arc_free_target: 32768
vfs.zfs.zio.use_uma: 1

I will see how it does through the next scrub cycle that happens mid month.

 

[MrToddsFriends]

I'm in the boat using matching vm.v_free_target and vfs.zfs.arc_free_target for a short time now, using slightly more deliberate or scary values for these tunables on my system

Code:

vm.v_free_target: 65536				  (262 144 kBytes)
vfs.zfs.arc_free_target: 65536		   (262 144 kBytes)

No swap usage during scrubbing so far. My system suffered from this problem since FreeNAS 9.10. @nick31, @toadman: Any news from your side? Setting vm.v_free_target and vfs.zfs.arc_free_target seems to be preferable to me over setting vfs.zfs.arc_max: No dependence on amount of installed memory and memory to be reserved for other stuff (Jails, VMs, Plugins).

 

[toadman]

I'm at 49152 currently (I saw regular swap at 32768). Saw 1 instance of swap used during scrub on the main server, which was then fixed by the script that runs 1x/hr. No swap at that value on the backup server (which do anything except receive backups). At 65536 I saw no swap, so I will probably go back to that value.

And the value is counting 4 KB memory pages. So the memory total at 65536 is 256 MB, not KB. Just FYI.

 

[toadman]

Also, fyi, my system left to it's own had vfs.zfs.arc_free_target = 28292 before I changed anything.

 

[MrToddsFriends]

So the memory total at 65536 is 256 MB, not KB. Just FYI.

Essentially just what I wrote. Thanks for your answer. On my system the default values were

Code:

vm.v_free_target: 173678			   (694 712 kBytes)
vfs.zfs.arc_free_target: 56562		 (226 248 kBytes)

So not matching (if that's the goal that should be headed to).

Edit: Having used a '.' as thousands separator here might not have been a good idea. Eradicated that in my last two postings.

 

[toadman]

Essentially just what I wrote. Thanks for your answer. On my system the default values were

Code:

vm.v_free_target: 173678			   (694 712 kBytes)
vfs.zfs.arc_free_target: 56562		 (226 248 kBytes)

So not matching (if that's the goal that should be headed to).

Edit: Having used a '.' as thousands separator here might not have been a good idea. Eradicated that in my last two postings.

Gotcha, yea, reading quickly I didn't read it that way. :)

 

[nick31]

Another update. It is 11.1-Release. All the same 8GB RAM, UMA enabled by default, arc_free_target=vm.v_free_target=43275 and with 34 days uptime I have no swap usage. Scrub is set by default (every 35 days, Sunday) and I am not sure how to check if it ran already or not, there is nothing in the log.

 

[Stux]

zpool status will tell you when the last scrub was run.

I scrub more often than that. But its your choice :)

 

[nick31]

zpool status will tell you when the last scrub was run.

I scrub more often than that. But its your choice :)

True I need to run it more often. Anyhow scrub was done on Feb. 11, no swap used.

 

[MrToddsFriends]

I'm in the boat using matching vm.v_free_target and vfs.zfs.arc_free_target for a short time now, using slightly more deliberate or scary values for these tunables on my system

Code:

vm.v_free_target: 65536				  (262 144 kBytes)
vfs.zfs.arc_free_target: 65536		   (262 144 kBytes)

Just giving a short feedback after using these tunables for about 10 weeks and several FreeNAS 11.1 versions now (-U1 ... -U4).

- No swap usage under any circumstances on my system so far. My system suffered from an excess swap usage problem since the very fist release of FreeNAS 9.10.
- IIRC (*) the difference between Wired Memory and ARC Size increased after setting these tunables, but this doesn't really harm. Eventually I will try 32768 instead of 65536 as values as proposed by @toadman to see if this gives more room for ARC without using swap.

(*) Due to an already reported bug (in combination with my system settings) I'm losing my reporting history with each and every 11.1 update but that's off-topic here.

 

[toadman]

Good to hear your success to date. After testing a while I ended up at 65535 for both values.

Now dealing with another issue (continued memory leaks in 11.1-U4).