Swap with 9.10

[MrToddsFriends]

Its possible that U2 fixed the issue. Its also possible (although perhaps more unlikely) that the systems haven't been worked hard enough to trigger it.

No change for me with with U2, swap gets still used during scrubbing as with 9.10.1 and before. Scrubs are scheduled twice per month on my system, so only one instance since upgrading to U2 so far, running your script three times per day.

 

[toadman]

Same here. Swap showed up on U2. (I just turned it off entirely vs run the script. No issues in months on a lightly loaded server with it off.)

 

[Stux]

It's in an enterprise environment, so it gets one of our VMware cluster backups via Veeam as well as a few MSSQL db backups and their associated transaction logs. Which, is then snapshotted and replicated to a "sister" NAS at our DR site. Luckily we only retain 90 days of active data and archive the rest otherwise I'd have to grow the pools significantly. :)

Unless the activity consists of random/varied block sizes (like rsyncing thousands of small files) it might not trigger the issue. I suspect veeam efficiently streams fixed sized blocks to the server, which means the UMA system works as intended recycling freed memory to be reused without t being released/reallocated

 

[Mlovelace]

I suspect veeam efficiently streams fixed sized blocks to the server, which means the UMA system works as intended recycling freed memory to be reused without t being released/reallocated

Yes, Veeam depending on the optimization chosen, will use block sizes of 4096Kb, 1024Kb, 512Kb or 256Kb. Once the optimization is selected it doesn't vary the block size of the data it sends to the NAS. Also, the data is deduped and compressed by Veeam so the lz4 pool compression won't result in varying block sizes of the backup data, as it's already compressed. I'm curious as to why we were seeing the behavior previous to the U2 patch. I suspect it is from the MSSQL backups and transaction logs as that is compressible data and would result in varying block sizes on the pool.

 

[Stux]

I just received an email from my primary system (running U2) reporting some swap usage... checking the reports

You can see that Inactive has grown steadily to 11.5GB over 4 weeks. Which reduces ARC and creates memory pressure.

Code:

#	arc_summary.py
System Memory:

	0.22%	70.92	MiB Active,	38.11%	11.81	GiB Inact
	60.45%	18.74	GiB Wired,	0.77%	245.41	MiB Cache
	0.43%	137.66	MiB Free,	0.01%	3.56	MiB Gap

	Real Installed:				32.00	GiB
	Real Available:			99.57%	31.86	GiB
	Real Managed:			97.28%	31.00	GiB

	Logical Total:				32.00	GiB
	Logical Used:			61.92%	19.81	GiB
	Logical Free:			38.08%	12.19	GiB

Kernel Memory:					366.51	MiB
	Data:				92.66%	339.62	MiB
	Text:				7.34%	26.89	MiB

Kernel Memory Map:				31.00	GiB
	Size:				53.45%	16.57	GiB
	Free:				46.55%	14.43	GiB

 

[Stux]

So I disabled ZFS UMA. Guess I'll have to wait another 4 weeks to see if it makes a difference ;)

I had to add it as a Loader tunable in order for it to take.

 

[Stux]

So, an update.

I've been getting a few instances of swapping over the last 24 hours. Inactive has been stable at 2.7GB for the last few days. I did notice that I actually saw sustained traffic at 950mbps for a while.. never seen that before... normally it was sustained at 800mbps. Might be a coincedence.

So, I'm trying this again:

Code:

#	sysctl vfs.zfs.arc_free_target vm.v_free_target
vfs.zfs.arc_free_target: 56375
vm.v_free_target: 173129
#	sysctl vfs.zfs.arc_free_target=173129
vfs.zfs.arc_free_target: 56375 -> 173129

 

[toadman]

Update - A scheduled scrub started today and the system was showing some minimal swapping again. (I was notified using the script above - thanks for that script!). I have the following sysctl settings on a lightly used 8GB system:

Code:

vfs.zfs.zio.use_uma: 0
vfs.zfs.arc_free_target: 43276
vm.v_free_target: 43276

 

[Stux]

So, I have some updates, related to long-run testing, and 11-RC3.

Firstly, with the sysctl loaders set, I saw my wired/inactive usage balance out at 50/50 of ram. Ie roughly 16GB of each.

This was over a few months of uptime.

Then I needed some functionality from 11-RC so I upgraded, and then to 11-RC2 and then to 11-RC3.

So as part of that upgrade I disabled the use_uma tunable I had, to see if the behaviour had changed.

Well, results are coming in...

So far, inactive has now grown to 18.3GB and ARC is down to 8.5GB.

The behaviour I was seeing previously before setting the tunable on the 9.10 series was that ARC usage would tend towards zero and Inactive tended toward 100%, given time.

I'm now seeing the same trend in 11... given time... I'll see where it ends up by the time RC4 comes out, but I suspect this issue is still live.

I also have 128GB on order.

This is the memory usage, zoomed out twice...

This is the ARC usage over the same period.

Once I confirm the same behaviour with 11-RC4/Release, I'll try enabling the disable UMA tunable again, and see if it at least prevents arc from droppping below 50%.

(weekly scrubs, which is why the impact happens weekly)

And before people tell me I don't have enough ram, realise that 18.6GB of the 32GB is "inactive"! This is memory which the kernel is setting aside for 'maybe' reallocating if the same allocation size ever gets requested.

 

[toadman]

I can report I have not seen the arc reduction issue in a long time. Since Sep 2016 I think. I can confirm I'm not running the uma tunable anymore and am currently on 9.10.2-U4. I do still see occasional swapping as reported by the script.

Watching the thread eagerly to see how Stux's experimentation goes.

 

[Stux]

So, I've been seeing some strange behaviour.

Firstly, the Wired/Inactive seemed to stabilize at 50/50 at about 15G each. Which is good I suppose, but today, I needed to pull a couple of 250GB files off the server

Wired usage plunged from 15GB to 6GB and Inactive surged up to 24GB. With ARC itself plunging to 3.8GB.

The transfers were made by AFP. The system is very lightly loaded at the moment, and has 32GB of RAM.

Swap jumped up to 36MB

I think its insane that this system is swapping when it has 24GB of inactive ram!

Now that the transfers have finished ARC is climbing up again, slowly. Its at 4.6GB.

 

[toadman]

Very weird. Though I have no idea what the algorithm is for managing inactive memory. I did see this while poking around. Might be worth a try?
https://forums.freebsd.org/threads/59513/

Code:

sysctl vm.defer_swapspace_pageouts=1

 

[Stux]

That symptom certainly sounds exactly like mine :)

Will play with it later

 

[danb35]

I probably haven't been paying enough attention here, but I'm also seeing odd behavior on my 9.10.2-U4 system. With 128 GB of RAM, I have a current ARC size of 75 GB, hit ratio of 12%, and 37 GB of RAM inactive. That seems more than a little inefficient.

 

[danb35]

In about two weeks, inactive has increased to 55 GB, and ARC has decreased to 57 GB, but hit ratio is up slightly to 13%. No swap is being used, though.

 

[Stux]

That's pretty much what I see. I end up with inactive/ARC approaching 50/50%

Tremendously inefficient.

I know you're not swapping but have you tried this?

https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-4#post-384491

I've been rebooting my system for misc hardware/software updates too often to diagnose further.

 

[danb35]

The sysctl vm.defer_swapspace_pageouts=1 command? Just ran that; I'll see if it changes anything.

 

[danb35]

The sysctl vm.defer_swapspace_pageouts=1 command?

Should I expect that this would free up some of the inactive memory on its own, or is there something else I need to do? Because there doesn't seem to be a change.

 

[toadman]

Having gone to 11.0 I can say I was still seeing some swap activity. Used the script to flush it. I was able to effectively stop it by limiting vfs.zfs.arc_max to 12GB on a 16GB system. Not sure that's the same issue you guys were having though. I'm not seeing the growth of inactive anymore.

 

[nick31]

I am on the freshly virtualized 11.1 and my system is swapping ~450mb practically idling with almost all arc wired and 30mb used. 10GB ram for VM.
I am going to disable uma and increase arc free target, will report how it goes.