vfs.zfs.arc_max tunable appears to be ignored - user error or bug

Status
Not open for further replies.

noprobs

Explorer
Joined
Aug 12, 2012
Messages
53
I have just increased my memory to 16GB on my FreeNAS 8.3.0 server as ARC was showing as being throttled at 8GB. After I added the memory I noticed that it did not appear to be being used. Looking at arc_summary output I could see
- ARC Max Size (High Water): 8:1 3.35 GiB
- vfs.zfs.arc_max = 3601681712

I have amended the tunable to '10392610498' (autotune recommendation) rebooted server but the above values (3601681712) remain. If I lower the tunable (to 2601681712) then arc_summary output shows that the new value has been picked up. note 16GB RAM is being recognised 'Real Installed: 16.00 GiB'

Is there a setting somewhere which is limiting vfs.zfs.arc_max to 3601681712 irrespective of tunable value?

Thanks,

Jon
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
What "arc_summary" are you referring to?

Edit: I'm doing a little reading, but doesn't that parameter set the limit, not set the actual size? In essence, you had set the ARC to not exceed 10GB, which it didn't. My guess is that based on your system, other settings and RAM usage that 3.6GB is all that is available. What other tunables do you have?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
What "arc_summary" are you referring to?

Presumably /usr/local/www/freenasUI/tools/arc_summary.py

Edit: I'm doing a little reading, but doesn't that parameter set the limit, not set the actual size? In essence, you had set the ARC to not exceed 10GB, which it didn't. My guess is that based on your system, other settings and RAM usage that 3.6GB is all that is available. What other tunables do you have?

Suggest deleting old tunables created by autotune script, rebooting, then re-running autotune. This sounds like prior autotune cruft getting involved in the decision-making process.

On a 16GB system here I get

Code:
ARC Size:                               84.26%  8.26    GiB
        Target Size: (Adaptive)         84.86%  8.31    GiB
        Min Size (Hard Limit):          12.50%  1.22    GiB
        Max Size (High Water):          8:1     9.80    GiB

        vfs.zfs.arc_min                         1315062385
        vfs.zfs.arc_max                         10520499087


which is what happens when you don't use autotune.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Arc_summary is pretty cool. I'm actually having random performance issues with my home FreeNAS server and I've been wondering how I could get alot of the info included in this. My issue is that if I open a random folder from Windows Explorer(via CIFS share) with 10-50 files it might take 1 second or it might take 10 seconds. If I look at the zpool iostat 1 during one of those 10 second waiting times it seems to read small 300k pieces each second until my directory listing appears. What is even stranger is that if I go back a directory and then double click on the same folder again i might have to wait 10 seconds all over again when I'd expect it to be cached. I thought it was a cache issue since the zpool is 18x2TB drives on a RAIDZ3 but has only 12GB of RAM. Now I can actually play around a little :P
 

noprobs

Explorer
Joined
Aug 12, 2012
Messages
53
Thanks for quick response. Yes '/usr/local/www/freenasUI/tools/arc_summary.py'

Noobsauce - yes I agree that sets the limit however the output of arc_summary is saying the limit is set to 3.35GB not the 10GB I have asked it to be set to in the GUi (and as show in /boot/loader.conf.local). In my case actual size= forced limit.

jgreco - I did try that from the GUI but it had no effect. Am I correct in thinking I can remove them from /boot/loader.conf.local? (Do you know if I am better to simply delete the file or just comment out the contents)?

Forgot to include:

ARC Size: 93.75% 3.14 GiB
Target Size: (Adaptive) 93.75% 3.14 GiB
Min Size (Hard Limit): 12.50% 429.35 MiB
Max Size (High Water): 8:1 3.35 GiB



Jon
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
noprobs - you can access the entries from the FreeNAS GUI by going to System/Tunables. Also make sure your checkbox for autotune is unchecked from System/Settings -> Advanced.
 

noprobs

Explorer
Joined
Aug 12, 2012
Messages
53
Thanks - I was editing the values via the GUI but as they don't appear to be taking effect I am reverting to CLI (the good old fall back). I will edit out values in loader.conf.local and see how it goes. Update in 30mins
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Thanks - I was editing the values via the GUI but as they don't appear to be taking effect I am reverting to CLI (the good old fall back). I will edit out values in loader.conf.local and see how it goes. Update in 30mins

I would delete the values from the GUI first. Nasty things may happen when the FreeNAS Database doesn't match the actual system settings. There's lots of stuff that you just don't do from the CLI.. so be careful. If things get out of sync its not always easy to fix.
 

noprobs

Explorer
Joined
Aug 12, 2012
Messages
53
Good News!

ARC Size: 0.05% 7.90 MiB
Target Size: (Adaptive) 100.00% 14.21 GiB
Min Size (Hard Limit): 12.50% 1.78 GiB
Max Size (High Water): 8:1 14.21 GiB

This time I disabled autotune BEFORE I deleted the tunable values (all from GUI) then I rebooted, enabled autotune, manually ran autotune and then rebooted again


Many thanks for everyone's advice. Noobsauce - thanks for warning; in this case /boot appears to be read-only.

I will re-run this scenario on my test system in the next couple of days to see if this is a one-off or a repeatable bug to be reported.

Jon
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
As much as I understand the desire for an "autotune" script, ...sigh... I see the opportunity for confusion like this as a result.
 

noprobs

Explorer
Joined
Aug 12, 2012
Messages
53
I see the opportunity for confusion like this as a result.

jgreco I am confused by your confusion, though I am a FreeNAS noob :D Am I missing something (before I do more testing to see if this is a repeatable situation). Personally I don't have the experience of FreeBSD/ZFS to know what is a good tune parameter (so I like the recommendation from the script) but I am au fait with *nix CLI/scripts etc - however my issue was that values above (but not below) 3.35GB for arc_max input into the GUI (and loaded into loader.conf.local) were being ignored on boot
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
noprobs - The entire USB stick is read-only. You have to mount it as writable to do anything manually. This is by design.

jgreco - What do you mean?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The "autotune" script is designed to take some values (at least some of which are already dynamically computed by FreeBSD) and adjust them to be more "compatible" with FreeNAS's purposes.

The problem is that even in the best case, you basically have to go through a reboot sequence to see the complete results of the tuning. Further, if you need to change them, you may need to go through another reboot sequence to see the result of changes. Worse, if you do something that changes the assumptions underpinning those tuned variables (like adding memory!), then you need to clear the old cruft out, reboot, run autotune, reboot again, and then verify that everything worked as intended.

This makes perfect sense if you understand what's going on under the sheets, but I maintain that it is ugly, counterintuitive to your average GUI user, and anything but "auto" tuning. Especially that last case, which is what this thread is all about.

A better fix for dynamically sized variables such as vm.kmem_size could be to identify where in FreeBSD those are set and alter the default FreeBSD calculation to what is appropriate for FreeNAS. If that happened, then when you stuck more RAM in your FreeNAS box, it'd just magically adjust - think of it as automatically tuning itself.

I just find it kind of ironic that static tuning is being applied to a BSD UNIX system that's spent decades developing dynamic sizing.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
WOW. That's disappointing. I was convinced that auto-tune would automatically adjust the values if you added more RAM. Ideally I'd expect that if you had autotune enabled with 12GB of RAM and then upgraded to 24GB it would autoadjust accordingly. If you needed a reboot it would reboot the system again(or at least tell you that you need a reboot and ask if you want to reboot now).

I was just looking in another thread at someone with horrible network performance and very odd tunables. Not sure if you've seen it....
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Ok, well I could be wrong there, but my recollection of autotune was that it kind of ran-once and was then mostly useless if the underlying platform changed, because it primarily adds new values to the loadables/tunables sections. A current reading of the manual does not make the actual current behaviour clear to me, but I feel comfortable saying that the manual documentation does suggest to me that "autotune" is not tightly integrated into the FreeNAS framework and that my criticism above is not without merit.

If you consider the suggested alternative, which is modifying the fundamental calculations used for automatic FreeBSD kernel sizing and configuration within the base system, well, that works "as intended" every time, unless someone specifies an override. To me, that's a win.

As for the other thread, I vaguely remember seeing something like that and that I thought you had it under control. ;-)
 

noprobs

Explorer
Joined
Aug 12, 2012
Messages
53
jgreco - I agree that the current implementation of "auto tune" if far from ideal. From my analysis it sets some variables once and then does not do so again as long as there are manual values for key tunables. Ie it is not really automatic and takes no account of any platform changes. A more dynamic solution would be preferable. The multiple reboots is not ideal but acceptable if the user is advised of the need for a reboot 'at a convenient time'.

NB This is actually all separate from my initial thread which was about manually set (but based on autotune algorithm) tunable values being ignored if they were a higher number then previously set buy autotune. As you said it looks like DB got itself in a mess and needed old values removing first.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yeah, the DB can really backfire for those of us with good command line skills. Pretty much if you can do something from the GUI you should.
 
Status
Not open for further replies.
Top