Just to reiterate what M0nkey has said above, and provide a few more details. We did, as M0nkey said, speak to Allan last night. He literally
wrote the book on ZFS, and we all walked through it and discussed to make sure everything was right.
Basically, the way things used to be done, pre 9.10.2, was that a certain cron job accessed the very same file(s) over and over and over again, all day long during the continuous maintenance of the FreeNAS. Kind of an oversight. I had actually noticed this before, a couple years ago, but I was like, "meh", and never reported it. But now they are doing it apparently in a more sensible way (the bug report itself is "private", so I don't know what the details are on the fix). As a result, these files, which on account of their constant presence in the MRU/MFU, were always "hits", and so, all things considered, well over 90 percent of the "files" requested were magically in the cache, because well over 90 percent of the "files" requested was this thing from the script. Lol? So that means we'd all get stupid numbers. 95% hits, 92% hits, 97% hits.
In hindsight, these numbers have always been ridiculous, and it's kind of odd we never questioned it in the past. Numbers like 40 or 50% are actually just fine. Allan, the ZFS master, on his 128GB super servers from Hell, we did the math on his hit ratio, and he was getting like 70-75%.
The thing to remember: We have been accustomed to seeing 90+%, and it's been an illusion. Hit ratios of more like 1 in 3 or 1 in 2, are fine, SOP, for a personal ZFS system. We have to "un-accustom" ourselves to seeing 90+%
If you want to test your ARC and make sure it's working (it is working---trust me), choose some good sized file, say, (I'm assuming everyone is smart enough to find an appropriate file on their NAS, and add the appropriate directory path) Dune.mp4. Then do this:
Code:
sysctl -a | grep arcstats > /tmp/before.txt
This saves your ARC stats before you attempt to "season" the ARC with Dune.
Code:
cat Dune.mp4 > /dev/null
This should load Dune into the ARC without doing anything else.
Code:
sysctl -a | grep arcstats > /tmp/after.txt
This saves your ARC stats after Dune is loaded into ARC.
Code:
cat Dune.mp4 > /dev/null
We do the same procedure, except this time, ZFS ought to know Dune is in the ARC, and not fetch it from disk.
Code:
sysctl -a | grep arcstats > /tmp/after2.txt
This saves your ARC stats after Dune is read out of the ARC.
Now, when you look at your three files, /tmp/before.txt, /tmp/after.txt, and /tmp/after2.txt, you should be able to verify your arc is working. The difference between after.txt and before.txt should be evident. You should see, kstat.zfs.misc.arcstats.prefetch_data_misses going up significantly, and kstat.zfs.misc.arcstats.prefetch_data_hits going up only mildly, if at all. When you look at after2.txt versus after.txt, though, assuming your file was loaded from ARC, you will see kstat.zfs.misc.arcstats.misses *not* go up, you should see kstat.zfs.misc.arcstats.prefetch_data_misses *not* go up, etc.
Bottom line (not) up front: The "real" ARC Hit Ratio has always really been 40% or 50% or whatever for most of us. A slightly sub-optimal way of doing ongoing maintenance as the FreeNAS run resulted in artificially inflated numbers. Everyone (including me) can relax.