Explanation of ARC hit graphs?

[Superman]

My understanding is reads can be served from either:

• RAM - ARC hit
• SSD - L2ARC hit
• General disks

I would NOT expect a read to be serviced by BOTH ARC and L2ARC, yet my graphs are showing ARC and L2ARC hit ratios, that when combined, far exceed 100%. How is this possible?

I thought one could be cumulative and include the other, but lines cross sometimes, so this is not possible.

Can anyone explain exactly what these graphs mean and how it's possible for them to combine to a seemingly theoretical maximum of 200%, rather than a theoretical maximum of 100% which I would have expected.

 

[kdragon75]

ZFS is magic.

 

[kdragon75]

On a serious note, the graphs are averages of time slots and therefore the math may not line up. Also FreeNAS is THE WORST with labeling units and scales on there reporting.

 

[DrKK]

Presumably, one doesn't go for an L2ARC hit unless one has already missed or ghosted on the L1ARC...so, I don't presume you would reasonable just add those numbers up sir.

 

[Stux]

As far as I know L1ARC is *evicted* to L2ARC. Thus L2ARC contains data that is NOT present in L1ARC.

 

[Superman]

I added my graph to a post, but I think I made some mistake and the post is not showing up, so here it goes again:

I really think there's bugs in this graphing.

Version:

Code:

cat /etc/version 
FreeNAS-9.10.2 (a476f16)

• How can my L2ARC size possibly be 20TB, when I only have a 500GB SSD?
• After restart a few weeks ago, why would L2ARC drop to 0 - it's an SSD so shouldn't lose contents at reboot.
• ARC hit ratio, after restart seems too high - I would have expected the ARC hit ratio to be very low initially, because being RAM, the ARC would be empty after reboot.

Can anyone explain these strange phenomenon?

 

[kdragon75]

How can my L2ARC size possibly be 20TB, when I only have a 500GB SSD?

If it was misconfigured, compression (but not that much!), or its just borked.

After restart a few weeks ago, why would L2ARC drop to 0 - it's an SSD so shouldn't lose contents at reboot.

The L2ARC is always reset on reboot. This is normal and expected as ZFS uses the RAM to map/lookup blocks in the L2ARC. If you reboot, you loose that map.

ARC hit ratio, after restart seems too high - I would have expected the ARC hit ratio to be very low initially, because being RAM, the ARC would be empty after reboot.

This depends on the read pattern. If initially the system is reading and rereading to same data or the prefetch (metadata) is working to its favor, you could see it jump then settle. Just speculation here.

 

[Superman]

Yeah, this graph seriously needs explanation. Two weeks after restart, graphs now look like attached. Same questions remain: How can ARC + L2ARC hit ratio exceed 100% and how can L2ARC be more than 1TB when SSD is only 512GB.

 

[MrToddsFriends]

Same questions remain: How can ARC + L2ARC hit ratio exceed 100% and how can L2ARC be more than 1TB when SSD is only 512GB.

Maybe you are affected by the same bug that popped up in this thread
https://forums.freenas.org/index.php?threads/l2arc-size-is-1tb-also-poor-performance.58225/
According to the bug report in question this problem was was fixed in 11.0-RC.

Before you upgrade to a 11.1 version of FreeNAS: These have other ARC Hit Ratio reporting problems that might affect you, see for example here:
https://forums.freenas.org/index.ph...ace-after-upgrading-memory.61527/#post-437480