ZFS Cache

[jcoffey0093]

I having problem with ZFS cache taking 11.9 of 14 MB of memory. How can I point ZFS cache to use ssd instead.
My Freenas box configuration

1. Dell Pentium(R) Dual-Core CPU E5700 @ 3.00GHz (2 threads)

2. 14.9 MB installed

3. FreeNAS-11.3-U3.2

4. 5 drives installed
4 WD RED 1.82 TB, 1 SSD 119 MB (cache)

Thanks

 

[Ericloewe]

You don't. If anything you may have a problem where very high memory pressure can cause the ARC to not reduce in size quickly enough, causing things to be swapped out to disk and creating suckiness, in which case you reduce the maximum size of the ARC.

With 16 GB of RAM, it is very unlikely that you will benefit from an L2ARC, as it consumes ARC to store its pointers. You would, most likely, see performance drop.

 

[Samuel Tai]

With 16 GB of RAM, it is very unlikely that you will benefit from an L2ARC, as it consumes ARC to store its pointers. You would, most likely, see performance drop.

Actually, I do see a speedup on my FreeNAS server with 16 GiB RAM and a 256 GiB L2ARC PCI-E SSD, especially for folders with thousands of images. Creating thumbnails with L2ARC available is pretty much real time, but there's noticeable delays with L2ARC absent.

I having problem with ZFS cache taking 11.9 of 14 MB of memory.

Why would you think this is a problem? FreeNAS isn't Windows, where having memory free is crucial for system stability. FreeNAS uses free memory for dynamic caching to improve ZFS performance. If a process needs more memory, FreeNAS automatically frees RAM from cache to allocate more memory to that process.

 

[jcoffey0093]

Actually, I do see a speedup on my FreeNAS server with 16 GiB RAM and a 256 GiB L2ARC PCI-E SSD, especially for folders with thousands of images. Creating thumbnails with L2ARC available is pretty much real time, but there's noticeable delays with L2ARC absent.



Why would you think this is a problem? FreeNAS isn't Windows, where having memory free is crucial for system stability. FreeNAS uses free memory for dynamic caching to improve ZFS performance. If a process needs more memory, FreeNAS automatically frees RAM from cache to allocate more memory to that process.

Th problem is with 16 gb of memory, zfs cache taking 11.9 - there not of memory for VM

 

[Ericloewe]

Actually, I do see a speedup on my FreeNAS server with 16 GiB RAM and a 256 GiB L2ARC PCI-E SSD, especially for folders with thousands of images. Creating thumbnails with L2ARC available is pretty much real time, but there's noticeable delays with L2ARC absent.

That's an interesting scenario that I'd be very interested in hearing more about, because you're exchanging a decent chunk of ARC for L2ARC. At ~25 MB of ARC per GB of L2ARC, that's 6.4 GB, which is what, half of your ARC, in practice?

Why would you think this is a problem? FreeNAS isn't Windows, where having memory free is crucial for system stability. FreeNAS uses free memory for dynamic caching to improve ZFS performance. If a process needs more memory, FreeNAS automatically frees RAM from cache to allocate more memory to that process.

I'll point out that Windows also caches things in RAM very aggressively (perhaps not as efficiently as the ARC strategy, but it's appreciated nonetheless). This is not a weird concept in any way, free RAM is wasted RAM. I have a couple of things open on my workstation (ZFS on Linux at the moment) and ~32 out of 64 GB used at the moment - and Firefox isn't that much of a memory hog, most of it is the ARC storing things for later.

Th problem is with 16 gb of memory, zfs cache taking 11.9 - there not of memory for VM

The ARC takes up available memory. It will shrink with memory pressure to make room for other things.

That said, you may be undersizing your system. 16 GB is a good place to be without VMs, but if you start adding VMs to that, you'll quickly eat up RAM. I guess you could theoretically have some small VM running a DHCP server or something, but I imagine that is not the case. Don't forget that 8 GB of RAM is the minimum for FreeNAS, so adding VMs (plural) can get you pretty close to that rather quickly.

 

[Samuel Tai]

Th problem is with 16 gb of memory, zfs cache taking 11.9 - there not of memory for VM

Actually, how much memory does the Virtual Machines tab show as available? This includes ARC memory that can be released to launch VMs.

 

[Dan Tudora]

Don't forget that 8 GB of RAM is the minimum for FreeNAS

hello
really ??
I see many smartphone with much RAM :D

5 or 6 VM with 2 Gb RAM each
is enough for everyone
do not lie people
if you need to do something, buy hardware, in special RAM

 

[Samuel Tai]

hello
really ??
I see many smartphone with much RAM :D

Yes. This is explicitly stated in the Guide, Section 1.4.1.

General guidelines for RAM:​

• A minimum of 8 GiB of RAM is required.
  Additional features require additional RAM, and large amounts of storage require more RAM for cache. An old, somewhat overstated guideline is 1 GiB of RAM per terabyte of disk capacity.
• To use Active Directory with many users, add an additional 2 GiB of RAM for the winbind internal cache.
• For iSCSI, install at least 16 GiB of RAM if performance is not critical, or at least 32 GiB of RAM if good performance is a requirement.
• Jails are very memory-efficient, but can still use memory that would otherwise be available for ZFS. If the system will be running many jails, or a few resource-intensive jails, adding 1 to 4 additional gigabytes of RAM can be helpful. This memory is shared by the host and will be used for ZFS when not being used by jails.
• Virtual Machines require additional RAM beyond any amounts listed here. Memory used by virtual machines is not available to the host while the VM is running, and is not included in the amounts described above. For example, a system that will be running two VMs that each need 1 GiB of RAM requires an additional 2 GiB of RAM.
• When installing FreeNAS® on a headless system, disable the shared memory settings for the video card in the BIOS.
• For ZFS deduplication, ensure the system has at least 5 GiB of RAM per terabyte of storage to be deduplicated.

 

[Dan Tudora]

a 256 GiB L2ARC PCI-E SSD

hello
a real good L2ARC PCI-E SSD doubles the price of that system/NAS (Dual-Core CPU E5700)
if I recall correct is a DDR2 CPU, nor ECC
be serious

 

[Dan Tudora]

some small VM running a DHCP server

hello
that can do a decent router (Asus or ubiquiti or etc.)
NAS IS NAS
do not mix things

 

[danb35]

be serious

If you're trying to communicate something, complete sentences and paragraphs are really helpful. Paying attention to context is also helpful--the part you quoted was from a different user (not OP) who's using very different hardware (not the antique CPU that OP is using), but offering his experience where a L2ARC device improves his performance despite having only 16 GB of RAM.

NAS IS NAS
do not mix things

Right. And that's why FreeNAS has jails and VMs, so they won't be used. OP's system isn't very suitable for doing much in the way of jails or VMs, but that's a matter of his hardware, not of FreeNAS' capabilities.

 

[Samuel Tai]

If you're trying to communicate something, complete sentences and paragraphs are really helpful. Paying attention to context is also helpful--the part you quoted was from a different user (not OP) who's using very different hardware (not the antique CPU that OP is using), but offering his experience where a L2ARC device improves his performance despite having only 16 GB of RAM.

Right. And that's why FreeNAS has jails and VMs, so they won't be used. OP's system isn't very suitable for doing much in the way of jails or VMs, but that's a matter of his hardware, not of FreeNAS' capabilities.

Thanks, @danb35, but @Dan Tudora appears to be posting from Romania, so I made allowance for him being a non-native English speaker, and didn't let it disturb me.

 

[Ericloewe]

Wait a minute, it just dawned on me that the CPU doesn't even fully support bhyve, it predates Extended Page Tables or whatever the other more obscure feature was called.

hello
really ??
I see many smartphone with much RAM :D

5 or 6 VM with 2 Gb RAM each
is enough for everyone
do not lie people
if you need to do something, buy hardware, in special RAM

I don't follow your line of reasoning.

hello
that can do a decent router (Asus or ubiquiti or etc.)
NAS IS NAS
do not mix things

My point is that there are few things you would want to run in a VM on a NAS where you have a scenario with multiple VMs happily coexisting on a server with just 16 GB of RAM. One VM? Sure, I can see that. Two would be weird, three or more absurd.

 

[Samuel Tai]

@jcoffey0093, your hardware probably isn't capable of running VMs, as @Ericloewe pointed out. You can still run plugins and jails, however.

 

[HoneyBadger]

That's an interesting scenario that I'd be very interested in hearing more about, because you're exchanging a decent chunk of ARC for L2ARC. At ~25 MB of ARC per GB of L2ARC, that's 6.4 GB, which is what, half of your ARC, in practice?

L2ARC headers are "per record" not "per GB" so it's more complex than any simple number - but it's not that bad, L2ARC now costs you (I believe) 70 bytes per record (OpenZFS and Oracle ZFS might be getting muddled here)

For the 128G SSD used by @jcoffey0093 - even if it was totally filled with 4K records, it would only consume 2240M of RAM. That might have an impact as it would represent a big chunk of the default metadata allotment, but if you're pulling that much small-block you really ought to have more than 16G of RAM to steal from. ;)

If it's storing a mixed workload and average out to 32K records, that's 280M. Much more manageable and highly recommended.

For both Dan and @Samuel Tai - I'd be interested in seeing the current values for:

kstat.zfs.misc.arcstats.l2_size
kstat.zfs.misc.arcstats.l2_hdr_size

 

[Samuel Tai]

kstat.zfs.misc.arcstats.l2_size
kstat.zfs.misc.arcstats.l2_hdr_size

kstat.zfs.misc.arcstats.l2_size=46428709888
kstat.zfs.misc.arcstats.l2_hdr_size=35144824

 

[HoneyBadger]

kstat.zfs.misc.arcstats.l2_size=46428709888
kstat.zfs.misc.arcstats.l2_hdr_size=35144824

So doing the calculation from bytes, we've got 43G of L2ARC being indexed, costing us 33M of RAM.

I think you can spare it. ;)

 

[Dan Tudora]

hello

back in business(forum), after a unpleasant/hard/stressful day of work
we make computer hardware/networking maintenance and consulting for city hall/hospital/pharmacy and in this day IS hard

I do not want to offend nobody, just a technical discussion about !!
and YES Samuel, is not my native language, BUT I have not excuse for my ironical/acid/not gentle commentary

sorry for all YOU people

 

[HoneyBadger]

@Dan Tudora - No offense taken on my part.

@jcoffey0093 - As suggested by others, your system is likely undersized for running plugins/jails, and is likely not suitable for running VMs at all based on the older/weaker CPU utilization. Very light usage (eg: DHCP/DNS) may be possible but I would hesitate to use it for anything compute-intensive.

 

[Dan Tudora]

have a good night
I am go to sleep
tomorrow is another day (f**ng working day)
succes