Buy or not buy SSD for cache

[Njanja]

Hello everybody

Here's a simple question regarding SSD for cache.
Should I buy an SSD or not, or should I invest in more RAM?
I have Dell R720 with 64GB RAM, 10GB network and 6x4TB HDDs.
Freenas I use mostly for storage and for iSCSI (Esxi and my PC).


Njanja

 

[sretalla]

Should I buy an SSD or not, or should I invest in more RAM?

Don't even question it.... More RAM is always better.

What are you hoping to see behave differently with "cache"? (are you thinking about L2ARC or SLOG?)

 

[Njanja]

Hi

L2ARC, but i can put in 64MB, if it's better

Njanja

 

[jgreco]

64GB of RAM is the point where L2ARC tends to become at least somewhat useful, but it is important to remember that the ARC itself has to be big enough that ZFS can spot repetitive accesses and learn what is good to push out to the L2ARC. Otherwise it just pushes nearly random blocks out to the L2ARC which may or may not be helpful. Consider the amount of data that you are accessing and make sure that you are not completely swamping the ARC. If you have two terabytes of actively accessed data, for example, I am skeptical that 64GB+L2ARC would work well, but if you only have 128GB of actively accessed data, I think L2ARC would probably be fine (but someone will point out that the additional 64GB RAM would still be a better move).

The nice thing is that SSD prices have plummeted, and you can potentially buy a small ~256GB L2ARC device to see how well it does, and then make a decision as to whether or not more RAM would also be helpful. The 256GB isn't a technical suggestion, it's a cost-based (low cost) suggestion.

 

[Constantin]

L2ARC indexing eats some RAM, Ixsystems recommends at least 32 GB RAM and L2ARC < 10x RAM. Thankfully, L2ARC can be recreated easily, is not system critical, so any SSD failures have no lasting impact. Use case vs. benefit is important, however. For example, I currently use L2ARC only for metadata storage to speed up rsync and browsing in general.

I initially installed a 1TB SSD drive for my system as the arcstats back then suggested a 40:1 ratio between metadata needs vs. pool size. That ratio may have improved since then because I waged a lengthy campaign to reduce the # of files on my server by consolidating old system backups into disk images (that use bands with many embedded files vs. individual files). Hence, my metadata needs likely shrunk a lot in the process.

I found significant benefits using a metadata only L2ARC with Rsync for backups. Yes, the rsync has to be executed at least 3 times to get the maximum benefit but after that, the system responsiveness is way up (the default system tuneables fill the L2ARC slowly, hence it takes a few backups/traverses before the cache is "hot"). ServeTheHome has some L2ARC SSD recommendations but pretty much anything that allows fast reads is OK.

Remember that L2ARC is currently not persistent and hence has to be "reheated" every time the server is rebooted. A metadata-only L2ARC should also not be confused with a metadata sVDEV (now available in TrueNAS 12.x), as the L2ARC is expendable while a failed metadata sVDEV will take the associated pool with it. I am planning on opening my L2ARC up to all files once I transition my system to 12.x and institute a 3-way mirrored metadata-only sVDEV.

 

[sretalla]

Remember that L2ARC is currently not persistent and hence has to be "reheated" every time the server is rebooted.

I think we are almost there in terms of the persistent L2ARC option...

Or maybe even already there?

How do I enable persistent L2ARC?

Keep seeing, "L2ARC rebuild disabled" on startup. Is there a flag I need to set to enable this new OpenZFS feature?

www.truenas.com

 

[Constantin]

Very cool. I imagine it’s a TrueNAS 12.x feature only though?

 

[sretalla]

Very cool. I imagine it’s a TrueNAS 12.x feature only though?

It's dropping in from OpenZFS2, so, yes.