Cache and/or RAM? Am i dumb?

[rdcustom]

Hello all,

referring to my first posts on this forum, I finally completed and happily run my new NAS for about a month.

This is my configuration:

Ryzen 7 5700g
128gb ram
2x 128gb SSD as boot-pool
2x 256gb SSD as Jails/Plugins pool
4x 4TB HDD as Backup Pool
4x 4TB HDD as Plex library
1x 1TB SSD as Plex pool cache

now, cause (as expected) the PLEX pool is running out of space and the Backup is almost empty, I am going to a 8x 4TB HDD Pool for PLEX + additional 2x 4TB mirror for Backup

Given that I need to delete and re-create the pool, is the SSD cache worth it with 128gb ram?
(this question is because I don't understood the difference between L2ARC and CACHE yet, at least for my usage)

If yes, I have a spare 2TB SSD, would you replace the 1 with the 2TB one?

Thanks in advance

 

[Arwen]

First, L2ARC is the technical name for Cache device. They are the same thing. I wish the writers of ZFS used the same word in all places.

A L2ARC / Cache is only useful for certain use cases.

For example, Plex movie or T.V. show playing, a L2ARC / Cache is worthless. The reason is that L2ARC / Cache is useful for repeated reads of the same object, (file, access control list or directory), not for single one off reads. Even backups won't benefit directly from a L2ARC / Cache device.

On the other hand, if you have a child that plays a single cartoon over and over again, then that might / would likely be cached. The trick is, can it be cached in RAM? And if so, a L2ARC / Cache is not needed, even for the repeated playing.

However, backups might benefit from an Metadata only L2ARC / Cache device. It depends on if your ARC, (which is in RAM), fills up. So, you next task is to check your ARC statistics and see if it is full. That can be complicated to read from the command line script output. However, the GUI has a rough guide of how much memory is in use. Start their.

 

[winnielinnie]

I wish the writers of ZFS used the same word in all places.

Do you realize it wasn't until a couple days ago I found out that "Fusion Pool" is just a term iXsystems uses, which is not an actual ZFS term?

 

[Arwen]

Well, to be fair, I think I saw "fusion pool" on the OpenZFS forums or GitHub. But, yes, it's not a common term.

 

[rdcustom]

First, L2ARC is the technical name for Cache device. They are the same thing. I wish the writers of ZFS used the same word in all places.

A L2ARC / Cache is only useful for certain use cases.

For example, Plex movie or T.V. show playing, a L2ARC / Cache is worthless. The reason is that L2ARC / Cache is useful for repeated reads of the same object, (file, access control list or directory), not for single one off reads. Even backups won't benefit directly from a L2ARC / Cache device.

On the other hand, if you have a child that plays a single cartoon over and over again, then that might / would likely be cached. The trick is, can it be cached in RAM? And if so, a L2ARC / Cache is not needed, even for the repeated playing.

However, backups might benefit from an Metadata only L2ARC / Cache device. It depends on if your ARC, (which is in RAM), fills up. So, you next task is to check your ARC statistics and see if it is full. That can be complicated to read from the command line script output. However, the GUI has a rough guide of how much memory is in use. Start their.

So, if I understand correctly, my SSD as a cache is totally useless for PLEX media, right?
Is 128gb RAM enough to get rid of that?

 

[Patrick M. Hausen]

Well, to be fair, I think I saw "fusion pool" on the OpenZFS forums or GitHub. But, yes, it's not a common term.

Apple coined that for their "Fusion Drives", so probably iX wanted to profit from the marketing momentum.

 

[Etorix]

So, if I understand correctly, my SSD as a cache is totally useless for PLEX media, right?

Correct.

Is 128gb RAM enough to get rid of that?

Running arc_summary should confirm that.
(If you want help in deciphering the output, open a SSH session to your NAS, run arc_summary there so you can copy the result and post it here within CODE Tags.)

 

[Morris]

Video is much slower than a hard disk. No need for cache for Plex.

 

[rdcustom]

Video is much slower than a hard disk. No need for cache for Plex.

yes, but disk spin up on an 8 disk pool takes a while...

 

[Morris]

yes, but disk spin up on an 8 disk pool takes a while...

All disks will spin up at the same time. You need a power supply that can support this. It's not that much surge power, 5 or 6w per disk.

 

[Arwen]

yes, but disk spin up on an 8 disk pool takes a while...

Yes, but even a L2ARC that was completely populated with stuff, it won't have anywhere near as much as the main data vDevs. Thus, it is almost certain that what you want won't even be in the L2ARC. So you would have to wait for the HDDs to spin up anyway.

Remember, ARC, (in memory), and L2ARC, (SATA SSD or NVMe), will only cache items that are read more than once, or very recently. That does not apply to normal T.V. show or movie watching. (Except for the kid example of repeating the same cartoon over and over again...)


This is why some people are moving their video storage to SATA SSDs or NVMes. They want silent video storage, or fast response time for initial buffering.

 

[winnielinnie]

That does not apply to normal T.V. show or movie watching. (Except for the kid example of repeating the same cartoon over and over again...)

This is why I set primarycache=metadata for my dedicated "seed" dataset, since there's no point is caching data in the ARC where the reads are scattered, random, happen across a large dataset*, and practically never hit the same data within a timespan of hours or even days.

* Historic collection of Linux ISOs and public domain videos pre-1930s.

 

[Morris]

Yes, but even a L2ARC that was completely populated with stuff, it won't have anywhere near as much as the main data vDevs. Thus, it is almost certain that what you want won't even be in the L2ARC. So you would have to wait for the HDDs to spin up anyway.

Remember, ARC, (in memory), and L2ARC, (SATA SSD or NVMe), will only cache items that are read more than once, or very recently. That does not apply to normal T.V. show or movie watching. (Except for the kid example of repeating the same cartoon over and over again...)


This is why some people are moving their video storage to SATA SSDs or NVMes. They want silent video storage, or fast response time for initial buffering.

TiVo, and every cable companies DVR use hard drives. Having used these as well as HDD in TrueNAS with Plex and other video software, the wait is miniscule if you use drives that spin up quickly. I switched my video storage to SATA SSDs to save power and I don't notice a difference in video start time.

 

[awasb]

You could try

Code:

sysctl vfs.zfs.l2arc.noprefetch=0

From zfs.4

Do not write buffers to L2ARC if they were prefetched but not used by applications. In case there are prefetched buffers in L2ARC and this option is later set, we do not read the prefetched buffers from L2ARC. Unsetting this option is useful for caching sequential reads from the disks to L2ARC and serve those reads from L2ARC later on. This may be beneficial in case the L2ARC device is significantly faster in sequential reads than the disks of the pool.
Use 1 to disable and 0 to enable caching/reading prefetches to/from L2ARC.

Check L2ARC hits before and after applying the sysctl with arc_summary.

 

[HoneyBadger]

Note that allowing L2ARC to store prefetched buffers will increase the write workload on the device(s) in the cache vdev.

 

[Constantin]

L2ARC did wonders for me re: rsync once I set it to "metadata-only" and persistent. It took about three passes for the cache to get "hot". After that, I got performance close to the "fusion pool" sVDEV setup I have now. sVDEV is even faster (the SSDs take over small files) but trickier to implement and riskier than L2ARC. (see here and here)

iXsystems yet has to ship the GUI elements / menus / dashboards for sVDEV planning, maintenance, etc. so I would stay away from sVDEVs unless you want to put in the time to learn about them, then use the CLI to plan the right small file size cutoff, occasionally confirm that the sVDEV still has adequate available space for small files, etc.

When implemented correctly, sVDEV works fantastically (see above), but you better understand the risks also - i.e. if the sVDEV dies, so does your pool. L2ARC can deliver many of the sVDEV benefits re: fast browsing yet there is no risk re: failure since a bad L2ARC can be ignored by the file system and then the slower pool is used instead.

As usual, the use case case matters also.