Adding cache and log disks

[Lucas Rey]

Hi, I'm planning to expand my pool from 4x4Tb to 4x12Tb disks in RAID-Z2. TrueNAS runs unders Proxmox, and this is the hardware specs for VM

• RAM: 64GB
• Main MoBo: Q370M D3H GSM PLUS
• Using Onboard SATA controller (6 SATA ports)

Because I still have 2 more free SATA ports, I'm thinking to the possibility to add 2 more SSD drive (maybe 1TB each) to use as Cache and Log to increase performance.
I already know that I should use mirrored VDEV, but I prefer RAID-Z2 for its data security (supports up to 2 disks fail).

I read somewhat on the wiki page and around the web regarding ZFS Cache and Slog, but I think a street test will be the best. So I just created a TrueNAS VM and use Virtual Disks to emulate the main TrueNAS. Pool was built in the following way, so 4x4Tb disks in RAID-Z2 and another 2 disks one for cache and other for log:

I did just some test with and without cache/log, and this is the result:
WITHOUT CACHE AND LOG:

Code:

WRITE:
dd if=/dev/zero of=test bs=1024 count=10m   
10737418240 bytes transferred in 43.568624 secs (246448415 bytes/sec) ==> 235 MB/Sec

READ:
dd of=/dev/null if=test bs=1024 count=10m
10737418240 bytes transferred in 6.626467 secs (1620383459 bytes/sec) ==> 1545 MB/Sec

WITH CACHE and LOG

Code:

WRITE:
dd if=/dev/zero of=test bs=1024 count=10m   
10737418240 bytes transferred in 15.031233 secs (714340488 bytes/sec) ==> 680 MB/Sec

READ:
dd of=/dev/null if=test bs=1024 count=10m
10737418240 bytes transferred in 7.234284 secs (1484240620 bytes/sec) ==> 1415 MB/Sec

So it seems only write speed is affected (log disk).
What do you think guys of this approach? Pros, Cons?

Thank you.

 

[LarsR]

L2arc is just a read cache, so it won't speed up write operations, and a log is only usefull with sync writes, e.g. ISCSI or NFS. If you're using smb then a slog is of no use to you.

 

[Lucas Rey]

L2arc is just a read cache, so it won't speed up write operations, and a log is only usefull with sync writes, e.g. ISCSI or NFS. If you're using smb then a slog is of no use to you.

Hi, and thanks for reply. The test in first post has been done on TrueNAS CLI directly. And yes, my main nas use will be via samba (transfer file and/or media stream.
So increased speed in my test is useful only from cli, but not in my usecase right?

 

[LarsR]

Yes

 

[Lucas Rey]

Yes

Thank you, most probably I'll go with mirrored vdevs to increase write speed. And maybe considering only the cache disk to increase read speed

 

[Davvo]

Generally you want L2ARC and SLOG to be M.2 due to reduced latency.
1TB L2ARC is way too big for 64GB of ARC (you want a ratio of 1:4 or 1:6 ARC:L2ARC), and gigantic for a SLOG.
You want a SLOG if you enforce syncwrites (always).
You want a drive able to delivee good performance in mixed operations (concurrent reads and writes) for your SLOG.
You can easily check whether you need a L2ARC by monitoring the ARC's hit rate: if it goes below 90% it's worth considering it.

Since you are virtualizing, remember to pass the whole controller to TN.

Please read the following resources.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

SLOG Devices

Provides general information on ZFS intent logs (ZIL) and separate intent logs (SLOG), their uses cases and implementation in TrueNAS.

www.truenas.com

SLOG Overprovisioning

Describes how to configure SLOG over-provisioning on TrueNAS CORE.

www.truenas.com

L2ARC

Provides information on L2ARC, caches drives, and persistent L2ARC implementations in TrueNAS CORE and SCALE.

www.truenas.com

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.truenas.com

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

 

[Jailer]

Something else of concern is if you are using the onboard SATA ports, how did you pass those disks through to the TrueNAS VM?