Is there an option for "real" write caching?

[thex]

When will openzfs 2.1.10 be available as an update?

I'm on Truneas Core 13 and current investigations of some kernel panics indicate that I might be affected by this bug: https://github.com/openzfs/zfs/pull/14454 (also discussed here: https://www.truenas.com/community/threads/sos-kernel-panic-help.104524/) It looks like the code was merged into...

www.truenas.com

Aborts and sometimes even Kernel panicking whenever I copied large files from it. Which seems to be a ZFS bug related to a corrupted pool.

New exos disks should be here tomorrow

 

[JossBrown]

Hello everyone… my first post, and I'm immediately going to chime in with a piece of heresy. You could definitely create a write cache, a real one, but it would still (of course) be a workaround, and it would (I think) only work for SMB copies and other files copies from your clien'ts UI file manager. It's kinda similar to how unRAID does it.

I don't have TrueNAS yet, not even a NAS/server, but will eventually (see my plans below). But I would do it like this…

You have your main array… let's call the pool "store_main". To your TrueNAS hardware you then add two fast (and large enough) M.2 NVMe SSDs (mirrored), as a new pool… let's call it "write_cache". On your client machine you then always need to copy files to the write_cache SMB share, and on your TrueNAS you create a scheduled job (script) that will regularly move new files from write_cache to store… well, "move", i.e. copy and unlink original from write_cache after successful copy to store_main.

The tricky part is that the mover script needs to know, which directory path on store_main to copy the file into. So I would obviously have to handle this on the client side. Below is a macOS script (rrcp) that I just hacked together, and it would (sort of) redirect a copy-paste file operation originally meant for a path on store_main to write_cache, and write the eventual destination subpath as an extended attribute. (You'd need a different hotkey for this kind of paste operation, e.g. CMD-CTRL-V, and I only tested this with Nimble Commander.) The scheduled mover script on TrueNAS would then just look for new files, read the destination from the XA, copy it to the main pool, and unlink the cached version.

Code:

#!/bin/zsh
# shellcheck shell=bash

# rrcp 0.1
# macOS
# reroute file copy & paste to a TrueNAS write cache instead of the main storage pool
#
# TrueNAS write cache must be mounted in addition to the TrueNAS main storage pool
# TrueNAS write cache can be mounted hidden (I assume -- untested!)
#
# Nimble Commander: set rrcp tool parameters as "%r %-P"
# Nimble Commander: set paste hotkey for rrcp tool to e.g. CMD-CTRL-V
# NOTE: Nimble Commander target directory must be opened in its own panel for CMD-C > CMD-CTRL-V
# NOTE: target panel must be active
#
# macOS Finder? IDGAFF

export LANG=en_US.UTF-8
export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/opt/local/bin:/opt/local/sbin:"$HOME"/.local/bin:"$HOME"/.local/sbin

_beep () {
    osascript -e 'beep' -e 'delay 0.5' &>/dev/null
}

# BEGIN USER SETUP
username=$(id -un) # OR: username="foo" etc.
writecachemountpoint="/private/mnt/remotes/write_cache" # always full path independent of firmlinking
storagemountpoint1="/private/mnt/remotes/store_main"
storagemountpoint2="/mnt/remotes/store_main"
# NOTE: storagemountpoint1 and storagemountpoint2 have to be the same, if no firmlink was set by user for this directory
# END USER SETUP

# logging
process="rrcp"
processid="local.$username.$process"
logloc="$HOME/Library/Logs/$processid.log"
exec > >(tee -a "$logloc") 2>&1

# get main storage target directory first (from Nimble Commander parameter %r)
# remove local mountpoint from target directory path
targetdirectorypath="$1"
targetsubpath=$(echo "$targetdirectorypath" | sed "s*^$storagemountpoint1/**" | sed "s*^$storagemountpoint2/**")

shift

currentdate=$(date)
echo -e "\n*** NEW RRCP PROCESS: $currentdate ***\nEventual storage target: $targetsubpath"

# access filepaths to be copied (from Nimble Commander parameter %-P)
xaerrors=false
cperrors=false
for sourcepath in "$@"
do
    posixdate=$(date +%s)
    echo -e "***\nrrcp to write_cache: $posixdate\nsource: $sourcepath"
    # create POSIX-dated subdirectories for file copies: better sorting options for the TrueNAS mover script
    cachetargetdir="$writecachemountpoint/$posixdate"
    ! [[ -d "$cachetargetdir" ]] && mkdir "$cachetargetdir"
    # actual copy
    sourcename=$(basename "$sourcepath")
    cachetargetpath="$cachetargetdir/$sourcename"
    if cp "$sourcepath" "$cachetargetpath" ; then
        echo "Success: file copied to write cache"
        # write extended attribute containing the destination subpath in the TrueNAS main storage pool
        if xattr -w "$processid" "$targetsubpath" "$cachetargetpath" ; then
            echo "Success: XA written"
        else
            echo "WARNING: failed to write XA!" >&2
            _beep
            xaerrors=true
        fi
    else
        echo "ERROR: copy process failed!" >&2
        _beep
        cperrors=true
    fi
done

# final output
if ! $xaerrors && ! $cperrors ; then
    echo -e "***\nDone"
else
    echo "***"
    if $xaerrors ; then
        echo "WARNING: there were errors writing metadata!"
    fi
    if $cperrors ; then
        echo "ERROR: there were fatal copy errors!"
    fi
fi

exit

# NOTES:
# TrueNAS needs a scheduled mover script (similar to unRAID)
# mover script would transfer new files from write cache to main storage
# mover script would sort POSIX-dated subdirectories, oldest first
# mover script would copy (cp -r) those subdirectories' contents to the correct path on the main storage pool
# mover script would determine the correct path from the contents of the extended attribute
# mover script would unlink the cached version of the file after successful copy to the main storage pool

As for my own plans for a NAS/server, it would be for extended and wide home use, incl. audio production with iSCSI, and one option is a DIY build, which would probably be something like: dual SFP28 25GbE NIC, 128 GB DDR5 ECC RAM, 8 x 4 TB SATA SSDs (Samsung EVO 870) in RAID-Z2. I do not think that with this kind of setup you would need SLOG or L2ARC—or a true "write cache", for that matter—, though I am thinking about L2ARC for metadata only (single NVMe).

 

[Davvo]

As for my own plans for a NAS/server

Please open a new thread.

 

[jgreco]

definitely create a write cache, a real one,

It's not a real one. How would you cache permissions, ownership, or metadata updates for example?

 

[JossBrown]

Good point. You couldn't use this to cache metadata updates, I think. But as I wrote above: it would just be a workaround, a purely hypothetical workaround, for plain SMB file copies, basically a glorified dropbox. You might be able to deal with permissions & ownership, though, but I hadn't thought about it, because this was just a quick hack. To stay with the dropbox analogy: maybe you could set up this "write cache" in this way. To be on the safe side, you could also write the owner and permissions to the file's extended attribute, so the mover script will have additional data available to know how to set them, not only where to put them. The actual owner in TrueNAS terms could be determined from the client machine's df output: //foo@192.x.x.x/bar … and permissions directly from the file. But it's a complicated mess, and in the end you don't need a write cache: RAM is, for all intents & purposes, our write cache in TrueNAS, if I recall correctly. I think the way that TrueNAS (or rather ZFS) handles it all is pretty awesome.

 

[jgreco]

The problem is that you're thinking about the NAS as a device that simply stores files. That's okay, some of us use it that way, but for a lot of users, it is carefully and closely integrated into a network environment with Active Directory, ownership, permissions, and then stuff like snapshots and clones, all of which really require real write caching, not just simulated. There are some users who've used some sort of Windows-side caching package to accelerate their gameplay (etc) and that may actually be a good compromise, allowing you to queue up a bunch of workload quickly on the client side and then letting the network work at its generally low speed to complete the work over time.

 

[sretalla]

What you're talking about is storage tiering (for writes) more than caching. Things like autotier or the full version of glusterfs (sadly not the one included in SCALE) can already do this.