They sound like they perform different functions, but it there a point of diminishing returns? Are there any guidelines when a metadrive should be used over ZIL and L2ARC drives?
All three have a different purpose and are intended to solve different problems.
Do you have too much "hot data" to fit in your RAM, and can't fit or afford more? Deploy L2ARC as a second-level read cache. It's not RAM fast but it beats spinning disk.
Do you have synchronous writes (eg: NFS clients, hypervisors, OLTP DBs) that have to return fast while remaining safe? Add an SLOG device, which will accelerate the response of the "write to stable storage" data flow.
Do you do a lot of metadata-heavy operations (directory listing, scans, small updates to many dozens/hundreds of thousands of files?) and find that they take far too long? Here's where a dedicated metadata vdev may help - rather than your back-end data vdevs spending time handling metadata reads and writes (especially if they're spinning disks) push this data to separate flash devices which are much faster at the random I/O that's inherent to metadata work.
If your "metadata-heavy" operations primarily result in
metadata reads then you can achieve most of the same results with adding L2ARC and setting the
secondarycache=meta property. It doesn't help
metadata writes - for that, you need the special vdevs. But an important note though is that unlike L2ARC, where all contents are volatile and loss of the device just results in slowdown,
a metadata vdev needs redundancy - metadata copies only exist on this vdev, and if you lose it, the whole pool is toast. (Edit: This also means you can't remove it after it's been added. Edit2: Apparently you can, but unless checksum on removal is implemented now, you could end up in a bad spot if you have a read error during a copy of pool metadata.) Mirrors will be heavily recommended and a triple-mirror wouldn't be unreasonable. Depending on your write workload you'll also want to use decently well-rated drives in terms of endurance. While it isn't directly "drinking from the firehose" like an SLOG is for sync writes, you will want to scale it based on how update-heavy your workload is. I'd say "mixed use" SSDs with a 1-3 DWPD rating depending on size are where you'd want to land. Not cheap 0.1 DWPD QLC, but not 25+ DWPD Optane either. (Edit3: Of course, if you can afford Optane, it's the best solution. It also doesn't suffer from increase read latency in a mixed-workload scenario.)
StorageReview did a YouTube podcast with
@Kris Moore where he talks a bit about the metadata-only vdevs (sorry, "Fusion Pools" ;) ) and I've linked to that timestamp (hopefully) below.
In this week’s podcast Brian sits down with Kris Moore from iXsystems. Kris gives an update on the progress of merging FreeNAS and the TrueNAS code bases tog...
youtu.be
Edit: You'll notice I didn't mention deduplication here. While you can add separate vdevs for metadata and in fact dedup tables explicitly in TN12 it doesn't relieve the additional memory pressure or extra considerations that arise from enabling deduplication. If you truly needed it before, you'll be very happy to have these vdevs available as it will increase performance (possibly significantly) but if you
didn't use it before, don't think that you can just drop a couple SSDs into a Fusion Pool as
special type=dedup and enable it globally. It's still a recipe for pain if you do it wrong.
