Storage Recommendations

[patmage]

I'm putting together a new server and would like some recommendations on the storage setup. I've got a new Xeon w-1370, motherboard and 64GB of ram.
For storage though I was hoping to reuse parts I already had. Those include 4 10TB HDDs, 2 1.8TB Intel DC SSDs, and a 1TB pci-e gen4 M.2 SSD. At first I was thinking 2 mirror vdevs for the 4 spinning drives in one pool for mass storage and another mirror vdev with the SSDs for a faster pool to put any VM's in. But then I was looking into special allocation and thinking maybe I should try that instead with the SSDs. It would definitely be nice to not have to worry about 2 separate pools. I'm not 100% sure how that works, hence this post, but I figure I'd need to set special_small_blocks to actually really take advantage of the 1.8tb. I'm not sure how much storage just metadata would take up. Is there a way to estimate that?
The fast 1TB M.2 drive seems overkill for the OS drive of SCALE, but I already have it. I could use it for L2ARC or SLOG, but from most of what I've read I don't think that would be very useful? So I'm not sure what to do with that. Maybe use some of it as the transcode directory for the plex container.

The main duty of the server will be for fileshare, plex, and openhab. But it will also host a few dev VMs for my webapp projects (Hopefully LXC or LXD containers at some point). And a handful of other containers as well. The main purpose of putting together a new setup though is to be able to transcode 4k content in plex. From what I've read here passing through the iGPU to a container should be working. As well as to get familiar with TrueNAS SCALE. I've got FreeNAS Mini's deployed at several clients sites as backup storage and there are definitely a few that could take advantage of having a linux host instead of FreeBSD.

 

[ornias]

There are multiple resources here and on the rest of the internet such as Level1-Tech. About these subjects.
I suggest you do some research first, read the primers/introductions here and come back with some more... "precise" questions...

Primarily showing what you already know, which articles you read and how you came up with these conclussions yourself.

 

[patmage]

I've probably read all of the general goto guides here, Level1-Tech, and servethehome covering ZFS and I'm fairly certain of what my end setup is going to be. Mostly I was just wondering what users experience has been so far with TrueNAS Scale and the special allocation devices as I haven't actually tried it yet for myself. Still waiting for my motherboard to show up. Other than that I was really just looking to see if anyone had an opinion on what they would do with a 1tb nvme drive in this setup if they had one and nothing else to put it in.

 

[ornias]

There's nothing inherently special about SCALE when it comes to storage. Both use the same version of OpenZFS...

 

[patmage]

When you say both I assume you are talking about SCALE and Core? Those are on different versions of OpenZFS right? 2.0.4-3 for Core and 2.1-rc6 for SCALE. And the two best resources I could find about special allocation were from Napp-it and the level1techs post that I'm not sure what versions of OpenZFS those were based on.

But honestly I was just looking for a general opinions on whether special allocation made sense for my use case, with any distro, or if I'd be better served with separate pools of ssd and hdd storage. I feel like my use case is probably a fairly standard one these days. I'll probably just end up trying both. It is just one of those things that takes forever when you are talking about transferring terabytes of data back and forth to try out different setups. And if carving out a bit on the NVMe for slog or something else made sense. I'll probably end up pxe booting a few raspberry pi's at some point and maybe a slog would be handy for that.