After a lot of thinking, I came down a little bit. I tuned down my requirements a lot. I got a little bit lost, because the 25Gbit my ISP offers me sounded so cool and I wanted to try out new technologies. But to be perfectly honest, 1Gbit speed is probably enough. So my current sober thinking is this:
I could get a A2SDi-4C-HLN4F as TrueNAS as recommended by @Patrick M. Hausen. Thanks for the tip! This board would offer 8 SATA Ports and a NVM or USB header for the boot disk. It has 4 NICs, so trunking would provide enough performance for multiple 1Gbit clients accessing data.
For RAM I would start with two 64GB RDIMMs. My current TrueNAS has 64GB and ARC hit ratio is around 98%, lowest I saw was 92%.
Unfortunately, I will not be able to try out special vdevs, because I don't have enough SATA ports, but I am not sure if that would have really helped in practice. As far as I understand it, serving metadata from ARC is faster anyway.
For my Hypervisor, I will stay with my X11SCM-F Proxmox with two nvme drives in mirror. For the unimportant VMs I don't really care, I could use TrueNAS as the destination and force async. For the other VMs, I am currently happy with the performance I have right now. If I wan't more read performance, I could add more RAM to get a bigger ARC, to gain write performance, I could add a SLOG later on.
I could get a A2SDi-4C-HLN4F as TrueNAS as recommended by @Patrick M. Hausen. Thanks for the tip! This board would offer 8 SATA Ports and a NVM or USB header for the boot disk. It has 4 NICs, so trunking would provide enough performance for multiple 1Gbit clients accessing data.
For RAM I would start with two 64GB RDIMMs. My current TrueNAS has 64GB and ARC hit ratio is around 98%, lowest I saw was 92%.
Unfortunately, I will not be able to try out special vdevs, because I don't have enough SATA ports, but I am not sure if that would have really helped in practice. As far as I understand it, serving metadata from ARC is faster anyway.
For my Hypervisor, I will stay with my X11SCM-F Proxmox with two nvme drives in mirror. For the unimportant VMs I don't really care, I could use TrueNAS as the destination and force async. For the other VMs, I am currently happy with the performance I have right now. If I wan't more read performance, I could add more RAM to get a bigger ARC, to gain write performance, I could add a SLOG later on.