Advice for homelab build

[ehsab]

Hi,
First post after many threads read, and i would need some advice since this is my first FreeNAS/ZFS build.
I do have some background running other datacenter storage, i'm a Nimble user in the datacenter.

This build is gonna be for my vsphere homelab storage but i was also thinking of running a couple of VMs for my household aswell. I'm more for high iops then large capacity.
I have two different options i'd like to run by you and see what i could improve or think different on.

First option is an Intel 2U system.
Supermicro SYS-2029U-TN24R4T (https://www.supermicro.com/products/system/2U/2029/SYS-2029U-TN24R4T.cfm)
Motherboard: Super X11DPU (https://www.supermicro.com/en/products/motherboard/X11DPU)
CPU: 2pcs Xeon Silver 4112 4C/8T 2.6Ghz
RAM: 12pcs SMC 16GB DDR4-2666 2Rx8 LP REG ECC DIMM (M393A2K40CB2-CVF)
Disk: 2pcs Samsung SM683 240GB SATA 2.5" (mirrored FreeNAS boot)
Disk: 20pcs Samsung PM983 960GB NVMe U.2 2.5" (mirrored iSCSI storage for VMs)
Disk: 1pcs Intel Optane SSD DC P4801X 200GB NVMe M2 (ZIL/SLOG)
NIC: 2pcs Intel XL710 40Gbps

Here is a 24-slot chassi with dual cpus, and i would like to keep the TDP to a minimum, but i wonder if the CPU will be a bottleneck here? Should i go with more and slower cores?
Any ideas/suggestions for the primary disks (PM983)? From what i can read if you have a fast and reliable ZIL/SLOG you can get by with TLC disks, right?


The second option is an AMD 1U system.
Supermicro AS-1113S-WN10RT (https://www.supermicro.com/Aplus/system/1U/1113/AS-1113S-WN10RT.cfm)
Motherboard: Supermicro H11SSW-NT (https://www.supermicro.com/en/products/motherboard/H11SSW-NT)
CPU: 1pcs AMD EPYC 7262 8C/16T 3.2/3.4GHz 128MB 155W
RAM: 8pcs SMC 32GB DDR4-3200 2Rx4 LP ECC REG DIMM (HMA84GR7CJR4N-XN)
Disk: 2pcs SuperDOM 64GB (mirrored FreeNAS boot)
Disk: 10pcs Samsung PM983 1.9TB NVMe U.2 2.5" (mirrored iSCSI VMs)
Disk: 1pcs Intel Optane SSD DC P4801X 200GB NVMe M2 (ZIL/SLOG)
NIC: 2pcs Intel XL710 40Gbps

This is a smaller server, and a little cheaper then the first alternative. However i have not seen many AMD Epyc builds, and wonder about the compatability in FreeNAS with EPYC cpus? I have also noted that the general recommendation for iops is many disks and i get that. But how big is the penalty with this type of disks, they already can perform alot of iops by themself.

I have som general questions aswell, like is it possible to grow a mirrored pool with adding more disks or do i need to replace the old disks completely with bigger disks?
What is the recommended amount of disks in a mirrored setup? is there a min/max?
I was thinking of overprovisioning the optane disk since its 200GB, any suggestions for the OP?

Thanks for taking the time.
Kind Regards
ehsab

 

[ehsab]

I'm leaning towards the AMD build with 10x 1,9TB disks.
There are two different motherboardoptions, H11SSW-NT and H12SSW-NTR (which is newer).
However when reading the motherboard manual two things strikes me.

1. On two of the PCIe lanes from the motherboard (H11SSW-NT) to the nvme slots there is a "re-driver" inbetween. Whats this for? Is this something to consider or will it has no impact on the nvme performance?
The newer motherboard does not have this re-driver.

2. There is a switch(?) between the CPU and the two M2 slots on H11SSW-NT, i assume this is for enabling x4 to one slot if the other is not being used. Is this something to consider or will it have no impact if i only use one M2 disk?
The H12SSW-NTR does not have this switch, but on the other hand both M2 slots are PCIe 4.0 only. What impact will this have for my P4801X?
I would however like to mirror two P4801X disks so either way i'm stuck at x2, but the old is PCIe 3 x2 and the new is PCIe 4 x2. Is this a bad idea?