Got lucky and now planning to go insane and replace the all in one ESXi and True..

[Herr_Merlin]

Hi all,
I am planning to replace my current system: https://www.truenas.com/community/t...s-a-vm-or-freenas-with-vms.86760/#post-602758

New System:

• 2U Supermicro
• X11DPU-Z+
• 512GB DDR4 reg. ECC 2666
• 2x Intel Xeon Platinum 8168
• 12x3.5 with 4 times NVMe

Now it comes all down to pool layout.. and the again limited IO (PCIe):

• PCIe X16 - Vega 56
• PCIe X16 - 5700XT
• PCIe X16 - AMD GPRO 4300
• PCIe X8 Adaptec or LSI RAID Card for OS
• PCIe X8 - 10Gbit dual port NIC

Leaves me with:

• 4x PCIe NVMe and cables I could connect to the backplane..
• 3x PCIe X8 in X16
• 1x PCIe X16

For disk I have:

• 8x 6TB SAS
• 7x 2TB SATA SSD Samsung Pro
• 4x 1TB NVMe M.2 Transcend
• 2x 480GB SATA SSD Samsung PM series
• 2x 240GB SATA SSD Samsung PM series
• a few times Intel Datacenter <120GB

For HBAs I have:

• Supermicro AOC-S2308L-L8i IT Mode
• StarTech PCIe X8 with PLX chip to 2xm.2 x4

For the VM I would go:

• Pass the onboard SATA Controller with 2 Intel SSDs for Boot
• Pass all other needed PCIe devices
• Give it 128GB of RAM
• Give it 8 or 12 CPU cores
• See that NUMA wise the PCIe devices and cores are on the same node

I want to leave all HDDs within the old server and use it "backup" and replicate data to that system. Same goes for the 2x 480GB SATA for the OS of that server.

With the limitation of free PCIe Slots as well as 3.5 Slots I am still trying to get my head an ideal pool layout.
Goals would be to keep the HDDs at the other system and replicated the data.
I need about 5TB for some dead data. Plus 3TB for VMs and those data.. I am missing some free slots for SLOG I assume. Has anyone tried to pass those Optane DIMMs to a VM? Is that even possible? I still got 8 free DIMM slots.. and CPUs capable off.. on ebay you can get them for about 130€ for 256GB.. same you would pay for an u.2 100GB optane P48x.

Or just buy 4 of those 1.6TB Intel NVMe SSDs and put them into RAIDZ1 to archive the free space goal.. or go closer to the edge and and set up mirror.. assuming performance is at least at 1/4 of a single disk... but those SSD would be about 1200€.. about double what I've paid for the system..

The other 8 free slots use up with the 2TB SATA SSDs in RAID Z2 for data... connected to the AOC-S2308L?

Any better idea?