Am I going the right way?

[Nodiaque]

Hello everyone,

I'm currently planning on building a new server. My current setup are pcs here and there that I'll merge under vm. My current needs is:

- Proxmox server on it running my vms
- Truenas VM for the storage of everything
- Windows 10 VM for emby server (like plex) with Nvidia GPU for transcoding (about 8gm ram, might raise to 16gb.)
- Windows 10 VM for conan exiled server (about 16gb ram)
- Docker for openhab, mosquitto-mqtt, nodered and mysql

The hardware I'm looking at is:
- LSI-9211-8i HBA card
- 5 or 6 Seagate Ironwolf 8TB in raid 5 (still unsure about 5 or 6 drive)
- 1 NVME WD Black SN750 500gb/1tb to host proxmon and the vm
- Intel i5-11600kf
- NVidia GPU for transcoding
- 64GB ram
- Asus Prime Z590-P

Is 64GB of ram enough? I'm reading that for ZFS, you need 1GB RAM/1TB Storage, where I plan on having between 40 to 48TB of storage.

Thank you

 

[jgreco]

Proxmox is not recommended. It is a very new entrant into the hypervisor game, and does not have a proven track record of being able to do PCIe passthru properly, especially on a non-server mainboard. ESXi is a better choice, with at least ten years of proven capability to do this.

The 1GB RAM/1TB storage rule of thumb is mainly to prevent people from doing stupid things, especially on the lower end of the memory spectrum. Do not expect to be able to have great success running 48TB of storage on 8GB of RAM. It might be okay at 16, will almost certainly be at 32. A lot of that relates to how performant you want/need it to be, and what the workload is. It is easier to build a machine and plan to allocate 32GB for FreeNAS, then find out it works great with 16, and have some leftovers for other nifty stuff, than it is to build a machine with 32GB, have all your RAM slots full, and then find out yes FreeNAS really needs 32GB for your usage model.

 

[Nodiaque]

ok. I just found out about ECC ram beeing highly recommended. Even if right now I only have an i3 with 4x hdd non raid storing the data, going to higher scale will be a good idea. I think looking for a server on ebay would be better?

 

[jgreco]

We do recommend ECC. There is a lot of debate on the topic, with anti-ECC fanatics cherry picking statements, even some from ZFS developers, suggesting that it is not needed. The topic is much more nuanced, but it boils down to the fact that ZFS has no pool "fsck" tools to fix a busted pool, and the primary strategy for pool integrity is to refrain from introducing errors, which means you always need redundancy, and that areas of the system such as main memory which are not checksummed independently by ZFS need to be 100% reliable (which is where ECC comes in). Otherwise, it may be 99.975% reliable, just like driving around in a car without a seatbelt, but you wear the seatbelt as insurance. Hopefully.

Buying a server on eBay is a bit of a nebulous thing unless you know what you are looking at. There is a Hardware Recommendations guide floating around here somewhere that talks about specifics. Please do not be tempted into buying any janky old Supermicro X7 or X8 platforms, which are cheaply available on eBay, but are power-hungry and quite ancient, with largely broken PCIe passthru support. X9 (Sandy Bridge, about a decade old now) and newer are recommended, with stuff like the X9SRL (E5) X9SCM (E3), and several others known to be solid performers. However, these are on the tail end of support in ESXi, so you may wish to stick to newer systems.

 

[Nodiaque]

Ok, so I'm getting different opinion everywhere, it's really a hard business case to deal with. I'll read again the hardware guide, I've saw it when I was building it with the hardware said over. While checking for the parts, I've gone to past 3000$ and that's when I though let's go the ebay server route.

Thanks!

 

[Nodiaque]

Just saw I cannot edit. I was looking at dell poweredge or hp servers. For the number of drive I want, in "desktop" format and with the gpu, I'll need a tower form factor. I'll check for the supermicro x11 or x12, unsure I'll find something of the like on ebay