overkill? - new ideas - go chinese? NEW SERVER!

[Oedzesg]

Hello all,

I've been a great satisfied user of Truenas Scale for a while now.

Over a year ago I built my server in the configuration as it currently is.

And since I'm not really one to do things halfway, I've handled it pretty well in my opinion :)

*Intel Xeon E-2256G
* Supermicro x12-STH-LN4F-P
* 64GB ECC memory
* Mellanox 10Gbe network card
* 8x 6TB HDD WD-RED = tank
* 2x Samsung Pro Nvme 1 TB = App/ VM data
* 2x Samsung Pro Nvme 1TB = Home folders
* 2x Kingston SSD 128Gb = boot drives

The 8 HDDs are neatly arranged on an HBA card.
The Nvme's on a LinkReal Pcie card.

All this is neatly packed in the silverstone cs381.

Now I'm actually stuck with the following:

For VMs and Docker/Kubernetes I think Proxmox just works better.

It is possible to upgrade my RAM to eg 128GB and run Truenas Core in a Proxmox VM, but since all my sensitive data is also stored here, including backups, etc., this just doesn't feel right.
(call me crazy or prove otherwise)

With the current setup, truenas never exceeds 20% processor load! whatever I do.
Streaming PLEX 4k movies with Atmos 7.1 files to the phone, the counter remains around 9%.

All in all, quite an overkill.

Since I want to build a new server for Proxmox, it seems to me to be a nice challenge to do this with a little less budget and, for example, use some older hardware.

But where the new proxmox server will probably be busier than my current Truenas server, especially when I also move PLEX to the proxmox server, it was my idea to convert the current setup into a new box and then a different motherboard + CPU to buy for my Truenas server.

When I look at nice used and affordable Xeon CPUs, for example, the 2667 V4 could be a nice option.

But then I have a bit of a problem that these X99 motherboards are all ATX or e-ATX and my silverstone fits a maximum of one mATX.
Until I came across Aliexpress.

Is his plate a good choice? what are your experiences?

I also have my linkreal NVME adapter from ali and it works great!

Awaiting your opinions and advice!

 

[danb35]

(call me crazy or prove otherwise)

I'm not going to argue the point, because I haven't run TrueNAS for serious work under Proxmox. But if you can pass through the HBA and the NVMe devices to the VM, and allocate enough RAM/CPU to it, I'd expect it to be generally a safe thing to do. And if you later decided you didn't want TrueNAS virtualized any more, returning it to bare hardware would be quite straightforward.

For VMs and Docker/Kubernetes I think Proxmox just works better.

I can't speak to Docker/k8s under Proxmox (though I do use some LXCs), but I certainly agree with respect to VMs. SCALE ought to run them well enough (both it and Proxmox use kvm under the hood), but Proxmox has a much more mature UI to handle them.

 

[Whattteva]

I'm not going to argue the point, because I haven't run TrueNAS for serious work under Proxmox. But if you can pass through the HBA and the NVMe devices to the VM, and allocate enough RAM/CPU to it, I'd expect it to be generally a safe thing to do. And if you later decided you didn't want TrueNAS virtualized any more, returning it to bare hardware would be quite straightforward.

This is exactly what I've done and so far it's been very solid running 24/7 the last 7 months. I'm still not going to give it a final stamp of approval though until I've run it for at least 2 years.

To OP: Worth noting that you should definitely be using CORE for Proxmox. My SCALE experimental VM is idling at 8-10% (mostly k3s process) for no apparent reason (it's running nothing and has no clients connected to it), whereas CORE idles at under 0-1% with actual clients connected to it. The server also runs a pretty beefy 10-core/20-threads Xeon Silver 4210T.

 

[Oedzesg]

Worth noting that you should definitely be using CORE for Proxmox.

Thanks for the tip!


I know it should be able to run TrueNAS stably in a vm.

But somehow it doesn't feel quite the right way to go.

In terms of hardware for a new server, I'm also a bit stuck.

Xeons lga-2011 are sufficient for a small price, but then the motherboards are not affordable, if they are available at all.

Everything that is still possible therefore seems to be a Chinese plate.
Anyone have any advice on this?

In addition, I'm also a bit on the trail of a Core i5-12600 eg.

I think fast enough for Truenas Core for storage without apps?

This one is still reasonably affordable, has ECC support and plenty of motherboards available.

Is that the better option?

 

[samarium]

If the 12600 has ECC support, make sure the motherboard supports ECC as well. Some allow ECC DIMMS, but don't actually enable ECC. Historically this has happened on some of the consumer boards. Make sure you have enough PCIe slots/lane for whatever you want to do.

I know what you meansby LGA-2011 good quality motherboards. I have a E5-2660 V4, and was looking for a SMC X10SRL-F for a reasonable price used, but can only find them for an unreasonable price new, and sometimes used is even more expensive. Utimately I found a EPYC 7302P/H11SSL-F used, and it can use the 2133 LRDIMMs I had for the E5 until I decide if I want to upgrade to 2933 or 3200. 7302P, H11SSL-F+IPMI, 4x LRDIMM=128GB, 1x m.2 NVME, 2x CPU FAN low RPM = 42W base idle at the wall if that is a consideration. I would prefer a H12SSL for PCIe 4.0, but more expensive.

 

[Oedzesg]

was looking for a SMC X10SRL-F for a reasonable price used, but can only find them for an unreasonable price new, and sometimes used is even more expensive

Yes, unbelievable, as if they do it on purpose: affordable CPU = priceless motherboard and vice versa.

So good, without trusting a Chinese board I installed Proxmox anyway :)

installation and setting up PCIE passthrough and then creating a Truenas core vm went great!

In addition, I read somewhere that it is not recommended to install plugins in a Truenas VM, so I made a separate VM in which I now run SabNZBD and PLEX! this is going great too.

Where I still have reservations about it is caching.

Both Truenas and PLEX use storage for temporary download and PLEX for metadata.

This also means that my Linux VM starts caching and then Truenas also starts caching since that's where the complete movies are moved to.

I know that temporary data in memory can be quickly freed up for other apps, but Proxmox simply sees the memory filled with temporary files in the VM as used up, so balooning is pointless.

For the temporary files I have now forwarded the motherboard NVME drive via PCIE passthrough to the vm running PLEX and SabNZBD and created a zfs pool in Proxmox for the virtual disks.

To prevent caching in Proxmox, could I do the following:

* Forward PCIE controller with 4 nvme disks via PCIE passthrough to Truenas and create a new pool in Truenas which I then link back to Proxmox for the VM disks?

* For PLEX and SabNZBD, use NFS shares directly in the docker container?



Linux VM with PLEX and SabNZBD:

VM in proxmox:

TrueNAS:

 

[WI_Hedgehog]

I know it should be able to run TrueNAS stably in a vm.
But somehow it doesn't feel quite the right way to go.

In my signature there's a link to: Resources List including Detailed Hardware and System Build Notes

In there you'll find: TrueNAS as a VM Under Proxmox or VMWare: Don't do this, Proxmox can operate as a SMB Server for Windows or NSF Server for Linux. VMWare sharing methods: (link1, link2)

The VM link leads to why you don't want to use those "free CPU cycles" according to someone other than myself, but that post has notes on memory and other useful information.

The real key however is in there under: Zebras All the Way Down (why simplicity is important)

You want to watch that. I have a 4-channel oscilloscope specifically for monitoring power in systems a.) being built, b.) doing something flaky/unpredictable. Limiting noise is a key in helping systems run reliably, and that doesn't account for bus timing being pushed to its limits, aging CPU power filter capacitors, special unexpected/unhandled rare exceptions, poorly written/malicious software, etc. I also constantly strive for cool-running systems to minimize component aging, thereby reducing the probability of zebras in the system.

You have to realize systems are being pushed to the limits of what they can do because everyone wants "faster, for free." Errors happen, and if the data is really important you want them to happen as infrequently as practical. (conversely, if you're gaming and crash every 6 months that may be acceptable IF it's not in a tournament; sometimes data loss is not that important)

ZFS and TrueNAS are about reliability because data corrupts, so much so hard drives with ECC aren't always reliable, RAID isn't reliable enough, and ZFS with end-to-end checking gets us there, though only with ECC memory.

 

[Oedzesg]

Proxmox can operate as a SMB Server for Windows or NSF Server for Linux

I never really thought about it, but Proxmox is of course also just Linux.

ZFS from command line takes some getting used to but is doable.

After moving a lot of data back and forth over the past few days, it all seems to be stable.

Now also a good option to run plex in an LXC and link the zfs dataset directly.
Unlike NFS or Samba, the library is now automatically updated when the media folders change.

Thanks a lot for pointing me in this direction!