Real world experiences with TrueNAS on ESX

[truenas_user210127]

I know that running TrueNAS on ESX (in a VM) is not ideal, was strongly advised against for a long time but is considered as "working" now when taking into account the right measures as explained in several articles (no scrub jobs, ...).

I have a use case at this moment where I cannot dedicate a physical machine to it but a server with VMware vSphere/ESXi 6.7 on it has 6 SATA disks that I want to use to serve NFS and SMB shares and have the option open to accelerate it using some ssd storage in the server and use some of the nice features of TrueNAS. In my case I would be able to expose all 6 individual RAID0 disks as RDM to the VM. Hardwarewise it is a Dell Poweredge R730xd with 24 SFF disk-slots. It has a Dell PERC H730P 12Gb SAS w/2GB NV Raid Controller.

My question is: Does anyone has this setup running in a prod environment for a longer time and has seen this running without big problems? Or do you think I will always at some point run into issues with this setup? If you have it running without probs what do you think are the most important changes to get right?

 

[sretalla]

the right measures as explained in several articles (no scrub jobs, ...).

I don't recall having seen the idea not to run scrubs... sounds like a terrible idea to me, scrubs are valuable and if not used, bit-rot will go un-corrected, potentially until there are no valid copies to recover from.

expose all 6 individual RAID0 disks as RDM to the VM

That also sounds like the completely wrong approach. You need to pass in the entire SATA controller/HBA with those disks attached or it's not direct access for TrueNAS.

a Dell PERC H730P 12Gb SAS w/2GB NV Raid Controller.

If I understand it correctly, this is a RAID controller... also not good.

Make sure you read and understand at least both of these before you go ahead (I don't expect you will find many folks willing to respond that they are running with success, since the ways you're proposing don't support success):

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[Herr_Merlin]

Yes it works, if done RIGHT. You need to have an HBA with the disk and pass that to the VM. Use only SMB / NFS for fileshare to the ESXi.
Know what you are doing.
Really know what you are doing.
Know a bit how trueNAS works.
And to everyone who wants to to do RDM disk or use a raid controller: https://www.youtube.com/watch?v=CXArovLJ60A

 

[truenas_user210127]

OK, I know the best option is to put the (non-RAID) controller in passthrough. But in this case I don't have that option. The server is a 730xd with an array of 25 SFF disks that have one bundled SAS connection to the RAID adaptor. The 19 other slots have disks in them that serve datastores to the ESXi as local storage. So I don't think I have the option of connecting just these 6 disks to a separate HBA controller.

The way I understood is that it also worked if all individual disks were put in a R0 (just one disk) and then exposed to the VM as RDM. I guess I understood it wrong. I can't find back the article.

 

[Herr_Merlin]

Just don't

 

[truenas_user210127]

OK, that is clear. Only way to do it in prod is have a dedicated HBA (non-RAID) that you can passthrough to the VM. Makes sense.

Do you know of any other software appliance that you can use as VM (so no ZFS) but that can leverage an SSD disk as acceleration. Because honestly that is the main reason why we want to use a product like TrueNAS. It allows us to have SATA (or SAS 10k) storage as capacity tier and SSD/NVMe as front-end acceleration/cache. Any suggestions?

 

[Herr_Merlin]

ZFS does not really tier.. slog is not a write cache...
L2ARC is a read cache but memory is usually more useful.

Windows Server with Storage Spaces and ReFS? but same applies use an HBA..