hypervisor + virtualized Truenas

[phier]

Hello,
i was planning to install truenas natively on supermicro+xeon 3, but i saw some discussion that ppl run truenas virtualize and therefor can use HW for virtualization of other VMs - basically no need to have another HW as hypervisor.

Is it possible to say if such a solution is supported by truenas, if its also safe / stable etc? I saw they virtualize it using ESXi.
Also i dont have any HBA card, so will such a virtualization somehow passthrought mainboard SATA directly to virtualized environment?

thank you!

 

[Alecmascot]

I think this is documented in the "Resources" section.

 

[phier]

@Alecmascot which section? https://www.truenas.com/docs/core/ i cant see that.

 

[Patrick M. Hausen]

This is the best write up on the topic so far:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

You can get away without an extra HBA if your mainboard allows for the SATA controller to be passed through to the VM in ESXi and you have some other means of booting - a small NVMe device for example.
The point is that you MUST pass through the controller, not individual drives.

Alternatively while the hypervisor builtin in TrueNAS is by no means as capable as ESXi it is more than sufficient for e.g. a single Linux or FreeBSD VM at home. I actually run a dozen production VMs @work on TrueNAS including our active directory controllers. We have replaced ESXi with TrueNAS almost entirely.

 

[phier]

@Patrick M. Hausen agree, well i thought there is some official documentation regardin that.
how can i find out if mb allows SATA controller to be passed? I have that one https://www.supermicro.com/en/products/motherboard/X11SSM-F

but once SATA can be passed through - is that safe/stable setup? thanks

Well what hypervisor is there bhyve? In that case is there a point to have that esxi setup?

I need to run lets say ~5 VMs... but i am running bhyve on my old freenas9x and its not much.

thanks

 

[anodos]

This is just personal reply, not as ix-employee (like most of my forums replies - I'm not a moderator).

There have been a a few of bug tickets recently (performance / stability related) that were assigned to me where users were virtualizing TrueNAS on top of proxmox (and a few other virtualization platforms) and presented the TrueNAS VM with limited resources and a single virtual HDD for the storage pool. I closed the tickets and asked the authors to reopen if they can reproduce on a supported configuration. Generally speaking we try to help users in our bug ticketing system (even though it's not a support system), but there are limits to how I will go when the setup is pretty far outside of what we consider "supported".

One of our kernel developers runs / tests all sorts of weird hardware and software combos because they can tickle interesting bugs. If you're not into that sort of thing, it's probably better to stick with well-vetted hardware / software configurations.

 

[Patrick M. Hausen]

how can i find out if mb allows SATA controller to be passed?

Install ESXi in a scratch device and try. X11 is a pretty recent and well supported platform so that might work. You need a completely alternative path to some storage device, e.g. NVMe, to boot ESXi - all SATA connectors will probably end up "inside the VM" once you enable pass through.

Well what hypervisor is there bhyve? In that case is there a point to have that esxi setup?

The hypervisor is bhyve and I am really content with management and performance running Ubuntu 16.04 up to 20.04, Windows 10 and Windows Server 2016. I don't see much of a point of another hypervisor layer for that reason. Other people seem to disagree and prefer a more mature product over byhve.

Positively put, bhyve is the "legacy free" hypervisor. That keeps the code base small, performance (hopefully) high, and enabled FreeBSD to enter the hypervisor "market" as late as they did in the first place.
On the downside: no 32 bit guests, no BIOS boot, best with paravirtualised (VirtIO) disk and network, debatable stability when not using VirtIO. Some need some of these features and so decide to pick ESXi or Proxmox. Then there's the quirky network setup - no vSwitches with VLANs, for example. Although you can create one bridge interface per VLAN to reach a configuration with VLANs. But it's quirky.

If you already know your VMs run fine in bhyve, there really is no point. Just stick to TrueNAS and bhyve. It's great

Kind regards,
Patrick

 

[jgreco]

virtualizing TrueNAS on top of proxmox (and a few other virtualization platforms) and presented the TrueNAS VM with limited resources and a single virtual HDD for the storage pool.

So, this. As the author of the linked guide to "how to virtualize" above, this is really the same situation I was seeing almost ten years ago. People make choices when virtualizing that turn out to be ill-advised. The fact that my article is nearly ten years old does not make it irrelevant. We know that Proxmox has immature PCIe passthru, even if it works for many people. We know that virtualization platforms tempt people to under-resource their NAS. We know that most hypervisors cope poorly with disk errors on virtual disks. I've been trying to help people make this stuff work for a decade now. Listen to the advice linked to above.

I can tell you that the path listed in that document has worked very well over the years. It's the golden path to FreeNAS/TrueNAS virtualization success.

well i thought there is some official documentation regardin that.

No, though Josh Paetzel did post a blog post some years back trying to push back on the common forum guidance to avoid unnecessary virtualization. The developers have always maintained a bit of an attitude that users should be able to do as they like, so I've gotten various blowback over the years over stuff like my anti-RAID-controller stance, raising the minimum memory requirement to 8GB, and perhaps the most offensive of these was the virtualization pushback. Especially since I already had posted the "how to virtualize" guidance. The main problem is that the developers sometimes forget that people come to ZFS for stability and get All P***ed Off when the thing eats their data. Here in the forums, we are frontline support for the product, and so we're really quite focused on advocating for what is known to work well, while strongly discouraging those things that are known to be problematic, because we want the positive outcomes.

 

[phier]

hello,
i went thru the guide to "how to virtualize", but i think its so generic?

"DO use PCI-Passthrough for a decent SATA controller or HBA. We've used PCI-Passthrough with the onboard SAS/SATA controllers on mainboards, and as another option, LSI controllers usually pass through fine. Get a nice M1015 in IT mode if need be."
1)no idea how to find out whats a decent SATA controller

2) as only suggested hypervisor - its looks like its ESXi only? So do i have to use it .. as its not for free or also vSphere Hypervisor can be used?

thanks

 

[Patrick M. Hausen]

no idea how to find out whats a decent SATA controller

Try the one on your mainboard. If it works, it's decent. If it doesn't, reconsider your virtualisation approach or buy an HBA.

as only suggested hypervisor - its looks like its ESXi only? So do i have to use it .. as its not for free or also vSphere Hypervisor can be used?

Yes, because that is the one known to work. And ESXi is the vSphere hypervisor. And it's free for development or home lab. Not for commercial installations. For these VMware essentials for three ESXi nodes can be purchased for less than one grand every three years.

 

[phier]

i can see that its free for home use; https://www.cbtnuggets.com/blog/cer...xi-free-vs-paid-a-look-at-license-limitations

@Patrick M. Hausen - define if its work?:)

 

[jgreco]

hello,
i went thru the guide to "how to virtualize", but i think its so generic?

You would prefer I get very specific in unnecessary ways?

"DO use PCI-Passthrough for a decent SATA controller or HBA. We've used PCI-Passthrough with the onboard SAS/SATA controllers on mainboards, and as another option, LSI controllers usually pass through fine. Get a nice M1015 in IT mode if need be."
1)no idea how to find out whats a decent SATA controller

Well, now, that really is the trick, now, isn't it. Intel AHCI, Intel PCH SCU, non-knockoff ASM106x *without* a SATA port multiplier, LSI HBA with the proper firmware, and a few other random things are known to work well. Unfortunately the PC marketplace is full of crappy gear, so lots will not. The real crap of it all is that lots of stuff will work sorta-okay until some adverse situation comes along, at which point things will go sideways, sometimes terribly.

2) as only suggested hypervisor - its looks like its ESXi only? So do i have to use it .. as its not for free or also vSphere Hypervisor can be used?

thanks

As noted above, vSphere Hypervisor *IS* ESXi. Given a few more years, Proxmox will probably make the list too, but right now too many people are trying to use dodgy old kit and it doesn't have all the sanding and polishing that VMware has put into ESXi to *make* it work. Proxmox today is about where ESXi was maybe ten years ago. Risky, potentially doable, but sort of off-roading it.

 

[Patrick M. Hausen]

define if its work?:)

Install ESXi, activate pass through for the device, install TrueNAS, create pool from the disks connected to that controller - if all of this works, things are looking good, already. Then destroy the pool and do a proper burn-in of the disks in TrueNAS. If the system survives that, go.

If any of the early steps fails, you still have not spent any additional money on hardware, and you can try installing TN on the metal and using the bhyve hypervisor.

In my experience these virtualisation workloads fail early and quite spectacularly. You will notice. I have never had creeping data corruption - yet

 

[jgreco]

In my experience these virtualisation workloads fail early and quite spectacularly. You will notice. I have never had creeping data corruption - yet

That's a bit optimistic. I've had stuff fail after partial days of uptime as recently as last year. The older LSI 2008 controllers seem to have some issues with larger memory NAS VM's, Someone mentioned that this isn't a problem on the 2308's or newer 3008's etc. It's still best practice to treat it like you were burning in a bare metal platform and test the crap out of it.

 

[Patrick M. Hausen]

In this particular thread I recommend to the OP to install TrueNAS on the hardware and try how far they'll get with bhyve. With all of my VMs I concede that setup and installation might be quirky, but I don't have VMs crashing or anything like that at all. All running 24x7.

 

[Stux]

Alternatively while the hypervisor builtin in TrueNAS is by no means as capable as ESXi it is more than sufficient for e.g. a single Linux or FreeBSD VM at home. I actually run a dozen production VMs @work on TrueNAS including our active directory controllers. We have replaced ESXi with TrueNAS almost entirely.

I have done a similar thing to my old ESXI/TrueNAS hosts too. They're now TrueNAS bare-metal again, with BHyve for the VMs, even including pfSense with hardware ethernet pass-thru.

 

[elvisimprsntr]

I ran CORE on QNAP QTS Virtualization Station of all things, but that was just to kick the tires before completely replacing QTS with CORE. Although, I was able to get a Windows VM Guest working on CORE, the one thing I missed from QTS VS was USB device passthough.
I haven't tired SCALE VM, because I installed Proxmox (which supports USB device passthrough) on one of my QNAP NAS and migrated my Windows VM Guest (not fun) to Proxmox. Windows VM Guest runs noticeable better under Proxmax that it ever did under QTS VS or CORE.

 

[phier]

In this particular thread I recommend to the OP to install TrueNAS on the hardware and try how far they'll get with bhyve. With all of my VMs I concede that setup and installation might be quirky, but I don't have VMs crashing or anything like that at all. All running 24x7.

well i am not sure,
I wanted to run also Router, pfsense (firewall) etc,
another Home-assistant; rest of the VM can do bhyve... but afraid about these mentioned, that there wont be images for bhyve etc.

@stuck - why you are back on truenas as baremetal - any issues? do you run pfsense as image for bhyve and how did u done that hardware ethernet pass-thru?



thanks

 

[Patrick M. Hausen]

Doesn't pfSense have a way to bootstrap over stock FreeBSD? OPNsense does. But you are right, for a firewall passthrough of network devices would be preferrable. I guess your best bet is ESXi for that.

 

[phier]

well still not sure here, in case i will run ESXi then where i will store all the images? i need another drive for ESXi images?
thanks!