bhyve, ESXi, Jails

[thomas-hn]

Currently, I have some questions about the virtualization features of TrueNAS CORE. Maybe, someone could answer them.

Why are a lot of people here using ESXi to virtualize TrueNAS CORE? Why not simply using the virtualization via bhyve in TrueNAS?
What are the disadvantages of bhyve? So far I found some comments here and there which told "Virtualization in TrueNAS is not very stable", is this true? If there are any issues, what are the reasons? Would you recommend bhyve for virtualization of some (2-4) Linux and Windows machines?

If using Jails, is it possible that a Jail directly (direct file access; not as block device) accesses the data on a ZFS dataset outside of the Jail (is it possible to provide a storage location / dataset from outside the Jail into the Jail without using a network share?)

 

[Patrick M. Hausen]

Why are a lot of people here using ESXi to virtualize TrueNAS CORE? Why not simply using the virtualization via bhyve in TrueNAS?

I don't know. I run bhyve in production.

What are the disadvantages of bhyve? So far I found some comments here and there which told "Virtualization in TrueNAS is not very stable", is this true? If there are any issues, what are the reasons?

It only supports 64 bit OSs as guests. It only supports UEFI boot, yet does not handle persistent boot variables. Virtual Console leaves a lot to be desired. No guest additions, although clean shutdown via ACPI does work. Managing VMs is a bit awkward at times. Networking can get complex if VLANs, LACP etc. come into play and is not well documented - although all of this does work.

Would you recommend bhyve for virtualization of some (2-4) Linux and Windows machines?

We run around two dozen virtual machines - Ubuntu, CentOS, Windows 10, Windows Server 2016 - without issues. I'd recommend enabling serial console and deleting VNC for Linux guests, and not messing with VNC for Windows after installation and using RDP instead.

If using Jails, is it possible that a Jail directly (direct file access; not as block device) accesses the data on a ZFS dataset outside of the Jail (is it possible to provide a storage location / dataset from outside the Jail into the Jail without using a network share?)

But of course. That is the point of jails. You can have local mounts of arbitrary directories in arbitrary locations of a jail. Multiple jails even. So e.g. 5 jails all mounting one and the same directory/dataset. One jail running MySQL server and creating the local socket in there. The other 4 running MySQL client applications and accessing the socket without the need to go over TCP ...

HTH,
Patrick

 

[sretalla]

Why not simply using the virtualization via bhyve in TrueNAS?

Limitations to passthrough of devices. Lack of enterprise management tools. Lack of available knowledge on the marketplace to design/manage/run bhyve. Lack of Hypervisor maturity/cluster functions/guest tools functionality. Not a Tier 1 Hypervisor. (also see the limitations list from @Patrick M. Hausen )

 

[zizzithefox]

Currently, I have some questions about the virtualization features of TrueNAS CORE. Maybe, someone could answer them.

Why are a lot of people here using ESXi to virtualize TrueNAS CORE? Why not simply using the virtualization via bhyve in TrueNAS?
What are the disadvantages of bhyve? So far I found some comments here and there which told "Virtualization in TrueNAS is not very stable", is this true? If there are any issues, what are the reasons? Would you recommend bhyve for virtualization of some (2-4) Linux and Windows machines?

If using Jails, is it possible that a Jail directly (direct file access; not as block device) accesses the data on a ZFS dataset outside of the Jail (is it possible to provide a storage location / dataset from outside the Jail into the Jail without using a network share?)

Because jails? Good (not as good as containerization in linux but good). Bhyve? BAD.
You don't want to use bhyve for virtualization unless you literally don't give a s**t about performance whatsoever.

Pick one: Esxi, hyper-v, xcp-ng/xenserver. They are all better than bhyve in every respect (well maybe hyper-v depends because you know, NTFS vs ZFS).

Don't get me wrong, I am using bhyve for my windows domain controller at home and linux machine, because it is running all the time and I don't want to spend other money on electricity: it works fine. It is really stable, once it is running. But the performance you can get from the other platforms, you can only dream of. The difference is appaling.

Just to give you an example, I had nextcloud running in linux in a bhyve vm on truenas 12. I switched it to a jail. The performance roughly quadrupled.

 

[Morris]

bhyve virtualization is simple to setup and stable. Unless you run into the limitations the others have mentioned it works well. The VNC console is very limited yet enough for you to watch your system boot and make simple changes. I agree with others that say to connect another way once the system is up. I'm running two Windows 10 VMs with Plex and network monitoring on one and Channels DVS running on another. Performance is fine for all applications. I've run Ubuntu as well and it worked fine.

 

[Patrick M. Hausen]

Just to give you an example, I had nextcloud running in linux in a bhyve vm on truenas 12. I switched it to a jail. The performance roughly quadrupled.

You would need to compare with an identical Linux VM in ESXi. Of course a jail is way faster than both.

 

[zizzithefox]

You would need to compare with an identical Linux VM in ESXi. Of course a jail is way faster than both.

That is exactly what I did. I was running that, before switching to bare metal Truenas: everything as a vm under esxi. There is rarely any noticeable performance penalty for running a vm in esxi, whatever the os, whatever the configuration. I mean, there is, but you CANNOT see it with your naked eyes, you need to measure it and it's pretty small.
You have never used esxi, have you?
A jail is NOT way faster than a virtualized linux vm under esxi. It is MARGINALLY faster.

I did it all. I measured everything every time (but I repeat, you could see it with your naked eyes):
- at least 25% drop in network performance
- at least 50% drop in storage performance
- at least 50% drop in CPU performance.
Talking same hardware all the time here.
Tested about the same performance drop with:
- nextcloud
- plex
- sqlserver (yes I run sqlserver under linux and windows)

I am not shitting on bhyve: it is a promising piece of software, as I stated before, it DOES work fine.
But you need not care about performance AT ALL.
Please don't start telling me about virtio, whatever. Tried all the tricks in the book for a year than I had enough.

I bet 100 € against anybody that can prove to me you are not leaving around 50% of perfomance on the ground by using bhyve against vmware or hyper-v. This probably holds for XCP-ng, too. Virtualbox, even.
OK, maybe it's my hardware. Maybe on newer hardware it is better, I don't know. I don't care, I have already wasted too much time on bhyve with little return. Please don't mislead people into thinking it is as GOOD as other products.

 

[zizzithefox]

bhyve virtualization is simple to setup and stable. Unless you run into the limitations the others have mentioned it works well. The VNC console is very limited yet enough for you to watch your system boot and make simple changes. I agree with others that say to connect another way once the system is up. I'm running two Windows 10 VMs with Plex and network monitoring on one and Channels DVS running on another. Performance is fine for all applications. I've run Ubuntu as well and it worked fine.

Yes, it is probably fine for your needs. Did you read the part where I said it is OK, but you are loosing a ton of performance? That's just what it is.
By the way, done plex too, sorry, and PLEX + WINDOWS + Bhyve is the worst configuration I have ever tried.
Are you trolling? You must be trolling... Come on.