TrueNAS as Xen Dom0

[zithro]

Hello, sorry to hijack another post title but it was in the archive and couldn't reply to it !

If you don't want to read it, here is a quick summary:
- Xen is available on FreeBSD
- bhyve and Xen can coexist

As I know that installing packages from the command line is not recommended, especially things like Xen which alter the OS deeply (kernel, boot), I wanted to know if this has been tested, and what the caveats I may run into !
Also, is Xen a virtualization platform considered at iX ? The cool thing is that contrary to KVM, Xen is multi platform and would work the same on either Core or Scale, and with nested virtualization could work on the cloud providers using Xen (like Amazon, last I read about).

Using TrueNAS as a dom0 would have some nice advantages: no need for another dom0, direct access to hardware (esp. drives controllers), ZFS-backed domUs without the need for NFS/iSCSI between dom0 and TrueNAS, easy migration of domU between Scale and Core, etc.

Thanks, and have a good day !

PS:
for context, I'm a home user/enthousiast (IT pro by trade) running two TrueNAS domU flawlessly since 4 years (one being the backup of the other). Both dom0 are Debian (9->10->11). All network activity is protected via two pfsense domU.
I run it on consumer hardware as "Network-in-a-box" setups. Yes, I know why "running virtualized is not recommended", I read all the posts.
I use vanilla Xen, and don't use any management tools for Xen, only small home made shell scripts, which I'd happily share for UI integration !

 

[Kris Moore]

I looked at that briefly a while back, but it would be somewhat difficult for us to make happen properly in a supported manner. Long term the plan is to focus VM work on SCALE with KVM, and not rock the boat on CORE.

 

[Patrick M. Hausen]

If you are interested in the ongoing work with virtualisation on FreeBSD, you might want to contact Michael Dexter (@michaeldexter on Twitter). He's holding regular bhyve developer and production users calls. He runs a mailing list to announce the dates, just ask, if you'd like to join.

 

[zithro]

Thank you for the answer, even if ... oh no ^^
I thought that a common virtualization platform would be easier to dev/support for you, not the oppposite !

somewhat difficult for us to make {happen properly}(1) in a {supported manner}(2)

May I ask more precisions ? Is 1 or 2 the problem, or rather 1+2 ? I mean, is the problem purely technical (1), or would it be a hassle to maintain it automagically (2), or both ?
I already see one major problem, the updates : I may be wrong but I imagine I'd have to re-install the Xen package after each update (as TrueNAS updater doesn't use pkg but rather pre-prepared packages, and the kernel is part of them) ?
You gonna tell me to try it myself, but is it technically feasible though ? Installing Xen isn't messing too much with the base install ?
Will the modified kernel alter or remove functionnality ?
Also, is it because KVM gets more attraction than Xen ? So has more users, more support from the industry, etc.

not rock the boat on CORE

Poor devil ;)
(or not, not sure to understand in which way you meant the expression, I'm no native EN speaker)

 

[zithro]

If you are interested in the ongoing work with virtualisation on FreeBSD

I am, even if my main FreeBSD use cases are for now only through TrueNAS core and pfSense !

He runs a mailing list to announce the dates, just ask, if you'd like to join

That would be really interesting, i'll try to contact him through this website though as I have no twitter account (but he seems inactive here).

Thanks for the pointer, Nachbar !

 

[Patrick M. Hausen]

He's not active here. You can DM me your email address and I'll ask him to add you. No spam, not even discussion, just announcements of the zoom meetings. And "production users" and "developers" are two separate groups with separate dates, although of course there is some overlap.

 

[Deleted User #319254452]

I confess I do far more reading than writing on the Forums!

Thank you Patrick and hello zithro. I have reached out to you by email and added you to the weekly bhyve call list, plus sent you a link to my script on GitHub that converts a FreeNAS 13.0 system into a Xen host. I am working through some issues on 14 with royger@ and am pleased to confirm that FreeBSD/Xen is indeed a best kept secret.

While ClonOS does support bhyve and Xen, it does not support them at the same time. I just confirmed this by trying to load bhyve's vmm.ko on a running Xen system. Fortunately, it is a graceful error rather than the panic that VirtualBox would cause.

One thing that Xen brings to the table is SeaBIOS, which supports far more operating systems than bhyve ever will (by design). I got TempleOS to run under Xen, but that is not a popular request, if in fact it has ever been requested.

You can find the weekly call agenda and minutes linked from bhyve.org and it is tracking active bhyve feature development.

Persistent EFI variables have landed in 14-CURRENT, which should help with Ubuntu guests, as they rely more heavily than most distributions on saved boot parameters. Other features like VGA and USB pass-through, save/restore, a paravirtualized clock, and many more are all in the works at their best pace in years.

None the less, Linux KVM is the go-to open source hypervisor for nesting guests within guests. Xen and Hyper-V have basic support for virtualizing themselves, but it was never a design goal of bhyve to add the complexity of nesting.

In the bigger picture... I love how TrueNAS SCALE is the CORE competitor that Rockstor never was. Proxmox and Ubuntu both offer Linux root on ZFS but iX has the best chance of getting it right!

 

[Deleted User #319254452]

I confess I do far more reading than writing on the Forums!

Thank you Patrick and hello zithro. I have reached out to you by email and added you to the weekly bhyve call list, plus sent you a link to my script on GitHub that converts a FreeNAS 13.0 system into a Xen host. I am working through some issues on 14 with royger@ and am pleased to confirm that FreeBSD/Xen is indeed a best kept secret.

While ClonOS does support bhyve and Xen, it does not support them at the same time. I just confirmed this by trying to load bhyve's vmm.ko on a running Xen system. Fortunately, it is a graceful error rather than the panic that VirtualBox would cause.

One thing that Xen brings to the table is SeaBIOS, which supports far more operating systems than bhyve ever will (by design). I got TempleOS to run under Xen, but that is not a popular request, if in fact it has ever been requested.

You can find the weekly call agenda and minutes linked from bhyve.org and it is tracking active bhyve feature development.

Persistent EFI variables have landed in 14-CURRENT, which should help with Ubuntu guests, as they rely more heavily than most distributions on saved boot parameters. Other features like VGA and USB pass-through, save/restore, a paravirtualized clock, and many more are all in the works at their best pace in years.

None the less, Linux KVM is the go-to open source hypervisor for nesting guests within guests. Xen and Hyper-V have basic support for virtualizing themselves, but it was never a design goal of bhyve to add the complexity of nesting.

In the bigger picture... I love how TrueNAS SCALE is the CORE competitor that Rockstor never was. Proxmox and Ubuntu both offer Linux root on ZFS but iX has the best chance of getting it right!

Also in the big picture... I have updated my lab for testing bhyve, Xen, Hyper-V, and TrueNAS SCALE in parallel. I am adding statistics to the agenda doc I mentioned as I collect them.