Bhyve - seeking best practices

[olddog9]

I've used the TrueNAS Core GUI to install and test several OSs on Bhyve and so far; Bhyve seems robust.
I'm comfortable with virtual environments and tested and experimented with lots of OSs on ESXi (vSphere 3, 4, 5, 6), VirtualBox, and some others.

The TrueNAS Core documentation is helpful but basic. I'm looking for more in-depth Bhyve commands & operational best practices pertinent to iXsystems' implementation in TrueNAS 12. I've looked at some of the FreeBSD virtualization docs and searched Bhyve.org, Klara Systems, and SmartOS (Joyent) for recent information.
The command line looks like the place to be ... for more granular control of Bhyve.

Can someone point the direction to, ... a link or reference (that I probably overlooked) ... for more technical administration & operation of Bhyve ?

Thank you to all for all constructive suggestions, recommendations, and general feedback.

 

[Patrick M. Hausen]

Can you give some examples of what you would want to achieve that is not covered by the UI?

I frequently

• create VMs
• boot from CD image to install Windows or Linux
• run these VMs in production

One hint that is probably not covered by the docs: use VirtIO devices for virtual disks and network interfaces.

 

[olddog9]

Hello Patrick, Thanks for the quick reply !

Examples:
1- Understand capabilities and best use of VirtIO.
Yes, more detailed information about the VirtIO devices would be very helpful, as currently implemented on TrueNAS Core 12.

2- Fix a mouse.
My most immediate puzzle is ... I installed FreeBSD 12.2 p7 on TrueNAS Bhyve 12 U5.1.
X11 was Installed with twm, but can't get mouse to work correctly. No buttons work, there is a movable pointer, but no focus. When the pointer is moved away from an open window to 'background' and then left button pressed, a menu appears, but nothing can be selected. To exit X11 and return to command line, I must stop the VM then reboot. ... Not exactly an elegant exit method. If you can offer suggestions, I'd be most grateful.
(FYI - Installed several Linux based distributions and ... Yes, the mouse works as expected on those OSs. Did not try any Microsoft product)

Some posts on FreeBSD.org strongly imply the mouse can only be configured for FreeBSD guests via command line at boot time. OK, I'm comfortable doing that, but Bhyve commands issued from command line impose possible conflicts with the Bhyve GUI settings. I can probably manage that, only IF I know where those GUI settings reside.

3- Salvage dead virtual machines. (vSphere was reliable, some developers were not.)
In past work with VMware vSphere (ESXi), when a virtual machine had corrupted snapshots, using the command line I could carefully edit those files to resurrect the VM from the dead. Or, when the *.vmx configuration file was deleted, as long as I had the valid image file (or files) I could carefully recreate the *.vmx file and consequently bring that VM back to life.

Is that type of technical documentation available for TrueNAS Core users .. like me ?
Does another open source project have similar Bhyve technical documentation that is applicable to TrueNAS Core 12 ?

I'm treading lightly ... to avoid trashing my nice new FreeBSD based TrueNAS Mini X+. (;<)

 

[Patrick M. Hausen]

OK - I never bothered to use any graphical UI on the VNC based console. That just sucks. In Windows 7 the mouse does not work at all, so you need to install with keyboard only. After installation I use RDP exclusively for Windows guests. My Linux guests don't have a UI at all. There is something working, modern and fast for X11, is there? I seriously don't know. Remote X11 sessions I remembers exist, but work only so-so ...

The virtual machine config is really only the stuff you entere in the UI. Take note of that and you are good to go. Everything else is in the Zvol on disk. So I would recommend reading on how to manage these - snapshots, copying, extending, ... you can start with just a copy of the Zvol (virtual disk) and recreate the machine from scratch. Bhyve and TrueNAS do not persist memory or anything else but the disk state across reboots and in snapshots.

HTH,
Patrick

 

[olddog9]

Patrick,
Thank you sir ! That is helpful.
I'll go exploring on a 'test' Zvol. I expect I'll learn a lot from that exercise.

For years, I built virtual machines for Microsoft, Linux, and Solaris developers, and they wanted a pretty desktop UI.
So, my habit was to always build a windowing interface for them.

Looks like I'll refresh my friendship with old friends, ... SSH and RDP ..., for primary administration tasks.

HTH = Yes, your guidance help a lot.

Thanks again !

 

[Ericloewe]

There is something working, modern and fast for X11, is there? I seriously don't know. Remote X11 sessions I remembers exist, but work only so-so ...

X11 forwarding over a GbE network works surprisingly well, to the point where I had a few people use our internal tools (think GIS/mapping for time-sensitive analysis of large images) over an SSH connection to gather feedback... And nobody thought it any different from the local machine. It is extremely latency-sensitive, though, and applications that do GUI stuff synchronously with real work will crawl to a halt over the internet.

x11vnc works well, but is finicky to setup. Old versions are buggy, but there has been development since ~late 2018ish.

 

[Patrick M. Hausen]

X11 forwarding over a GbE network works surprisingly well, to the point where I had a few people use our internal tools (think GIS/mapping for time-sensitive analysis of large images) over an SSH connection to gather feedback

Performance is ok on a LAN, but I vaguely remember colour palette problems, keyboard mapping problems and such things. Same for VNC. No built-in file transfer, either. RDP "just works" for Windows as if you were working at the console. Isn't there a modern equivalent for a remote Unix graphical environment?

 

[Ericloewe]

Isn't there a modern equivalent for a remote Unix graphical environment?

You'd think the Wayland people would have had that as a priority, wouldn't you? It's even worse there, incredibly. I spent some time researching the state of this stuff a few months back, in preparation for moving some workstation applications to containers running on servers, but nobody really seems to care that much.

but I vaguely remember colour palette problems, keyboard mapping problems and such things

That shouldn't be a problem for straight X forwarding, but I can imagine there being some edge cases...

 

[Samuel Tai]

You could also give xrdp a go, which installs an RDP server on Linux.

 

[Patrick M. Hausen]

You could also give xrdp a go, which installs an RDP server on Linux.

Since RDP is the ubiquitous de-facto standard on the client side I would definitely give that a try if I wanted a remote Unix desktop. Great, thanks!

 

[olddog9]

Patrick, Eric, and Samuel,

Thank you all ! Each of you shared valuable insights.

Patrick,
You gave me an idea about testing some other remote graphical solutions. If my tests yield decent results, I'll report back.

Eric,
"X11 forwarding over a GbE network works surprisingly well, to the point where I had a few people use our internal tools (think GIS/mapping for time-sensitive analysis of large images) over an SSH connection to gather feedback"
That is impressive ! ... Years ago, since I last used X11 forwarding ... I found it slow but functional ... or at best ... adequate. Never experienced the performance level you describe. Now remembering,... that X11 forwarding was probably over a 100 MB or 10 MB network. (I'm showing my age.)

Samuel,
Thank you. XRDP is an interesting suggestion. I will test that.

Best regards to all.
Chuck

 

[Phil1295]

Fell on this by case while looking for bhyve advise
VNC with xorg or x11 works perfectly over lan even at 4k. You just need a third party vnc server
I documented the steps for both methods in below thread. The thread also provides my proper startup script for vncserver on boot, which is really not documented

GUIDE: Setup secure GUI control on a jail using either VNC or X11

It is not easy to find guides for FreeBSD. In this guide, I explain with a near step by step how to: - install a KDE5 or GENOME 3 GUI in a jail - remote control the jail using either VNC or the X11 forwarding - install VNC as a service started on jail boot - properly setup a secure SSH...

www.truenas.com