Ubuntu VM do not work after 11.3 upgrade

[koff1]

Hi
I am fairly new with Freenas and recently I upgraded to 11.3 (thought this was a minor thing… my mistake), everything seems to be working except bhyve. I had an Ubuntu Desktop 18.04 LTS VM running, but after the upgrade, it does not come farther than the GRUB menu on start-up. Whatever I choose here the screen only stays black, nothing happens, tried to leave it overnight but nothing. I also tried to make a new VM and install Ubuntu one more time; however, I cannot get past the GRUB Install menu, same thing happens, whatever I choose the screen just go blank.

I am kind of stuck her, any help would be appreciated.

 

[KrisBee]

It could be a permission problem, i.e not being able to read the zvol used by the virtual machine. What is the path ( /mnt/....... ) to the dataset holding the zvol? Can root traverse the entire path?

You might get some clues from the output of using this command at the FreeNAS shell tail -f /var/log/middlewared.log when you start your VM.

 

[koff1]

Hi. Thanks for the reply and sorry for my late reply. The zvol is in /dev/zvol/Bjornbert/VM/Ubunrtu-z9p6sd. I seem to able to transverse the path, but just to be sure I lifted all restrictions on the zvol, but didn't work. Nothing jump out at me in the log, except that in the start it say middlewared.service_exception.CallError: [EFAULT], google wasn't particularly helpful on that one. I have attached the log file.

 

[koff1]

And in case you dont want to open the file..

Last login: Mon Feb 24 17:56:32 on pts/9
FreeBSD 11.3-RELEASE-p5 (FreeNAS.amd64) #0 r325575+8ed1cd24b60(HEAD): Mon Jan 27 18:07:23 UTC 2020

FreeNAS (c) 2009-2020, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.

For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@freenas[~]# tail -f /var/log/middlewared.log
middlewared.service_exception.CallError: [EFAULT] VM Ubunrtu failed to start: 24/02/2020 17:56:53 tight : 92 | 5552/ 41473104 (100.0%)
24/02/2020 17:56:53 RichCursor : 1 | 255/ 255 ( 0.0%)
24/02/2020 17:56:53 TOTALS : 103 | 5867/ 41473419 (100.0%)
24/02/2020 17:56:53 Statistics events Received/ RawEquiv ( saved)
24/02/2020 17:56:53 KeyEvent : 2 | 16/ 16 ( 0.0%)
24/02/2020 17:56:53 PointerEvent : 56 | 336/ 336 ( 0.0%)
24/02/2020 17:56:53 FramebufferUpdate : 6 | 60/ 60 ( 0.0%)
24/02/2020 17:56:53 SetEncodings : 1 | 68/ 68 ( 0.0%)
24/02/2020 17:56:53 SetPixelFormat : 1 | 20/ 20 ( 0.0%)
24/02/2020 17:56:53 TOTALS : 66 | 500/ 500 ( 0.0%)
[2020/02/24 17:58:26] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update-master.ixsystems.com:443
[2020/02/24 17:58:27] (DEBUG) urllib3.connectionpool._make_request():437 - https://update-master.ixsystems.com:443 "GET /FreeNAS/trains_redir.json HTTP/1.1" 200 153
[2020/02/24 17:58:27] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 -TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/trains.txt', 'https://update-master.ixsystems.com/FreeNAS/trains.txt'])
[2020/02/24 17:58:27] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update.ixsystems.com:443
[2020/02/24 17:58:28] (DEBUG) urllib3.connectionpool._make_request():437 - https://update.ixsystems.com:443 "GET /FreeNAS/trains.txt HTTP/1.1" 301 169
[2020/02/24 17:58:28] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update.freenas.org:443
[2020/02/24 17:58:28] (DEBUG) urllib3.connectionpool._make_request():437 - https://update.freenas.org:443 "GET /FreeNAS/trains.txt HTTP/1.1" 200 528
[2020/02/24 17:58:28] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 -TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/trains.txt', 'https://update-master.ixsystems.com/FreeNAS/trains.txt']): Read 528 bytes total
[2020/02/24 17:58:28] (DEBUG) freenasOS.Configuration.AvailableTrains():1001 - Input line `' is unparsable
[2020/02/24 17:58:28] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 -TryGetNetworkFile(['https://update-master.ixsystems.com/FreeNAS/FreeNAS-11.3-STABLE/LATEST'])
[2020/02/24 17:58:28] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update-master.ixsystems.com:443
[2020/02/24 17:58:29] (DEBUG) urllib3.connectionpool._make_request():437 - https://update-master.ixsystems.com:443 "GET /FreeNAS/FreeNAS-11.3-STABLE/LATEST HTTP/1.1" 200 1982
[2020/02/24 17:58:29] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 -TryGetNetworkFile(['https://update-master.ixsystems.com/FreeNAS/FreeNAS-11.3-STABLE/LATEST']): Read 1982 bytes total
[2020/02/24 17:58:29] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 -TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem'])
[2020/02/24 17:58:29] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update-master.ixsystems.com:443
[2020/02/24 17:58:30] (DEBUG) urllib3.connectionpool._make_request():437 - https://update-master.ixsystems.com:443 "GET /updates/ix_crl.pem HTTP/1.1" 200 1028
[2020/02/24 17:58:30] (DEBUG) freenasOS.Configuration.CheckFreeSpace():77 - CheckFreeSpace(path=/tmp/tmp3_mwgqii.pem, pool=None, required=1028)
[2020/02/24 17:58:30] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 -TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem']):Read 1028 bytes total
[2020/02/24 17:58:30] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 -TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt', 'https://update-master.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt'])
[2020/02/24 17:58:30] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update.ixsystems.com:443
[2020/02/24 17:58:31] (DEBUG) urllib3.connectionpool._make_request():437 - https://update.ixsystems.com:443 "GET /FreeNAS/Validators/ValidateUpdate-0frmasef.txt HTTP/1.1" 301 169
[2020/02/24 17:58:31] (DEBUG) urllib3.connectionpool._new_conn():959 - Startingnew HTTPS connection (1): update.freenas.org:443
[2020/02/24 17:58:31] (DEBUG) urllib3.connectionpool._make_request():437 - https://update.freenas.org:443 "GET /FreeNAS/Validators/ValidateUpdate-0frmasef.txt HTTP/1.1" 200 185
[2020/02/24 17:58:31] (DEBUG) freenasOS.Configuration.CheckFreeSpace():77 - CheckFreeSpace(path=/tmp/tmpd3zar47m, pool=None, required=185)
[2020/02/24 17:58:31] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 -TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt', 'https://update-master.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt']): Read 185 bytes total
[2020/02/24 17:58:58] (INFO) VMService.__set_guest_vmemory():753 - ===> SettingARC FROM: 65858416640 TO: 61563449344
[2020/02/24 17:58:58] (DEBUG) VMService.vm_5.run():179 - ====> NIC_ATTACH: igb1
[2020/02/24 17:58:58] (DEBUG) VMService.vm_5.run():291 - Starting bhyve: bhyve -A -H -w -c 4 -m 4096 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm5A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4,e1000,tap0,mac=00:a0:98:49:e2:88 -s 29,fbuf,vncserver,tcp=192.168.1.39:5716,w=1920,h=1080,, -s 30,xhci,tablet-s 3:0,virtio-blk,/dev/zvol/Bjornbert/VM/Ubunrtu-z9p6sd 5_Ubunrtu
[2020/02/24 17:58:58] (DEBUG) VMService.vm_5.run():304 - ==> Start WEBVNC at port 5616 with pid number 21081
[2020/02/24 17:58:58] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:58:58 Listening for VNC connections on TCP port 5716
[2020/02/24 17:58:58] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:58:58 Listening for VNC connections on TCP6 port 5716
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 other clients:
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 Client Protocol Version 3.8
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 Protocol version sent 3.8, using 3.8
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 rfbProcessClientSecurityType: executing handler for type 1
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 rfbProcessClientSecurityType: returning securityResult for client rfb version >= 3.8
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 Pixel format for client 192.168.1.39:
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 32 bpp, depth 24, little endian
[2020/02/24 17:59:09] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:09 true colour: max r 255 g 255 b 255, shift r 16 g 8 b 0
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 Using image quality level 6 for client 192.168.1.39
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 Using JPEG subsampling 0, Q79 for client 192.168.1.39
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 Using compression level 2 for client 192.168.1.39
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 Enabling NewFBSize protocol extension for client 192.168.1.39
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 Enabling LastRect protocol extension for client 192.168.1.39
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEFE)
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFECC)
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEC8)
[2020/02/24 17:59:10] (DEBUG) VMService.vm_5.run():313 - Ubunrtu: 24/02/2020 17:59:10 rfbProcessCl

 

[KrisBee]

Not sure my brain's in gear today. The virtual machine looks to have booted OK and WEBVNC started normally. I seem to remember coming across this when messing around with VMs myself but my memory is failing me right now. Help, I need re-charging!

 

[koff1]

I have just had some more time trying to experiment with this.
I have now tried Ubuntu, Lubuntu and arclinux. They all behave the same way, I can’t get pass the first grub/install screen, However I tried to install windows 10 and that worked, but had to run it on 1 virtual processor, I got the fault “multiprocessor configuration not supported” whenever I increased the number of processors.

So far it looks like this is a linux related problem, have any of you had any luck in installing a linux distro on freenas 11.3 bhyve?

 

[KrisBee]

Should have aked before but what is your hardware? I did get around to creating a fresh install of FN11.2-7U in a VM and installed both ubuntu 18 server and desktop in side that. After updgrading to FM11.3 both bhyve VMs run OK. What browesr are you using? It's not something stupid like the resolution of the VNC device?

 

[koff1]

Hi, thanks for your test. Hardware is dual Opteron 6328 on a Tyan S8236 motherboard. 64GB RAM and 4 8TB HDD + system drive.

The data pool is encrypted if that matters. I have tried in Opera, Vivaldi and Firefox. I am also using Ultra VNC Viewer.

I do not think it is a resolution problem as I can see something in cent OS that I just tried to install. I get a – in the upper LH corner of the screen after selecting install, I believe it is supposed to be flashing but it just freeze.

But again, Windows is working which is a bit weird (but a nice test as it proves that it is not a permission problem ), as I am unable to install any of the linux OSes.

 

[KrisBee]

Encrypted pool - so this is unlocked in Fn11.3 as expected?

 

[koff1]

Yes, I can access all data, and I have created new zvols for testing.

 

[KrisBee]

There seem to be a few other reports of VM problems in FN11.3, e.g:

VM not working after 11.3 upgrade | "Host does not exist"

Ubuntu VM installed on 11.2-U7 was working well, but after I upgrade to 11.3-STABLE, I can no longer connect via VNC or PuTTY. Freenas > Virtual Machines, shows it to be on. I've restarted the VM, powered off then on again. I also upgraded my pools which had no affect. Seeing this in the...

www.ixsystems.com

11.3 Bug? - VMs crashing when initiating a VNC session

FreeNAS 11.3 Supermicro X9DRi-LN4F+ 2x E5-2680v2 128GB Reg ECC DDR3-1600 Don't know where this should go. please advise if this is better posted somewhere else. I just upgraded my install from 11.2-U8 to 11.3 via the manual upgrade option from 11.3 installation media (USB stick). Neither...

www.ixsystems.com

 

[pttymuth]

I have just had some more time trying to experiment with this.
I have now tried Ubuntu, Lubuntu and arclinux. They all behave the same way, I can’t get pass the first grub/install screen, However I tried to install windows 10 and that worked, but had to run it on 1 virtual processor, I got the fault “multiprocessor configuration not supported” whenever I increased the number of processors.

Thank you for posting this.

Your reply here is what I found when I googled "windows bhyve multiprocessor configuration not supported" after seeing my Windows Server 2016 VMs no longer booted after upgrading to 11.3. Unfortunately for me, my VMs still do not boot after adjusting to only 1 CPU. The hardware is a SuperMicro H8DGU-F with 2x Opteron 6366HE CPUs and 64GB RAM. So I've rolled back to the 11.2 boot environment.

Perhaps this is only a problem for older AMD machines? I have Windows Server 2016 bhyve VMs working fine with multiple CPUs on Supermicro X10SLM/Intel Xeon E3-1231v3 & Supermicro X9DRT/Intel Xeon E5-2628Lv2 FreeNAS 11.3 installations.

 

[pul718]

Thank you for posting this.

Your reply here is what I found when I googled "windows bhyve multiprocessor configuration not supported" after seeing my Windows Server 2016 VMs no longer booted after upgrading to 11.3. Unfortunately for me, my VMs still do not boot after adjusting to only 1 CPU. The hardware is a SuperMicro H8DGU-F with 2x Opteron 6366HE CPUs and 64GB RAM. So I've rolled back to the 11.2 boot environment.

Perhaps this is only a problem for older AMD machines? I have Windows Server 2016 bhyve VMs working fine with multiple CPUs on Supermicro X10SLM/Intel Xeon E3-1231v3 & Supermicro X9DRT/Intel Xeon E5-2628Lv2 FreeNAS 11.3 installations.

Full disclosure, I am technical but not great in the freebsd or linux worlds. That said, I have been having this issue (Debain 10 bhyve VM not running, cannot connect to VNC either) along with my iocage jail becoming corrupt. My specs are below and from what @pttymuth has posted makes me wonder if it is trully related to the AMD Opterons.

I have tried upgrading to the following version 11.3-RELEASE, 11.3-U1 and 11.3-U2 from 11.2-U8. Each of which I have my debian VM show as started/running from the FreeNAS web interface, but I cannot ssh to it nor connect via VNC. I have even tried pausing the reboot waiting on a VNC connection.

My PMS iocage jail will show as corrupt and in accessible. I have tried to fetch the latest iocage version and tried to go back to 11.2, neither works. I neglected to capture the output of trying to start the iocage from my FreeNAS cli, but can do that if needed.

The only way to get back to a working state has been to revert back to the 11.2-U8 environment. This immediately resolves my debian VM issue, but I have to copy the defaults_backup.json to defaults.json in my iocage to get the PMS jail functioning again. Any thoughts or help would be appreciated. I am willing to submit a bug if that would be helpful, but might need some direction on what information I should supply.

OS Version:
FreeNAS-11.2-U8
(Build Date: Feb 14, 2020 15:55)

Processor:
AMD Opteron(tm) Processor 6380 (16 cores)

Memory:
32 GiB

Motherboard:
SuperMicro H8SGL

 

[Basil Hendroff]

A couple of issues surfaced for Ubuntu VMs in FreeNAS 11.3-U2. Review these posts for possible workarounds:

Ubuntu Virtual Machine No Longer Works
Mouse stops working in VM (ubuntu 18.04.4)

 

[pul718]

I checked am I am already using virtio vs AHCI, but the issue(s) persists. Also, those potential solutions do not address the IOcage issue where it comes up corrupt after moving to 11.3, I am willing to try any further troubleshooting.

 

[pul718]

I hate to necro an old thread, but I am still in the same situation and unable to upgrade. Not sure if pttymuth was able to find a solution or not. I guess I will just wait for the next release and hope I can update ok. Perhaps someone has a suggestion other than what we have thus far attempted.