Running Linux Virtual Machines

[NASbox]

After wasting almost 2 days trying to get a Linux Virtual Machine to run, I set out to document the process so that hopefully others will be spared from the frustrations of this task.

From what I can see I have no choice but to use UEFI if I'm going to use the GUI as the serial doesn't seem to do a very good job of emulating an old BIOS style machine (or am I missing something here) and most common Linux distros are not set up for serial console.

After trying to boot a Tunkey Linux CD (DEBIAN-I don't think it's UEFI), I tried a Ubuntu Server 16.04LTS network install, which I was able to get to boot and install. The problem came when I attempted to boot the VM. It simply hung.

I used the rescue mode of the install CD to document the resulting configuration and experiment.

Here are the steps I followed:

1. Create Virtual Machine and Devices:

Install Ubuntu using DEFAULT settings: (The insert was made with the rescue mode and shows the key contents of the EFI boot partition.

Finally I attempted to use the Boot Maintenance Manager to boot the various EFI files.

I tried running: BOOT/bootx64.efi, ubuntu/grubx64.efi, ubuntu/shimx64.efi, ubuntu/nmx64.efi. The result was a message that flashed too quickly to see and a black screen that never recoved.

When I ran nmx64.efi I got the Shim UEFI Key Management screens - I went as far as displaying the Select a Key screen shown here, but backed out before applying the change.

I've gone as far as I can with this, I'm hoping someone can help me get this going.

Thanks in advance for any assistance.
---

 

[dlavigne]

This is a bug that has been fixed for BETA1, due out next week: https://redmine.ixsystems.com/issues/26708 .

 

[NASbox]

This is a bug that has been fixed for BETA1, due out next week: https://redmine.ixsystems.com/issues/26708 .

Thanks Dru for the "heads up" on that.... If I understand RedMine correctly [11.2 (Due in about 2 months (03/09/2018))], I'm looking at the beginning of September (at best) before I see this fix in something stable? Did I get that right, or would this be rolled into 11.1U6? I depend on my FreeNAS server and I'm a FreeBSD noob, so I avoid beta like the plague.

I saw the thread:
[HOWTO] How-to Boot Linux VMs using UEFI
https://forums.freenas.org/index.php?threads/howto-how-to-boot-linux-vms-using-uefi.54039/
which seems to be mainly Centos but there is some Ubuntu mixed in.
Using the rescue disk I executed cp -v ubuntu/grubx64.efi BOOT/bootx64.efi from /target/EFI
I'm wondering if the directory efi is missing, or if if the EFI Partition contains EFI at the root, and then mounts the partition on the root filesystem at /boot/efi?

Based on the information provided (ask if you need more), can anyone suggest how I can work around this issue till it's fixed, or do I need to wait until I get an update in 2-3 months?

Thanks in advance for any advice/guidance/hints/suggestions.

 

[dlavigne]

That particular ticket (the fix you want) is targetted for BETA1 which is due out on Monday, but I suspect it might be a few days after that as we're still QAing that release. That particular ticket did pass QA.

 

[KrisBee]

@NASbox Your screen caps & text clearly show you have tried to to use Ubuntu 16.04 LTS. Unfortunately @dlavigne has referred you to a bug which relates to Ubuntu 18.04 LTS which uses a later kernel.

So, your VM should boot post-install. If you had used the default LVM disk layout during install, IIRC the correct files should be created in /boot/efi/EFI/BOOT/ and you'd expect the VM to boot post-install without intervention. For other disk layouts you'd want to use the expert install which at the grub-install stage gives the option to Force grub-efi installation to the removable media path.

If you really need to use a non-uefi debian based iso you could use iohyve at the CLI.

 

[NASbox]

Thanks for the quick replies guys!

That particular ticket (the fix you want) is targetted for BETA1 which is due out on Monday, but I suspect it might be a few days after that as we're still QAing that release. That particular ticket did pass QA.

@dlavigne Correct me if I misunderstand - the fix is only for the beta version, so it won't show up in the stable branch until when 11.1U6, 11.2?

As I said earlier, anything but stable is NOT an option for me. My values for my FreeNAS are Data Integrity, Stability, Security, and then if possible extra features. I'm no FreeBSD guru and simple problems can waste a ton of time which I don't have.

@NASbox Your screen caps & text clearly show you have tried to to use Ubuntu 16.04 LTS. Unfortunately @dlavigne has referred you to a bug which relates to Ubuntu 18.04 LTS which uses a later kernel.

So, your VM should boot post-install. If you had used the default LVM disk layout during install, IIRC the correct files should be created in /boot/efi/EFI/BOOT/ and you'd expect the VM to boot post-install without intervention. For other disk layouts you'd want to use the expert install which at the grub-install stage gives the option to Force grub-efi installation to the removable media path.

If you really need to use a non-uefi debian based iso you could use iohyve at the CLI.

@KrisBee Do I need to use LVM? I just used the default/use whole disk configuration.
Here's what I have.

The system didn't boot after installation, so I copied grubs64efi from EFI/ubuntu to EFI/BOOT
That didn't boot either, so I added .deactivated so that I could get back into the EFI Shell.
(Is there anyway to force the system to boot to the shell so I don't need to do this?)

Is there a fix for this as it stands, or is there a special way I should reinstall?

Questions for someone with a lot of FreeBSD experience/knowledge of the FreeNAS development roadmap:
Is the whole VM infrastructure so immature/unstable that I am going to have a lot of problems with updates/changes?
Is a future design change likely to break a VM so that it has to be recreated?
Everytime I update FreeNAS am I going to be faced with figuring out how to fix what the upgrade broke?
(Is this likely to be a frequent occurance, and is the fix likely to be simple and well documented, or am I going to have to spend hours researching?)

 

[KrisBee]

@NASbox No you don't have to use LVM, but if you don't you may need to use this command as root post-install on your VM:

grub-install --efi-directory=/boot/efi --boot-directory=/boot --removable

If you started from scratch you could just use the default LVM disk layout, or use the expert install, or boot into recovery mode and force EFI installation as I previously mentioned.

The problem you have experienced is simply a result of how bhyve's UEFI firmware interacts with linux distros. Bhyve's UEFI firmware does not yet support emulation of nvram so you cannot use the EFI boot manager to add a boot option which will persist between boots of the VM. The UEFI-CSM variant of the bhyve firmware does not work with vnc. There are various strategies to ensure a linux VM will boot post install. Many linux distros need no intervention to boot after installation using the UEFI boot option.

 

[NASbox]

@NASbox No you don't have to use LVM, but if you don't you may need to use this command as root post-install on your VM:

grub-install --efi-directory=/boot/efi --boot-directory=/boot --removable

@KrisBee When/How do I run this? If the system won't boot - I assume I have to do some sort of chroot or something, but I don't know exactly what to do. Can you add a few more details.

 

[KrisBee]

Rather than explain how to use the recovery mode, it will be quicker just to attach a cdrom device to our vm using a refind iso ( see: https://sourceforge.net/projects/refind/ ). That should enable you to boot your VM, then execute the command as root.

 

[NASbox]

Rather than explain how to use the recovery mode, it will be quicker just to attach a cdrom device to our vm using a refind iso ( see: https://sourceforge.net/projects/refind/ ). That should enable you to boot your VM, then execute the command as root.

@KrisBee Good news/bad news - Good news, I was able to get the VM to boot using refind. Bad news, the command didn't work. I can't get the machine to boot without refind. Here's what I have so far:

I also get a bunch more boot entries in the refind menu.
At least I can still get the VM to boot, but instead of the first boot entry, it's the last entry that boots.

Any suggestions?

 

[KrisBee]

Well the command works in as far as it appears to create the correct files in the correct directory. But you left the .deactivated file in place, I'd remove it. What's output of fdisk -l in your booted VM? You could try re-installing grub in the booted VM e.g grub-install /dev/vda followed by update-grub - assuming your VM's hdd is /dev/vda. Then repeat the other grub command before finally rebooting.

 

[NASbox]

Well the command works in as far as it appears to create the correct files in the correct directory. But you left the .deactivated file in place, I'd remove it. What's output of fdisk -l in your booted VM? You could try re-installing grub in the booted VM e.g grub-install /dev/vda followed by update-grub - assuming your VM's hdd is /dev/vda. Then repeat the other grub command before finally rebooting.

@KrisBee Here's what I did:
(grub-install/update-grub, remove extra file, grub-install --removable)

Here is the refind menu i ended up with:

and the four large icons provided the following choices and results:

1. Boot EFI\ubuntu\grubx64.efi from EFI System Partition [FAILED]
2. Boot EFI\BOOT\grubx64.efi from EFI System Partition [FAILED]
3. Boot Fallback boot loader from EFI System Partition [FAILED]
4. Boot boot\vmlinuz-4.4.0-128-generic.efi.signed from 18 GiB ext4 volume [OK]

the only option that successfully booted was the one that booted directly from the kernel.
All other choices resulted in a black screen (after a message flashed too quick to read).

Here is the contents of /boot/efi/EFI/BOOT/grub.cfg: (and /boot/efi/EFI/ubuntu/grub.cfg - both files have the same content):

Code:

search.fs_uuid e38931d7-9976-4530-af4e-ccbc5be86119 root
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg

• What does ($root) refer to in this context?
  
• Am I correct that /boot/efi/EFI/ubuntu/ isn't used - it is simply a model to use to create /boot/efi/EFI/BOOT/?

Contents of file /boot/grub/grub.cfg:

Code:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
	saved_entry="${chosen}"
	save_env saved_entry
  fi
}
function recordfail {
  set recordfail=1
  if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
	insmod all_video
  else
	insmod efi_gop
	insmod efi_uga
	insmod ieee1275_fb
	insmod vbe
	insmod vga
	insmod video_bochs
	insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root  e38931d7-9976-4530-af4e-ccbc5be86119
else
  search --no-floppy --fs-uuid --set=root e38931d7-9976-4530-af4e-ccbc5be86119
fi
	font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
	set timeout_style=hidden
	set timeout=0
  # Fallback hidden-timeout code in case the timeout_style feature is
  # unavailable.
  elif sleep --interruptible 0 ; then
	set timeout=0
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
		set gfxpayload="${1}"
		if [ "${1}" = "keep" ]; then
				set vt_handoff=vt.handoff=7
		else
				set vt_handoff=
		fi
}
if [ "${recordfail}" != 1 ]; then
  if [ -e ${prefix}/gfxblacklist.txt ]; then
	if hwmatch ${prefix}/gfxblacklist.txt 3; then
	  if [ ${match} = 0 ]; then
		set linux_gfx_mode=keep
	  else
		set linux_gfx_mode=text
	  fi
	else
	  set linux_gfx_mode=text
	fi
  else
	set linux_gfx_mode=keep
  fi
else
  set linux_gfx_mode=text
fi
export linux_gfx_mode
menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e38931d7-9976-4530-af4e-ccbc5be86119' {
		recordfail
		load_video
		gfxmode $linux_gfx_mode
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  e38931d7-9976-4530-af4e-ccbc5be86119
		else
		  search --no-floppy --fs-uuid --set=root e38931d7-9976-4530-af4e-ccbc5be86119
		fi
		linux   /boot/vmlinuz-4.4.0-128-generic.efi.signed root=UUID=e38931d7-9976-4530-af4e-ccbc5be86119 ro  splash quiet $vt_handoff
		initrd  /boot/initrd.img-4.4.0-128-generic
}
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-e38931d7-9976-4530-af4e-ccbc5be86119' {
		menuentry 'Ubuntu, with Linux 4.4.0-128-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-128-generic-advanced-e38931d7-9976-4530-af4e-ccbc5be86119' {
				recordfail
				load_video
				gfxmode $linux_gfx_mode
				insmod gzio
				if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
				insmod part_gpt
				insmod ext2
				if [ x$feature_platform_search_hint = xy ]; then
				  search --no-floppy --fs-uuid --set=root  e38931d7-9976-4530-af4e-ccbc5be86119
				else
				  search --no-floppy --fs-uuid --set=root e38931d7-9976-4530-af4e-ccbc5be86119
				fi
				echo	'Loading Linux 4.4.0-128-generic ...'
				linux   /boot/vmlinuz-4.4.0-128-generic.efi.signed root=UUID=e38931d7-9976-4530-af4e-ccbc5be86119 ro  splash quiet $vt_handoff
				echo	'Loading initial ramdisk ...'
				initrd  /boot/initrd.img-4.4.0-128-generic
		}
		menuentry 'Ubuntu, with Linux 4.4.0-128-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.0-128-generic-recovery-e38931d7-9976-4530-af4e-ccbc5be86119' {
				recordfail
				load_video
				insmod gzio
				if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
				insmod part_gpt
				insmod ext2
				if [ x$feature_platform_search_hint = xy ]; then
				  search --no-floppy --fs-uuid --set=root  e38931d7-9976-4530-af4e-ccbc5be86119
				else
				  search --no-floppy --fs-uuid --set=root e38931d7-9976-4530-af4e-ccbc5be86119
				fi
				echo	'Loading Linux 4.4.0-128-generic ...'
				linux   /boot/vmlinuz-4.4.0-128-generic.efi.signed root=UUID=e38931d7-9976-4530-af4e-ccbc5be86119 ro recovery nomodeset
				echo	'Loading initial ramdisk ...'
				initrd  /boot/initrd.img-4.4.0-128-generic
		}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
menuentry 'System setup' $menuentry_id_option 'uefi-firmware' {
		fwsetup
}
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

Directory listing of /boot/grub:

Code:

drwxr-xr-x 5 root root	4096 Jun 24 12:05 .
drwxr-xr-x 4 root root	4096 Jun 22 12:30 ..
drwxr-xr-x 2 root root	4096 Jun 22 12:30 fonts
-r--r--r-- 1 root root	6164 Jun 24 12:05 grub.cfg
-rw-r--r-- 1 root root	1024 Jun 24 21:03 grubenv
drwxr-xr-x 2 root root	4096 Jun 24 11:33 locale
-rw-r--r-- 1 root root 2398585 Jun 22 12:30 unicode.pf2
drwxr-xr-x 2 root root   12288 Jun 24 11:33 x86_64-efi

Contents of boot/grub/grubenv

Code:

# GRUB Environment Block
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################root@ubuntu164:/boot/grub#

Any hints/suggestions are much appreciated.

 

[KrisBee]

I don't know why you are getting this failure, you'd be quicker to re-create your Ubuntu VM using the expert install where you can force UEFI installation and Force grub-efi installation to the removable media path before the VM reboots.

 

[NASbox]

I don't know why you are getting this failure, you'd be quicker to re-create your Ubuntu VM using the expert install where you can force UEFI installation and Force grub-efi installation to the removable media path before the VM reboots.

I'm assuming that this is what you mean:

This screen came up for a few seconds when I selected yes.

I was able to get the grub menu, as long as I was starting after "Power Off".
I tried both the normal and the advanced options:

I got a persistent message when I selected advanced, but didn't change any options

I was able to get grub started, but was not able to boot the system.

I tested again with refind, and refind, was able to boot the kernel directly, but the normal EFI option failed to boot.

I don't know if this gives any clues or not, but here is the grub configuration:

Do I need to choose another kernel? (I used the default Linux 4.4.0-128 generic signed.)

IIUC bhyve will not boot a grub image, but refind is able to boot the kernel directly.

Any suggestions anyone?

 

[KrisBee]

Curious, everything seems to be in place. Right now, I think I'd be using another ubuntu iso, such as: ubuntu-16.04.3-server-amd64.iso

With this iso, if you used the same disk layout ( whole disk guided and no LVM) after the install has completed leave the cdrom device attached to your VM, reboot then use the rescue mode to get to the "Force UEFI install to removable device" option. Just make sure you pick the correct root device in the step prior to this.

 

[ovizii]

So basically, I can't simply use teh freenas GUi to create a Debian VM, install and run it, right?
This whole long post seems to indicate one needs different tricks and the console to "fix" things?

Just wanted to spare myself some trouble as after reading the guide for 11.1U4 it sounded like the GUI vor VMs would work just fine.

 

[KrisBee]

@ovizii Debian 9.4 install using "expert" option and at the grub install steps answer yes to
Force grub-efi installation to the removable media path

 

[ovizii]

@ovizii Debian 9.4 install using "expert" option and at the grub install steps answer yes to
Force grub-efi installation to the removable media path

Thanks, but following the GUI guide, I can't even boot from an ISO. I will open my own thread but following the guide, ISO won't boot, doing a "vm vm_name show" gives me: "$vm_enable is not enabled in /etc/rc.conf"

 

[NASbox]

I've given up till things get a bit more stable, but refind was the only way I could get the VM to boot.

 

[radar]

NASbox, thanks for your troubles. Were you ever able to setup a Linux Virtual Machine? I definitely do not want to go through days of headache but a virtual linux server is pivotal in my network.