rancher migration in 11.3

ykhodo

Explorer
Joined
Oct 19, 2017
Messages
52
I realize that 11.3 has removed rancher support. Is there a way to migrate an existing rancher raw disk to a new VM where I install rancher manually? Is there a better way to do this?
 

TheOnyx

Cadet
Joined
Jan 29, 2020
Messages
2
And even worse yet, for some reason my 11.3 wouldn't properly create a new grub vm. It kept complaining that it had no boot disk "[EFAULT] There is no boot disk for vm". Making an uefi vm worked fine however rancheros doesn't work on uefi and uefi-csm wouldn't boot either, and no way to tell why as far as I can tell as vnc doesn't work with anything but uefi.

If anyone figured out an upgrade path from 11.2 rancheros vms to 11.3 I'm eager to find out how.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
Did you read the release notes before upgrading? Its been clear for some time that this facility would be deprecated with no upgrade path provided.

There's no indication in the FreeNAS 11.3 how a "grub" boot might work for a VM. In any case AFAIK the rancheros.iso is syslinux based and is not recoginsed by grub and is not grub bootable. Hence FN previously downloaded a special rancheros image and created the boot mechanism as a background task. Looks like a case of saving all docker configs, checking how your persitent data has been stored, etc before upgrading and then re-creating containers after installing docker in a new Linux VM as the new FreeNAS guide suggests. If you've used the Rancher server UI then I wouldn't know if some kind of migration is possible between stuff running under rancheros and another instance running in a linux VM ( all in FN 11.2-U7 of course).
 

TheOnyx

Cadet
Joined
Jan 29, 2020
Messages
2
Thanks for taking the time to reply @KrisBee, I was aware that the "docker" vm functionality is being deprecated, and all my volumes are well persisted and backed up. I have no intention of reusing the old vms in any way as I can easily redeploy them. My issue is that I couldn't get the rancheros to boot in any of the 3 options grub/uefi-csm/uefi. I'm aware that rancheros doesn't support uefi but I expected one of the other two to work. The current issue is that I have no idea how to debug the reason why booting a fresh uefi-csm with the rancheros.iso is not doing anything, just hangs. No way to vnc into it and using serial I get no output whatsoever.

I'm not complaining that it's not working, I'm well aware that I'm doing something stupid here, question is, what? Why can't I get the rancheros.iso to boot?
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
See my previous reply. I assume you have downloaded a rancheros.iso, perhaps from here: https://github.com/rancher/os/releases - bhyve can't boot it full stop, the iso does not support UEFI or grub.
 
Last edited:

Jeud

Cadet
Joined
Nov 12, 2013
Messages
3
After migrating to Freenas 11.3 my Docker Host (Rancher) won't start any more with error "[EFAULT] grub-bhyve timed out, please check your grub config.".
1) The path of grubconfig was in wrong dataset in freenas db (/data/freenas-v1.db , table: vm_vm, field: grubconfig).
2) The timeout in vm.py is too short (2s) change to 20s (and reboot) (line 280 in /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py)

After this my Docker Host work again.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@Jeud Useful info, but hacking system files is hardly a "migration path", nor is there any use friendly way to update rancheros. It's pretty clear ixsystems expected the use of their "docker host" VM to wither on the vine.
 

ykhodo

Explorer
Joined
Oct 19, 2017
Messages
52
A workaround I had was to copy over the raw disk to another machine. I then converted the raw disk to a virtual box disk:

Code:
VBoxManage internalcommands createrawvmdk -filename foo.vmdk -rawdisk rancher_rancher


You can then start the VM on a different machine and copy your settings / reproduce your environment on a new VM on freenas (I chose fedora 31 server).
 

Bytesplit

Dabbler
Joined
Jul 24, 2017
Messages
12
I've migrated to 11.3 over the weekend and was surprised that the Docker container was not migrated to generic Bhyve but actively disabled by removing the .bhyve_containers folder (thanks @Jeud ). Strange decision and needless!

I restored the .bhyve_containers folder and the container came up without an issue again.

@KrisBee while I agree that for iXsystems Docker support was a pain and not their business the container is working fine and I would have wished for them to just leave it alone. I did upgrade my RancherOS manually before and FreeNAS did always kill my setup by wiping it with the default template.

It is no problem to run non-EFI containers with Bhyve on FreeNAS. It's just always a painful process to find the grub details and setup the launch process. As that is outside of the container Linux updates may seem to fail as grub is not being updated. With EFI everything is inside the VM and can be handled from there.

For anyone interested in keeping the Rancher containers for the moment:
1. copy your grubconfig to a safe place. It points to the vmlinuz and initrd inside the VM so they must be accurate.
Code:
set timeout=0
set default=RancherOS
menuentry "bhyve-image" --id RancherOS {
set root=(hd0,msdos1)
linux /boot/vmlinuz-4.9.80-rancher rancher.password=rancher printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait rancher.resize_device=/dev/sda
initrd /boot/initrd-v1.3.0
}

2. When you are able to again launch the VM you may try to update it.
Code:
sudo ros os version
v1.3.0
sudo ros os upgrade
# DO NOT RESTART YET!
Continue with reboot [y/N]: n
# Instead go into the RancherOS host to find out what vmlinuz and initrd are now in /boot
sudo system-docker run --rm -it -v /:/host alpine /bin/ash
cd /host/boot
ls
initrd-v1.5.5
vmlinuz-4.14.138-rancher

3. With the new initrd and vmlinuz you may update your grubconfig.
4. Reboot RancherOS and verify ros os version is uptodate

Then I upgraded Rancher (https://rancher.com/docs/rancher/v1.6/en/upgrading/#single-container) which needed renaming the Docker config folder (see https://github.com/rancher/os/issues/2300#issuecomment-390456535). Finally I had to clean the overlay structure:
Code:
docker system prune --all --volumes --force


Tada! 10 GB freed ... updated Rancher and RancherOS while still booting with bhyve grub. No issues!

Now I can setup another container host with any of: Rancher 2.0, k3OS or Fedora CoreOS and start migrating my containers to Kubernetes.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@Bytesplit Thanks for the useful info for those wanting to continue using rancheros/rancherUI.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@Kevk See FreeNAS guide 17.2
 

romracer

Cadet
Joined
Aug 5, 2016
Messages
7
See my previous reply. I assume you have downloaded a rancheros.iso, perhaps from here: https://github.com/rancher/os/releases - bhyve can't boot it full stop, the iso does not support UEFI or grub.
@KrisBee Since you think you're the expert on this, can you please explain to me what UEFI-CSM has to do with GRUB?

I understand completely that RancherOS uses syslinux internally to boot. bhyve-grub runs outside of the VM environment and simulates what a GRUB inside of the VM would have done (mount a filesystem, read an initial ramdisk and kernel, load them into memory at the appropriate address). I also know that RancherOS does not include UEFI support.

But this is exactly the use case UEFI-CSM is designed to cover. CSM stands for Compatibility Support Module and it's used to boot operating systems that only support legacy BIOS-based booting. As long as a bootloader is installed to the boot sector inside the VM (syslinux in the case of RancherOS), the CSM should find it and start the boot process. Notice how I never mentioned GRUB in this paragraph. It's irrelevant in this discussion.

So the real questions are, why won't bhyve's UEFI-CSM boot a syslinux-based boot loader and how does one troubleshoot that without VNC?

And just to clear the air, FreeNAS did not create a special RancherOS image or perform some background magic in 11.2. They simply provided a known version of RancherOS (v1.1.3) and a grub.cfg to match it, because as @Bytesplit noted, the grub.cfg contains version specific items (kernel and ramdisk). They likely deprecated the whole thing because it's difficult to maintain. How do you update an external grub.cfg with the appropriate versions for a kernel and ramdisk technically only visible inside the VM? The only way I can think to do it, is to provide known versions of RancherOS and the associated grub.cfg and only permit those (eye opener, it's what FreeNAS did for the official 11.2 release). In 11.3, they can wash their hands completely by saying "deploy an Ubuntu VM, install Docker, update it yourself."
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@romracer All this has been discussed before, ever since the "Docker VM" function was introduced. The implementation never kept up to date with sometimes rapid release cycle of rancheros, and you can speculate as to whether the main individual responsible for packaging the rancher vmlinuz, initrd and roofs, plus relevant code in FreeNAS, is even still associated with ixsystems.

It's not that FreeNAS "did some magic" - it's that fact that is done in the background, leaving the end user to understand how it's put together and how, as @Bytesplit has decribed, it might be continued to be used.

As to your question re: UEFI-CSM, I'm not a developer so better to ask ixsystems or seek an answer over at the freebsd-virtualization mailing list. If you can attract the attention of Rodney Grimes perhaps he can tell you why VNC still does not work with the uefi-edk2-bhyve-csm firmware or work with syslinux.

IIRC M. Araujo did a lot of work on libvncserver sponsered by ixsystems ( https://www.freshports.org/net/libvncserver/ ) and was also responsible for introducing the "Docker Vm" function into FreeNAS.
 

Bytesplit

Dabbler
Joined
Jul 24, 2017
Messages
12
Out of curiosity I just spun up a new Grub based VM. It does not work at all ... because nothing for grub is created. No device.map and no grub.cfg. Without knowing what FreeNAS is expecting to launch a VM even knowing how to format these files I don't know where to put them.

We seem to get off topic but in general I would have hoped for some more background on how FreeNAS is handling VMs. I saw people using iohyve and/or vm-bhyve but as long as the section is in the UI I would prefer to just use that.

Any hints on where FreeNAS is saving this information? Docker is deprecated I understood... but Grub altogether? Why was it not pulled from UI?
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@Bytesplit Do you mean using the UI with a boot mode of GRUB, as introduced in FN11.3 (not present in FN.112 series)? I wondered about how this is supposed to work. My guess would be in the case of linux with an install iso that allows access to the GRUB terminal over a serial connection.
 

Bytesplit

Dabbler
Joined
Jul 24, 2017
Messages
12
(this should perhaps go into a new topic ...)

@KrisBee ... boot mode Grub yes! I learned that VMs are handled in middleware plugin vm.py. There is a section which writes a (temporary) device.map and grub.cfg to kind of automate the Grub boot sequence.

Sadly it's not working for me. It was horribly painful to debug the process as the device.map gets deleted on bhyve close. The temporary device.map seems to be the main issue. There is a routine to just use the existing one but I did not get around why that is not taken. So I was stuck in the creation of said /tmp/(tempname) device.map. To appear in the device.map vm.py checks a bootable flag from freenas-db. I cannot find where to set this flag in the web UI so I manually adjusted the database sqlite. Next it only detects AHCI raw devices. As my boot device is a ZFS dataset I've modified the vm.py selection process accordingly. The CD should also appear there but for me never did. As I had booted bhyve manually for install and don't need the CD I ignored the issue for now.

I went to Jira to create a ticket and found this "NAS-101049". So Grub support seems new and untested. Looking at the Github vm.py has changed so much I'm not sure a pull request for 11.3 makes sense.

Since then things are working pretty stable. I've rebooted to check if things persist and all is well. I've also come across the overcommit issue which was discussed in other threads (VM memory warning with 8 of 16GB free). Then I have a feature request to set jail/VM boot sequence as I have services relying on other VMs existence.

So that was my weekend basically ...
 
Top