Resource icon

[How-To] A guide to migrate FreeBSD VM from ESXi to FreeNAS (bhyve)

I moved my FreeBSD VM (v7) from ESXi 5.5.0 to bhyve under FreeNAS-11-MASTER-201709210509. Below are short instructions with my example.
  1. Prepare VM on ESXi. With your preferred method:
    - shutdown VM
    - remove all snapshots
    - remove from ESXi inventory to avoid problems. Do not delete data files
  2. Copy ESXi VM virtual disk files. Mine had 5 virtual disks attached
    - ssh root@esxihost
    - scp -r /vmfs/volumes/datastore2/srv_10 root@freenas:/mnt/tank/vm
  3. Convert. I used jail under same FreeNAS box for this
    - create a standard type jail. I named it VirtualBoxJail. Do not confuse it with legacy VirtualBox template
    View attachment 20748
    - add storage with a copy of ESXi virtual disks
    View attachment 20740
    - start this jail and install VirtualBox inside it using shell
    pkg update
    pkg install virtualbox-ose-nox11
    - convert all vmdk hard drive files to raw image files
    cd /mnt/srv_10/
    VBoxManage clonehd srv_10.vmdk srv_10_bhyve_volume_1.raw --format RAW
    VBoxManage clonehd srv_10_1.vmdk srv_10_bhyve_volume_2.raw --format RAW
    ...

    - create ZVOLs for each VM disk same size like original + one 100 MB for EFI. We will need it to fix VM boot
    View attachment 20741
    - copy data from RAW files to ZVOLs repectively. I added a pipe via pv to see some progress but you can ommit that
    dd if=/mnt/tank/vm/srv_10/srv_10_bhyve_volume_1.raw | pv -petr | dd of=/dev/zvol/tank/vm/srv_10_vol_1
    dd if=/mnt/tank/vm/srv_10/srv_10_bhyve_volume_2.raw | pv -petr | dd of=/dev/zvol/tank/vm/srv_10_vol_2
    dd if=/mnt/tank/vm/srv_10/srv_10_bhyve_volume_3.raw | pv -petr | dd of=/dev/zvol/tank/vm/srv_10_vol_3
    dd if=/mnt/tank/vm/srv_10/srv_10_bhyve_volume_4.raw | pv -petr | dd of=/dev/zvol/tank/vm/srv_10_vol_4
    dd if=/mnt/tank/vm/srv_10/srv_10_bhyve_volume_5.raw | pv -petr | dd of=/dev/zvol/tank/vm/srv_10_vol_5
  4. Assemble new bhyve VM
    - create new VM
    - configure nic
    - add virtual CD. You will need to download and mount FreeBSD image on it to fix EFI boot
    curl https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso
    - add earlier created ZVOLs (EFI and all other VM disks)
    View attachment 20742
  5. Fix EFI , mounting points and NIC names
    - start VM. It should boot from attached FreeBSD iso. Start VNC session and Select Live CD. Now we will create EFI partition with image files.
    - login with root. Check virtual disk names
    dmesg | grep ada
    - on that EFI 100 MB disk create GUID partition table
    gpart create -s GPT /dev/ada0
    - create a GUID Partition Table (GPT) scheme:
    gpart create -s GPT /dev/ada0
    - Add the EFI partition with the label "EFI" aligned to 1MB:
    gpart add -t efi -a 1M -l EFI /dev/ada0
    - create the FAT16 filesystem for the EFI partition, with a label of FreeBSD_EFI:
    newfs_msdos -F 16 -L FreeBSD_EFI /dev/ada0p1
    - mount it:
    mount -t msdosfs /dev/ada0p1 /mnt
    - create the directory for the FreeBSD UEFI bootloader:
    mkdir -p /mnt/efi/EFI/BOOT
    - copy the FreeBSD UEFI first-stage bootloader into that directory with the name which firmware expects:
    cp /boot/boot1.efi /mnt/efi/EFI/BOOT/BOOTX64.EFI
    - unmount the filesystem:
    umount /mnt
    - now mount your virtual disk with root (/) and edit your fstab as your drive names must have changed. Alternatively you can fix nic name in rc.local
    mount /dev/ada1p2 /mnt
    vi /mnt/etc/fstab
    vi /mnt/etc/rc.conf
    umount /mnt


    my changes were
    da0p2 -> ada1p2
    da1p1 -> ada2p1
    da2p1 -> ada3p1
    da3p1 -> ada4p1
    da4p1 -> ada5p1
    vmx0 -> vtnet0
  6. After all this is done remove CD from VM and reboot it. You should be good to go

p.s. You can remove vmdk, VirtualBox converted raw files and VirtualBoxJail. Now VM data lives inside ZVOLs

used material from
https://forums.freebsd.org/threads/51393/
http://justinholcomb.me/blog/2016/03/26/migration-to-freebsd-part4.html
m0nkey_ thanks for comments
  • Like
Reactions: Artion
Author
Myka
Views
4,112
First release
Last update
Rating
0.00 star(s) 0 ratings
Top