grub boot HOW-TO

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
There was very little documentation on how to create a VM that boots from GRUB in the GUI. I had to go digging in /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py to figure out how to do this successfully. I found there are 2 required parameters that aren't accessible from the GUI, which have to be set via the REST 2.0 API to achieve a successful installation.

1. First, starting at line 142 of /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py:

Code:
                # Get grub devices to be used in grub-bhyve
                if device['dtype'] == 'RAW' and device['attributes'].get('boot'):
                    grub_devices.append(device['attributes']['path'])


This means that grub VMs can only boot from a RAW device, and the device needs to have the attribute "boot"="true" set. If the RAW device doesn't have the "boot"="true" attribute set, you'll get the error "There is no boot disk for vm: <name of grub VM>" when you try to boot the VM in the GUI, as described starting at line 241 of /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py:

Code:
        # grub-bhyve support
        device_map_file = tempfile.NamedTemporaryFile()
        grub_dir = None
        if self.vm['bootloader'] == 'GRUB':

            if not grub_devices:
                raise CallError(f'There is no boot disk for vm: {self.vm["name"]}')

            for i, device in enumerate(grub_devices):
                device_map_file.write(f'(hd{i}) {device}\n'.encode())
            device_map_file.flush()


Unfortunately, the only way to set this attribute is via the REST 2.0 API:

Code:
        curl --basic -u root -k -X PUT "http://<FreeNAS GUI IP>/api/v2.0/vm/device/id/<device id>" -H "accept: */*" -H "Content-Type: application/json" -d '{"dtype":"RAW","vm":<vm id>,"attributes":{"boot":true,"path":"<path to image>"}}'


The device ID for the RAW file is visible in the GUI, and the VM ID can be found in the number of the COM port /dev/nmdm<VM ID>B. Note the path to the RAW file is a mandatory field for the API call. Also, password login to root has to be enabled for the API call to succeed, as you'll authenticate the API call with root's password.

What this also means is that it's not possible to install from a CD-ROM device to a RAW file. You'll need to export a temporary zvol via iSCSI to an external hypervisor, like VirtualBox or Hyper-V, and install from the ISO to the zvol from outside FreeNAS. Once the install is complete, and the temporary external VM is verified to boot from the zvol, then you can use dd to convert the temporary zvol to the RAW file:

Code:
       dd if=<path to zvol> of=<path to RAW file>


2. Second, you'll need to make grub-bhyve happy. As described in /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py, starting at line 253:

Code:
            if (
                self.vm['grubconfig'] and
                self.vm['grubconfig'].startswith('/mnt/') and
                os.path.exists(self.vm['grubconfig'])
            ):
                grub_dir = os.path.dirname(self.vm['grubconfig'])
            else:
                grub_dir = f'/tmp/grub/{self.vm["id"]}_{self.vm["name"]}'
                os.makedirs(grub_dir, exist_ok=True)
                grub_file = os.path.join(grub_dir, 'grub.cfg')
                with open(grub_file, 'w') as f:
                    f.write(self.vm['grubconfig'])


The VM itself needs to have the "grubconfig" attribute set. This is either a path to the grub.cfg, starting from /mnt, or the contents of a working grub.cfg. This is where I found the documentation for grub-bhyve contradictory. Grub-bhyve does NOT use the grub.cfg inside the RAW file, even though it can navigate within the RAW file and load the kernel and initrd; rather, grub-bhyve relies on a grub.cfg on the host.

I recommend using a path to grub.cfg for the "grubconfig" attribute, as it's easier to modify the grub.cfg in vi instead of having to edit the attribute via API:

Code:
        curl --basic -u root -k -X PUT "http://<FreeNAS GUI IP>/api/v2.0/vm/id/<vm id>" -H "accept: */*" -H "Content-Type: application/json" -d '{"grubconfig":"<path to grub.cfg>/grub.cfg"}'


The simplest thing here is to just cut & paste the grub.cfg from inside the RAW file to a new grub.cfg on the host. Note, use single quotes instead of parentheses for the grub set root command.

If you've typoed anything inside the grub.cfg, you'll get the error "grub-bhyve timed out, please check your grub config." when you try to boot the VM.
 
Last edited:

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Update for TrueNAS Core:

This HOW-TO still works, but the API call to set the boot attribute on the RAW file now has both path and size as mandatory parameters.

Code:
curl --basic -u root -k -X PUT "https://<TrueNAS IP>/api/v2.0/vm/device/id/<device id>" -H "accept: */*" -H "Content-Type: application/json" -d '{"dtype":"RAW","vm":<vm id>,"attributes":{"boot":true,"path":"<path to image>","size":"<size of image in GiB>"}}'


Note that this may overwrite your carefully crafted RAW file with a new empty file of the stated size, so use a dummy path for the API call and match the size of your RAW file. Then adjust the path in the GUI to the actual path of your RAW file after you've confirmed the RAW device has the boot=true attribute. You can delete any dummy files created.
 

revoman

Dabbler
Joined
Jan 11, 2012
Messages
13
Thank you so much for this walkthrough! Exactly what I was looking for. Not sure why "grub" is exposed in the GUI after reading this.

Also, when I use the new API call the physical and logical sector size values get lost. I had to actually use

Code:
curl --basic -u root -k -X PUT "https://<TrueNAS>/api/v2.0/vm/device/id/<raw-device-id>" -H "accept: */*" -H "Content-Type: application/json" -d '{"dtype":"RAW","vm":<vm-id>,"attributes":{"boot":true,"path":"/mnt/tank/volumes/esxi8.raw","size":"50","physical_sectorsize":4096,"logical_sectorsize":4096}}'
 
Last edited:
Top