People who got AMD GPU passthrough to work, how did you do it?

[rdfreak]

Let me walk you through the rabbit hole I found myself in. It starts off with needing a VM with AMD GPU passthrough.

Creating a VM with the GUI doesn't quite work: if I stop the VM (no shutting down), I get an error, Unknown PCI header type '127'. This is supposedly common for AMD GPUs, and will persist until the NAS is restarted. Should be able to be fixed by creating a VM of type i440fx. (Source for the fix: https://forums.unraid.net/topic/76510-execution-error-internal-error-unknown-pci-header-type-127/)

Creating an i440fx-type VM is not possible using the GUI, but I can do so with the API (https://www.truenas.com/docs/api/scale_websocket_api.html#vm). Actually I can't, if I specify devices: the POST request to the /vm endpoint fails if the request's body contains a "devices" property, even if it's the default value ([] or empty array):

Code:

422 Unprocessable entity
{
  "vm_create.devices": [
    {
      "message": "Field was not expected",
      "errno": 22
    }
  ]
}

Creating an i440fx-type VM succeeds without including a devices property. But when adding a disk later, qemu fails:

Code:

[EFAULT] internal error: qemu unexpectedly closed the monitor: 2023-05-15T12:07:44.558730Z qemu-system-x86_64: -device ide-hd,bus=sata0.0,drive=libvirt-1-format,id=sata0-0-0,bootindex=1,logical_block_size=4096,physical_block_size=4096,write-cache=on: discard_granularity must be a multiple of logical_block_size

In fact, the above error will occur every time a VM has its primary disk removed and re-added, even if it's the same zvol! Or if a disk is added after the VM has been created with the GUI. The only way I have managed to get any VM to run is by creating it from the GUI, and adding its disk during the initial creation options.

All of this can be fixed, if

1. a i440fx-type VM could be added from the GUI
2. being able to specify devices when creating a VM using the API
3. being able to boot a VM when a disk has been added after its creation

So, how does one get an AMD GPU to pass through on an i440fx machine? Maybe more importantly, how does one fix that qemu error?

 

[rdfreak]

Never mind. I managed to create a VM via the GUI, then I changed its type to pc-i440fx-5.2 via the /vm/id/{id} endpoint. Confirmed with a GET request to see whether changes took effect. The VM initially caused the whole appliance to become unresponsive, a restart later turned out that using a different machine type doesn't solve the initial problem. Not to mention the Fedora installation would just freeze shortly after a boot option would be selected. GPU passthrough on AMD doesn't work as far as I'm concerned.