SOLVED GPU Passthrough Fails if the HDMI port is plugged in on boot

[ULTIoNT]

Hi there. I have a Ubuntu VM configured to start automatically once powered on and when I reboot TrueNAS, I find CPU utilization on one thread is constantly 100% and the VM is not up. Upon checking

Code:

/var/log/messages

I find the following error endlessly repeating:

Code:

vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0x6000000000-0x600fffffff 64bit pref]

If I turn off the auto start and unplug the display (a dummy HDMI plug in actual), then I can plug it back in and manually start the VM with the GPU correctly after the system loads. I have come across a few posts regarding the same error and it seems that it is due to the GPU not being properly released. I have set up the GPU to be isolated and I do find the monitor (a real monitor which I connected to the same port for troubleshooting) freezes after TrueNAS boots and prints out a line saying that the VGA device is switched or something like that (sorry I forgot to write that down).
The GPU involved here is:

Code:

01:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

and it appears in the list of isolated GPU PCI Ids as

Code:

NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A]

One solution that I have read about is to add the following kernel parameters:

Code:

midclt call system.advanced.update '{"kernel_extra_options": "vga=off"}'

I presume that is going to work because it basically disabled outputting to a monitor completely, but I haven't tried because I am worried I would have no easy way of changing it back without the display.

Edit:
Also confirmed that running

Code:

echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove
echo 1 > /sys/bus/pci/rescan

does solve the problem. Then the next question is, how to make sure this runs before the VM starts? The original post (targeting Proxmox) sharing this solution uses a cron job at reboot but I am not sure if it starts before the VM in TrueNAS.

Edit 2:
TrueNAS doesn't allow special cron job triggers like

Code:

@reboot

anyway. Wil init script do the same thing?

Edit 3:
Confirmed that adding the following command to the post-init script solves the problem

Code:

echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove && echo 1 > /sys/bus/pci/rescan

Remember to change the PCI id to match your GPU if you need to do the same thing.

 

[artlessknave]

please read through the forum rules and post your hardware.

as you haven't posted your hardware, the best I can do now is a quick guess:

it sounds to me like you are trying to use the video card for both truenas AND PCIe passthrough? if so, I am quite sure the answer is 'no', you cannot do this. you need a display for TrueNAS. you cant just move the truenas display to a VM: truenas is USING it.

if I have misunderstood, please post the hardware and a better description of your setup.

 

[ULTIoNT]

please read through the forum rules and post your hardware.

as you haven't posted your hardware, the best I can do now is a quick guess:

it sounds to me like you are trying to use the video card for both truenas AND PCIe passthrough? if so, I am quite sure the answer is 'no', you cannot do this. you need a display for TrueNAS. you cant just move the truenas display to a VM: truenas is USING it.

if I have misunderstood, please post the hardware and a better description of your setup.

Thank you for the reply, although I have marked this as solved. As to the hardware:

• Motherboard: Asrock B660M Pro RS
• CPU: i3 13100
• RAM: 16Gx4 at 3200
• Hard drive: Silicon Power 1T SP001TBP34A60M28 for boot; Kingston KC3000 1T stripe; Seagate Exos X14 14Tx4 in RAIDz-1
• Hard disk controllers: N/A
• Network cards: on-board I219-v; Intel X540-AT2

I will make sure to add this to my signature later, but I don't think they are relevant to the issue I was encountering. The GPU is a modded version of 2080Ti so it has 22G of VRAM and all I can tell is that the PCB comes from Gigabyte. I was just trying to dedicate this GPU to a Ubuntu VM and use the iGPU for TrueNAS and the docker apps. I have set it up as an isolated device, enabled simultaneous output from iGPU and dGPU in UEFI, but I didn't set the iGPU as the default and instead leave it for the MOBO to decide which to use because I may sometimes just connect a monitor and forget to change this setting. For all monitor/displays I mentioned in the original post, they are physical devices not the virtualized display you add to the VM through the web UI.