VM (Win11) Freezes with GPU Passthrough

[raidflex]

Hello All,

So I just setup a Windows 11 VM on TrueNAS CORE 13.0-U3.1 with passthrough for a GTX 1050 Ti and I am finding the VM freezes right after installation with GPU passthrough enabled. I have added the required tunables to Truenas and have no issues adding the device to the VM.

If I disconnect the GPU from the VM there is no issue and the system does not freeze. Has anyone else successfully setup GPU passthrough on CORE with Windows 11?

 

[HoneyBadger]

I have added the required tunable to Truenas

Can you please show the exact tunables as well as the output of pciconf -lv to ensure that you've matched up the PCI IDs with your tunable?

Note that VGA/GPU devices are unsupported for passthrough in bhyve, so your results may vary. You'll also have to deal with the NVIDIA Code 43 issue inside of the VM.

 

[raidflex]

Can you please show the exact tunables as well as the output of pciconf -lv to ensure that you've matched up the PCI IDs with your tunable?

Note that VGA/GPU devices are unsupported for passthrough in bhyve, so your results may vary. You'll also have to deal with the NVIDIA Code 43 issue inside of the VM.

I have also tried installing Win11 without the GPU and then attached it after. I can see the 1050 TI show up in device manager, but the system will freeze at some point as well.

Code:

root@Atlantis:~ # pciconf -lv
hostb0@pci0:0:0:0:      class=0x060000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          0c08 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v3 Processor DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
pcib1@pci0:0:1:0:       class=0x060400 rev=0x06 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                                                                                          0c01 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Control                                                                                                                                                                                                                          ler'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:1:1:       class=0x060400 rev=0x06 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                                                                                          0c05 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controll                                                                                                                                                                                                                          er'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:0:20:0:      class=0x0c0330 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c31 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB xHCI'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:26:0:      class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c2d subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
pcib3@pci0:0:28:0:      class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c10 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:28:2:      class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c14 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:0:28:3:      class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c16 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:0:28:4:      class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c18 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
ehci1@pci0:0:29:0:      class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c26 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:      class=0x060100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c52 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = 'C222 Series Chipset Family Server Essential SKU LPC Controller                                                                                                                                                                                                                          '
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:      class=0x010601 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c02 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [                                                                                                                                                                                                                          AHCI mode]'
    class      = mass storage
    subclass   = SATA
ichsmb0@pci0:0:31:3:    class=0x0c0500 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c22 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
none0@pci0:0:31:6:      class=0x118000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          8c24 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'Intel Corporation'
    device     = '8 Series Chipset Family Thermal Management Controller'
    class      = dasp
ppt0@pci0:1:0:0:        class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x                                                                                                                                                                                                                          1c82 subvendor=0x1028 subdevice=0x3512
    vendor     = 'NVIDIA Corporation'
    device     = 'GP107 [GeForce GTX 1050 Ti]'
    class      = display
    subclass   = VGA
none1@pci0:1:0:1:       class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x                                                                                                                                                                                                                          0fb9 subvendor=0x1028 subdevice=0x3512
    vendor     = 'NVIDIA Corporation'
    device     = 'GP107GL High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA
mps0@pci0:2:0:0:        class=0x010700 rev=0x05 hdr=0x00 vendor=0x1000 device=0x                                                                                                                                                                                                                          0086 subvendor=0x15d9 subdevice=0x0691
    vendor     = 'Broadcom / LSI'
    device     = 'SAS2308 PCI-Express Fusion-MPT SAS-2'
    class      = mass storage
    subclass   = SAS
pcib4@pci0:3:0:0:       class=0x060400 rev=0x03 hdr=0x01 vendor=0x1a03 device=0x                                                                                                                                                                                                                          1150 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'AST1150 PCI-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
vgapci0@pci0:4:0:0:     class=0x030000 rev=0x30 hdr=0x00 vendor=0x1a03 device=0x                                                                                                                                                                                                                          2000 subvendor=0x15d9 subdevice=0x0804
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'ASPEED Graphics Family'
    class      = display
    subclass   = VGA
igb0@pci0:5:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          1533 subvendor=0x15d9 subdevice=0x1533
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb1@pci0:6:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x                                                                                                                                                                                                                          1533 subvendor=0x15d9 subdevice=0x1533
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
mlx4_core0@pci0:7:0:0:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x                                                                                                                                                                                                                          1003 subvendor=0x15b3 subdevice=0x0055
    vendor     = 'Mellanox Technologies'
    device     = 'MT27500 Family [ConnectX-3]'
    class      = network
    subclass   = ethernet

 

[HoneyBadger]

Looks as if the masking is correct. You might also need to pass through the 1/0/1 device (NVIDIA Audio controller) into the same VM though, perhaps it's looking for that and seizing up the guest when the drivers load?

Afraid I can't offer much help from experience on this, as my current host GPUs are Tesla/GRID.

 

[raidflex]

Looks as if the masking is correct. You might also need to pass through the 1/0/1 device (NVIDIA Audio controller) into the same VM though, perhaps it's looking for that and seizing up the guest when the drivers load?

I just tried again, this with both the GPU and HDMI audio passthrough but it still results in freezing. I know technically Bhyve does not support GPU passthrough, but I am curious if anyone was able to get it to work on Windows. Otherwise maybe this is why its not supported.