VMs (Windows 10 or Linux Mint) not starting with Tesla P4 Passthrough

[farout]

Hi
First time poster here :)

My system:

TrueNAS SCALE 23.10.1
ASRock EPC621D8A Motherboard
Intel Xeon 3136 CPU
96 GB ECC RAM
1x 500 GB Ironwolf pro SSD (Bootpool)
3x 8 TB Ironwolf pro High WRL HDD (Pool 1 RAIDZ1)
2x 2 TB WD Red SN700 NVME (Pool 2 Mirror)
Intel X550-T2 10G NIC

Whenever I try to passthrough a Tesla P4, the VM, Windows 10 OR Linux Mint, does not start. Truenas just displays the spinning circle forever. Refreshing the page shows the VM as not started. I dont find any error messages in the VM logfile.

The result is the same, if I isolate the GPU first, use the GPU allocation menu in the VM creation setting or if I add the Tesla P4 via "PCI Passthrough Devices".

The Tesla P4 is in its own IOMMU group.
The result is the same, if I use another PCI slot.
I even have a second Tesla P4 --> same problem with this card aswell.

The Tesla is working properly inside Truenas, I can use it for Apps.

Passing through an old GTX 970 works perfectly.

Any ideas ?

farout

 

[farout]

I solved it !

If you have this problem, do not create a VM with "Ensure display device" on !

It works if you add the display later under devices, but the VM will not start if you choose the option from the drop down menu while creating the VM!
Weird !