VM not working after adding PCI Passthrough

[Arcabilbao]

Hello, I have a Home Assistant VM running on TrueNas Core. I've been trying to get a usb stick to passthrough to the VM by adding a PCI device to it.

I´ve already set up both tunables (vmm_load & pptdevs). And the device can be added to the VM.

The problem is, once the device is added to the VM, it no longer boots up. If I delete the PCI device from the list, it can boot up again no problem.

Any clue why this might be happening?

Thanks

 

[Patrick M. Hausen]

Depending on your IOMMU topology the USB controller shares resources with some other device used by the TrueNAS host OS. Your best bet is using a PCIe add on card if you have a free slot.

 

[Arcabilbao]

Would that happen with two different pci USB hosts? When i list all pci devices two different USB controllers appear and the problem happens with both of them.

I would assume at least one of them works independently from the rest. Could that not be the case?

 

[Patrick M. Hausen]

Not necessarily. Use pciconf -lv to check. Then on my system e.g:

Code:

igb0@pci0:8:0:0:    class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb1@pci0:8:0:1:    class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet

Both devices start with "0:8" which as far as I know means they share PCI bus #8. So you can pass both of them to a VM or none, but not just one of them.

My USB devices on this particular system seem to be fine:

Code:

ehci0@pci0:0:26:0:    class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c2d subvendor=0x15d9 subdevice=0x086d
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
ehci1@pci0:0:29:0:    class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c26 subvendor=0x15d9 subdevice=0x086d
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB

But then if I was to use them for a VM I would lose keyboard and mouse access (even the remote console via IPMI connects to the on board USB controller).

And last, expect bhyve to improve a lot in that regard with the upcoming TrueNAS CORE 13.2 - if you can wait for that. Or you might want to try a nightly release. But do so at your own risk, sorry.

Kind regards,
Patrick

 

[Arcabilbao]

Ok, I think I understand now what you mean.

You mentioned at the beggining that an easy fix could be installing a USB PCIe card and passing it directly. Would that just show up as a PCI device to passthrough without the need of any of the extra steps?

Sorry if I'm not very clear, but this is kind of new to me and I'm learning as I go.

 

[Patrick M. Hausen]

I am guessing - not knowing your exact hardware specs and not being a top notch expert in this particular field - that the PCIe slot(s) are routed completely independently of all on board devices so you have a better chance of a clean pass through.

Also:

- USB controllers are more or less dirt cheap
- If you buy online, plan ahead to have time to test, keep all the packaging and accessories in pristine condition, ... you can return the card if that does not work out

HTH
Patrick

 

[Arcabilbao]

Ok thanks, I'll try it out and report back with the results.

 

[friskens]

Ok kinda new to TrueNAS (but been running FreeBSD since 1995) but how about using console="nullconsole" so you never init the videocard at all. Requires you to have ssh setup previous to modifying the settings ofc. Also perhaps the serial console would work for imaginary security of access. I'm using the first one to PCIe passthrough in a Nvidia GTX 970 into a VM for running plex. I know this might not be an intended way to do things ofc and ppl like thier mouse and keyboard access to thier server ofc. Oh btw there's some error messages and stuff you need to sort out since someone has been lazy and piped errors to the videoconsole but that's minor fixes only. Happy Transcoding!

 

[Patrick M. Hausen]

They are trying to pass through a USB controller, not a GPU.

 

[friskens]

They are trying to pass through a USB controller, not a GPU.

My instruction's still valid since they get stuck on passing through a PCIe USB controller/port which is locked/initilized by FreeBSD to the videoconsole in the same manner as a PCIe GPU would be. Although I should have never included the GPU example for clarity.

 

[Patrick M. Hausen]

TIL - thanks.

 

[Arcabilbao]

I am guessing - not knowing your exact hardware specs and not being a top notch expert in this particular field - that the PCIe slot(s) are routed completely independently of all on board devices so you have a better chance of a clean pass through.

Also:

- USB controllers are more or less dirt cheap
- If you buy online, plan ahead to have time to test, keep all the packaging and accessories in pristine condition, ... you can return the card if that does not work out

HTH
Patrick

Hello.

Sorry for the late reply, I had other proyects to finish and this one got put on the back drawer.

I ended up getting the USB pci card as for your suggestion Patrick. I again located its pci device name, and maneged to generate the tunables accordingly. I had a little problem when rebooting the Home Assistant VM with the pci passthrough where it got stuck on a booting loop, but it was just a matter of altering the device order on the TrueNas VM options so that the USB card was last.

All in all it's been stable for the time being and I've been able to use USB devices on the VM without a problem. So the operation has been a success.

However, I am a bit intrigued by a small detail that maybe someone more experience can help me out with. My understanding is that by making the tunables for the PCI device passthrough, what you are doing is also blocking the host device from "seeing" said device in order to give the VM full access to it. So how come I still see the USB card when executing pciconf -v -l? Shouldn't it be "invisible" for TrueNas now?

It might be a stupid question but I thought it might help me learn a bit more about all of this.

Thanks for all the help

 

[friskens]

Hello.

Sorry for the late reply, I had other proyects to finish and this one got put on the back drawer.

I ended up getting the USB pci card as for your suggestion Patrick. I again located its pci device name, and maneged to generate the tunables accordingly. I had a little problem when rebooting the Home Assistant VM with the pci passthrough where it got stuck on a booting loop, but it was just a matter of altering the device order on the TrueNas VM options so that the USB card was last.

All in all it's been stable for the time being and I've been able to use USB devices on the VM without a problem. So the operation has been a success.

However, I am a bit intrigued by a small detail that maybe someone more experience can help me out with. My understanding is that by making the tunables for the PCI device passthrough, what you are doing is also blocking the host device from "seeing" said device in order to give the VM full access to it. So how come I still see the USB card when executing pciconf -v -l? Shouldn't it be "invisible" for TrueNas now?

It might be a stupid question but I thought it might help me learn a bit more about all of this.

Thanks for all the help

Hello Arcabilbao
All you're doing is basically disabling kernel init of the device, pciconf -v -l still shows what hardware is in your system it hasn't turned invisible to the pci host bus, it is just not inited for use. Hope this was a acceptable explanation.

Greetings Hans

 

[Arcabilbao]

Hello Arcabilbao
All you're doing is basically disabling kernel init of the device, pciconf -v -l still shows what hardware is in your system it hasn't turned invisible to the pci host bus, it is just not inited for use. Hope this was a acceptable explanation.

Greetings Hans

Oh. Ok I think I understand now.

Thank you everyone. Wouldn´t have been able to do any of this without your help.