SOLVED No network access when VM is started via console, starting from UI works fine

[UnchartedBull]

Hi,

I'm currently trying to passthrough my USB3.0 PCI Hub to my HASSio VM to finally connect the Zigbee stick directly to HomeAssistant. I did some searching around on how to do that and got something that kind of works together. The big problem here is, that hassio can't connect to the network if I start the VM via the CLI (as root user). If I start the VM via the UI it works just fine though. The PCI passthrough is working fine and I can connect to the machine via VNC and interact normally, except no network connection.
I saw in the logs that a NIC_ATTACH was happening:

Code:

[2020/06/23 17:57:54] (DEBUG) VMService.vm_6.run():179 - ====> NIC_ATTACH: alc0
[2020/06/23 17:57:54] (DEBUG) VMService.vm_6.run():293 - Starting bhyve: bhyve -A -H -w -c 2 -m 1300 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm6A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 29,fbuf,vncserver,tcp=192.168.x.x:6011,w=1024,h=768,, -s 30,xhci,tablet -s 4,e1000,tap3,mac=00:a0:98:5a:xx:xx -s 3:0,ahci,hd:/dev/zvol/hassio/system 6_HassIO

So i was wondering, whether and how I need to do that if I start the VM from the cli. My other guess would be that the UI uses a different user to start the VMs. If I knew that user I could try starting the VM with it and see if something changes.

It would also be super helpful to know, whether I can adjust the command that is used to start the VM via the UI, maybe I can just inject my snippet there and see if that works.

Here is how I got to the current point (if this helps):

- Created VM via UI and installed everything
- Disabled PCI bridge via loader.conf
- Found bhyve command used by the UI:
Starting bhyve: bhyve -A -H -w -c 2 -m 1300 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm6A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 29,fbuf,vncserver,tcp=192.168.x.x:6011,w=1024,h=768,, -s 30,xhci,tablet -s 4,e1000,tap3,mac=00:a0:98:5a:xx:xx -s 3:0,ahci,hd:/dev/zvol/hassio/system 6_HassIO
- Added the -S option and -s 1:0,passthru,5/0/0 right behind hostbridge
- Executed that script as root
- If I connect to the hassio install via vnc I can see connected USB devices (to that PCI adapter) via the hardware info.

Would be great if someone could help.
Thanks!

 

[Patrick M. Hausen]

It could be that with the additional passthrough the virtual NIC ends up as a different PCI device and is not configured inside your guest OS. Linux is notorious in that regard.

 

[UnchartedBull]

From what I can tell it is the same PCI device. Doesn't -s 4,e1000,tap3,mac=00:a0:98:5a:xx:xx specifically tell bhyve to use PCIe Bus 4. I just sraight up copied that part, so that should be the same. And as far as I can tell the VM is trying to make a connection, that would indicate that it found a network card, but it doesn't have internet access ...

 

[Patrick M. Hausen]

Use ifconfig to find the tap interface for your VM and check if it is added to the bridge interface that is used to connect VMs (in most cases bridge0). If it isn't, use ifconfig bridgeX addm tapY.

 

[UnchartedBull]

Thanks for your quick reply! I did some further investigation and found some more details:
- If I start the VM via the UI a free tap interface is created and correctly attached to the bridge.
- If I start the VM via the Command Line the specified tap interface is created but is NOT attached to the bridge.
- Attaching it manually doesn't change anything though ...

I also listed all pci devices to make sure all the links are correct (first one is if started from UI, second one form Command Line). The all seem to be the same except Bus 0, which is the USB 3.0 controller:

Although a really weird thing I noticed is the output of the ip command, while it looks completely normal if it started via the UI with just 3 interfaces and eth0 correctly connected:

it is a mess if I start via the CLI:

I don't know what all these vethd4 things are, but I don't think, that they should be there. Also enp0s4 is the PCI Bridge to FreeNAS, but it isn't attached to an interface (eth0) am I right? Also the VM only supports the ip command and not ifconfig, since it is a slimmed-down version, only intended to run docker. Would be great to get your thought, since your linux skills seem to be way above mine. I would think that somehow linux messed up the interface creation if started from the CLI, but I'm not to familiar with the ip command.

 

[Patrick M. Hausen]

The enp0s4 is the emulated ethernet interface. With modern Linux versions there is no eth0, anymore. For $reasons. The other ones I don't know - possibly Docker? What does ip addr enp0s4 say? Or however this ip thingy works ;)

I guess the FreeNAS side is more interesting (and relevant), here. Is the bridge "up"? Is the tap interface "up"? Does the bridge have the physical interface of your FreeNAS server as a member as well as the tap interface? If not, do another ifconfig ... addm ...

P.S. And why does the forum software change "i p" to "IP"?

 

[UnchartedBull]

Regarding the FreeNAS side of things: bridge is up, tap3 is up, bridge is working correctly and connected to alc0 (connected to the router) (my PiHole is working without any issues during this whole time)

Regarding the docker system:
The only difference I can see during boot between the two systems is that the NetworkManager does timeout in the CLI case (which it doesn't if I boot via cli):

All the weird interface seem to be created from eth0 (at least I think that is what these logs mean):

The enp0s4 does not get a valid IPv4 address:

I found one very interesting log though. If I restart the Network Manager it will work just fine until it tries to get an IP from the router (Unifi USG). That operation does time out. Does this mean that the router is denying the request or could it also mean that FreeNAS just doesn't forward the request? Sorry for all the images, but I can't copy the logs out of the VNC viewer and that is currently the only way to access the machine.

Thanks!

 

[Patrick M. Hausen]

Use tcpdump -i bridgeX -n port bootpc to watch the DHCP traffic ...
About those interfaces: no clue, I never did what you are trying to do. Satisfied with running VMs from the UI. I just apply my general networking knowledge here.

 

[UnchartedBull]

the tcpdump does report a request if started over the UI, but doesn't report any if started via the CLI. So the problem isn't the router, but rather the interface bridge link not working I guess. I would be satisfied to start over the UI as well, if I can attach the PCI Controller there, which isn't possible I think.

I did notice one more thing though:
The auto generated tap interface if I start via the UI does have the flag "UP"

while the one created via the cli (and linked to the bridge) does not have this flag:

it is a member of bridge0 though:

Does that mean, that tap3 isn't actually active (up)? Sorry I somehow missed this flag last time, so I thought everything was alright. How would one "up" tap3?

 

[Samuel Tai]

You may be reusing tap3. I've noticed when I restart my VMs, they'll use the next consecutive tap interface, i.e., tap0->tap1->tap2, etc. Try ifconfig destroy tap3 before launching your script. Bhyve is supposed to create the tap interface and bring it up, and then add it to the bridge.

 

[UnchartedBull]

It's working now! I didn't reuse tap3 (I have 2 other vms (tap0, tap1 and UI HomeAssistant on tap3)). Problem was that I had to up the tap interface, now everything is working fine. Thanks so much for all the help! If I had read the logs correctly we probably would've been there earlier, so sorry for that!

For anyone wondering / encountering the same problem here is how it is working for me (using tap9 as an example here):

Code:

sudo ifconfig tap9 create
sudo ifconfig bridge0 addm tap9
sudo ifconfig tap9 up
sudo bhyve -A -H -S -w -c 2 -m 1300 -s 0:0,hostbridge -s 1:0,passthru,5/0/0 -s 31,lpc -l com1,/dev/nmdm6A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 29,fbuf,vncserver,tcp=192.168.5.10:6011,w=1920,h=1080,, -s 30,xhci,tablet -s 4:0,e1000,tap9,mac=00:a0:98:5a:xx:xx -s 3:0,ahci,hd:/dev/zvol/hassio/system 6_HassIO

Make sure to also use tap9 in your bhyve command (just before the mac address)

Now the last thing for me to do is to create a script with that commands and somehow run that at boot. Or ideally update the command that the UI uses. Does anyone know, if and how this works?