11.2-BETA3 New VM device order feature - problems/bugs?

[KrisBee]

11.2-BETA3 has introduced a new feature which allows the user to give VM devices an order number. The device order determines which PCI slot is used for a device when bhyve is invoked. Previously the user had no control over this ( see: https://redmine.ixsystems.com/issues/42371 ). Potentially this should allow VM configs via the new WebUI that could only be achieved using iohyve & the CLI in the past. For example, create a VM for FreeNAS master with multiple disks attached. Or make use of UEFI-CSM.

But, there is as yet, no documentation to say how this is supposed to function, nor from brief testing is it clear that there is a simple mapping between a VM device's "order number" and the bhyve pci slot it is attached to. For one thing, the behaviour is dependent on whether you use AHCI or VIRTIO mode for disk devices.

One obvious problem is that users moving from BETA2 to BETA3 are finding their linux VMs have lost network connectivity. For distros that use "predictable network interface names" like enp0s3, instead of eth0, the interface name can change (e.g becomes enp0s4) in BETA3 as the NIC is attached to a different pci slot. The solution is to change the NIC "device order" number in the WebUI.

Currently, there is a bug which prevents additional disk devices from being added to an existing VM via the new webUI. You have to go the old UI to add any additional disk devices, then return to the new webUI.

https://redmine.ixsystems.com/issues/46029

 

[Tim1962]

I briefly tried 11.2-Beta3 but found this issue affected my VW so reverted back to 11.2-Beta2. I find network interfaces a dark art that I have never mastered (yet). Could someone explain in simple terms how to achieve the following

"The solution is to change the NIC "device order" number in the WebUI."

Can it be done in 11.2-Beta2 or is it dependent on Beta3 features?

Many thanks Tim

 

[KrisBee]

My God, using 11.2-Beta3 affected your VW? So much for German engineering! ;)

Joking apart, I'll post a brief explanation later today.

 

[Patrick M. Hausen]

One obvious problem is that users moving from BETA2 to BETA3 are finding their linux VMs have lost network connectivity. For distros that use "predictable network interface names" like enp0s3, instead of eth0, the interface name can change (e.g becomes enp0s4) in BETA3 as the NIC is attached to a different pci slot. The solution is to change the NIC "device order" number in the WebUI.

Good grief! Thank you so much - exactly my current problem with BETA3. Waiting for your explanation, then ... ;)

Patrick

 

[Tim1962]

My God, using 11.2-Beta3 affected your VW? So much for German engineering! ;)

Joking apart, I'll post a brief explanation later today.

Have you not tested FreeNAS for emissions!
Thanks in advance

 

[KrisBee]

This example shows how a linux VM can loose network connectivity when upgrading from 11.2-BETA2 to 11.2-BETA3 and how it can be restored.

In 11.2-BETA3 all devices attached to VMs are given a "device order number" by the system, whether the VM is pre-existing or newly created. Images vm1 and vm2 below show the system allocated "order numbers" for a pre-existing simple debian VM. The bhyve command invoked when this VM is started can be seen by using the command tail -f /var/log/middleward.log .

Code:

2018/09/14 07:00:04] (DEBUG) VMService.run():259 - Starting bhyve: bhyve -A -H -w -c 2 -m 1024 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm16A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 4,virtio-net,tap0,mac=00:a0:98:71:dd:6f -s 29,fbuf,vncserver,tcp=192.168.0.151:5743,w=1024,h=768,,wait -s 30,xhci,tablet -s 3:0,ahci,hd:/dev/zvol/NasPool/VM/vmtest4 16_vmtest4
[2018/09/14 07:00:04] (DEBUG) VMService.run():272 - ==> Start WEBVNC at port 5643 with pid number 4377
[2018/09/14 07:00:04] (DEBUG) VMService.run():281 - vmtest4: 14/09/2018 08:00:04 Listening for VNC connections on TCP port 5743
[2018/09/14 07:00:04] (DEBUG) VMService.run():281 - vmtest4: 14/09/2018 08:00:04 Listening for VNC connections on TCP6 port 5743

Splitting the bhyve command over separate lines for clarity gives:

Code:

bhyve -A -H -w -c 2 -m 1024 /
-s 0:0,hostbridge -s 31,lpc /
-l com1,/dev/nmdm16A /
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd /
-s 4,virtio-net,tap0,mac=00:a0:98:71:dd:6f /
-s 29,fbuf,vncserver,tcp=192.168.0.151:5743,w=1024,h=768,,wait  /
-s 30,xhci,tablet \
-s 3:0,ahci,hd:/dev/zvol/NasPool/VM/vmtest4 16_vmtest4

Note in this case the NIC device has been allocated to bhyve pci slot 4.

Using VNC to access the debian VM shows its network interface is given the "predictable interface name" of enp0s4. But there is no network connectivity. Looking at the network config file shows this VM was using enp0s3 as its interface name. This is shown in image vm3 below.

To restore the VM's network connectivity you could: (A) reconfigure the VM's network, or (B) change the device order number via the new webUI.

The snag with option (B) is that this feature is so far undocumented and there is not necessarily a simple mapping between the device order number as seen in the webUI and the slots used when bhyve is invoked. However, in this case as we need the NIC attached to slot 3 and not slot 4, we can set the "order number" of the NIC device to that used by the disk device. Just changing the NIC "order number" to 1001 automatically changes those of other attached devices. See image vm4 below.

When the same debian VM is started after this simple change, the bhyve command becomes this:

Code:

bhyve -A -H -w -c 2 -m 1024 -s 0:0,hostbridge -s 31,lpc -l com1,/dev/nmdm16A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3,virtio-net,tap0,mac=00:a0:98:71:dd:6f -s 29,fbuf,vncserver,tcp=192.168.0.151:5743,w=1024,h=768,,wait -s 30,xhci,tablet -s 4:0,ahci,hd:/dev/zvol/NasPool/VM/vmtest4 16_vmtest4

Note the NIC device is now attached to slot 3! Image vm5 shows network connectivity is restored.

 

[Tim1962]

OK, I am a bit confused but (partially) sorted!

Tried to follow this but struggled to make progress. HOWEVER almost by accident I noticed via the UI if you access the VM page and click the 3 dots the bottom item is change device order. Selected the NIC and made it 1001 and the rest reorder. Rebooted the VM and it worked. Job done!!!!

Next issue (but new thread I think), is the new FreeNAS UI on Safari doesn't seem to work & is 99% unusable for me...

Best wishes. Tim

 

[vinistois]

Ok thanks for this!!

When upgrading to beta3, the nic in freenas "interface" section was deleted and I did not notice, which played absolute havok with everything.

Finally I reset the config and was able to re-add the nic properly and regain access to the box, only to find out my VMs had completely disappeared (storage pool was missing)

I went to storage import pool and got it back. VMs still missing.

I created a new VM and attached the appropriate disk from the pool. I used /nonexistant as a CDrom, then went to devices and deleted the CDrom device.

I managed to get my VM booting again, but still the networking was down. I noticed, as per this thread, that it was looking for enp0s3, but the nic in Ubuntu was named enp0s4. The nic wouldn't come "up" and wouldn't automatically dhcp anymore.

So I edited the cloud-init file:

Code:

 a $ sudo nano /etc/netplan/50-cloud-init.yaml

and simply changed the name in there from enp0s3 to enp0s4. Save and reboot.

NOW the VM boots properly, survives a reboot, and survives a reboot of the whole box.

THANKS for this post!

Now if I can only get my rancher VM back up....

 

[Patrick M. Hausen]

Thanks again, I just rebooted into BETA3 again, changed the configured NIC in my Ubuntu VM - everything up and running again.

Patrick

P.S. Yes, the UI feels even more quirky in Safari, now. No reproducible single bug to report, yet ... I already did report the missing menus in the VM section.

 

[ukos]

@KrisBee Thank you so much for the detailed explanation. I was having trouble finding the issue myself. Now all debian vms are up and running again!

It is also notable that changing the device number by using the edit button will not automatically reassign the ordering of other devices.