TrueNAS 13.0-U4 - Cannot create VM with more than 7 VirtIO disks

[aheid]

I'm getting the following error when I try to add an 8th VirtIO disk device (zvol) to a VM:

Unable to setup 'Debian' VM object: XML error: Invalid PCI address function=0x8, must be <= 7
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 1903, in initialize_vms
self.vms[vm_data['name']] = VMSupervisor(vm_data, self.middleware)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 96, in __init__
self.update_domain()
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 115, in update_domain
self.__define_domain()
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 130, in __define_domain
if not self.LIBVIRT_CONNECTION.defineXML(vm_xml):
File "/usr/local/lib/python3.9/site-packages/libvirt.py", line 4407, in defineXML
raise libvirtError('virDomainDefineXML() failed')
libvirt.libvirtError: XML error: Invalid PCI address function=0x8, must be <= 7

After I get this error, trying to start the VM fails with the following error:

application.call_method():174 - Exception while calling vm.start(*[3])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 139, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1247, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 1598, in start
self.vms[vm['name']].start(vm_data=vm)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 151, in __getattribute__
raise RuntimeError('Domain attribute not defined, please re-instantiate the VM class')
RuntimeError: Domain attribute not defined, please re-instantiate the VM class

If I edit the VM properties, say change its name, I can start the VM again. Without the new device of course.

Adding the new disk device as AHCI works as expected. FWIW the VM also has a VirtIO NIC attached.

Is this a known VirtIO limitation I'm unable to find?

 

[zithro]

Hello,

I don't have a direct answer to your problem, but may have a pointer.
AFAIK, Xen/QEMU does not allow more than 8 devices of the same type.
(I know you're not using this, but maybe guests limitations are the same on all virt platforms ?).
For example, I cannot use more than 8 network devices in a VM. It includes real (ie. PCI-PT) and virtual devices.
But on the bhyve manpage, I read :
Run a [...] virtual machine with 8 AHCI SATA disks, an AHCI ATAPI CD-ROM.

So, I don't know. As said, it's no more than a hint ! Not a real answer.
Sorry if that does not help, or does not apply to your situation.

EDIT: in fact I was true, the answer is in the log file : "Unable to setup 'Debian' VM object: XML error: Invalid PCI address function=0x8, must be <= 7". So, max 8 virtual devices of the same type