melloa
Wizard
- Joined
- May 22, 2016
- Messages
- 1,749
If someone is thinking about trying XCP-ng as a hypervisor and run FreeNAS or FreeBSD, here's some quick PCI pass through commands I've tested and worked:
Assuming XCP-ng is installed and the VM is created, the first step is to find out the addresses for your PCI interfaces. The way to do that is with lspci from the hypervisor shell.
Step 1 - Find the PCI interface address
You can be more specific using grep and the manufacture for your device. For instance, if you want to get the information for your LSI HBA:
Output:
Make a note of the interface address, 04:00.0 in the case above.
Step 2 - Exclude the interface from dom0
With that interface address handy, you can change XCP configuration using the following command: /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=<PCI Device Address>"
In this example:
Step 3 - Reboot is needed to ensure that device is no longer in use by XCP.
Step 4 - Obtain the VM UUID
From the shell command:
Output:
Make a note of the VM UID, bfc9ea79-431c-ebb2-6e90-051910d333b4, in this example.
Step 5 - Add the passed through PCI interface to the VM
Use the following command:
After those five steps you will be able to start your VM and use ZFS to manager your pool.
In case you have more than one HBA you'd like to assign to a VM, you will get the information for both HBAs and the command for step 5 will look like:
In the case above two PCI interfaces with addressed 01:00.0 and 04:00.0 are been assigned to the VM. That was tested with one HBA and one Chelsio T320:
Assuming XCP-ng is installed and the VM is created, the first step is to find out the addresses for your PCI interfaces. The way to do that is with lspci from the hypervisor shell.
Code:
[root@xcp-lab ~]# lspci 00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 22) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22) 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22) 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22) 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22) 00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22) 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22) 00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22) ...
Step 1 - Find the PCI interface address
You can be more specific using grep and the manufacture for your device. For instance, if you want to get the information for your LSI HBA:
lspci | grep LSI
Output:
Code:
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02
Make a note of the interface address, 04:00.0 in the case above.
Step 2 - Exclude the interface from dom0
With that interface address handy, you can change XCP configuration using the following command: /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=<PCI Device Address>"
In this example:
/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(04:00.0)"
Step 3 - Reboot is needed to ensure that device is no longer in use by XCP.
Step 4 - Obtain the VM UUID
From the shell command:
xe vm-list
Output:
Code:
uuid ( RO) : 827ba682-920e-40b2-9e7c-2860b68aaa76 name-label ( RW): Control domain on host: xcp-lab power-state ( RO): running uuid ( RO) : bfc9ea79-431c-ebb2-6e90-051910d333b4 name-label ( RW): FreeNAS11 power-state ( RO): running
Make a note of the VM UID, bfc9ea79-431c-ebb2-6e90-051910d333b4, in this example.
Step 5 - Add the passed through PCI interface to the VM
Use the following command:
Code:
xe vm-param-set other-config:pci=0/0000:<B:D.f> uuid=<vm uuid> Where: <B:D.f> is the PCI Device ID (04:00.0 in this example)
xe vm-param-set other-config:pci=0/0000:04:00.0 uuid=bfc9ea79-431c-ebb2-6e90-051910d333b4
After those five steps you will be able to start your VM and use ZFS to manager your pool.
In case you have more than one HBA you'd like to assign to a VM, you will get the information for both HBAs and the command for step 5 will look like:
xe vm-param-set other-config:pci=0/0000:01:00.0,0/0000:04:00.0 uuid=88c901ac-b98b-e576-9999-2671907dc298
In the case above two PCI interfaces with addressed 01:00.0 and 04:00.0 are been assigned to the VM. That was tested with one HBA and one Chelsio T320:
Code:
[root@xcp-lab ~]# lspci | grep Che 01:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter