hypervisor + virtualized Truenas

[phier]

VMware documentation link for 7.0 below:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-5B3CAB26-5D06-4A99-92A0-3A04C69CE64B.html

If you don't have vCenter you may need to use a different process from the vSphere Host Client:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.hostclient.doc/GUID-2B6D43A6-9598-47C4-A2E7-5924E3367BB6.html

Sam might have been working under the assumption that you are currently booting ESXi from a device attached to one of the SATA ports on this controller, which of course would make the SATA controller be "in use" by the hypervisor and therefore unavailable for passthrough. In this case, you are booting from a separate NVMe device and the SATA controller will not be in use.

thanks so much! in that case i will try it tomorrow once memtest finished/

 

[phier]

If you don't have vCenter you may need to use a different process from the vSphere Host Client:

https://docs.vmware.com/en/VMware-v...UID-2B6D43A6-9598-47C4-A2E7-5924E3367BB6.html

i dont think it will work .. @honeyeyo

Add a PCI Device to a Virtual Machine in the VMware Host Client​

DirectPath I/O allows the guest operating system of a virtual machine to access the physical PCI and PCIe devices connected to a host directly. By using this technology, you can connect each virtual machine to up to sixteen physical PCI devices. You can use Dynamic DirectPath I/O to assign multiple PCI passthrough devices to a virtual machine. Starting with vSphere 7.0, you can identify the PCI passthrough devices by their vendor and model name.



local SATA ports on mainboard are PCI device... ?

 

[HoneyBadger]

i dont think it will work .. @honeyeyo

Add a PCI Device to a Virtual Machine in the VMware Host Client​

DirectPath I/O allows the guest operating system of a virtual machine to access the physical PCI and PCIe devices connected to a host directly. By using this technology, you can connect each virtual machine to up to sixteen physical PCI devices. You can use Dynamic DirectPath I/O to assign multiple PCI passthrough devices to a virtual machine. Starting with vSphere 7.0, you can identify the PCI passthrough devices by their vendor and model name.



local SATA ports on mainboard are PCI device... ?

Yes, your SATA ports are controlled by a controller (Intel C232) which is attached to the PCIe bus.

 

[phier]

Yes, your SATA ports are controlled by a controller (Intel C232) which is attached to the PCIe bus.

oh okay! thanks! i thought its somehow connected somewhere else... no idea where :)

 

[phier]

@HoneyBadger still not clear sorry ...

There could also be a mixture of these options. A B550 motherboard could for instance have 2 SATA ports directly connected to the CPU, along with 6 off the South Bridge, and then however many more the designer of the motherboard wanted off additional SATA controllers chips connected to the PCIe bus.

so what if these are connected to the CPU? or based on the Intel C232 you know its attached to PCIe on board?

trying to find out it from spec... https://www.intel.com/content/www/us/en/products/sku/90578/intel-c232-chipset/specifications.html

maybe more straightforward from there> https://www.tomshardware.com/news/intel-c232-c236-chipsets-xeon,31051.html

 

[HoneyBadger]

@HoneyBadger still not clear sorry ...

There could also be a mixture of these options. A B550 motherboard could for instance have 2 SATA ports directly connected to the CPU, along with 6 off the South Bridge, and then however many more the designer of the motherboard wanted off additional SATA controllers chips connected to the PCIe bus.

so what if these are connected to the CPU? or based on the Intel C232 you know its attached to PCIe on board?

trying to find out it from spec... https://www.intel.com/content/www/us/en/products/sku/90578/intel-c232-chipset/specifications.html

maybe more straightforward from there> https://www.tomshardware.com/news/intel-c232-c236-chipsets-xeon,31051.html

We're digressing quite a bit more, but SATA ports hanging directly off a CPU package would mean there's an I/O controller on the package that speaks SATA and we're entering the SoC (System-on-Chip) family of computing. Not nearly as common in the x86 space.

PCIe lanes directly off the CPU though is a different and very common thing.

See attached block diagram for a B550 motherboard. Notice that the SATA ports all come from the chipset, which will expose the SATA controller as it's own device on the PCIe bus. Even the AMD G-series "SoC" chips still do this, I believe; it's just an internal PCIe link to the I/O controller rather than it being part of the CPU core logic.

 

[phier]

@HoneyBadger
hello, seems its doesnt work .. :(

in the host config ... its not availble to be passthrough ... apart of the Ethernet cards nothing can be passthrough.. even USB cant be .. .
any idea here?

thanks!

going thru the https://docs.vmware.com/en/VMware-v...UID-2B6D43A6-9598-47C4-A2E7-5924E3367BB6.html
> i am not able to create passthrough device ;/ the PCIe is grayed out ...

 

[Patrick M. Hausen]

Your ESXi installation is NOT on a drive connected to SATA, right?

 

[phier]

@Patrick M. Hausen yes on nvme drive installed in nvme->Pcie port and plugged into the pcie port.

 

[Patrick M. Hausen]

OK, so it seems ESXi and your mainboard in combination are not capable of pass through of that on board SATA controller.

So redesign/rethink time ...

- buy a PCIe HBA and use that to connect the drives
- use SATA for ESXi and go all NVMe for TrueNAS
- drop ESXi and use TrueNAS for virtualisation
- ...

I'm very satisfied with CORE without ESXi, why don't you give it a try?

 

[phier]

@Patrick M. Hausen

I made it work>

Help with SATA Controller Passthrough (supermicro) from ESXi 7 to Truenas

@HoneyBadger this is whats there: lspci 0000:00:17.0 SATA controller: Intel Corporation Sunrise Point-H AHCI Controller [vmhba0] thats what i added cat /etc/vmware/passthru.map 8086 a102 d3d0 false and thats how it looks like after reboot.... does it mean it works and its safe? thanks!

www.truenas.com

- use SATA for ESXi and go all NVMe for TrueNAS

dont get that option? could u pls elaborate?

 

[phier]

@Patrick M. Hausen

"You do not load the VM from the passed-through HBA. You load the VM from a VMFS datastore in ESXi. Which most probably will be a local SATA drive in your case."

So as far as i understand properly, nvme drive -> pcie will be attached (Accessed) by ESXi and VMFS datastore will be created on it.
Hope my understanding is correct.


edit:
maybe one more important ... Does Hypervisor ie ESXi - needs also some RAM for its operation/running? is it possible to say how many?

thanks

 

[Patrick M. Hausen]

So as far as i understand properly, nvme drive -> pcie will be attached (Accessed) by ESXi and VMFS datastore will be created on it.

Yes.

 

[phier]

Yes.

cool thanks.
and if there is a point to have that drive in mirror - raid1 in case i will snasphost VMs to truenas pool/dataset... ?

 

[Patrick M. Hausen]

ESXi cannot mirror drives, so no. You need to rely on backups to a TrueNAS backed ZFS pool.

 

[phier]

i see or run these on hw raid 1 .. but its nonsense i think. overload...

regarding the ESXi ram ... i thought 4-8 max is enough but as per spec ...
ESXi 7.0 requires a minimum of 8 GB of physical RAM. Provide at least 12 GB of RAM to run virtual machines in typical production environments.

so seems 12GB ram .. for ESXi?:(

 

[Patrick M. Hausen]

You need 8 G at least for TrueNAS and I'd recommend 16. And I would add another 16 if I wanted to run any more VMs beside TrueNAS which seems to be your intention. We have been talking in and out about a server system with NVMe, a dedicated LSI HBA, several SATA disk drives connected to that and you are debating 8 G of memory? You must be trolling, possibly all the time. Laptops have 32 G today ...

 

[phier]

i dont wanna run VM inside TRUENAS.
I am talking all the time about
baremetal -> ESXi ..... Truenas as VM (16G) , other VM as per need...
yes 8G for ESXi. as per spec recco is 12G.

 

[Patrick M. Hausen]

TrueNAS needs at the very least 8 G of memory to run. Yes, without any VMs in TrueNAS. And that memory needs to be reserved to the VM in the ESXi VM config and cannot be shared. Then you need memory for all other VMs you want to run. Unbuntu? 4 G or something? Two of them? 8 G. Windows 10? 8 G memory to be in any way responsive. So you need at least 16 G for ESXi and a couple of smaller VMs and an additional 8 G for the TrueNAS VM alone.

 

[phier]

dont get it ...i said i can reserver/not share for TruenasVM 8-12-16Gb if required.

yes ubuntu /windows etc will need some mem ... as per use/req ...

but i dont understand how did u come up to that number that i will at least 16G for Esxi?

using ESXi setup;
you need ram for VMs (Truenas/windows, linux etc... )

and question was ... how much ram needs ESXi itself. .. as per documentation it says: ESXi 7.0 requires a minimum of 8 GB of physical RAM. Provide at least 12 GB of RAM to run virtual machines in typical production environments.

even you are not reading it correctly or i am not able to write it properly ...