Halfway done low-budget ESXi build

[Shockwaver]

It's a proper virtualized switch

Problem is that OPNsense is having troubles with the drivers (both of the ones you may use in ESXi) when it comes to handling the PPPoE interface. No issues with the LAN interfaces though.

Could anyone kindly point me to some *budget saving* SAS HBA cards? I wish it not to be the bottleneck of my transfers to the SLOG (most likely an Intel S3500). I'd need just 4 ports!

 

[Shockwaver]

And that was that, dices are rolled. Final config:

Hypervisor: ESXi 6.7

MoBo: Supermicro MBD-X11SSH-LN4F
CPU: Intel Xeon E3-1230 v6
RAM: 64 GB DDR4-2400 ECC UDIMM (4x Crucial CT16G4WFD824A 16GB CL17)
Boot and VMs HDD: Samsung 960 PRO NVMe 512GB
CCTV HDD: 4TB 7200rpm (VMDK)
SAS HBA: LSI 9207-8i P20 IT Mode (passthrough) (thanks to this post it costed me just 100 bucks, with shipping to Italy, in place of 220)
NAS HDDs: 3x2TB Seagate Ironwolf (on SAS HBA)
SLOG: Intel DC S3500 80GB (on SAS HBA)

Very low end, still it will serve me well at home for some time being... I guess...

Thanks to everyone who helped me.
Feel free to throw your insults to this build :D

 

[Shockwaver]

If you want to pass the onboard SATA controller to FreeNAS that won't work on ESXi past 6.5 U2, 6.7 does not currently allow that. If you want to run 6.7 it would require an added controller that you can passthrough

Hi!
Could you point me to any resource stating this please? I don't seem to find any

 

[kdragon75]

Hi!
Could you point me to any resource stating this please? I don't seem to find any

This seems to be a generally known issue and has to do with the way the ESXi host filters devices internally. There may be a hacker way around this but in would involve modifying core configurations of ESXi and is far over my head.

 

[Shockwaver]

Ok, I mean, it's more like a bug introduced with the 6.5u2+ releases... It's not stated directly in the docs like "you won't be able to passthrough your onboard stuff", is it?

 

[kdragon75]

No it's not a bug. We home users are using PCI passthrough in unsupported configurations. If you want a system that you know for a fact will work as expected, ALL parts including the storage systems (iSCSI, FC, DAS, etc) need to be on VMware HCL.
If you suspect something in the docs, go look for them. VMware has amazing documentation.

 

[Shockwaver]

If you want a system that you know for a fact will work as expected, ALL parts including the storage systems (iSCSI, FC, DAS, etc) need to be on VMware HCL

Sure about that! Does that mean no embeded sata controllers are in the HCL for it not to work?
I mean if I was to find a chipset in the HCL (and I suspect there are... many...) it shouldn't matter whether it is in an HBA card or MoBo embedded, after all in the docs it is explained how to pass PCI devices and controllers, but nowhere is stated you cant (or shouldn't) pass an on-board controller

I could be wrong

 

[Sphinxicus]

nowhere is stated you can't (or shouldn't) pass an on-board controller

Have a read of this thread, post 17 & 23, where it is intimated that this should be resolved with 6.7u1

 

[Shockwaver]

No it's not a bug. We home users are using PCI passthrough in unsupported configurations. If you want a system that you know for a fact will work as expected, ALL parts including the storage systems (iSCSI, FC, DAS, etc) need to be on VMware HCL.
If you suspect something in the docs, go look for them. VMware has amazing documentation.

Take a look at this support for Sunrise Point AHCI https://www.vmware.com/resources/co..._interval=10&sortColumn=Partner&sortOrder=Asc. I have at least one of those on my old Asrock 90-MXGXA0-A0UAYZ Z170 where my ESXi 6.5 is still running waiting for the upfresh. It is passed-through AND supported in 6.7
Without specific stating of otherwise I EXPECT it to work on 6.7 while embedded, shouldn't I?

EDIT: sorry I wasn't able to make the link work... so you could just search for Sunrise Point AHCI at https://www.vmware.com/resources/compatibility/ within IO devices category

 

[Shockwaver]

Have a read of this thread, post 17 & 23, where it is intimated that this should be resolved with 6.7u1

I did see that thread, but I thought it just reffered to general/USB passthrough. Actually it could be a wider case, then... So it DOES smell like a bug...

 

[Sphinxicus]

I did see that thread, but I thought it just reffered to general/USB passthrough. Actually it could be a wider case, then... So it DOES smell like a bug...

not 100% if it does apply to your issue or not but since you have already committed to your hardware, there is no harm in installing 6.5uX and seeing if you can pass them through on that, upgrade to 6.7, if it breaks it, roll back to 6.5 again. Its only a home server so I'm assuming there is no call to have the latest and greatest version

 

[Shockwaver]

not 100% if it does apply to your issue or not but since you have already committed to your hardware, there is no harm in installing 6.5uX and seeing if you can pass them through on that, upgrade to 6.7, if it breaks it, roll back to 6.5 again. Its only a home server so I'm assuming there is no call to have the latest and greatest version

Right, I will anyway force my way with 6.7 when the hw will get to me. I will first try the onboard controllers just for curiosity and use the HBA in any case after that. I'm confident at least the latter will work with passthrough

 

[kdragon75]

Sure about that! Does that mean no embeded sata controllers are in the HCL for it not to work?
I mean if I was to find a chipset in the HCL (and I suspect there are... many...) it shouldn't matter whether it is in an HBA card or MoBo embedded, after all in the docs it is explained how to pass PCI devices and controllers, but nowhere is stated you can't (or shouldn't) pass an on-board controller

I could be wrong

There's a whole separate HCL for passthrough. and just because its not on the HCL doen not meant it wont work.

Without specific stating of otherwise I EXPECT it to work on 6.7 while embedded, shouldn't I?

Nope. Any two cards based on the same chip may have various tweaks to their firmwares or implementations from vendor to vendor or even between revisions.

 

[Shockwaver]

There's a whole separate HCL for passthrough

Do you have a spare link?

Any two cards based on the same chip may have various tweaks to their firmwares or implementations from vendor to vendor or even between revisions

Right, but in the HCL charts there is not more specific info about the HW. It shows just the chipset with its VID and DID and of course if I was to make a choice I'd pick one with same IDs (I didn't specify it but of course I was refferring to the case where ALSO those IDs are same between the 2 chipsets)

In the end I mean this:
They don't state you cannot perform on-board controllers passthrough? Alright, if it's in their HCL (with same device and vendor IDs) it must work. Otherwise sw or documentation will be needing a fix... I think that's quite peaceful

 

[kdragon75]

I stand corrected, there is not HCL for passthrough. There are technical limitations on some hardware though.
https://www.vmware.com/pdf/vsp_4_vmdirectpath_host.pdf
Problems with Device Assignment Dependencies