SOLVED PCI Passthrough failure - TrueNAS Host, PfSense Guest

[homer27081990]

Hello, I seem to have hit a wall in my effort to use PfSense with passthrough NICs. Specifically, after trying the recommended approach, after finding and implementing an alternative solution, together with some extensive log-digging sessions, I came to the conclusion that I have to ask for help. I am currently completely unable to passthrough the NICS in question, or, rather, 3 out of the 4 ports on the card.
My system specs, witch I am only posting because of a possible correlation to the underlying problem are:

HP ProLiant DL380P Gen 8 (8.1 software-firmware) second hand system
Intel Xeon E5-2640 @ 2.5 GHz 6 Core CPUs (2 of them)
(TrueNAS says Broadcom) HP Ethernet 1Gb 4-port 331FLR Adapter (iLO says embedded, but in TrueNAS withought passthrough is listed PCIe)
HP Smart Array P420i SAS-SATAIII Controller
250GB Test SSD, SATA III 2.5 inch Kingston drive
4 X 4GB DDR-III RAM RDIMMs (non HPE memory)
Patsburg 6 Port SATA AHCI Controller (having no ports, I imagine this is for the USB's, DVD and iLO)
TrueNAS-13.0-U1.1
VM1 - PfSense 2.6.0 x64
The Host is managed both from WebGUI and iLO remotely. The PfSense guest through VNC, UEFI boot (tried every other type, from both the installer and VM setup).

I first tried following the official guide for PCI passthrough, witch consists of the tunables vmm_load="YES" type LOADER and pptdevs="3/0/1 3/0/2 3/0/3" (those are the correct IDs) rebooting the host and starting the PfSense VM. No joy.
I then tried altering the scheme, using vmm_load=YES type LOADER and pptdevs2="3/0/1 3/0/2 3/0/3". Still no joy.
Interesting parts from Host log on boot:

Code:

Firmware Warning (ACPI): Invalid lenght for FADT/Pm1aControlBlock: 32, using default 16 (20201113/tbfadt-850)
Firmware Warning (ACPI): Invalid lenght for FADT/Pm2ControlBlock: 32, using default 16 (20201113/tbfadt-850)

acpi0: _OSC failed: AE_BUFFER_OVERFLOW

pcieb3: <ACPI PCI-PCI bridge> at device 2.0 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
pci0:3:0:0: VPD read timed out
pci0:3:0:0: failed to read VPD data.
bge0: <Broadcom unknown BCM5719, ASIC rev. 0x5719001> mem 0xf6bf0000-0xf6bfffff,0xf6be0000-0xf6beffff,0xf6bd0000-0xf6bdffff irq 32 at device 0.0 numa-domain 0 on pci3
bge0: APE FW version: NCSI v1.3.16.0
bge0: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E
miibus0: <MII bus> numa-domain 0 on bge0
brgphy0: <BCM5719C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Using defaults for TSO: 65518/35/2048
bge0: Ethernet address: xx:xx:xx:xx:xx:xx
pci0:3:0:1: VPD read timed out
pci0:3:0:1: failed to read VPD data.
ppt0 mem 0xf6bc0000-0xf6bcffff,0xf6bb0000-0xf6bbffff,0xf6b70000-0xf6b7ffff irq 36 at device 0.1 numa-domain 0 on pci3
pci0:3:0:2: no valid VPD ident found
pci0:3:0:2: failed to read VPD data.
ppt0 mem 0xf6bc0000-0xf6bcffff,0xf6bb0000-0xf6bbffff,0xf6b70000-0xf6b7ffff irq 32 at device 0.2 numa-domain 0 on pci3
pci0:3:0:3: no valid VPD ident found
pci0:3:0:3: failed to read VPD data.
ppt0 mem 0xf6bc0000-0xf6bcffff,0xf6bb0000-0xf6bbffff,0xf6b70000-0xf6b7ffff irq 36 at device 0.3 numa-domain 0 on pci3

Any help will be greatly appreciated.

Update 1
I am currently trying again, this time with virtio_load YES and virtio_pci_load YES in pfsense (guest) /boot/loader.conf. Btw, I know this could be construed as a pfSense question, but, seeing as this is about loader.conf, it is a FreeBSD question and thus, some of the digital sorcerers of the forum might provide some insight, what is hw.hh.use_if_start and hw.hh.vf_transparent? I cannot find anything about them anywhere. Only result in Google is a line of code in GitHub, with no explanation whatsoever.

Update 2
As expected, virtualized IO drivers had no effect. Will hail-Mary the hw.hh.use_if_start to 0.

Update 3
Changing hw.hh.use_if_start="1" to hw.hh.use_if_start="0" had no effect. I am now at a dead end.

Update 4
OPNSense not viewing the NICs. Trying linux to see if it is a bhyve issue or a freebsd guest issue.

 

[homer27081990]

Ubuntu did not see the NICs. Can anyone help with this?

 

[homer27081990]

Never mind. You can only pass full IOMMU groups. Please mark as solved.

 

[homer27081990]

Still I cannot, for the life of me, understand why the bleeping 1st port I did not passthrough kept on working...

 

[homer27081990]

Hahaha... After sooo many hours and all this debacle, right after installing one more Realtek Gigabit NIC (PCIe 4X), ... haha... the system hang after boot on middlewared loading plugins 6/60 (system) , at witch point, it failed... Some of the tunables must be the culprit, because the upgrade setup I am now performing (no more energy for debugging) is doing just fine.

 

[homer27081990]

Changed pci numbering, because of the addition of the realtek NIC, together with the passthrough tunables, was the culprit. Reinstalled 2 times because of this, one upgrade (database failed to transfer), one from the top. There has got to be a better way to handle the addition of hardware. Anyway, passed all 4 broadcom ports, pfsense sees them normally, I am going to sleep...