PCI pass through for TV tuner card stopped passing through.

[samurai336]

Good afternoon the tl;dr is that I cannot figure out why the PCI pass through in my TrueNAS scale install has stopped working. I've tried a lot of different things including reinstalling TrueNAS scale, moving the card to a different PCI slot and about a dozen other things which I have described in detail below.

I have also described all the different things I did that led up to this issue and all the solutions I've tried so far to diagnose and remedy. At this point though I have no idea why its now working.

--Back story--

In the beginning I had an old daily driving PC I replaced with some fairly good specs. It has an intel i7-4770 CPU and 32GB of ram. I built it as a gaming PC back in 2013 that I was using up until a few months ago when I replaced it after it had some issues with handling a newer graphics card.

Not wanting to throw it out or have it become a dust collector, I put linux mint on there bought a TV tuner card (Hauppauge 1196 WinTV HVR-1265 PCI Express), attached an antenna to it and hooked it up to my living rooms TV, with Plex and nextPVR. This worked great as a little media server for several months in my living room with no problems.

However, inspired to degooglify my life I decided it might be better to turn that machine into something that could handle such things better and I had read and heard about TrueNAS scale. In my research I saw that while it was not quite there to pass hardware through to a docker image, a Virtual Machine should be no problem. So I decided to get some extra HDDs and install TrueNAS scale.

--Discovering the Problem--

I got TrueNAS scale up and running on this machine very quickly and set up my first pool. After tooling around with the provided Plex app I decided a VM is what I really needed. So I set up an ubuntu server vm in TrueNAS. I tried to add hardware but the list was empty. After some research(https://www.truenas.com/community/t...ie-device-such-as-a-network-card-to-vm.95635/) I discovered there may be some settings not enabled on my motherboard. So into the bios I went, virtualization was enabled but then I dug around and found that vt-d was not enabled so I turned that on and booted back into TrueNAS. I was then able to see all the PCI devices and could add them to the VM.

Using `lspci` I found the TV tuner card and made sure to add that as a PCI passthrough device in the VM settings then booted up the VM, installed nextPVR and did a channel scan and sure enough I had broadcast TV coming through the VM to my web browser on my other desktop.

From here I went to set up the NFS mounting for this media server VM, but to do that I had to set up a network bridge. As an aside I must say that following these directions from TrueNAS was a complete PITA and required several reboots to get changes to stick (https://www.truenas.com/docs/scale/virtualization/accessingnasfromvm/). Finally the VM could ping the internet and the host machine.

At this point I wanted to add another pair of HDDs to the machine so I shut the machine down and plugged an additional pair of HDDs. Turned it back on and created a second Pool with them.

Happy with my creation, I decided to double check the media servers nextPVR instance to see if the channel guide had updated and when I clicked to watch a channel at the bottom I saw `Streaming Failed (transcoder exited)`. I slowly went through rebooting everything; the VM, the machine TrueNAS was on but kept running into the same issue.

--Diagnosis–

It was previously working. I had seen it with my own eyes, so I wasn’t sure what had changed. I started with the last non hardware thing I did which was the network bridge. So I deleted that and put it back to how it was and that did not solve the problem.

I then suspected it was a power draw problem so I opened the machine back up and unplugged the power from the two additional HDDs I added and the same problem when I booted back up.

Next to make sure it was not an actual hardware problem, I took a USB of Linux Lite and I booted that and installed nextPVR and told it to channel scan and it worked and I was once again able to stream live TV to my web browser. So the hardware was fine.

Next I moved the TV tuner PCI card to a different PCI slot and changed the VM to point to the new PCI address. Same problem.

Next I spun up a new VM and added the TV cards PCI address as a passthrough to that new VM and installed nextPVR there and still the same problem.

Next I started looking into dmesg to see if there were any problems. No errors reported.

Code:

[    5.967675] tveeprom: Hauppauge model 161111, rev A1I6, serial# 4036167623
[    5.967676] tveeprom: MAC address is 00:0d:fe:93:07:c7
[    5.967677] tveeprom: tuner model is SiLabs Si2157 (idx 186, type 4)
[    5.967677] tveeprom: TV standards NTSC(M) ATSC/DVB Digital (eeprom 0x88)
[    5.967678] tveeprom: audio processor is CX23888 (idx 40)
[    5.967678] tveeprom: decoder processor is CX23888 (idx 34)
[    5.967679] tveeprom: has no radio, has IR receiver, has no IR transmitter
[    5.967679] cx23885: cx23885[0]: hauppauge eeprom: model=161111
[    5.986754] Console: switching to colour frame buffer device 240x67
[    6.011266] amdgpu 0000:01:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[    6.036847] cx25840 8-0044: cx23888 A/V decoder found @ 0x88 (cx23885[0])
[    6.041001] b43-phy0: Broadcom 4352 WLAN found (core revision 42)
[    6.041634] b43-phy0 ERROR: FOUND UNSUPPORTED PHY (Analog 12, Type 11 (AC), Revision 1)
[    6.041987] b43: probe of bcma0:1 failed with error -95
[    6.042221] Broadcom 43xx driver loaded [ Features: PNLS ]
[    6.049063] [drm] Initialized amdgpu 3.40.0 20150101 for 0000:01:00.0 on minor 0
[    6.668160] cx25840 8-0044: loaded v4l-cx23885-avcore-01.fw firmware (16382 bytes)
[    6.697143] intel_rapl_common: Found RAPL domain package
[    6.697376] intel_rapl_common: Found RAPL domain core
[    6.697590] intel_rapl_common: Found RAPL domain dram
[    6.748200] cx23885: cx23885[0]: registered device video0 [v4l2]
[    6.748484] cx23885: cx23885[0]: registered device vbi0
[    6.748807] cx23885: cx23885[0]: alsa: registered ALSA audio device
[    6.748809] cx23885: cx23885_dvb_register() allocating 1 frontend(s)
[    6.749080] cx23885: cx23885[0]: cx23885 based dvb card
[    6.761028] si2157 7-0060: Silicon Labs Si2147/2148/2157/2158 successfully attached
[    6.761353] dvbdev: DVB: registering new adapter (cx23885[0])
[    6.761595] cx23885 0000:06:00.0: DVB: registering adapter 0 frontend 0 (LG Electronics LGDT3306A VSB/QAM Frontend)...
[    6.762261] cx23885: cx23885_dev_checkrevision() Hardware revision = 0xd0
[    6.762559] cx23885: cx23885[0]/0: found at 0000:06:00.0, rev: 4, irq: 16, latency: 0, mmio

: 0xf0800000


No errors reported.

The latest thing I’ve done is replace the battery on my machine's motherboard and after it reset the CMOS, turned virtualization and vt-d back on, and checked it turned on correctly which it looks like it did.

Code:

dmesg | grep -i -e DMAR -e IOMMU
[    0.000000] Command line: BOOT_IMAGE=/ROOT/22.02.RELEASE@/boot/vmlinuz-5.10.93+truenas root=ZFS=boot-pool/ROOT/22.02.RELEASE ro console=ttyS0,9600 console=tty1 libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1
[    0.007451] ACPI: DMAR 0x00000000DD955738 000080 (v01 INTEL  HSW      00000001 INTL 00000001)
[    0.007467] ACPI: Reserving DMAR table memory at [mem 0xdd955738-0xdd9557b7]
[    0.018013] Kernel command line: BOOT_IMAGE=/ROOT/22.02.RELEASE@/boot/vmlinuz-5.10.93+truenas root=ZFS=boot-pool/ROOT/22.02.RELEASE ro console=ttyS0,9600 console=tty1 libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1
[    0.018097] DMAR: IOMMU enabled
[    0.091269] DMAR: Host address width 39
[    0.091410] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.091604] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.091888] DMAR: RMRR base: 0x000000dd8c8000 end: 0x000000dd8d4fff
[    0.092114] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.092343] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.092539] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.093041] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.423257] iommu: Default domain type: Passthrough (set via kernel command line)
[    1.176592] DMAR: No ATSR found
[    1.176749] DMAR: dmar0: Using Queued invalidation
[    1.176967] pci 0000:00:00.0: Adding to iommu group 0
[    1.177164] pci 0000:00:01.0: Adding to iommu group 1
[    1.177357] pci 0000:00:14.0: Adding to iommu group 2
[    1.177552] pci 0000:00:16.0: Adding to iommu group 3
[    1.177744] pci 0000:00:19.0: Adding to iommu group 4
[    1.177937] pci 0000:00:1a.0: Adding to iommu group 5
[    1.178129] pci 0000:00:1b.0: Adding to iommu group 6
[    1.178321] pci 0000:00:1c.0: Adding to iommu group 7
[    1.178515] pci 0000:00:1c.1: Adding to iommu group 8
[    1.178707] pci 0000:00:1c.3: Adding to iommu group 9
[    1.178900] pci 0000:00:1d.0: Adding to iommu group 10
[    1.179104] pci 0000:00:1f.0: Adding to iommu group 11
[    1.179299] pci 0000:00:1f.2: Adding to iommu group 11
[    1.179538] pci 0000:00:1f.3: Adding to iommu group 11
[    1.179730] pci 0000:01:00.0: Adding to iommu group 1
[    1.179917] pci 0000:01:00.1: Adding to iommu group 1
[    1.180110] pci 0000:03:00.0: Adding to iommu group 12
[    1.180305] pci 0000:04:00.0: Adding to iommu group 13
[    1.180500] pci 0000:05:01.0: Adding to iommu group 14
[    1.180695] pci 0000:05:04.0: Adding to iommu group 15
[    1.180894] pci 0000:05:05.0: Adding to iommu group 16
[    1.181089] pci 0000:05:06.0: Adding to iommu group 17
[    1.181284] pci 0000:05:07.0: Adding to iommu group 18
[    1.181479] pci 0000:05:08.0: Adding to iommu group 19
[    1.181674] pci 0000:05:09.0: Adding to iommu group 20
[    1.181872] pci 0000:06:00.0: Adding to iommu group 21
[    1.182069] pci 0000:08:00.0: Adding to iommu group 22
[    1.182266] pci 0000:0a:00.0: Adding to iommu group 23
[    1.182463] pci 0000:0c:00.0: Adding to iommu group 24
[    1.182701] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.197929] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.
[    1.340976]     intel_iommu=on

And still nothing. Finally I thought I’d pull out a big hammer and reinstall the TrueNAS scale. Which I did but I didn’t nuke my old settings (it did not seem to give me the option) this did not work but I do wonder if I should escalate to that.

--conclusion–


So I am stumped both on what could be wrong and how to diagnose this.

Please let me know your thoughts or any additional information I might be able to provide

Thanks

 

[sretalla]

Not sure about this, but if you're passing it into a VM, I'm pretty sure you don't want to see the host loading drivers for it.

Maybe that's what "changed" in that a driver was somehow added to the build that wasn't there before and is now attaching to the card?

I haven't looked into having a driver not load or reserving a device for passthrough (I think it's the iommu groups), but you might want to look into that.

 

[HoneyBadger]

I know it's ProxMox related and not SCALE, but it's using the same Linux/KVM underpinnings - here's how to bind the vfio-pci driver (telling it to use the device for passthrough only) or blacklist it (don't load that driver at all) on the host:

PCI(e) Passthrough - Proxmox VE

pve.proxmox.com

See if this helps @samurai336

 

[samurai336]

I know it's ProxMox related and not SCALE, but it's using the same Linux/KVM underpinnings - here's how to bind the vfio-pci driver (telling it to use the device for passthrough only) or blacklist it (don't load that driver at all) on the host:

PCI(e) Passthrough - Proxmox VE

pve.proxmox.com

See if this helps @samurai336

That would make sense to me, however in practice that did not fix the issue. Here's how I went about following those instructions to try and do this.

As it said on that page add...

to a .conf file in /etc/modprobe.d/

I added a new `.conf` file called `tv-tuner.conf` to ' /etc/modprobe.d/' and added my devices ids. The file looked like this:

Code:

options vfio-pci ids=14f1:8880,0070:2a18

which was based on the output of `lspci -vnn`

Code:

06:00.0 Multimedia video controller [0400]: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb [14f1:8880] (rev 04)
        Subsystem: Hauppauge computer works Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb [0070:2a18]
        Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 21
        Memory at f0800000 (64-bit, non-prefetchable) [size=2M]
        Capabilities: [40] Express Endpoint, MSI 00
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vital Product Data
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [200] Virtual Channel
        Kernel driver in use: vfio-pci
        Kernel modules: cx23885

Next as it instructed I ran update `initramfs` with

Code:

`# update-initramfs -u -k all` 

and got the output below which seemed like a red flag to me but figured I'd press on anyway.

Code:

root@n45z0rr[/etc/modprobe.d]# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.10.93+truenas
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
grep: /etc/initramfs-tools/modules: No such file or directory
cryptsetup: ERROR: Couldn't resolve device boot-pool/ROOT/22.02.0.1
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: WARNING: target 'md126' not found in /etc/crypttab
I: The initramfs will attempt to resume from /dev/dm-1
I: (/dev/mapper/md126)
I: Set the RESUME variable to override this.

After completing I rebooted and now on start up I started seeing these very cranky messages that have never come up before around `mdadm: no arrays found in config file` repeating over and over until finally continuing the boot cycle.

Here's a screen shot since I could not find the logs:

In addition my machine freezes on boot a lot and I have to hard restart it at least twice before I get a successful boot.

After it finally booted into the OS checked `lspci -nnk` and saw `Kernel driver in use: vfio-pci`, however the nextPVR on the vm was still unable to watch TV.

So I tried deleting `tv-tuner.conf` and adding the values I had put in there into an existing file since the instructions on the page were ambiguous in its description of `to a .conf file in /etc/modprobe.d/` so I picked `dkms.conf` thinking well its a kernel module right? and inserted the same text as listed above based on lspci output for the card.

Code:

options vfio-pci ids=14f1:8880,0070:2a18

Ran `# update-initramfs -u -k all` and got the same blast of `: No such file or directory` rebooted which took awhile because now it freezes on random boot steps/process and saw that the vm still couldn't use the TV card.

So I had one more thing I could try which was the black list option in those instructions. So after undoing everything I checked what the Kernel driver in use was and saw it was `vfio-pci` still and so I added that to `owfs-common.conf` where I saw other drivers blacklisted by adding the line

Code:

blacklist vfio-pci

, then ran the `initramfs` update again.

I rebooted again with lots of freezing and restarts needed, this time however the vm its self could not start because it gave me an error about being able to connect to the PCI device. So I ran lspci to see what the card was using/doing and I saw that the driver was now listed as follows .

Code:

 Kernel driver in use: CX23887

So I decided to try one more thing, I changed the black list entry I did to `CX23887` and ran initramfs update again and rebooted.

After lots of freezes and restarts the system came up, checked lspci and saw `Kernel driver in use: vfio-pci`. Went to the nextPVR page hosted on the VM and still nothing.

So while I think we are on the right trail I don't think the steps in the link are doing what they claim they do on that website when it come to TrueNAS scale. Any thoughts on what I'm not doing right or I am missing would be greatly appreciated.

Thanks all for trying to help.

 

[asw2012]

Hello, I just have to ask, have you tried the Docker NextPVR in TrueNAS Scale, instead of using a VM? I am currently trying to get the PCI Tuner to pass through to the Docker container,