SOLVED NVIDIA 30 series & 40 series PCIe passthrough success?

[Basserra]

The issue I was experiencing may not have been related to the operating system (TrueNAS_Scale), but rather the hardware configuration. The GPU I wanted to passthru (3080Ti/3090Ti) is most likely favored due to the motherboard layout. I believe; its priority queues to be higher than the other GPU in my system. Therefore, the solution/workaround is to remove the display(s) from the passthru_GPU before the host(TrueNAS) boots/reboots & not reconnected until after the VM boots, so that the host doesn't access/utilize the passthru_GPU. Afterwards, I can shutdown the VM and leave the GPU in 'limbo' if desired or boot another VM with it, all without messing with the display(s) again until a host reboot.

Original Post:
Hello everyone! Has anyone met success with passingthrough a 30 or 40 series GPU in TrueNAS Scale? I cannot find success with passingthrough a RTX3080Ti. Quick mention that I don't have a 40 series GPU, just wondering if anyone has. You can see my current hardware in my signature TNScale_22.12.0, and while I can technically passthrough the 3080 to a VM and have it boot, it will ultimately fail by crashing/boot-looping on login AFTER installing the driver on Windows or the OS on Pop!_OS. I have attempted with Windows10_21H2(driver version 528.02) & Pop!_OS_22.04_nvidia_23. I am able to have the GPU passthrough on first boot while using VNC, install the OS, and install the driver on Windows; before the issue starts.
However, I have removed/replaced the 3080 and successfully passedthrough a GTX1660Super with no issues. I was able to install the driver, restart VM, and run Unigine_Heaven multiple times.
I think I read a thread or two on people successfully using a 3050 where they set the PCIe speed to Gen3, I don't believe I have that option.

I am not new to GPU passthrough
I run Proxmox on my Desktop PC (9900K&3090Ti) with PCIe passthrough of the GPU & a USB controller for a few years now, and used to use Unraid to do the same before. I've also done vGPU slicing on Windows 10 Hyper-V for multiplayer gaming. BUT please don't take this message as "I know what I'm doing!" I often make mistakes and always want to learn more. Just please no, "No one does that!" statements, I do understand that well from the resident mods past posts. This isn't a business situation.

 

[Straafe]

Uh oh, I have been planning my first TrueNAS Scale build and doing something like this was the whole point of the build (having multiple 3090's in it and assigning them/passthrough to different VM's).

 

[Basserra]

Yeah I want to swap my 3080 for the 3090 too, but I'm still fighting with it to get it working at all. I have tried to virsh edit a VM to Q35 which needed pci-root changed to pcie-root, but made no difference. Again, if anyone with the parts could try, it'd be appreciated; try with no settings changed vs setting the PCIe speed in the BIOS to Gen3. My mobo BIOS doesn't expose this option. Also, my GPU is currently on a riser, but I've also tried 3 different slots and directly in the board's slot/no riser. Edit: The 1660 was also passed thru on a riser no problem, but it's also Gen3 native. And my desktop mobo doesn't support Gen4 where the 3080 & 3090 do work.

I am planning on installing Proxmox to my scratch disks and passing thru the GPU just to verify it's not just my hardware's problem, hopefully by next weekend. I know it shouldn't be the GPU as it has been passed thru with Proxmox on my desktop system, and, it may be my mobo since it does have two integrated components that failed simultaneously recently, including its iGPU.

 

[Straafe]

I'd be really interested to hear the results of that

 

[rustigsmed]

Yes after lots of attempts (including lost VMS) I've now (last 24hrs) successfully got my truenas scale with win10 VM gaming working with an rtx 3050 - it works so well I was thinking of throwing my 3090 in. HDMI dummy plug made all the difference - its capable of 4k60 or 1080p120hz or 1440p60 (unfortunately can't get 1440p120 going). HDMI dummy plug essentially creates a second screen you can select.

Initially used remmina to access it but too laggy for gaming (mouse issues in game too) - then tried steam remote play which worked great - 60hz (no lag) but also no desktop. People say add notepad as a none steam game then press f1 but didn't really work for me - CPU-z did (minimised) but the thing crashes straight after playing a game's (as technically steam is playing two things at once). Tonight I installed Sunshine on the VM - and moonlight on the client (kubuntu) it allows 120hz - my limitation was the dummy plug compatibility (nv control panel wouldn't allow custom) - note these programs require nvidia cards.

Good luck

 

[Basserra]

Alright, I've done it. I have gotten the 3080 to work, hooked in a loaded iSCSI drive and ran Unigine_Superposition - Cyberpunk2077 Benchmark(UltraPreset&Ultra/DLSS/RT) - CitiesSkylines(few minutes) - GoatSimulator3(few minutes), each with no issues at all (ignoring ultrawide), streaming over sunshine-moonlight. The only issue I had was Blender Benchmark not installing which is unrelated I suppose.

Show : Story

I pulled an all nighter last night starting with installing ProxmoxVE and passing through to a Pop!_OS VM then later a Win10_21H2-driver_531.41 VM. To begin, the PVE VM was able to not crash as often, but every few seconds the screen would hang for a second then go black for a couple seconds then allow me use it for a few seconds again. I installed U_Heaven and launching it would crash the VM as usual, but the third time it threw an error about the audio & DirectX. So I threw another monitor at it with integrated audio and it managed to run! However, there was still an inconsistantcy with the VM, so for a quick sanity check, I swapped the 3080 for a QuadroT600. The T600 was more stable and consistant and after many reboots of both the VM & host, I found that I have to remove the display outputs of the passthru GPU before booting the host and only plugging one in AFTER starting the VM (even vfio-stub didn't help) (I think this is due to the host preferring to use the 3080/T600 as it is in slot_1, the furthest from the CPU ). After which, all seemed well and normal, I could even poweroff the VM for a while and fire it back up with no issues. I went back to the 3080 and it worked as expected. Finally I came back to TN, with audio-enabled monitor & not plugging it in until the VM was started, and viola! I did run this gambit of trying plugging in at different times and plugging after VM start is the only stable option. Now, as long as I remove the HDMI_dummy before host reboot, I have passed through the 3080, it performed as I expected. 1440p 75fps avg in Cyberpunk Benchmark UltraPreset (55fps DLSS_Quality/RT_Ultra) was better than I expected on an EPYC while still running everything else it regularly does.

System avg idle/operating power output 300w with peaks +/- 50w -- System+GPU engaged avg power output 750w-800w running U_Superposition

@rustigsmed Glad you got it working for you! And welcome to the joys & pains of virtualized gaming! Would you mind sharing your CPU/mobo models? I'm now thinking my problems are just my particular hardwares, but I'm not ruling out it's a platform problem yet. So I'd like to find more examples. Thanks for sharing!

its capable of 4k60 or 1080p120hz or 1440p60 (unfortunately can't get 1440p120 going)

In PVE I can get a HDMI dummy to run 1440p120 & 3440x1440_100fps and it's awesome (except U_Superposition was not obeying and rendering in 16:9 while the desktop was 21:9). But in TNScale I also can't modify it, in fact many options are missing from nvidiaCP in a TN VM vs a PVE VM for some reason (unless I'm mistaken and I only had the PVE VM plugged in a real monitor - I'm tired can't remember). I also can't remove the VNC from a TN VM yet else it won't boot, but a PVE VM runs better when without (none).

P.S. sunshine-moonlight are now compatible with nvidia/amd/intel hardware on either server/client side. I haven't tested this myself though. Oh yeah, I had the steam_play/notepad issue too, opening a 'new' instance of notepad from notepad got me out to the desktop, also just mstsc.exe worked for me.

 

[rustigsmed]

@rustigsmed Glad you got it working for you! And welcome to the joys & pains of virtualized gaming! Would you mind sharing your CPU/mobo models? I'm now thinking my problems are just my particular hardwares, but I'm not ruling out it's a platform problem yet. So I'd like to find more examples. Thanks for sharing!

In PVE I can get a HDMI dummy to run 1440p120 & 3440x1440_100fps and it's awesome (except U_Superposition was not obeying and rendering in 16:9 while the desktop was 21:9). But in TNScale I also can't modify it, in fact many options are missing from nvidiaCP in a TN VM vs a PVE VM for some reason (unless I'm mistaken and I only had the PVE VM plugged in a real monitor - I'm tired can't remember). I also can't remove the VNC from a TN VM yet else it won't boot, but a PVE VM runs better when without (none).

P.S. sunshine-moonlight are now compatible with nvidia/amd/intel hardware on either server/client side. I haven't tested this myself though. Oh yeah, I had the steam_play/notepad issue too, opening a 'new' instance of notepad from notepad got me out to the desktop, also just mstsc.exe worked for me.

Thanks! Yes - pretty happy with it now. Think it will remove the need for any dual booting on my desktop (I have a couple of obscure astrophotography programs that I use which still need windows - weening off slowly). Well weirdly, I have managed today to get custom resolution going, ufo test confirmed it has worked 1440p@120hz - I really didn't do anything differently or tweak anything so hopefully it remains stable (only thing i can think of was I had previously under "edit" - "CPU Model" set as 'blank/custom', but have since changed it to "EPYC" - perhaps that helped? I suppose 4k120hz is out of the question as the dummy plug is probably hdmi2.0 -not 2.1. yes, after looking at a lot of YT videos it seemed people were only getting success with proxmox - so I was kinda surprised truenas scale came through with the goods as I couldn't find a guide - and actually probably appears easier to configure. Oh wow, thanks for the heads up on sunshine-moonlight now accepting more than nvidia, didn't realise - that's awesome and gives me more options (also have a 6900xt). Specs for the TN server are 5950x/ASRock - X570-PHANTOM-GAMING-WIFI-AX (mobo model supports ecc memory and has 8x satas )/128gb ecc @3200. I only use it as a nas/valheim server/nextcloud/pihole and occasional win vm, so meets my needs nicely.

 

[rustigsmed]

Update, switched out the 3050 to a 3090 and it is working. Had to remove dummy as described by Bassera above and restart the entire system a couple of times. May have had the VM not starting issue a few people have mentioned in the forums. All good now.

 

[Basserra]

Update: I full/fresh installed TrueNAS_Scale_22.12.2 when it came out and reapplied my config. I wanted to run it for at least a week before posting back here. Absolutely no issues with new update, I skipped 22.12.1 for no reason in particular. Running current update since it came out, and it hasn't made any perceivable difference in regards to GPU passthrough; except for that I can now 'isolate' the 3080 (22.12.0 did not allow this, stating that it didn't have a GPU for host while the P400 was also there). I still have to remove the display/dummy plug from the passthru-GPU before host/VM boot. I am planning on swapping the GPUs from - 3080Ti & P400 → 3090Ti & T600. However, currently and more importantly I'm working on swapping/upgrading my main pool with new/bigger HDDs. Sidenote to any dissenters against my setup: I am not building a 'gaming server' to play video games myself, I aquired the 3090 for its VRAM capacity and sheer horsepower. While it may be used to run a gaming VM occasionally for friends & family visits, it is primarily for me to practice my CAD/rendering skills and start learning machine-learning. I still can't edit the nvidia CP for custom resolutions on a TNScale VM, but I was able to remove the VNC/display device by ticking the box for "Ensure Display Device" in the 'Edit VM' settings, then in the VM I just disable the display in 'device manager' on Windows. I'm going to try getting a different dummy plug that might support ultrawide for my desktop monitor & sanity.

@rustigsmed Regarding your "CPU Model." From what I've learned, you should always choose "Host/Passthrough" when using PCI passthrough, even when not a GPU. This is because the VM will utilize the full capabilities/features of your processor that it can support. Otherwise, you should choose a virtualized CPU model that also supports the capabilities/features of your CPU as close as possible. And, thanks for contributing to this thread!