How to recompile ZFS in TrueNAS for QAT support

[saloomy]

I have an Intel QuickAssist card (8970) that is showing up in my TrueNAS Scale system:

Code:

87:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
    Subsystem: Intel Corporation QuickAssist Adapter 8970
    Flags: bus master, fast devsel, latency 0, IRQ 49, NUMA node 1, IOMMU group 63
    Memory at fbd40000 (64-bit, non-prefetchable) [size=256K]
    Memory at fbd00000 (64-bit, non-prefetchable) [size=256K]
    Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [60] MSI-X: Enable+ Count=17 Masked-
    Capabilities: [6c] Power Management version 3
    Capabilities: [74] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1b0] Access Control Services
    Kernel driver in use: c6xx
    Kernel modules: qat_c62x

88:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
    Subsystem: Intel Corporation QuickAssist Adapter 8970
    Flags: bus master, fast devsel, latency 0, IRQ 50, NUMA node 1, IOMMU group 64
    Memory at fbc40000 (64-bit, non-prefetchable) [size=256K]
    Memory at fbc00000 (64-bit, non-prefetchable) [size=256K]
    Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [60] MSI-X: Enable+ Count=17 Masked-
    Capabilities: [6c] Power Management version 3
    Capabilities: [74] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1b0] Access Control Services
    Kernel driver in use: c6xx
    Kernel modules: qat_c62x

89:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
    Subsystem: Intel Corporation QuickAssist Adapter 8970
    Flags: bus master, fast devsel, latency 0, IRQ 51, NUMA node 1, IOMMU group 65
    Memory at fbb40000 (64-bit, non-prefetchable) [size=256K]
    Memory at fbb00000 (64-bit, non-prefetchable) [size=256K]
    Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [60] MSI-X: Enable+ Count=17 Masked-
    Capabilities: [6c] Power Management version 3
    Capabilities: [74] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [138] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [140] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1b0] Access Control Services
    Kernel driver in use: c6xx
    Kernel modules: qat_c62x

I also have a module already loaded which contains the driver, which I assume is coming with Kernel 5.15 in TrueNAS 22:

Code:

root@truenas[/proc/spl/kstat/zfs]# modinfo qat_c62x
filename:       /lib/modules/5.15.107+truenas/kernel/drivers/crypto/qat/qat_c62x/qat_c62x.ko
version:        0.6.0
description:    Intel(R) QuickAssist Technology
firmware:       qat_c62x_mmp.bin
firmware:       qat_c62x.bin
author:         Intel
license:        Dual BSD/GPL
srcversion:     A1CC81021D58DB37B83364E
alias:          pci:v00008086d000037C8sv*sd*bc*sc*i*
depends:        intel_qat
retpoline:      Y
intree:         Y
name:           qat_c62x
vermagic:       5.15.107+truenas SMP mod_unload modversions

I think at this point all I need to do is get the QAT options in ZFS so I can get QAT stats:

Code:

root@truenas[~]# cat /proc/spl/kstat/zfs/qat                                                                       
cat: /proc/spl/kstat/zfs/qat: No such file or directory

Is there a way to reconfigure ZFS package in TrueNAS with the QAT options enabled? Setting up a compile environment within Truenas seems daunting. Maybe build it in a Debian 12 environment and move the .deb package to TrueNAS?

 

[jgreco]

TrueNAS is an appliance firmware and is not designed for this kind of tinkering. See

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

The developers and the users here do not really want rogue firmware mods running around, because history has shown that these often end up being the source of problems later on for users, and that users are frequently reticent about admitting that they've hacked random stuff up. This makes for really unpleasant debugging and wastes everyone's time.

If you really think you know exactly what needs changing, set up a build environment (the instructions are included with the TrueNAS source code) and then build your own custom firmware. This is one of those "if you don't understand it, you probably shouldn't do it" things.

Otherwise, you are certainly encouraged to put in a feature request (see the "Report a Bug" at the top of every page or on your server). Be prepared for the possibility that your request may be rejected for a variety of reasons, most likely including that there are general issues with enabling QAT.

 

[saloomy]

Hi Jgreco and thank you for time you took to respond.

If you really think you know exactly what needs changing, set up a build environment (the instructions are included with the TrueNAS source code) and then build your own custom firmware. This is one of those "if you don't understand it, you probably shouldn't do it" things.

That is exactly what I am in the process of doing, but I was honestly looking for a solution to keep with the update mechanisms in place, rather than build again and again when new features come out. We have run CentosOS 7 + OpenZFS in our environment with QAT that has had performance gains for our use case, and are excited to migrate to TrueNAS. We just dont want to lose the performance for the added manageability that makes TrueNAS so attractive.

Be prepared for the possibility that your request may be rejected for a variety of reasons, most likely including that there are general issues with enabling QAT.

I have read through various forum posts in TrueNAS, the OpenZFS communities, and Intel information, but I have not seen a true "issue" articulated with enabling QAT. Is there some info you can shoot me or help point me in the right direction to uncover these issues? Are they strictly performance related under certain circumstances or stability related? Our experience has been a positive one, but I am always looking to learn from what others experiences may be.

Thanks again Jgreco!

 

[LarsR]

The latest stable release uses the 5.15.107 Debian Kernel. The next update, cobia, currently uses the 6.1 Kernel.
If its not currently enabled i doubt there is a possibility to do so.

 

[jgreco]

but I was honestly looking for a solution to keep with the update mechanisms in place, rather than build again and again when new features come out.

That won't work. The update mechanism for TrueNAS is image-based, it's a complete release of the OS platform (think: "like as in a tarball"). I am reading in between the lines from what you say that you might believe it is some sort of piecemeal "prepackaged bits" like apt or dpkg or whatever stuff Linux users use. It isn't. Your next upgrade of TrueNAS will dump an entire new boot environment (i.e. "OS") onto your platform and then boot from that. It will not conveniently integrate your custom changes or work around binary blobs that you have changed.

Still reading in between the lines, it sounds like you have the competency to pull off building your own release, and maybe that's the way to go if you cannot get any buy-in from iX to integrate whatever QAT stuff you need.

I have not seen a true "issue" articulated with enabling QAT. Is there some info you can shoot me or help point me in the right direction to uncover these issues?

I'm not aware of any. However:

Are they strictly performance related under certain circumstances or stability related?

You may have misunderstood this to mean "end user experience issues". Much of the time when iX is asked to do something that enhances the end user experience and has no other negative effects, they will do so. However, if, for example, enabling QAT in the common platform were to make compromises that hurt other applications, they will typically refuse. Examples of this COULD include for example if QAT support being enabled in the binaries caused the inclusion of code that would not execute on older Intel or on AMD platforms, or caused performance issues or required other workarounds on various platforms. This is similar to how when we had crypto devices in the early days that required SSL to be compiled differently depending on whether or not crypto hardware existed in the system, this turned into a large problem for distributing and properly installing OpenSSL.

 

[saloomy]

You may have misunderstood this to mean "end user experience issues". Much of the time when iX is asked to do something that enhances the end user experience and has no other negative effects, they will do so. However, if, for example, enabling QAT in the common platform were to make compromises that hurt other applications, they will typically refuse. Examples of this COULD include for example if QAT support being enabled in the binaries caused the inclusion of code that would not execute on older Intel or on AMD platforms, or caused performance issues or required other workarounds on various platforms. This is similar to how when we had crypto devices in the early days that required SSL to be compiled differently depending on whether or not crypto hardware existed in the system, this turned into a large problem for distributing and properly installing OpenSSL.

You make an excellent point. I am in the process now of building from source in a build-environment with the change to the ZFS compile options. I will perform some testing, and I am requesting XI to enable QAT, on the hope that they do so (once I justify my request with before / after performance benchmarks). I will take your comment to heart, and add in testing enabled QAT option with no QAT present, and do benchmarking on older CPUs like a Xeon E5. Perhaps there is performance impact, but perhaps not. I have a handy bit of hardware spanning generations I can dedicate for a time to test this solution.

Thank you for all your input.