Best way to make a feature request? LXC + QAT?

[ajgnet]

Hi guys — What is the best way to request that OpenZFS w/ QAT support be compiled into a future version of TrueNAS Scale? Would be really nice to offload dataset encryption, checksums, and gzip compression. Wish I could “wake up” the existing hardware functionality on my MiniXL+ NASs and decrease CPU utilization.

Also, does TrueNAS Scale support LXC?

Thank you and keep up the amazing work.

 

[morganL]

Have you tries TrueNAS 12.0 yet?... NIGHTLY also has Zstandard compression. 1st step is to transition those 12.0 features.

LXC is built into SCALE, but is not necessarily exposed to users. The goal is just reliable Docker and Kubernetes capability. Is there a specific capability you see as a "must have"?

 

[ornias]

What is the best way to request x be compiled into a future version of TrueNAS Scale?

In order of effectiveness:
Option 1: Send out a PR yourself
Option 2: Make a convincing argument why your feature is a must-have for the majority of users
Option 3: Make a suggestion on the Jira tracker

OpenZFS w/ QAT support...[on] TrueNAS Scale

It should already be available on TrueNAS SCALE, are you having any issues using it via the CLI?

Would be really nice to offload dataset encryption, checksums, and gzip compression

I agree this would be nice for people that already have QAT support, although Zstandard compression already is a huge step forward and something I think you would be quite statisfied using... (and we, the people who made the PR to get zstd into zfs, would very much like as much feedback as possible!)

TrueNAS Scale support LXC?

I do agree LXC support is something that should be put into the UI eventually (It would make competitor Proxmox even more irrelevant), But I also agree with Morgan about focus: You can't add everything at once at launch, you need to make choices. Not doing LXC at launch would be an acceptable compromise imho.
But the ability to have lightweight, fully functional (OS container instead of application container) "virtual" linux containers would be awesome!

 

[ajgnet]

You guys are awesome--thanks.

I'm looking forward to trying ZStd compression. Will definitely test and give feedback.

Regarding QAT, since all the Atom (included on my Mini XL+s) and many of the Denverton/Xeon-D processors have an integrated co-processor, including the driver and compiling ZFS w/ QAT support could really enable some nice performance gains while minimizing CPU. The hardware support is already there. Checksums, encryption, and compression could all be offloaded. Even OpenVPN AES-CBC can be offloaded. If you haven't seen already, the performance gains are significant:

GZIP compression offloading with QAT accelerator by wli5 · Pull Request #5846 · openzfs/zfs

This patch implement the hardware accelerator method in GZIP compression in ZFS. When the ZFS pool is enabled GZIP compression, the compression API will be automatically transferred to the hardware...

github.com

I have one Nvidia GPU that I want to share between docker containers and virtual machines. With a VM I can only pass-through the device, excluding it from the docker containers. So LXC seems like a lighter-weight option to access the shared GPU. I'm using Proxmox on one of my servers for this exact reason, but will give TrueNAS scale a try now that I know there is support through the console. Hopefully the UI picks everything up when the features are further developed.

 

[ornias]

@ajgnet Are you sure the version IX includes doesn't include QAT support? (aka did you test it)?
If so, I doubt there are many downsides just including it in the build. Primarily because IX actually sells hardware with QAT support afaik.

 

[ajgnet]

Have you tries TrueNAS 12.0 yet?... NIGHTLY also has Zstandard compression. 1st step is to transition those 12.0 features.

I really need docker containers and LXC w/ Nvidia support. Otherwise I'd be on TrueNAS 12.0 right now.

LXC is built into SCALE, but is not necessarily exposed to users. The goal is just reliable Docker and Kubernetes capability. Is there a specific capability you see as a "must have"?

Thanks this is helpful. I need LXC to have multiple "VMs" access a shared Nvidia card.

@ajgnet Are you sure the version IX includes doesn't include QAT support? (aka did you test it)?
If so, I doubt there are many downsides just including it in the build. Primarily because IX actually sells hardware with QAT support afaik.

I can confirm there is currently no ZFS QAT support. I agree it would be really nice if it were included in the build for the exact reasons you mention :).

 

[ornias]

I can confirm there is currently no ZFS QAT support. I agree it would be really nice if it were included in the build for the exact reasons you mention :).

You can file a suggestion in the Jira...
I might be able to take a look myself :)

 

[morganL]

I really need docker containers and LXC w/ Nvidia support. Otherwise I'd be on TrueNAS 12.0 right now.
I need LXC to have multiple "VMs" access a shared Nvidia card.

So, the need is to share a GPU card across containers..... that is part of the plan. Not sure that LXC is needed to accomplish that.

 

[ajgnet]

You can file a suggestion in the Jira...
I might be able to take a look myself :)

Thanks. I added a suggestion:

https://jira.ixsystems.com/browse/NAS-107334

So, the need is to share a GPU card across containers..... that is part of the plan. Not sure that LXC is needed to accomplish that.

Agree containers are mostly sufficient. But there are some cases where we need something less temporal, closer to a Virtual Machine. For example, database and web server on one host with GPU compute support. LXC works really well in that case.

 

[ornias]

Agree containers are mostly sufficient. But there are some cases where we need something less temporal, closer to a Virtual Machine. For example, database and web server on one host with GPU compute support. LXC works really well in that case.

To be clear in terminology:
LXC and Docker are both containers, just a different kind.

@morganL
The thing is: Docker is able to do this, but isn't able to do this for containers that need to be fully persistant.
Docker does (primarily) application containers, what Ajgnet wants is the option to have system containers (lxc) and still being able to share the GPU across many (which VM's would prohibit. Using docker as system container would be somewhat of a hack tbh.


Update:
I've looked into QAT.

Intel has kinda neglected it so it isn't Kernel version 5.6 compatible.
Luckily I managed to source a patch elsewhere.

I do have hacky patch ready to test building ZFS with QAT support on SCALE:
https://github.com/truenas/truenas-build/pull/41


TLDR: QAT is not compatbile with TrueNAS SCALE, but I hacked something together

*Edit*
Got it to work without Intel support.

 

[ornias]

@morganL About LXC, I just noticed this thread as part of a reddit post and I think it very well highlights the way potential smaller scale (pun intended ;) ) customers think about the LXC vs Docker debate and how having LXC available might push people in favor of TrueNAS SCALE:

Reddit - Dive into anything

www.reddit.com

 

[ajgnet]

To be clear in terminology:
LXC and Docker are both containers, just a different kind.

@morganL
The thing is: Docker is able to do this, but isn't able to do this for containers that need to be fully persistant.
Docker does (primarily) application containers, what Ajgnet wants is the option to have system containers (lxc) and still being able to share the GPU across many (which VM's would prohibit. Using docker as system container would be somewhat of a hack tbh.


Update:
I've looked into QAT.

Intel has kinda neglected it so it isn't Kernel version 5.6 compatible.
Luckily I managed to source a patch elsewhere.

I do have hacky patch ready to test building ZFS with QAT support on SCALE:
https://github.com/truenas/truenas-build/pull/41


TLDR: QAT is not compatbile with TrueNAS SCALE, but I hacked something together

*Edit*
Got it to work without Intel support.

This is the fastest I have ever seen the transition from forum post to working prototype. Thank you so much. I really hope this gets included in the future builds. Even if the feature is not exposed through the UI -- CLI is more than sufficient. The performance improvement is significant, and for low-power processors like Atom, can really take pressure off the CPU.

 

[ajgnet]

@morganL About LXC, I just noticed this thread as part of a reddit post and I think it very well highlights the way potential smaller scale (pun intended ;) ) customers think about the LXC vs Docker debate and how having LXC available might push people in favor of TrueNAS SCALE:

Reddit - Dive into anything

www.reddit.com

I agree with you 100%.

 

[ornias]

This is the fastest I have ever seen the transition from forum post to working prototype.

To be clear: This is building, but i'm not 100% sure it's working as I don't have QAT hardware.
If you build the ISO yourself (clone the git branch, install dependencies, run "Make", as instructed on the readme), you can test it yourself.

After you've installed TrueNAS SCALE using this patched installer ISO, you still need to install the QAT drivers manually on your TrueNAS SCALE machine. But you don't have to patch the ZFS if my hack worked.

*Edit*
How to install the QAT drivers:

Code:

# Install dependencies
apt -qyy install \
    linux-image-${KERNEL_VERSION} \
    linux-headers-${KERNEL_VERSION} \
    pciutils \
    wget \
    pkg-config \
    git \
    libudev-dev
    
# Setup QAT driver
mkdir -p /QAT/FETCH
wget https://01.org/sites/default/files/downloads/qat1.7.l.4.10.0-00014.tar.gz -O /QAT/FETCH/qat.tar.gz
wget https://raw.githubusercontent.com/spdk/spdk/master/test/common/config/pkgdep/patches/qat/0001-timespec.patch -O /QAT/FETCH/0001-timespec.patch
cd /QAT
tar zxvf /QAT/FETCH/qat.tar.gz -C /QAT
git apply FETCH/0001-timespec.patch
rm -Rf /QAT/FETCH
chown -R root:root  /QAT
./configure --enable-kapi
make
make install
cd -

 

[morganL]

@morganL About LXC, I just noticed this thread as part of a reddit post and I think it very well highlights the way potential smaller scale (pun intended ;) ) customers think about the LXC vs Docker debate and how having LXC available might push people in favor of TrueNAS SCALE:

Reddit - Dive into anything

www.reddit.com

It's an interesting questions whether LXC should be viewed as an alternative to KVM or to Docker/Kubernetes. Most applications can be run on all 3 of them. Suggest we review it after we release the 1st version of SCALE.

 

[beagle]

+1 for LXC. It can be seen as a lightweight VM and offering all 3 alternatives over time would cover all bases.

I'm also looking into SCALE as an all in one solution for home use. I'm currently running a CentOS server because VM features are not "mature" on CORE and ZFS features are not mature on Proxmox.

 

[ajgnet]

To be clear: This is building, but i'm not 100% sure it's working as I don't have QAT hardware.
If you build the ISO yourself (clone the git branch, install dependencies, run "Make", as instructed on the readme), you can test it yourself.

After you've installed TrueNAS SCALE using this patched installer ISO, you still need to install the QAT drivers manually on your TrueNAS SCALE machine. But you don't have to patch the ZFS if my hack worked.

*Edit*
How to install the QAT drivers:

Code:

# Install dependencies
apt -qyy install \
    linux-image-${KERNEL_VERSION} \
    linux-headers-${KERNEL_VERSION} \
    pciutils \
    wget \
    pkg-config \
    git \
    libudev-dev
   
# Setup QAT driver
mkdir -p /QAT/FETCH
wget https://01.org/sites/default/files/downloads/qat1.7.l.4.10.0-00014.tar.gz -O /QAT/FETCH/qat.tar.gz
wget https://raw.githubusercontent.com/spdk/spdk/master/test/common/config/pkgdep/patches/qat/0001-timespec.patch -O /QAT/FETCH/0001-timespec.patch
cd /QAT
tar zxvf /QAT/FETCH/qat.tar.gz -C /QAT
git apply FETCH/0001-timespec.patch
rm -Rf /QAT/FETCH
chown -R root:root  /QAT
./configure --enable-kapi
make
make install
cd -

Confirming this works.

 

[ornias]

Confirming this works.

Just to be sure:
You build the ISO from my PR, then you installed used that ISO, then you installed the drivers on the new system using the code I added to my post and QAT started working?

 

[ajgnet]

Yes that's correct. The only issue is I could not get QAT acceleration to work on the root ZFS device since the ZFS modules load before the QAT module is loaded. But I re-load the module after booting and everything works as expected.

 

[ornias]

Yes that's correct.

Thats great news! :)
As soon as a new version of the QAT drivers comes out (which should include kernel 5.6 support), I'm going to officially submit the PR for always building ZFS with QAT support. That should atleast make it trivial to add QAT for endusers, because the current blocker for enabling QAT on SCALE is mostly the need to rebuild ZFS on a system running ZFS as root.

I could not get QAT acceleration to work on the root ZFS device

How does it function on the non-root zfs device? Or is this the case for all ZFS devices?
That being said: Driver addition was not part of my PR, so i'm actually pretty happy my PR did work.