TrueNAS Scale performance difference with apps from different catalogs/repositories.

[buckweet]

I've been testing out different app images of sabnzbd/sonarr/radarr and I'm finding performance differences when using the Truecharts 3rd party app catalog vs just launching a docker container from Docker hub such as linuxserver/sabnzbd..

I have 2 x 1gig circuits @ home and with the Truecharts Sabnzbd I can pull over 100MB/sec consistently, with the linuxserver/sabnzbd image I'm only getting 70ish.. So I was curious why this would be? Each image is using the exact same settings (I have the /config mapped the same, so I just stop/start each one to test).

After doing a docker inpsect on each of the containers, I see the CPU shares are set to 2 for any image that you launch manually via the 'launch docker image' button, while the Truecharts ones have 10 shares.. Memory doesn't seem to be limited though, in fact the Truecharts ones has less memory allocated. I'm not able to find a setting any where to adjust this, does anyone else know? Using these same linuxserver/X images on another host doesn't have this cpu share flag set, so it's something that TrueNAS k3s orchestration is doing I believe.

Now with what I've found, I've asked this same question on reddit and had my hand slapped for even trusting docker inspect output.. All I can say is this is easily repeatable if I flip between these docker images. So if this isn't a reliable source (the docker inspect) then how can I validate this?

 

[sammael]

I have seen similar behaviour (though not network traffic related) when deploying valheim/minecraft server manually and via truechart app.

We only get this screen on apps, not on manual docker images:

When I deployed one of the servers manually they were sluggish while nas cpu sat mostly idle with tens of Gb free ram, with above settings deploying same server via Truechart app the server is markedly more responsive with noticeable increase in resource usage on nas.

One thing you could try is to install Truecharts "docker-compose" app, adjust it's resource limits, and then run your own image inside it and see if it makes difference (I ran several containers like this previously though as they ran satisfactorily (read about as fast as the host I moved them from) I never bothered to do any benchmarks).

 

[buckweet]

Well that's a bummer, that other message got posted accidentally and I'm not able to edit it..

Anyways, here's what I see using docker inspect..

This is from the Truecharts image..
"CpuShares": 10,
"Memory": 8589934592,
<few lines of irrelevant information omitted>
"CpuPeriod": 100000,
"CpuQuota": 400000,


This is from the linuxserver/sabnzbd

"CpuShares": 2,
"Memory": 0,
<few lines of irrelevant information omitted>
"CpuPeriod": 100000,
"CpuQuota": 0,


In Docker the default for CpuPeriod (based on this doc https://docs.docker.com/config/containers/resource_constraints/ ) says that it is 1000000 like shown above. The default cpushares is 1024 it says too... What's interesting though is on another Ubuntu with Docker machine I have, these are all set to 0 when I run the image.

Sure would be nice if these settings could be exposed.

 

[mervincm]

Be careful with setting Kubernetes limits and requests. CPU limits are in millicore and are the max that the pod can use. If it tries to use more it will be throttled to that amount. Ram limits are a similar max but tend to come with the added bonus in that if your pod attempts to use more it will be evacuated and restarted. Requests are the minimum you need to run your pod. It is evaluated when your pod is started and if there is not enough on your host, it won't even start. so if you have 8 cores and 8 apps each with 1000 millicore requests running, and try to start another it will fail. Quotas and deployment choices make it even more tricky.

 

[truecharts]

Sure would be nice if these settings could be exposed.

They cannot be exposed. SCALE does NOT use docker, it uses kubernetes which itself utilises (controls) docker.

You cannot make per-app tweaks on the docker-engine layer from within kubernetes byond the features exposed by kubernetes itself (which are not docker settings), or settings that happen to trigger certain engine-level changes.

Please remember: Docker is just one of the many container engines used by Kubernetes, with most other engines you as end user never actually get to see the info you managed to fetch using docker commands.

Just because you can look at internals, doesn't mean you're able to validate if they are correct nor does it mean there actually exists anything to directly set things on that layer.

In short: You're just confusing yourself by bugtracing something too deeply at the moment.
Unless you're a kubernetes specialist and know yourself to be able to evaluate mistakes in dockershim, in which case it would be odd to discuss things here, because you would've been discussing it with the k3s/k8s/dockershim teams instead ;-)

 

[buckweet]

They cannot be exposed. SCALE does NOT use docker, it uses kubernetes which itself utilises (controls) docker.

You cannot make per-app tweaks on the docker-engine layer from within kubernetes byond the features exposed by kubernetes itself (which are not docker settings), or settings that happen to trigger certain engine-level changes.

Please remember: Docker is just one of the many container engines used by Kubernetes, with most other engines you as end user never actually get to see the info you managed to fetch using docker commands.

Just because you can look at internals, doesn't mean you're able to validate if they are correct nor does it mean there actually exists anything to directly set things on that layer.

In short: You're just confusing yourself by bugtracing something too deeply at the moment.
Unless you're a kubernetes specialist and know yourself to be able to evaluate mistakes in dockershim, in which case it would be odd to discuss things here, because you would've been discussing it with the k3s/k8s/dockershim teams instead ;-)

If it can't be exposed on the launch docker workflow, then how is it exposed in the Truecharts questions when launching an app?

 

[truecharts]

If it can't be exposed on the launch docker workflow, then how is it exposed in the Truecharts questions when launching an app?

Lets hope 3th time is the charm:
Kubernetes controls how docker is used, not iX Systems not us. Neither of us have any precise control over how kubernetes sets certain docker container parameters, there is a translator/controler in between kubernetes and docker (dockershim) that controls that. Kubernetes does not and cannot directly change any parameters there.

We also don't set these, we set kubernetes cpu/ram limits and requests, which might or might not get translated into certain parts of docker. Again: That's not something we or iX Systems builds, that's dockershim doing its magic.

You simply need to stop thinking about docker (and docker commands to troubleshoot), you need to limit yourself to kubernetes, that's the only part you, us or iX has any direct control over.

IF you want to bugtrace something with dockershim, you'll need to contact the folks at dockershim. There is literally no one here on this forum or within our community with experience on that front. Because it's not normal knowhow when using kubernetes.

 

[buckweet]

I understand what you're saying in the grand scheme of things. You're missing my point too, so please hear me out.. In the Truecharts app launching workflow, there are options to set the resource consumptions to limit and give minimum amounts.. This is what I would like exposed for the general workflow when creating a docker container..

I fully understand what Kubernetes is, it's an orchestration engine. At the end of the day it's instructing docker (used in Scale) to do XYZ. Or it could tell other container engines too, when used in other infrastructures. What I see is that the settings that is being applied in the Truecharts workflow IS being propagated to the docker instance, which I believe is allowing this instance to perform faster than a general app that we can launch.

All I'm trying to do is get apples to apples here.. There should be no reason that the Truecharts apps perform faster than any other app.

 

[truecharts]

I understand what you're saying in the grand scheme of things. You're missing my point too, so please hear me out.. In the Truecharts app launching workflow, there are options to set the resource consumptions to limit and give minimum amounts.. This is what I would like exposed for the general workflow when creating a docker container..

I fully understand what Kubernetes is, it's an orchestration engine. At the end of the day it's instructing docker (used in Scale) to do XYZ. Or it could tell other container engines too, when used in other infrastructures. What I see is that the settings that is being applied in the Truecharts workflow IS being propagated to the docker instance, which I believe is allowing this instance to perform faster than a general app that we can launch.

Those minimums might not what you think they are, they are minimum amounts kubernetes uses to determine of an App can start.
It's also not related to "CPU shares" directly, as far as we're aware. And this was the specific setting you where most interested in.

But yes, these settings can, in theory, also be exposed by iX.

All I'm trying to do is get apples to apples here.. There should be no reason that the Truecharts apps perform faster than any other app.

Well there are many cases where our Apps/Charts outperform those by others. In the end design differences do mater. It's not "just" a GUI or option difference, we do also make other design decisions that differ from Apps and Charts made by others. Some of them will significantly impact performance in the grand scheme of things.

 

[buckweet]

Well there are many cases where our Apps/Charts outperform those by others. In the end design differences do mater. It's not "just" a GUI or option difference, we do also make other design decisions that differ from Apps and Charts made by others. Some of them will significantly impact performance in the grand scheme of things.

And this could very well be true, I just can't prove or disprove it.. The linuxserver/sabnzbd is very fast on an ubuntu server running docker. So something is just 'off' about it.