Apps not working after update

[danb35]

please report a bug.

[NAS-122345] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[pookNast]

The thread was here: https://www.truenas.com/community/threads/looking-for-bluefin-systems-where-apps-dont-start.106968/

We're going to make 22.12.3 available next week. There is one bugfix that is likely to be relevant.

[NAS-120955] - iXsystems Jira​

This bugfix forces Apps starting to wait for the cgroups in kernel to be available. We think this is the cause of many customer issues. The behaviour varied based on the system build.

As a "hall of Famer"... please report a bug.. if you are willing to use the pre-release nightly, we can verify and otherwise help you resolve.

Hello

I updated to 22.12.3 in order to resolve the issue:

"
CRITICAL
Failed to configure kubernetes cluster for Applications: Missing 'cpuset, cpu' cgroup controller(s) which are required for apps to function
2023-04-21 09:36:48 (America/Los_Angeles)

"

Good news: that missing "cpuset,cpu" error is gone
Bad news: when I set pool I get a huge CPU usage spike and CPU temp spike and new error*

to note: my 'ix-applications- data set is unencrypted but the parent pools are encrypted. I don't know if that is supposed to be that way, and/or is an artifact from the previous (22.12.2) attempt to get the Apps to run properly.

Any advice? Thank you

Edit: I get a new error:

The following datasets are not encrypted but are within an encrypted dataset: 'data/ix-applications' which is not supported behaviour and may lead to various issues.​

Edit 2: Ah I see now the issue I am having now does not pertain to the "cpuset, cpu" issue and that is resolved with the new update. Thank you.

 

[frunkAf]

This started happening for me after I upgraded to 22.12.3.

I tried all of the proposed fixes to no avail.

If I'm reading this pull request correctly, https://github.com/truenas/middleware/pull/10838

Since 22.12.2, it first checks to see if the following cgroup controllers are available, 'cpu', 'cpuset', 'memory', 'hugetlb', 'pids', before it allows apps to start.

In the commit that's listed on github, I see the following...

@private def ensure_cgroups_are_setup(self): supported_controllers = {'cpu', 'cpuset', 'memory', 'hugetlb', 'pids'} cgroup_root_path = '/sys/fs/cgroup' system_supported_controllers_path = os.path.join(cgroup_root_path, 'cgroup.controllers')

Which appears to reference the '/sys/fs/cgroup/cgroup.controllers' file, but this file does indeed have those controllers listed.

So I was thinking that perhaps there's another file it's referencing?

The error says "Missing 'cpu, cpuset, hugetlb' cgroup controller(s)", so I did the following to find the cgroup.controllers...

root@TrueNAS[~]# find /sys/fs/cgroup -name cgroup.controllers

I get 48 different instances of 'cgroup.controllers' throughout this parent directory. I gather that the Apps module references one of these files for the necessary controllers so I inspected them and found that following 31 had only the 'memory' and 'pids' listed...

/sys/fs/cgroup/user.slice/user-0.slice/user@0.service/app.slice/dbus.socket/cgroup.controllers /sys/fs/cgroup/system.slice/syslog-ng.service/cgroup.controllers /sys/fs/cgroup/system.slice/systemd-udevd.service/cgroup.controllers /sys/fs/cgroup/system.slice/cron.service/cgroup.controllers /sys/fs/cgroup/system.slice/wsdd.service/cgroup.controllers /sys/fs/cgroup/system.slice/nginx.service/cgroup.controllers /sys/fs/cgroup/system.slice/rpcbind.socket/cgroup.controllers /sys/fs/cgroup/system.slice/system-modprobe.slice/cgroup.controllers /sys/fs/cgroup/system.slice/systemd-journald.service/cgroup.controllers /sys/fs/cgroup/system.slice/winbind.service/cgroup.controllers /sys/fs/cgroup/system.slice/ssh.service/cgroup.controllers /sys/fs/cgroup/system.slice/rrdcached.service/cgroup.controllers /sys/fs/cgroup/system.slice/nfs-blkmap.service/cgroup.controllers /sys/fs/cgroup/system.slice/boot-grub.mount/cgroup.controllers /sys/fs/cgroup/system.slice/tmp.mount/cgroup.controllers /sys/fs/cgroup/system.slice/rpcbind.service/cgroup.controllers /sys/fs/cgroup/system.slice/middlewared.service/cgroup.controllers /sys/fs/cgroup/system.slice/nscd.service/cgroup.controllers /sys/fs/cgroup/system.slice/run-rpc_pipefs.mount/cgroup.controllers /sys/fs/cgroup/system.slice/collectd.service/cgroup.controllers /sys/fs/cgroup/system.slice/smartmontools.service/cgroup.controllers /sys/fs/cgroup/system.slice/nut-driver.service/cgroup.controllers /sys/fs/cgroup/system.slice/zfs-zed.service/cgroup.controllers /sys/fs/cgroup/system.slice/ntp.service/cgroup.controllers /sys/fs/cgroup/system.slice/nut-monitor.service/cgroup.controllers /sys/fs/cgroup/system.slice/smbd.service/cgroup.controllers /sys/fs/cgroup/system.slice/dbus.service/cgroup.controllers /sys/fs/cgroup/system.slice/system-getty.slice/getty@tty1.service/cgroup.controllers /sys/fs/cgroup/system.slice/system-getty.slice/cgroup.controllers /sys/fs/cgroup/system.slice/avahi-daemon.service/cgroup.controllers /sys/fs/cgroup/system.slice/systemd-logind.service/cgroup.controllers

Now the question is, does anyone have any idea which of these 'cgroup.controllers' file is referenced for the Apps to determine the controllers are available? I have a feeling it's

/sys/fs/cgroup/system.slice/middlewared.service/cgroup.controllers

...however, I don't seem to have write access to this.

 

[morganL]

There is a 22.12.3.1 update coming which addresses some of these issues.

 

[omeganot]

There is a 22.12.3.1 update coming which addresses some of these issues.

@morganL - I'll keep an eye out for 22.12.3.1, but personally 22.12.3 got me back up and running again.

To ensure nothing with the ix-applications dataset was misconfigured (I read the PR about incorrect configuration of it over time) I did fully unset the pool for apps, delete ix-applications, and then reset the pool (after update. I was going to do it prior but forgot). This has me up and running again it seems for now. I even got a message in alerts that the missing config information alert was cleared upon this reset.

 

[atom5ive]

Still having the below error when trying to launch "cloudflared" app. Any update on this? Completely updated system running latest update TrueNAS-SCALE-22.12.4 Bluefin Release.


[EFAULT] Unable to connect to kubernetes cluster

More info...
Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body rv = await self.method(*([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 397, in do_create await self.middleware.call('kubernetes.validate_k8s_setup') File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1368, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 513, in validate_k8s_setup raise CallError(error) middlewared.service_exception.CallError: [EFAULT] Unable to connect to kubernetes cluster

 

[morganL]

Still having the below error when trying to launch "cloudflared" app. Any update on this? Completely updated system running latest update TrueNAS-SCALE-22.12.4 Bluefin Release.


[EFAULT] Unable to connect to kubernetes cluster

More info...
Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body rv = await self.method(*([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 397, in do_create await self.middleware.call('kubernetes.validate_k8s_setup') File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1368, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 513, in validate_k8s_setup raise CallError(error) middlewared.service_exception.CallError: [EFAULT] Unable to connect to kubernetes cluster

Is your pool encrypted?
Please spec any hardware or relevant setup info.