Issue with Apps Starting/Removing/Re-installing

[BoostedIS300]

Hello,
Yesterday I started noticing an issue with one of my apps (uptime-kuma) not starting after a system reboot. It would go through the loading screen of starting, reaching 100% but then never deploy. I attempted this multiple times to no avail. I rebooted my system in an attempt to get a clean start and try again, same out come.

When clicking on the app where it shows "app events" the below error was listed:

MountVolume.MountDevice failed for volume "pvc-d46767f6-b96b-4608-91cd-6739b1eac12b" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers

Decided to just try and delete the app and reinstall, but it would get stuck at 50% and not pass the "scaling down to 0 pods", or what ever message. Waited for a good while then rebooted again (full power down). The app is gone but will not let me install.

After this reboot, another app (thelounge) is doing the same thing (same MountVolume error). Doing the above steps again for thelounge, but now attempting to reinstall thelounge gives me the below error. It says something about it being terminated but the app is already gone from the apps page and will not let me install it.

[EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: create: failed to create: secrets "sh.helm.release.v1.thelounge.v1" is forbidden: unable to create new content in namespace ix-thelounge because it is being terminated
More info...

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 486, in do_create
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1368, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1328, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/helm.py", line 44, in helm_action
raise CallError(f'Failed to {tn_action} chart release: {stderr.decode()}')
middlewared.service_exception.CallError: [EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: create: failed to create: secrets "sh.helm.release.v1.thelounge.v1" is forbidden: unable to create new content in namespace ix-thelounge because it is being terminated

Not sure if it is related, but I did update to TrueNAS-SCALE-22.12.4.2 about a week ago from (I believe) 22.12.3

Thanks for any help in advance.

OS Version: TrueNAS-SCALE-22.12.4.2
Product :X9DRi-LN4+/X9DR3-LN4+
Model: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Memory: 126 GiB

 

[morganL]

Is this the only App on the system?
Are the other Apps OK?

 

[BoostedIS300]

Is this the only App on the system?
Are the other Apps OK?

No, I have 12 apps total (including those). I was able to install these apps again under a different name ("thelounge" before, but worked using "lounge"). Those apps are fine, other than my Plex app did have the MountVolume issue at one point but it is currently up and running. I am starting to worry my other apps may have this issue if I reboot. I'm not noticing any data loss or hardware issues either.

 

[morganL]

Is this a TrueCharts App?

Looks like it is trying to use an openebs-zfs driver which is not the TrueNAS SCALE recommended approach.
Openebs zfs driver has quirks and has never been tested or supported.
That driver is being removed in dragonfish to avoid this type of issue.

In the meantime, Apps on the system should not use it. All the standard TrueNAS apps have had it removed from use.

 

[BoostedIS300]

They are indeed all TrueCharts apps.

It's very annoying that their apps cause so many problems. I've had to reset all my apps after their last breaking change with 'values.scalesGPU' a few months back.

I will head over to their discord tomorrow and see if any one can offer any insight as to why this is happening all of a sudden.

Cheers.

 

[truecharts]

[Mod edit - Keep it professional, please.]

In short:
- We've had writhen instructions on how we needed to use iX' heavily customised OpenEBS installation to stay within their support scope
- They silently retracted that support scope
- So we requested(!) it being removed, so we can roll our own without iX modifications
- And tries to shift blame to both OpenEBS and TrueCharts, instead of looking at themselves.
- We will release actually stock OpenEBS with instructions, including options for user-accessible storage etc. Q1 2024. Initial testing has already finished.

 

[truecharts]

Hello,
Yesterday I started noticing an issue with one of my apps (uptime-kuma) not starting after a system reboot. It would go through the loading screen of starting,...

In contrast to what Morgan was saying, this is 100% sure a part of either kubernetes (managed by iX Systems) or iX-Systems middleware.
We touch nothing in the error loop of your error.

 

[Kris Moore]

We will keep it short, as we'll publish a full write-down later of how badly iX messed up OpenEBS, which is a trusted industry standard.

In short:
- We've had writhen instructions on how we needed to use iX' heavily customised OpenEBS installation to stay within their support scope
- They silently retracted that support scope
- So we requested(!) it being removed, so we can roll our own without iX modifications
- And tries to shift blame to both OpenEBS and TrueCharts, instead of looking at themselves.
- We will release actually stock OpenEBS with instructions, including options for user-accessible storage etc. Q1 2024. Initial testing has already finished.

Please keep this professional, no need for that kind of language. You've been warned and banned before for unprofessional behavior, lets keep this on the up and up.

As as been discussed elsewhere, iX does not use, therefore we do not test, therefore does not support the OpenEBS driver. This is why we have "big scary warnings" about using 3rd party catalogs in general on TrueNAS. It's perfectly expected and within bounds that over time unused code-paths will break down, bit-rot, or otherwise exhibit new and "fun" behaviors. This is why its called "unsupported" in the first place.

To set proper expectations, anytime anything third party is loaded in, there is a assumption of responsibility being made that the third party will have to own the support and testing. That means if the third party uses libraries / code-paths not directly used by iX/TrueNAS itself, there is a high risk of regressions occurring at some random point. Even shipping a stock OpenEBS (or any other external code) is a high risk activity that is almost guaranteed to cause future breakage and regressions. Caveat Emptor!

 

[truecharts]

We stand with our previous comments about this and are sorry for the inconvenience this is causing our users.
Users are advised to contact our support staff if they have any issues with OpenEBS, as our staff does have means to dealing with these issues.

In 2024 we will be releasing OpenEBS ourselves and increasing support for other platforms for users with production requirements.

We will refrain from commenting on this on the iX forums any further.