"Initializing Apps Service" stuck after Cobia upgrade

[bakamonogatari]

I updated from Bluefin to Cobia yesterday.

Some of the apps were stuck in "deploying" state (even though they actually seemed to be running). I chose to "update all" and this is where it all went to hell.

I rebeooted, and now I'm at an empty app list and "Initializing Apps Service" stuck.

I ran `k3s kubectl get nodes`

and got this

Code:

ix-truenas   Ready    control-plane,master   542d   v1.26.6+k3s-e18037a7-dirty

and then this
`k3s kubectl get pods --all-namespaces`

Code:

NAMESPACE             NAME                                       READY   STATUS    RESTARTS        AGE
kube-system           csi-nfs-node-964b9                         3/3     Running   9 (6m16s ago)   13h
kube-system           svclb-pihole-dns-tcp-eda4b62a-8jtf4        1/1     Running   0               6m4s
kube-system           svclb-jellyfin-0c7964fe-p7g7w              1/1     Running   0               6m4s
kube-system           snapshot-controller-546868dfb4-hcxt7       1/1     Running   0               6m56s
kube-system           csi-smb-node-7rgpv                         3/3     Running   9 (6m16s ago)   13h
kube-system           snapshot-controller-546868dfb4-kp9jg       1/1     Running   0               6m56s
kube-system           svclb-pihole-codeserver-ac8ee1ca-hcnqn     1/1     Running   0               6m4s
kube-system           csi-smb-controller-7fbbb8fb6f-sdcgw        3/3     Running   0               6m55s
ix-redis              redis-ix-chart-fbf45dbcb-6tpm8             1/1     Running   0               6m55s
kube-system           coredns-59b4f5bbd5-kk5z7                   1/1     Running   0               6m56s
prometheus-operator   prometheus-operator-5dcffb7cb8-w7d8h       1/1     Running   0               6m57s
kube-system           svclb-qbittorrent-torrent-543f5661-4cbmh   2/2     Running   0               6m4s
kube-system           svclb-qbittorrent-7c903860-b5cfk           1/1     Running   0               5m48s
cert-manager          cert-manager-cainjector-ffb4747bb-886p2    1/1     Running   0               6m56s
cert-manager          cert-manager-8444f6f86b-s657w              1/1     Running   0               6m57s
kube-system           csi-nfs-controller-7b74694749-cjs5t        4/4     Running   0               6m56s
metallb-system        controller-84d6d4db45-62qzs                1/1     Running   0               6m55s
kube-system           svclb-pihole-ed0de4f1-kzlfb                1/1     Running   0               6m4s
cert-manager          cert-manager-webhook-545bd5d7d8-68b9m      1/1     Running   0               6m56s
cnpg-system           cnpg-controller-manager-5d74bc79fb-g66zl   1/1     Running   0               6m57s
kube-system           svclb-readarr-7ea31063-9pbfn               1/1     Running   0               6m4s
kube-system           openebs-zfs-controller-0                   5/5     Running   0               6m4s
kube-system           svclb-pihole-dns-453102d0-9sh87            1/1     Running   0               5m58s
metallb-system        speaker-fwz42                              1/1     Running   0               6m4s
kube-system           openebs-zfs-node-clxlh                     2/2     Running   0               5m42s
ix-pihole             pihole-857c5d8845-dtspz                    2/2     Running   0               6m55s
ix-readarr            readarr-5665587784-lt8xz                   1/1     Running   0               6m55s
ix-jellyfin           jellyfin-749877fcd5-sxcxg                  1/1     Running   0               6m55s
ix-qbittorrent        qbittorrent-6c5d46c8b4-ldgv7               1/1     Running   0               6m55s

The only one that actually seems to be working is Redis (which isn't the truecharts version but a docker image manually deployed to a container). The others aren't accessible by their respective web uis.

I also did attempt to un-set the pool, and then reselect it, as well as toggle GPU support to force a Kubernetes restart, two things I saw suggested elsewhere here.

Not sure where to go from here and would appreciate any help.

 

[bakamonogatari]

FWIW another reboot brought the apps list back up, however some of them are still stuck "deploying" despite seeming to be running.

 

[ABain]

FWIW another reboot brought the apps list back up, however some of them are still stuck "deploying" despite seeming to be running.

We've seen issues with the browser cache, have you tried clearing your browser cache, if they seem to be running then that might well explain the problem. Which version are you running?

 

[bakamonogatari]

Version: TrueNAS-SCALE-23.10.1.3

I actually even went as far as trying a different browser, one that hadn't been used for Truenas before. Seems that doing yet a 3rd reboot solved everything, and all apps updated and running normally now.

 

[opensourceFTW]

I had the same problem. Rebooting my install 3x fixed itself. Hmmm.

 

[crk1918]

I have a similar issue but after reboot. However, I ran `k3s kubectl get nodes`, and mine is not ready, it is taking forever to deploy.

 

[UnluckyGuy]

Same issue here. not able to solve.

 

[anodos]

Same issue here. not able to solve.

There was an issue related to a tmpfs filesystem used for catalog validation / setup that could cause apps to fail to initialize (along with many other problems). It was addressed in 23.10.2.

 

[davev]

Update to 23.10.2 did not help :(

 

[marshalleq]

I've just done a new install - decided to migrate from unraid. VM running, but apps are stuck on Initialising apps service. As it is a new install I went so far as to stop the middleware service and delete the ix-applications dataset, reboot and start again.. Perhaps I just need to wait longer. How long is this meant to take?

 

[crk1918]

I upgraded to 23.10.2 with no problem and rebooted a couple more times, forcing Initializing Apps Service a couple more times by turning GPU support on and off. All my apps are deploying again, with that being said, some apps still don't like to be upgraded due to new permission issues, so I had to stay with the old version for now.

 

[soliverum]

1. After updating to version 23.10.2, if you encounter the issue described:
2. Navigate to the "Applications" tab in the settings.
3. Select "Unset pool".
4. Once you've done that, observe where it says "Initializing Apps Service"; it should stop.
5. Next, reselect the pool where your applications are located.
6. Now, observe that "Initializing Apps Service" should work correctly.

 

[davev]

1. After updating to version 23.10.2, if you encounter the issue described:
2. Navigate to the "Applications" tab in the settings.
3. Select "Unset pool".
4. Once you've done that, observe where it says "Initializing Apps Service"; it should stop.
5. Next, reselect the pool where your applications are located.
6. Now, observe that "Initializing Apps Service" should work correctly.

Apps started much quicker after this. Thank you

 

[georgelza]

met of my apps seem to get back to running state, except metallb which is used for ingress. without it none of my external users can get to the services hosted on my NAS.
G

 

[davev]

Apps started much quicker after this. Thank you

This helped for 1 day :( Back to Initializing app service

 

[Thumb6729]

Apps started much quicker after this. Thank you

I tried this, no dice. Also tried restarting middlewared, waiting, rebooting etc.

My ix-applications folder is still intact.

I'd appreciate some pointers

 

[rol.dob]

I have a similar problem. The 23.10.2 ran for 2 weeks then suddenly got stuck on "Initializing applications in service" and deploying... I've tried rebooting the system more times, changing settings, set/unset the app pool, but no effect.

Any suggestions?

 

[rol.dob]

I have upgraded the system to Dragonfish beta-1, It has not fixed the problem.

 

[crk1918]

I have upgraded the system to Dragonfish beta-1, It has not fixed the problem.

which application do you have an issue with? All of them? Maybe try to deploy a new app (for testing) and see what happens.

 

[rol.dob]

which application do you have an issue with? All of them? Maybe try to deploy a new app (for testing) and see what happens.

All of them (e.g. Jellyfin) can't start (got stuck in deploying).
When it is trying to "initializing app service", the CPU load is constantly above 90% (that's why I used to unset the pool to avoid unnecessary power consumption and I wait for the fix..... )

I have installed NetData from both (TrueNAS and TrueCharts) catalog in both case are stuck in deploying.

When I try to open "shell" or "view logs" of app, the container combo box is empty...

History of NetData:

2024-03-21 09:26:15 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
2024-03-21 09:21:03 Created pod: netdata-5b8ddcd454-94x6d
2024-03-21 09:21:02 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
2024-03-21 09:20:55 Scaled up replica set netdata-5b8ddcd454 to 1