Apps stuck deploying

[alieg]

I've just migrated from Core to Scale 2 days ago. I thought this would just be a case of re-installing my plugins and copying over the config but every app I install is stuck in Deploying. I've waited on them for several hours.

I've tried to delete the apps and start again to get something working, some have got stuck in "Waiting for pods to terminate" I've rebooted several times and none of them are working.

 

[indivision]

Are you installing these apps with no configuration changes?

What hardware are you using?

 

[alieg]

I initially tried installing Plex with custom paths to data but after that wasn't deploying I tried some more apps with no changes to the default settings from both Official and TrueCharts

Setup is:
TrueNAS-SCALE-22.02.3
Intel(R) Core(TM) i5-3470T CPU @ 2.90GHz
12GB RAM
1.05 TiB (60%) Used | 721.02 GiB Free

Currently I've been trying to reset by deleting the ix-applications folder in the same way as your thread here to see if that would work

Possible to "reset" Kubernetes?

I seem to have broken my Kubernetes by deleting snapshots. Explained here: https://www.truenas.com/community/threads/how-to-get-rid-of-12000-snapshots.101048/ So, now I have a new question: Is there a way to restore Kubernetes functionality? Perhaps by removing all apps and snapshots and...

www.truenas.com

Unfortunately I'm also having a problem where on the apps page, if I press settings > advanced settings / choose pool nothing happens

 

[daniel-]

I have the same issue on my server. I was able to deploy apps a couple of weeks go. Now, I'm just trying to install Syncthing, and it always failed during preinstall.

I saw that every time the server restart, I get this message from the alerts menu

Code:

Failed to start kubernetes cluster for Applications: (101, 'Network is unreachable')
2022-08-16 19:22:27 (America/Toronto)

OS Version:TrueNAS-SCALE-22.02.3
Product:Z9PE-D16 Series
Model:Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Memory:63 GiB

 

[indivision]

Hm. If you can't even change the pool that way, it sounds like something bigger is wrong.

12GB of RAM is not very much for a server. Are you sure that you aren't running into hardware-related issues?

 

[indivision]

How invested are you into the current state of the server?

In other words, would it be catastrophic to re-install Scale from scratch?

 

[alieg]

I have the ability to go up to 16GB, the only issue I've had is a repeating error
"Device /dev/gptid/6c870662-da40-11ea-8ad9-1cc1defa4b32 is causing slow I/O on pool Storage."

But other than that I haven't experienced any issues, it generally sits at 3GB Used, 6.5GB free.
It's mainly a Plex and Home Assistant server. I don't do much with file sharing anymore. It can be a bit slow to install things and I assume my impatience with it has led to some of this.

It wouldn't be too much of an issue to re-install if I had to.
A more preferable option to me would be to change the boot environment to my previous Core 13.0-U1 (I have a config file I downloaded before upgrading if needed), and perhaps try again immediately or at a later date - is changing the boot environment something that can be done to roll back?

I can get access to the settings from ssh for the settings, but it's been difficult to find an example of how it should be written out.

If I could unset pool and choose pool from ssh that would get around it.

I've managed to delete most of the ix-applications folder but it's giving an error of "/releases: pool or dataset is busy and cannot unmount"

 

[indivision]

It wouldn't be too much of an issue to re-install if I had to.

That's what I would do. Something may have been messed up during upgrade or messing around with apps. If your system is really slow, you might have reset or some other interruption before a process completed.

A more preferable option to me would be to change the boot environment to my previous Core 13.0-U1 (I have a config file I downloaded before upgrading if needed), and perhaps try again immediately or at a later date - is changing the boot environment something that can be done to roll back?

I think this is possible in theory. But, someone else would have to confirm...

I've managed to delete most of the ix-applications folder but it's giving an error of "/releases: pool or dataset is busy and cannot unmount"

I think you're probably just digging a deeper hole with this type of thing. It's always better to be able to troubleshoot from a state of functionality. And the distance from where you are to that is a re-install. It feels like piecing it together from the current state will be a longer adventure than that.

 

[alieg]

I've fully deleted the ix-applications folder after a reboot and setup the apps pool settings again. I've currently installed Home Assistant and Plex and I'm waiting to see if given time it will start.

With Home Assistant, It's not showing in the installed applications tab and the activity monitor is showing this (as it did before):
Installing catalog item 75%
Waiting for 'home-assistant' pods to terminate 75%
Is this normal for setup?
(the 2 failed items are where I tried to use upper case characters in the name)

 

[indivision]

It's difficult to know for sure what would be normal on a very minimal system. It would definitely not be normal on my system. Apps typically install in less than 60 seconds.

Did you try looking at the logs and events for the apps?

 

[alieg]

Yes, initially I could see some logs saying something along the lines of "BadRequest code: 400" but on my later attempts it there was no available container in the dropdown to select, so I couldn't see any logs.

From starting installing Plex at 10:50 I waited until 14:20 and it was still stuck in deploying.
I decided to try changing the boot environment back to core. It worked and successfully booted back into my old config seemingly without issue.

I thought this would have been something fairly simple when starting out but it's been very frustrating to troubleshoot as there's so little info out there, most of the threads I've come across have been dead ends.

 

[indivision]

As successful as its already been, Scale is still pretty new. So, there may not be as many edge-case scenarios covered in the forums.

It may be that Core is able to slog along with an under-powered system easier than Scale can.

Years ago, I used to experiment with Core on some pretty dated hardware. And it actually did pretty well most of the time. But, by the time I switched to Scale I had invested in better hardware. So, I can't really compare the two in that metric.

 

[alugowski]

You must configure some Kubernetes networking settings else you'll get that 'Network is unreachable' error. Under Apps -> Settings -> Advanced Settings select the "Route v4 Interface" to be the network interface you use, and also the gateway. Just make these match your NAS's networking. The interface is the network card your ethernet cable is plugged into, and the gateway is just your router IP.

I don't understand why these don't get populated for you, but they don't. Feel free to submit a Jira ticket for it.

You must also specify a default gateway in Network -> Global Configuration -> Settings. Again you're just copying what DHCP returns, but for some reason you must duplicate it again manually.

 

[alugowski]

I have the ability to go up to 16GB, the only issue I've had is a repeating error
"Device /dev/gptid/6c870662-da40-11ea-8ad9-1cc1defa4b32 is causing slow I/O on pool Storage."

But other than that I haven't experienced any issues, it generally sits at 3GB Used, 6.5GB free.
It's mainly a Plex and Home Assistant server. I don't do much with file sharing anymore. It can be a bit slow to install things and I assume my impatience with it has led to some of this.

It wouldn't be too much of an issue to re-install if I had to.
A more preferable option to me would be to change the boot environment to my previous Core 13.0-U1 (I have a config file I downloaded before upgrading if needed), and perhaps try again immediately or at a later date - is changing the boot environment something that can be done to roll back?

I can get access to the settings from ssh for the settings, but it's been difficult to find an example of how it should be written out.

If I could unset pool and choose pool from ssh that would get around it.

I've managed to delete most of the ix-applications folder but it's giving an error of "/releases: pool or dataset is busy and cannot unmount"

You have plenty of RAM for your use case. That slow I/O might be a sign that disk is dying, but that's unrelated to your other issues.

Officially there is no way to migrate from SCALE to CORE. Reinstall and reload your saved CORE configuration. If you didn't wipe your iocage volume then you might be able to boot it back as it was.

Honestly unless you have a compelling reason to use SCALE, for you I'd say go back to CORE. There are a lot of little bugs here and there in SCALE that can set you back. One is that there is a race condition somewhere in the app startup, so sometimes apps don't come up after a reboot. You must reboot again to fix. It's rare-ish, but it happened to me several times already and it would be infuriating if my home automation depended on it. That said, having easy access to the Docker Hub is a big deal (and is the reason I moved over and dealing with all the teething pains).

 

[daniel-]

@alieg , I was able to fix a deployment error on my TrueNAS scale. Before deploying an app, I did go to the "Manage Docker Images" tab. There I saw 3 updates for rancher and other thing. I updated them and was able to deploy app or docker afterward.

Hope it helps!

 

[Hurlikus]

Than

You must configure some Kubernetes networking settings else you'll get that 'Network is unreachable' error. Under Apps -> Settings -> Advanced Settings select the "Route v4 Interface" to be the network interface you use, and also the gateway. Just make these match your NAS's networking. The interface is the network card your ethernet cable is plugged into, and the gateway is just your router IP.

View attachment 57917

I don't understand why these don't get populated for you, but they don't. Feel free to submit a Jira ticket for it.

You must also specify a default gateway in Network -> Global Configuration -> Settings. Again you're just copying what DHCP returns, but for some reason you must duplicate it again manually.

View attachment 57919

Thank you, this fixed it for me. My apps were stuck deploying after a reboot (evem though they worked fine before) but deployed again after your fix.

 

[d1gz0r]

I am having a little different issue. When I click install on any app I just get "Please wait" pop up with no way to close it and it stays active permanently. I can only exit it by refreshing the page. No error or anything like that. Logs are clear too...

 

[d1gz0r]

I am having a little different issue. When I click install on any app I just get "Please wait" pop up with no way to close it and it stays active permanently. I can only exit it by refreshing the page. No error or anything like that. Logs are clear too...

 

[d1gz0r]

I just got everything working. This will not work for everyone, but worth the try:
At first I thought to re-install TrueNAS (it's just a some server for fun, so this is no big deal). But instead I upgraded to BETA release. Apps started to install, but fail on 75%. I than rollback to stable version and everything just started working!

 

[alieg]

I've come back to try again with Scale because I want to update Home Assistant but I'm having this problem again.