Apps not showing up or working after reboot ...

[pixelwave]

Hi,

I am on the Truenas Scale release version (TrueNAS-SCALE-22.02.0).

Yesterday I rebooted my system and noticed that the apps disappeared:

The pool is still there and data as well:

I also tried to reboot many times ... after the last one let it stay online for about 10 hours and the last thing I did was a middleware restart in the CLI with:

"service middlewared restart"

All without success. This is super weird? How can they just disappear without error ... pool still there ... data still there ...?

I had various Offical Chart apps (Deconz, MQTT, ..) as well as Truecharts apps installed (homeassistant, resilio, ...).

 

[pixelwave]

Ok ... weird ... I rebooted now again and they show up.

 

[Kris Moore]

Ok ... weird ... I rebooted now again and they show up.

Wonder if it some race condition or similar. If you can reproduce it again later, please file a ticket on https://jira.ixsystems.com with your debug file so we can dig into it.

 

[dffvb]

I had the same yesterday too, and eventually they came back...

 

[K.J]

Wonder if it some race condition or similar. If you can reproduce it again later, please file a ticket on https://jira.ixsystems.com with your debug file so we can dig into it.

@Kris Moore @dffvb @pixelwave

Here also the necessary problems after the update from Core to Scale.
See the tread:

Problems/questions after updating from Core to Scale.

Was glad I finally got Truenas Core running last year with help from you guys without any problems. After the update from Core to Scale I still run into some problems. (1) automatic update doesn't work anymore. (2) I don't get to see apps anymore. Like plex etc.... (3) In mail alerts now only...

www.truenas.com

 

[pixelwave]

@Kris Moore @dffvb @pixelwave

Here also the necessary problems after the update from Core to Scale.
See the tread:

Problems/questions after updating from Core to Scale.

Was glad I finally got Truenas Core running last year with help from you guys without any problems. After the update from Core to Scale I still run into some problems. (1) automatic update doesn't work anymore. (2) I don't get to see apps anymore. Like plex etc.... (3) In mail alerts now only...

www.truenas.com

I have a fresh Scale install .. no migration from core or anything if you refer to that ... ?

 

[K.J]

I have a fresh Scale install .. no migration from core or anything if you refer to that ... ?

I did an update from Core

 

[madmattco]

I've encountered an issue similar to yours but my apps never went missing. I am just stuck on deploying. This happened after my first reboot of the system after I had everything working and configured properly. Tried rebooting, restarting the k3s service, and I'm almost to the point of just saving my config and reinstalling scale. Really dont understand why they wanted to overcomplicate the whole docker vs kubernetes thing. Docker would have worked fine for these releases.

I did file a ticket on jira with debug information so hopefully it will get resolved.

 

[BoomShaka]

Just wanted to chime in on this too to draw some more attention. I have a nightly scheduled reboot and for the past 2 days the apps don't show up after that. However rebooting from the GUI has always brought them back.

I am running the reboot bash command to reboot. Are there extra processes in the GUI reboot function? Is there a way to trigger this kind of reboot from either a bash script or the Cron job section of the GUI? If it helps, all of my apps are from the Truecharts catalog.

 

[K.J]

@BoomShaka

What helped with me to get the network working again so I could load the app again is to change the lag configuration from lacp to failover.
Later I changed it back to lacp anyway.
So far the network remains up and running.

 

[gsrcrxsi]

i had the same issue. it seems to be an intermittent issue. some reboots all the apps are gone, and some reboots they come back fine. there seems to be no way to know.

@Kris Moore , which debug would be most helpful? grabbing the debug on the times when it boots up with no apps? or when you reboot and the apps come back? or both?

 

[cyrus104]

I've gotten this several times, a reboot fixed it but since I'm using it as a VM share, my external hypervisor starts all VMs which then all need to be restarted again as well. It's not a smooth or fun process.

 

[BoomShaka]

So I haven't checked the logs of the failed startups yet, but I had an idea of what might be related to this issue.

30 minutes before my nightly reboot, I have a Cron job for Truetool to backup, upgrade the Truecharts apps and update the catalog. Instead of being time based, I moved the reboot command to the end of the Truetool script and changed the command to shutdown -r now instead of reboot. Not a very scientific approach but I'll try more step by step things if the problem persists. For now, I tried manually running this new Cron job a couple times and there was no issues. It also ran last night with no issues.

What I suspect was the issue is that Truecharts was in the middle of running a catalog update or something and the shutdown script didn't properly stop it. Since the shutdown happens immediately after these tasks are manually run now, it's pretty much guaranteed that these tasks are not being run during reboot.

I'll monitor this further and maybe bring the question up over at Truecharts if it comes back.

 

[BoomShaka]

So for anyone following this, my fix didn't work and now I have 2 days of data, one where apps disappeared and one where everything worked. I'm gonna go through the logs to find any differences.

I'll post updates and link the Truecharts forum post I start. Maybe I'll link any other thread with the same issue.

 

[Fred Hardy]

I have the same problem, just giving some info if it can help solving this issue.
My apps reappeared after a clean power off/power on.
When they reappeared where was no automatic catalogs "refresh all" at startup and when they disappeared there was one.

 

[stavros-k]

Wonder if it some race condition or similar. If you can reproduce it again later, please file a ticket on https://jira.ixsystems.com with your debug file so we can dig into it.

We've seen plenty of users having this issue in Truechart's discord. I've had a "simillar"(NAS-114677) issue some days before release. Which was a race condition, @waqarahmed found what it was and patched. Not sure if it got in the Release.

Note: I'm on nightlies since this patch and didn't have this or similar problem.
Note2: I don't advice anyone switching to nightlies for production use.

 

[Sparx]

Any hints on this? I have the same issue. And since there is a lot of VMs to start and get various stuff rolling for each reboot, i would prefer a fix not involving a reboot. middleware restart didnt help. Docker ps shows that some of the apps are up.

 

[Kulmala]

TrueNAS-SCALE-22.02.0.1 Same, occasionally app not showing but running other times not showing and not running, weird indeed.

 

[HarryMuscle]

TrueNAS-SCALE-22.02.0.1 Same, occasionally app not showing but running other times not showing and not running, weird indeed.

I suggest creating a ticket for IX Systems to properly investigate and fix the issue.

 

[blacksteel75]

@BoomShaka did you ever find a solution to the problem? I am experiencing the same.

I have a weekly "reboot" cron job and the applications do not start when the server comes back up. If I do a manual restart however, all of the applications do startup normally as expected.

If there's an IX Systems JIRA ticket I should follow, could you let me know?

I'm running TrueNAS-SCALE-22.02.2.1