SOLVED Kubernetes not running

[sdgenxr]

I'm having problems with the dataset that my apps are on. The dataset got filled up to 100% from all the OS upgrades and improper Sabnzb download folder configuration on my part. I have since been able to delete the "iocage" directory freeing up some space. Unfortunately, Kubernetes will not restart with a system reboot.

The dataset is on a pair of partitioned ssd's for my boot drives and apps. Should I wipe the dataset and start over with my apps, or can I get Kubernetes to start working again and continue with the apps I've already installed? I'm currently running the latest version of Cobia.

 

[sretalla]

I'm currently running the latest version of Cobia.

Probably not as this hotpatch 23.10.1.1 was released to address the network issues on startup with kubernetes in 23.10.1.

TrueNAS SCALE 23.10.1.1 has been released!

We are pleased to release TrueNAS SCALE 23.10.1.1! This is a small hotpatch with fixes for web interface issues discovered after the 23.10.1 release. The full 23.10.2 maintenance release is anticipated later in February. Changes: Fix non-physical interface (Link Aggregation, VLAN, Bridge)...

www.truenas.com

You should be able to get it working with the same dataset if you can get enough free space in the pool.

 

[sdgenxr]

Probably not as this hotpatch 23.10.1.1 was released to address the network issues on startup with kubernetes in 23.10.1.

Thanks for the info about this hot patch. Unfortunately, Kubernetes is still not running and the app page is still showing "Initializing Apps Service".

 

[Belperite]

Probably not as this hotpatch 23.10.1.1 was released to address the network issues on startup with kubernetes in 23.10.1.

TrueNAS SCALE 23.10.1.1 has been released!

We are pleased to release TrueNAS SCALE 23.10.1.1! This is a small hotpatch with fixes for web interface issues discovered after the 23.10.1 release. The full 23.10.2 maintenance release is anticipated later in February. Changes: Fix non-physical interface (Link Aggregation, VLAN, Bridge)...

www.truenas.com

You should be able to get it working with the same dataset if you can get enough free space in the pool.

This didn't fix the issue for me. I still have to toggle the GPU setting in order for k8s / truenas to find the bridge on reboot.

 

[sdgenxr]

Any other ideas on how I can get Kubernetes running again?

 

[sretalla]

Does the pool designated for apps still have 0b available?

How much free space is there?

I would suggest if there's enough, the service will be able to start.

 

[sdgenxr]

I have since been able to delete the "iocage" directory freeing up some space. Unfortunately, Kubernetes will not restart with a system reboot.

I moved the applications pool to my HDD tank dataset and kubernetes still does not start. When I try and move the applications pool back to my ssd dataset, I get the following errors:

Error: force
Apps have been partially initialized on 'ssd' pool but it is missing 'ssd/ix-applications/releases, ssd/ix-applications/catalogs, ssd/ix-applications/default_volumes' datasets. Specify force to override this and let system re-initialize applications.

Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 427, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 465, in __run_body rv = await self.method(*([self] + args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf res = await f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 461, in do_update await self.validate_data(config, 'kubernetes_update', old_config) File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 355, in validate_data verrors.check() File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 70, in check raise self middlewared.service_exception.ValidationErrors: [EINVAL] kubernetes_update.force: Apps have been partially initialized on 'ssd' pool but it is missing 'ssd/ix-applications/releases, ssd/ix-applications/catalogs, ssd/ix-applications/default_volumes' datasets. Specify force to override this and let system re-initialize applications.

Error: [EINVAL] kubernetes_update.force: Apps have been partially initialized on 'ssd' pool but it is missing 'ssd/ix-applications/releases, ssd/ix-applications/catalogs, ssd/ix-applications/default_volumes' datasets. Specify force to override this and let system re-initialize applications.

 

[sretalla]

It's certainly telling you that some parts of ix-applications is missing, so you can use the "Force" option under Apps | Settings | Advanced Settings to get the system to put it back (I'm not clear if you already tried that or not).

Also: this will cause redeployment of all apps, so if you haven't been careful, some things in apps can go missing.

 

[sdgenxr]

Once I deleted all the snapshots, I was able to delete the application directories and reinstall all the apps.