Some APPs Failing after Angelfish to Bluefin Upgrade

[achas008]

I've got a system with multiple apps, two of which are failing after upgrading from angelfish to bluefin. The error looks like this:

Code:

: Process 29(postgres) has RLIMIT_CORE set to 1
: Aborting core

My system:
Motherboard: ASUSTeK Z11PA-U12
CPU: Intel Xeon Bronze 3104 (6 cores @ 1.7GHZ)
RAM: 124.5 GiB
Hard Drives:
tank: 11 disks, RAIDZ3
5 x 6TB (Model WDC_WD60EFZX-68B3FN0)
5 x 4TB (Model HGST_HDN724040ALE640)
1 x 4TB (Model Hitachi_HUS724040ALE641)
1 x 1TB [cache] (Model SPCC M.2 PCIe SSD)
boot: 1 Disk
1 x 512 GB (Model SanDisk_SD8SB8U512G1122)
other?
1 x 0B (Virtual_HDisk1)
1 x 0B (Virtual_Floppy0)
SATA Controller: C620 Series Chipset Family SSATA Controler [AHCI mode]
Network Cards:
I210 Gigabit Network Connection [Ethernet Interface] (Integrated, x2)
BCM57406 NetXtreme-E 10GBASE-T Ethernet Controller (1 x PCIe, 2 ports)

I have 5 apps:
PostgreSQL (fails to deploy)
unifi
openhab
plex
wikijs (fails to deploy)

 

[morganL]

Thanks

The code with "aborting core"... where did you get that info from?

In any case, please report a bug.
Can you confirm the same error survives a reboot?

 

[achas008]

The aborting core messages appear in the console output at the bottom of the UI.

So far, this error is pretty robust, surviving reboots and changing app settings. Turning off all other apps and VMs and trying to start the app produces the same error.

I'll get a bug report pushed out.

 

[morganL]

Please share the bug ID/ticket

 

[achas008]

Bug ID NAS-119963

[NAS-119963] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[impestrator]

Hi, I upgraded from Angelfish to Bluefin today and have been having similar problems since. Some apps hang in a deployment loop (Plex, Home Assistant etc), some are deployed but the web interface is not accessible (Pihole). I have now set all settings to "load balancer", selected a nic and configured it with DHCP. When I call the IP of the Pihole instance, I see Lighttpd. So the web server itself is there, but how do I now get to the webinteface of the application?

As for the problem with the deployment loop. Is it the same problem: Do you have any idea? I can't find anything in the logs either.

I would be very grateful for your help.

Greetings
Martin

 

[achas008]

It seems like the devs are looking for systems having this problem. It's difficult to duplicate the issue. See this post:

Looking for Bluefin systems where Apps don't start

(repost from another discussion) Adding a status note on TrueNAS SCALE Bluefin.... There are now about 15,000 Bluefin systems up and active. Most of iX TrueNAS test systems have been running Bluefin very reliably. Apart from the bugs identified in the release notes, the Apps have had the most...

www.truenas.com

 

[browntiger]

Yeah but it sounds like number of You have the issues with TrueCharts PostgreSQL pods. I don't know if TrueNas folks address those..
In addition the ugh: "selected a nic and configured it with DHCP"... K8s with static IP - sure. K8s bound to DHCP... not me.

 

[morganL]

It seems like the devs are looking for systems having this problem. It's difficult to duplicate the issue. See this post:

Looking for Bluefin systems where Apps don't start

(repost from another discussion) Adding a status note on TrueNAS SCALE Bluefin.... There are now about 15,000 Bluefin systems up and active. Most of iX TrueNAS test systems have been running Bluefin very reliably. Apart from the bugs identified in the release notes, the Apps have had the most...

www.truenas.com

This issue may be a specific App issue. However, you indicated plex was also having problems with permissions.

Are you sharing datasets via SMB or NFS? Have you removed Host Path validation?

Can you describe the permissions you have?

 

[impestrator]

This issue may be a specific App issue. However, you indicated plex was also having problems with permissions.

Are you sharing datasets via SMB or NFS? Have you removed Host Path validation?

Can you describe the permissions you have?

I share (in that case) the dataset via SMB. Host Path Validation within Kubernetes settings is disabled, I enabled that for testing without any sucess.

What exactly do you meant with "Can you describe the permissions you have?"? Do you mean SMB permissions on the share?

 

[morganL]

I share (in that case) the dataset via SMB. Host Path Validation within Kubernetes settings is disabled, I enabled that for testing without any sucess.

What exactly do you meant with "Can you describe the permissions you have?"? Do you mean SMB permissions on the share?

Yes, just looking for anything that might prevent the App users from accessing the files.

 

[impestrator]

I would like to share my SMB permissions to you but my Truenas Scale WebUI is set to german and if I try to set it to English I´ll get the following messeage.

I´ve tried to create a new certificate, but that does not do the trick, Anyways, here are my SMB persmissions, I hope you can handle the settings with German GUI:

Share ACL:

Filesystem ACL:

I noticed really strange stuff with the apps since I updated to Bluefin:

The app "Home Assistant" (no SMB shares) runs flawless: I set the network configuration to static IP, defined the port -> Web UI is accessable

PIhole (no SMB shares): Container runs, set the network configuration to static IP, defined the port -> Web UI is not accessable via ip:port. Without the port, I can verify that a webserver ist running.

All other containers (plex, paperless, nextcloud) with SMB shares are in the deploying loop. Are there any logs I can provide to you? All the logs I can see within the specified app container do not show any helpful entries...

Edit: I got Pihole up and running, webui is reachable. I can confirm, that the deploying loop is related to the smb share. When I change "Type of Storage" from "Home Path" to "PVC", the app runs. When I change back to "Host Patch" (and then the path to the share) it fails again. So something is wrong with the share. But what? I didn´t change anything. Is that maybe related to this one? Bluefin what to get rid of root user, doesn´t it?

 

[LarsR]

With bluefin a hostpath validation was implemented, which prevents apps from starting if the same path is used for smb and hostpath.
There's two solutions:
1. Disable Hostpath validation in the advanced apps pool settings.
2. Create a child Dataset, Copy your Data inside that Child Dataset, edit the permissions and use the Child Dataset as the hostpath.
This way the smb and hostpath are different an your apps should start. I'm doing this with my plex app

 

[impestrator]

Thank you very much for that advice. Disabling Hostpath validation solved the issue for Plex, but the app "paperless" is still in the deployment loop, although I set (for testing) the storage settings from Host path to PVC or even when I use the same configuration (beside the exact path) then plex. Could you also help me with that?

 

[LarsR]

I'm afraid i can't help you with paperless, i've never used it.
And i'm not an well versed in the art of troubleshooting apps myself.
If you're using the truecharts version of paperless your best bet would be the truecharts discord.
But they won't accept tickets when hostpath validation is turned off since it's a security feature.

 

[impestrator]

Thanks for your explainations, but what I still don´t understand. If I enable Hostpath validation, Plex App won´t start although I do not share the folder in "hostpath" via SMB or anything else.

And as I said, Paperless does not start, even if I set "PVC" and it worked before I upgrated to bluefin.
Anyways, thank you again for your effort and time you spent for supporting me...

 

[JmarcSyd]

With bluefin a hostpath validation was implemented, which prevents apps from starting if the same path is used for smb and hostpath.
There's two solutions:
1. Disable Hostpath validation in the advanced apps pool settings.
2. Create a child Dataset, Copy your Data inside that Child Dataset, edit the permissions and use the Child Dataset as the hostpath.
This way the smb and hostpath are different an your apps should start. I'm doing this with my plex app

Thanks Lars. Did you have to recreate your Plex library? Such a hassle ...

 

[LarsR]

No i didn't have to recreate anything.

 

[JmarcSyd]

No i didn't have to recreate anything.

Thanks Lars. It took a while to move the Plex files to the child dataset but then it worked like a charm.

 

[JmarcSyd]

Well this doesn't work anymore in 22.12.1 . Looks like it was a bug that allowed the use of a child dataset and the bug has been fixed. I've just disabled Hostpath validation to make it work because I really can't be bothered now.