paperless-ngx issues after Bluefin update

indivision · Jan 18, 2023

I've had paperless-ngx running well for some time. But, after updating to Bluefin, it stopped processing new documents. It gives a vague error about the server closing the connection unexpectedly. The full error is:

Code:

[2023-01-18 12:54:49,917] [ERROR] [paperless.handlers] Creating PaperlessTask failed: server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute

    return self.cursor.execute(sql, params)

psycopg2.OperationalError: server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/usr/src/paperless/src/documents/signals/handlers.py", line 548, in before_task_publish_handler

    PaperlessTask.objects.create(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 671, in create

    obj.save(force_insert=True, using=self.db)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 812, in save

    self.save_base(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 863, in save_base

    updated = self._save_table(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1006, in _save_table

    results = self._do_insert(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1047, in _do_insert

    return manager._insert(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1791, in _insert

    return query.get_compiler(using=using).execute_sql(returning_fields)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql

    cursor.execute(sql, params)

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute

    return self._execute_with_wrappers(

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers

    return executor(sql, params, many, context)

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute

    return self.cursor.execute(sql, params)

  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__

    raise dj_exc_value.with_traceback(traceback) from exc_value

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute

    return self.cursor.execute(sql, params)

django.db.utils.OperationalError: server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

Has anyone else run into this issue and found a fix?

Thank you for any help you can offer!

browntiger · Jan 20, 2023

TrueCharts paperless-ng? Mine works fine - no complaints. I can post my settings if needed.

indivision · Jan 20, 2023

browntiger said:
TrueCharts paperless-ng? Mine works fine - no complaints. I can post my settings if needed.

Thank you. Yes. Please do post the external app settings from the TrueNAS UI.

I'm sure it will work again from a fresh install. But, maybe I can revive with some settings adjustment that happened.

Restoring the documents into a new install can be done. But, it's a PIA. Hoping to avoid! :)

browntiger · Jan 20, 2023

You should be able to reconfigure it without any effect on the data or documents.
I know my permissions are too loose 770, or even 750 would have been fine.

indivision · Jan 21, 2023

browntiger said:
You should be able to reconfigure it without any effect on the data or documents.
I know my permissions are too loose 770, or even 750 would have been fine.

Thank you. I think my permissions are ok as they were working for about a year before the Bluefin update.

Can you share the other settings on the app page? I have seen those require adjustments before. So, I'm wondering if that's what happened this time...

browntiger · Jan 22, 2023

Sure but it is all standard stuff. It looks like @truecharts completely redesigned and revamped their app.
I can not recognize it. I thought I forked it just a few weeks back.

Here it is LOL https://github.com/truecharts/catalog/tree/main/stable/paperless-ng/
Check your postgresql and the permissions... There is not much to go.
May be there is some sort of issue with postgresql, shell to it or check the logs

But here are my settlings

indivision · Jan 23, 2023

Thank you. It looks like mine were already the same.

I ended up re-installing with the same settings and it works again. Though, a few of the largest documents that it had processed before will no longer process. Maybe the file size limit setting was changed in that over-haul?

browntiger · Jan 23, 2023

Paperless has param DEFAULT_WAIT_TIMEOUT is typically set to 30 limiting the max file size.

indivision · Jan 25, 2023

Hm. Is this changeable as an environment variable in TrueNAS? Or, changing config file somewhere?

I tried adding it as an environment var with 120. But, the same files still failed...

browntiger · Jan 25, 2023

Should be able to add env, DEFAULT_WAIT_TIMEOUT and 120 right in the chart. May be ask @truecharts ...

truecharts · Jan 27, 2023

browntiger said:
Should be able to add env, DEFAULT_WAIT_TIMEOUT and 120 right in the chart. May be ask @truecharts ...

Not sure, other (support)staff members have this specific app as their specialty.

browntiger · Jan 27, 2023

So let me try.
1) Add it to Extra Environment Variables
Variable: DEFAULT_WAIT_TIMEOUT
Value: 120

After the pods deployed: select paperless-ng, hit three vertical dots hamburger icon. Select shell. In the shell select paperless-ng-XXXXXX pod.
After shell opens type: set
check if variable DEFAULT_WAIT_TIMEOUT is set.

impestrator · Feb 3, 2023

Hello,

after I upgraded to bluefin two weeks ago I had some issues with several apps. I could make them running again after disabling host validation and reconfiguring network settings, but paperless-ngx is still in the deployment loop. Maybe you have an idea how to fix it.

I´ve got the following logs, which does not help me...to be honest:

Code:

2023-02-03T13:37:05.375901177Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m
2023-02-03T13:37:05.377689708Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m[1mWelcome to the Bitnami postgresql container[0m
2023-02-03T13:37:05.379794524Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
2023-02-03T13:37:05.382002640Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
2023-02-03T13:37:05.384010481Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0m
2023-02-03T13:37:05.401347313Z [38;5;6mpostgresql [38;5;5m13:37:05.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL setup **
2023-02-03T13:37:05.428629817Z [38;5;6mpostgresql [38;5;5m13:37:05.42 [0m[38;5;2mINFO [0m ==> Validating settings in POSTGRESQL_* env vars..
2023-02-03T13:37:05.435553271Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Cleaning stale /bitnami/postgresql/data/postmaster.pid file
2023-02-03T13:37:05.439262138Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Loading custom pre-init scripts...
2023-02-03T13:37:05.443801228Z [38;5;6mpostgresql [38;5;5m13:37:05.44 [0m[38;5;2mINFO [0m ==> Initializing PostgreSQL database...
2023-02-03T13:37:05.461285848Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> pg_hba.conf file not detected. Generating it...
2023-02-03T13:37:05.463490028Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> Generating local authentication configuration
2023-02-03T13:37:07.280880781Z [38;5;6mpostgresql [38;5;5m13:37:07.28 [0m[38;5;2mINFO [0m ==> Deploying PostgreSQL with persisted data...
2023-02-03T13:37:07.292909728Z [38;5;6mpostgresql [38;5;5m13:37:07.29 [0m[38;5;2mINFO [0m ==> Configuring replication parameters
2023-02-03T13:37:07.326803311Z [38;5;6mpostgresql [38;5;5m13:37:07.32 [0m[38;5;2mINFO [0m ==> Configuring fsync
2023-02-03T13:37:07.335811278Z [38;5;6mpostgresql [38;5;5m13:37:07.33 [0m[38;5;2mINFO [0m ==> Configuring synchronous_replication
2023-02-03T13:37:07.356447148Z [38;5;6mpostgresql [38;5;5m13:37:07.35 [0m[38;5;2mINFO [0m ==> Loading custom scripts...
2023-02-03T13:37:07.361128640Z [38;5;6mpostgresql [38;5;5m13:37:07.36 [0m[38;5;2mINFO [0m ==> Enabling remote connections
2023-02-03T13:37:07.372642799Z [38;5;6mpostgresql [38;5;5m13:37:07.37 [0m[38;5;2mINFO [0m ==> ** PostgreSQL setup finished! **
2023-02-03T13:37:07.372718490Z
2023-02-03T13:37:07.398219936Z [38;5;6mpostgresql [38;5;5m13:37:07.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL **
2023-02-03T13:37:07.429331329Z 2023-02-03 13:37:07.429 GMT [1] LOG:  pgaudit extension initialized
2023-02-03T13:37:07.441928608Z 2023-02-03 13:37:07.441 GMT [1] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-02-03T13:37:07.442469695Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-02-03T13:37:07.442499074Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2023-02-03T13:37:07.456260299Z 2023-02-03 13:37:07.456 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-02-03T13:37:07.476475240Z 2023-02-03 13:37:07.476 GMT [87] LOG:  database system was interrupted; last known up at 2023-02-03 13:35:42 GMT
2023-02-03T13:37:07.532778664Z 2023-02-03 13:37:07.532 GMT [87] LOG:  database system was not properly shut down; automatic recovery in progress
2023-02-03T13:37:07.548605395Z 2023-02-03 13:37:07.548 GMT [87] LOG:  invalid record length at 0/239F0AC0: wanted 24, got 0
2023-02-03T13:37:07.548644004Z 2023-02-03 13:37:07.548 GMT [87] LOG:  redo is not required
2023-02-03T13:37:07.610310774Z 2023-02-03 13:37:07.610 GMT [1] LOG:  database system is ready to accept connections

Code:

2023-02-03T13:37:05.375901177Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m
2023-02-03T13:37:05.377689708Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m[1mWelcome to the Bitnami postgresql container[0m
2023-02-03T13:37:05.379794524Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
2023-02-03T13:37:05.382002640Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
2023-02-03T13:37:05.384010481Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0m
2023-02-03T13:37:05.401347313Z [38;5;6mpostgresql [38;5;5m13:37:05.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL setup **
2023-02-03T13:37:05.428629817Z [38;5;6mpostgresql [38;5;5m13:37:05.42 [0m[38;5;2mINFO [0m ==> Validating settings in POSTGRESQL_* env vars..
2023-02-03T13:37:05.435553271Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Cleaning stale /bitnami/postgresql/data/postmaster.pid file
2023-02-03T13:37:05.439262138Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Loading custom pre-init scripts...
2023-02-03T13:37:05.443801228Z [38;5;6mpostgresql [38;5;5m13:37:05.44 [0m[38;5;2mINFO [0m ==> Initializing PostgreSQL database...
2023-02-03T13:37:05.461285848Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> pg_hba.conf file not detected. Generating it...
2023-02-03T13:37:05.463490028Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> Generating local authentication configuration
2023-02-03T13:37:07.280880781Z [38;5;6mpostgresql [38;5;5m13:37:07.28 [0m[38;5;2mINFO [0m ==> Deploying PostgreSQL with persisted data...
2023-02-03T13:37:07.292909728Z [38;5;6mpostgresql [38;5;5m13:37:07.29 [0m[38;5;2mINFO [0m ==> Configuring replication parameters
2023-02-03T13:37:07.326803311Z [38;5;6mpostgresql [38;5;5m13:37:07.32 [0m[38;5;2mINFO [0m ==> Configuring fsync
2023-02-03T13:37:07.335811278Z [38;5;6mpostgresql [38;5;5m13:37:07.33 [0m[38;5;2mINFO [0m ==> Configuring synchronous_replication
2023-02-03T13:37:07.356447148Z [38;5;6mpostgresql [38;5;5m13:37:07.35 [0m[38;5;2mINFO [0m ==> Loading custom scripts...
2023-02-03T13:37:07.361128640Z [38;5;6mpostgresql [38;5;5m13:37:07.36 [0m[38;5;2mINFO [0m ==> Enabling remote connections
2023-02-03T13:37:07.372642799Z [38;5;6mpostgresql [38;5;5m13:37:07.37 [0m[38;5;2mINFO [0m ==> ** PostgreSQL setup finished! **
2023-02-03T13:37:07.372718490Z
2023-02-03T13:37:07.398219936Z [38;5;6mpostgresql [38;5;5m13:37:07.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL **
2023-02-03T13:37:07.429331329Z 2023-02-03 13:37:07.429 GMT [1] LOG:  pgaudit extension initialized
2023-02-03T13:37:07.441928608Z 2023-02-03 13:37:07.441 GMT [1] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-02-03T13:37:07.442469695Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-02-03T13:37:07.442499074Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2023-02-03T13:37:07.456260299Z 2023-02-03 13:37:07.456 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-02-03T13:37:07.476475240Z 2023-02-03 13:37:07.476 GMT [87] LOG:  database system was interrupted; last known up at 2023-02-03 13:35:42 GMT
2023-02-03T13:37:07.532778664Z 2023-02-03 13:37:07.532 GMT [87] LOG:  database system was not properly shut down; automatic recovery in progress
2023-02-03T13:37:07.548605395Z 2023-02-03 13:37:07.548 GMT [87] LOG:  invalid record length at 0/239F0AC0: wanted 24, got 0
2023-02-03T13:37:07.548644004Z 2023-02-03 13:37:07.548 GMT [87] LOG:  redo is not required
2023-02-03T13:37:07.610310774Z 2023-02-03 13:37:07.610 GMT [1] LOG:  database system is ready to accept connections

Code:

2023-02-03T13:37:03.079347632Z [38;5;6mredis [38;5;5m13:37:03.07 [0m
2023-02-03T13:37:03.080619237Z [38;5;6mredis [38;5;5m13:37:03.08 [0m[1mWelcome to the Bitnami redis container[0m
2023-02-03T13:37:03.081882544Z [38;5;6mredis [38;5;5m13:37:03.08 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
2023-02-03T13:37:03.083417482Z [38;5;6mredis [38;5;5m13:37:03.08 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
2023-02-03T13:37:03.084734015Z [38;5;6mredis [38;5;5m13:37:03.08 [0m
2023-02-03T13:37:03.086506965Z [38;5;6mredis [38;5;5m13:37:03.08 [0m[38;5;2mINFO [0m ==> ** Starting Redis setup **
2023-02-03T13:37:03.108488071Z [38;5;6mredis [38;5;5m13:37:03.10 [0m[38;5;3mWARN [0m ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
2023-02-03T13:37:03.112259963Z [38;5;6mredis [38;5;5m13:37:03.11 [0m[38;5;2mINFO [0m ==> Initializing Redis
2023-02-03T13:37:03.126320218Z [38;5;6mredis [38;5;5m13:37:03.12 [0m[38;5;2mINFO [0m ==> Setting Redis config file
2023-02-03T13:37:04.502526188Z [38;5;6mredis [38;5;5m13:37:04.50 [0m[38;5;2mINFO [0m ==> Configuring replication mode
2023-02-03T13:37:04.548437011Z
2023-02-03T13:37:04.548495502Z [38;5;6mredis [38;5;5m13:37:04.54 [0m[38;5;2mINFO [0m ==> ** Redis setup finished! **
2023-02-03T13:37:04.566827863Z [38;5;6mredis [38;5;5m13:37:04.56 [0m[38;5;2mINFO [0m ==> ** Starting Redis **
2023-02-03T13:37:04.585027734Z 1:C 03 Feb 2023 13:37:04.580 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2023-02-03T13:37:04.585085944Z 1:C 03 Feb 2023 13:37:04.580 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
2023-02-03T13:37:04.585128302Z 1:C 03 Feb 2023 13:37:04.580 # Configuration loaded
2023-02-03T13:37:04.585151428Z 1:M 03 Feb 2023 13:37:04.581 * monotonic clock: POSIX clock_gettime
2023-02-03T13:37:04.585173759Z 1:M 03 Feb 2023 13:37:04.582 * Running mode=standalone, port=6379.
2023-02-03T13:37:04.585206322Z 1:M 03 Feb 2023 13:37:04.582 # Server initialized
2023-02-03T13:37:04.585228974Z 1:M 03 Feb 2023 13:37:04.583 * Reading RDB base file on AOF loading...
2023-02-03T13:37:04.585250233Z 1:M 03 Feb 2023 13:37:04.583 * Loading RDB produced by version 7.0.5
2023-02-03T13:37:04.585271059Z 1:M 03 Feb 2023 13:37:04.583 * RDB age 2712424 seconds
2023-02-03T13:37:04.585302370Z 1:M 03 Feb 2023 13:37:04.583 * RDB memory usage when created 1.71 Mb
2023-02-03T13:37:04.585325919Z 1:M 03 Feb 2023 13:37:04.583 * RDB is base AOF
2023-02-03T13:37:04.585346998Z 1:M 03 Feb 2023 13:37:04.583 * Done loading RDB, keys loaded: 3, keys expired: 0.
2023-02-03T13:37:04.585367850Z 1:M 03 Feb 2023 13:37:04.583 * DB loaded from base file appendonly.aof.5.base.rdb: 0.000 seconds
2023-02-03T13:37:04.700782827Z 1:M 03 Feb 2023 13:37:04.699 * DB loaded from incr file appendonly.aof.5.incr.aof: 0.117 seconds
2023-02-03T13:37:04.700854818Z 1:M 03 Feb 2023 13:37:04.699 * DB loaded from append only file: 0.117 seconds
2023-02-03T13:37:04.700875969Z 1:M 03 Feb 2023 13:37:04.700 * Opening AOF incr file appendonly.aof.5.incr.aof on server start
2023-02-03T13:37:04.700916292Z 1:M 03 Feb 2023 13:37:04.700 * Ready to accept connections
2023-02-03T13:38:05.041545420Z 1:M 03 Feb 2023 13:38:05.040 * 10000 changes in 60 seconds. Saving...
2023-02-03T13:38:05.043314065Z 1:M 03 Feb 2023 13:38:05.043 * Background saving started by pid 160
2023-02-03T13:38:05.083038706Z 160:C 03 Feb 2023 13:38:05.082 * DB saved on disk
2023-02-03T13:38:05.083513366Z 160:C 03 Feb 2023 13:38:05.083 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
2023-02-03T13:38:05.143821439Z 1:M 03 Feb 2023 13:38:05.143 * Background saving terminated with success

truecharts · Feb 3, 2023

impestrator said:

Hello,

after I upgraded to bluefin two weeks ago I had some issues with several apps. I could make them running again after disabling host validation and reconfiguring network settings, but paperless-ngx is still in the deployment loop. Maybe you have an idea how to fix it.

I´ve got the following logs, which does not help me...to be honest:

Code:

2023-02-03T13:37:05.375901177Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m
2023-02-03T13:37:05.377689708Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m[1mWelcome to the Bitnami postgresql container[0m
2023-02-03T13:37:05.379794524Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
2023-02-03T13:37:05.382002640Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
2023-02-03T13:37:05.384010481Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0m
2023-02-03T13:37:05.401347313Z [38;5;6mpostgresql [38;5;5m13:37:05.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL setup **
2023-02-03T13:37:05.428629817Z [38;5;6mpostgresql [38;5;5m13:37:05.42 [0m[38;5;2mINFO [0m ==> Validating settings in POSTGRESQL_* env vars..
2023-02-03T13:37:05.435553271Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Cleaning stale /bitnami/postgresql/data/postmaster.pid file
2023-02-03T13:37:05.439262138Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Loading custom pre-init scripts...
2023-02-03T13:37:05.443801228Z [38;5;6mpostgresql [38;5;5m13:37:05.44 [0m[38;5;2mINFO [0m ==> Initializing PostgreSQL database...
2023-02-03T13:37:05.461285848Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> pg_hba.conf file not detected. Generating it...
2023-02-03T13:37:05.463490028Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> Generating local authentication configuration
2023-02-03T13:37:07.280880781Z [38;5;6mpostgresql [38;5;5m13:37:07.28 [0m[38;5;2mINFO [0m ==> Deploying PostgreSQL with persisted data...
2023-02-03T13:37:07.292909728Z [38;5;6mpostgresql [38;5;5m13:37:07.29 [0m[38;5;2mINFO [0m ==> Configuring replication parameters
2023-02-03T13:37:07.326803311Z [38;5;6mpostgresql [38;5;5m13:37:07.32 [0m[38;5;2mINFO [0m ==> Configuring fsync
2023-02-03T13:37:07.335811278Z [38;5;6mpostgresql [38;5;5m13:37:07.33 [0m[38;5;2mINFO [0m ==> Configuring synchronous_replication
2023-02-03T13:37:07.356447148Z [38;5;6mpostgresql [38;5;5m13:37:07.35 [0m[38;5;2mINFO [0m ==> Loading custom scripts...
2023-02-03T13:37:07.361128640Z [38;5;6mpostgresql [38;5;5m13:37:07.36 [0m[38;5;2mINFO [0m ==> Enabling remote connections
2023-02-03T13:37:07.372642799Z [38;5;6mpostgresql [38;5;5m13:37:07.37 [0m[38;5;2mINFO [0m ==> ** PostgreSQL setup finished! **
2023-02-03T13:37:07.372718490Z
2023-02-03T13:37:07.398219936Z [38;5;6mpostgresql [38;5;5m13:37:07.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL **
2023-02-03T13:37:07.429331329Z 2023-02-03 13:37:07.429 GMT [1] LOG:  pgaudit extension initialized
2023-02-03T13:37:07.441928608Z 2023-02-03 13:37:07.441 GMT [1] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-02-03T13:37:07.442469695Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-02-03T13:37:07.442499074Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2023-02-03T13:37:07.456260299Z 2023-02-03 13:37:07.456 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-02-03T13:37:07.476475240Z 2023-02-03 13:37:07.476 GMT [87] LOG:  database system was interrupted; last known up at 2023-02-03 13:35:42 GMT
2023-02-03T13:37:07.532778664Z 2023-02-03 13:37:07.532 GMT [87] LOG:  database system was not properly shut down; automatic recovery in progress
2023-02-03T13:37:07.548605395Z 2023-02-03 13:37:07.548 GMT [87] LOG:  invalid record length at 0/239F0AC0: wanted 24, got 0
2023-02-03T13:37:07.548644004Z 2023-02-03 13:37:07.548 GMT [87] LOG:  redo is not required
2023-02-03T13:37:07.610310774Z 2023-02-03 13:37:07.610 GMT [1] LOG:  database system is ready to accept connections

Code:

2023-02-03T13:37:05.375901177Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m
2023-02-03T13:37:05.377689708Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0m[1mWelcome to the Bitnami postgresql container[0m
2023-02-03T13:37:05.379794524Z [38;5;6mpostgresql [38;5;5m13:37:05.37 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
2023-02-03T13:37:05.382002640Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
2023-02-03T13:37:05.384010481Z [38;5;6mpostgresql [38;5;5m13:37:05.38 [0m
2023-02-03T13:37:05.401347313Z [38;5;6mpostgresql [38;5;5m13:37:05.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL setup **
2023-02-03T13:37:05.428629817Z [38;5;6mpostgresql [38;5;5m13:37:05.42 [0m[38;5;2mINFO [0m ==> Validating settings in POSTGRESQL_* env vars..
2023-02-03T13:37:05.435553271Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Cleaning stale /bitnami/postgresql/data/postmaster.pid file
2023-02-03T13:37:05.439262138Z [38;5;6mpostgresql [38;5;5m13:37:05.43 [0m[38;5;2mINFO [0m ==> Loading custom pre-init scripts...
2023-02-03T13:37:05.443801228Z [38;5;6mpostgresql [38;5;5m13:37:05.44 [0m[38;5;2mINFO [0m ==> Initializing PostgreSQL database...
2023-02-03T13:37:05.461285848Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> pg_hba.conf file not detected. Generating it...
2023-02-03T13:37:05.463490028Z [38;5;6mpostgresql [38;5;5m13:37:05.46 [0m[38;5;2mINFO [0m ==> Generating local authentication configuration
2023-02-03T13:37:07.280880781Z [38;5;6mpostgresql [38;5;5m13:37:07.28 [0m[38;5;2mINFO [0m ==> Deploying PostgreSQL with persisted data...
2023-02-03T13:37:07.292909728Z [38;5;6mpostgresql [38;5;5m13:37:07.29 [0m[38;5;2mINFO [0m ==> Configuring replication parameters
2023-02-03T13:37:07.326803311Z [38;5;6mpostgresql [38;5;5m13:37:07.32 [0m[38;5;2mINFO [0m ==> Configuring fsync
2023-02-03T13:37:07.335811278Z [38;5;6mpostgresql [38;5;5m13:37:07.33 [0m[38;5;2mINFO [0m ==> Configuring synchronous_replication
2023-02-03T13:37:07.356447148Z [38;5;6mpostgresql [38;5;5m13:37:07.35 [0m[38;5;2mINFO [0m ==> Loading custom scripts...
2023-02-03T13:37:07.361128640Z [38;5;6mpostgresql [38;5;5m13:37:07.36 [0m[38;5;2mINFO [0m ==> Enabling remote connections
2023-02-03T13:37:07.372642799Z [38;5;6mpostgresql [38;5;5m13:37:07.37 [0m[38;5;2mINFO [0m ==> ** PostgreSQL setup finished! **
2023-02-03T13:37:07.372718490Z
2023-02-03T13:37:07.398219936Z [38;5;6mpostgresql [38;5;5m13:37:07.39 [0m[38;5;2mINFO [0m ==> ** Starting PostgreSQL **
2023-02-03T13:37:07.429331329Z 2023-02-03 13:37:07.429 GMT [1] LOG:  pgaudit extension initialized
2023-02-03T13:37:07.441928608Z 2023-02-03 13:37:07.441 GMT [1] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-02-03T13:37:07.442469695Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-02-03T13:37:07.442499074Z 2023-02-03 13:37:07.442 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2023-02-03T13:37:07.456260299Z 2023-02-03 13:37:07.456 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-02-03T13:37:07.476475240Z 2023-02-03 13:37:07.476 GMT [87] LOG:  database system was interrupted; last known up at 2023-02-03 13:35:42 GMT
2023-02-03T13:37:07.532778664Z 2023-02-03 13:37:07.532 GMT [87] LOG:  database system was not properly shut down; automatic recovery in progress
2023-02-03T13:37:07.548605395Z 2023-02-03 13:37:07.548 GMT [87] LOG:  invalid record length at 0/239F0AC0: wanted 24, got 0
2023-02-03T13:37:07.548644004Z 2023-02-03 13:37:07.548 GMT [87] LOG:  redo is not required
2023-02-03T13:37:07.610310774Z 2023-02-03 13:37:07.610 GMT [1] LOG:  database system is ready to accept connections

Code:

2023-02-03T13:37:03.079347632Z [38;5;6mredis [38;5;5m13:37:03.07 [0m
2023-02-03T13:37:03.080619237Z [38;5;6mredis [38;5;5m13:37:03.08 [0m[1mWelcome to the Bitnami redis container[0m
2023-02-03T13:37:03.081882544Z [38;5;6mredis [38;5;5m13:37:03.08 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
2023-02-03T13:37:03.083417482Z [38;5;6mredis [38;5;5m13:37:03.08 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
2023-02-03T13:37:03.084734015Z [38;5;6mredis [38;5;5m13:37:03.08 [0m
2023-02-03T13:37:03.086506965Z [38;5;6mredis [38;5;5m13:37:03.08 [0m[38;5;2mINFO [0m ==> ** Starting Redis setup **
2023-02-03T13:37:03.108488071Z [38;5;6mredis [38;5;5m13:37:03.10 [0m[38;5;3mWARN [0m ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
2023-02-03T13:37:03.112259963Z [38;5;6mredis [38;5;5m13:37:03.11 [0m[38;5;2mINFO [0m ==> Initializing Redis
2023-02-03T13:37:03.126320218Z [38;5;6mredis [38;5;5m13:37:03.12 [0m[38;5;2mINFO [0m ==> Setting Redis config file
2023-02-03T13:37:04.502526188Z [38;5;6mredis [38;5;5m13:37:04.50 [0m[38;5;2mINFO [0m ==> Configuring replication mode
2023-02-03T13:37:04.548437011Z
2023-02-03T13:37:04.548495502Z [38;5;6mredis [38;5;5m13:37:04.54 [0m[38;5;2mINFO [0m ==> ** Redis setup finished! **
2023-02-03T13:37:04.566827863Z [38;5;6mredis [38;5;5m13:37:04.56 [0m[38;5;2mINFO [0m ==> ** Starting Redis **
2023-02-03T13:37:04.585027734Z 1:C 03 Feb 2023 13:37:04.580 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2023-02-03T13:37:04.585085944Z 1:C 03 Feb 2023 13:37:04.580 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
2023-02-03T13:37:04.585128302Z 1:C 03 Feb 2023 13:37:04.580 # Configuration loaded
2023-02-03T13:37:04.585151428Z 1:M 03 Feb 2023 13:37:04.581 * monotonic clock: POSIX clock_gettime
2023-02-03T13:37:04.585173759Z 1:M 03 Feb 2023 13:37:04.582 * Running mode=standalone, port=6379.
2023-02-03T13:37:04.585206322Z 1:M 03 Feb 2023 13:37:04.582 # Server initialized
2023-02-03T13:37:04.585228974Z 1:M 03 Feb 2023 13:37:04.583 * Reading RDB base file on AOF loading...
2023-02-03T13:37:04.585250233Z 1:M 03 Feb 2023 13:37:04.583 * Loading RDB produced by version 7.0.5
2023-02-03T13:37:04.585271059Z 1:M 03 Feb 2023 13:37:04.583 * RDB age 2712424 seconds
2023-02-03T13:37:04.585302370Z 1:M 03 Feb 2023 13:37:04.583 * RDB memory usage when created 1.71 Mb
2023-02-03T13:37:04.585325919Z 1:M 03 Feb 2023 13:37:04.583 * RDB is base AOF
2023-02-03T13:37:04.585346998Z 1:M 03 Feb 2023 13:37:04.583 * Done loading RDB, keys loaded: 3, keys expired: 0.
2023-02-03T13:37:04.585367850Z 1:M 03 Feb 2023 13:37:04.583 * DB loaded from base file appendonly.aof.5.base.rdb: 0.000 seconds
2023-02-03T13:37:04.700782827Z 1:M 03 Feb 2023 13:37:04.699 * DB loaded from incr file appendonly.aof.5.incr.aof: 0.117 seconds
2023-02-03T13:37:04.700854818Z 1:M 03 Feb 2023 13:37:04.699 * DB loaded from append only file: 0.117 seconds
2023-02-03T13:37:04.700875969Z 1:M 03 Feb 2023 13:37:04.700 * Opening AOF incr file appendonly.aof.5.incr.aof on server start
2023-02-03T13:37:04.700916292Z 1:M 03 Feb 2023 13:37:04.700 * Ready to accept connections
2023-02-03T13:38:05.041545420Z 1:M 03 Feb 2023 13:38:05.040 * 10000 changes in 60 seconds. Saving...
2023-02-03T13:38:05.043314065Z 1:M 03 Feb 2023 13:38:05.043 * Background saving started by pid 160
2023-02-03T13:38:05.083038706Z 160:C 03 Feb 2023 13:38:05.082 * DB saved on disk
2023-02-03T13:38:05.083513366Z 160:C 03 Feb 2023 13:38:05.083 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
2023-02-03T13:38:05.143821439Z 1:M 03 Feb 2023 13:38:05.143 * Background saving terminated with success

Maybe actually contact the support staff of the project building your App...

browntiger · Feb 3, 2023

@impestrator, Your paperless-ngx postgres container looks good. No errors. Your are missing paperless-ngx pod btw.
To search for an error open shell cd /var/log/, find k3s.daemon.log; see (tail -100 ) if there is an error that relates or it may direct you to a log.
Check the log cd /var/log/pods/ix-XXXX,

But like @truecharts said (visit TrueCharts Discord https://discord.gg/tVsPTHWTtr) and give them a try.

impestrator · Feb 3, 2023

browntiger said:
@impestrator, Your paperless-ngx postgres container looks good. No errors. Your are missing paperless-ngx pod btw.
To search for an error open shell cd /var/log/, find k3s.daemon.log; see (tail -100 ) if there is an error that relates or it may direct you to a log.
Check the log cd /var/log/pods/ix-XXXX,

But like @truecharts said (visit TrueCharts Discord https://discord.gg/tVsPTHWTtr) and give them a try.

Thanks for the advices. I´ve checked k3s.daemon.log and searched for paperless. Maybe this interesting:

Code:

43039:Feb  3 22:20:47 truenas k3s[670340]: E0203 22:20:47.491007  670340 configmap.go:197] Couldn't get configMap ix-paperless/paperless-redis-health: failed to sync configmap cache: timed out waiting for the condition
43040:Feb  3 22:20:47 truenas k3s[670340]: E0203 22:20:47.491472  670340 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698faf9f-f37f-46b2-8a4a-7cefe73f0bc5-redis-health podName:698faf9f-f37f-46b2-8a4a-7cefe73f0bc5 nodeName:}" failed. No retries permitted until 2023-02-03 22:20:47.991088974 +0100 CET m=+94688.264817852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "redis-health" (UniqueName: "kubernetes.io/configmap/698faf9f-f37f-46b2-8a4a-7cefe73f0bc5-redis-health") pod "paperless-redis-0" (UID: "698faf9f-f37f-46b2-8a4a-7cefe73f0bc5") : failed to sync configmap cache: timed out waiting for the condition
43041:Feb  3 22:20:47 truenas k3s[670340]: I0203 22:20:47.495445  670340 request.go:682] Waited for 1.004513548s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6443/api/v1/namespaces/ix-paperless/configmaps?fieldSelector=metadata.name%3Dpaperless-redis-health&resourceVersion=16915577
43042:Feb  3 22:20:47 truenas k3s[670340]: time="2023-02-03T22:20:47+01:00" level=info msg="Will attempt to re-write config file /mnt/HDD/ix-applications/docker/containers/e43a0b53b7bef5e2343efb1a298351cc8e0445079cb85c950b14a80eb03da3fe/resolv.conf as [nameserver 172.17.0.10 search ix-paperless.svc.cluster.local svc.cluster.local cluster.local options ndots:1]"
43044:Feb  3 22:20:47 truenas k3s[670340]: I0203 22:20:47.926061  845696 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-paperless", Name:"paperless-postgresql-0", UID:"50ad9e81-d94a-4ad2-85e7-01e33f880ffe", APIVersion:"v1", ResourceVersion:"16916625", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.3.55/16] from ix-net
43046:Feb  3 22:20:48 truenas k3s[670340]: time="2023-02-03T22:20:48+01:00" level=info msg="Will attempt to re-write config file /mnt/HDD/ix-applications/docker/containers/db8f9f4521df507f7b859d5d6c927fc2b6c498e8faf9e82bc6c7a055b88f12cc/resolv.conf as [nameserver 172.17.0.10 search ix-paperless.svc.cluster.local svc.cluster.local cluster.local options ndots:1]"
43048:Feb  3 22:20:49 truenas k3s[670340]: I0203 22:20:49.030281  845844 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-paperless", Name:"paperless-redis-0", UID:"698faf9f-f37f-46b2-8a4a-7cefe73f0bc5", APIVersion:"v1", ResourceVersion:"16916635", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.3.56/16] from ix-net
43050:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.055102  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43051:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.107456  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43052:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.107702  670340 event.go:294] "Event occurred" object="ix-paperless/paperless-paperless-ng-manifests" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: paperless-paperless-ng-manifests-xcft6"
43053:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.121842  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43055:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.132076  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43056:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.135706  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43057:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.178225  670340 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cllv\" (UniqueName: \"kubernetes.io/projected/7dc87929-92e1-4bce-8de4-4f1dfc2b8968-kube-api-access-9cllv\") pod \"paperless-paperless-ng-manifests-xcft6\" (UID: \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\") " pod="ix-paperless/paperless-paperless-ng-manifests-xcft6"
43058:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.178316  670340 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"paperless-paperless-ng-manifests-temp\" (UniqueName: \"kubernetes.io/empty-dir/7dc87929-92e1-4bce-8de4-4f1dfc2b8968-paperless-paperless-ng-manifests-temp\") pod \"paperless-paperless-ng-manifests-xcft6\" (UID: \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\") " pod="ix-paperless/paperless-paperless-ng-manifests-xcft6"
43059:Feb  3 22:20:58 truenas k3s[670340]: time="2023-02-03T22:20:58+01:00" level=info msg="Will attempt to re-write config file /mnt/HDD/ix-applications/docker/containers/260338f86c2e2ce85025756911ff5d7f1ba317ead307d3c6e5772172fc393845/resolv.conf as [nameserver 172.17.0.10 search ix-paperless.svc.cluster.local svc.cluster.local cluster.local options ndots:5]"
43060:Feb  3 22:20:58 truenas k3s[670340]: I0203 22:20:58.276311  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43061:Feb  3 22:20:58 truenas k3s[670340]: I0203 22:20:58.263704  847840 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-paperless", Name:"paperless-paperless-ng-manifests-xcft6", UID:"7dc87929-92e1-4bce-8de4-4f1dfc2b8968", APIVersion:"v1", ResourceVersion:"16916686", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.3.57/16] from ix-net
43063:Feb  3 22:20:59 truenas k3s[670340]: I0203 22:20:59.282775  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43064:Feb  3 22:21:00 truenas k3s[670340]: I0203 22:21:00.298318  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43068:Feb  3 22:21:05 truenas k3s[670340]: I0203 22:21:05.382604  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43069:Feb  3 22:21:06 truenas k3s[670340]: I0203 22:21:06.392643  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43070:Feb  3 22:21:07 truenas k3s[670340]: I0203 22:21:07.257697  670340 reconciler.go:211] "operationExecutor.UnmountVolume started for volume \"paperless-paperless-ng-manifests-temp\" (UniqueName: \"kubernetes.io/empty-dir/7dc87929-92e1-4bce-8de4-4f1dfc2b8968-paperless-paperless-ng-manifests-temp\") pod \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\" (UID: \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\") "
43072:Feb  3 22:21:07 truenas k3s[670340]: W0203 22:21:07.258273  670340 empty_dir.go:523] Warning: Failed to clear quota on /var/lib/kubelet/pods/7dc87929-92e1-4bce-8de4-4f1dfc2b8968/volumes/kubernetes.io~empty-dir/paperless-paperless-ng-manifests-temp: clearQuota called, but quotas disabled

I´ve also checked /var/log/pods/ix-XXXX. I´ve got two paperless folders starting with ix-paperless, but both are empty.

impestrator · Feb 7, 2023

Ok, unfortunately the guys from TrueCharts Discord told me to raise a ticket for my issue. Before I do that I would ask you, if help me with that:

Code:

adding uncounted pods to status: Operation cannot be fulfilled on jobs.batch "paperless-paperless-ng-manifests": the object has been modified; please apply your changes to the latest version and try again

What changes do I have to apply and where?

I also checked the application events and found that:

Code:

error killing pod: failed to "KillContainer" for "paperless-redis" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: ac53a045ea79b143f95560064693f1aa91fea3e8b1137ce24af5e18916d08e74"

And I noticed a

Code:

Startup probe failed: dial tcp 172.16.7.167:8000: connect: connection refused

in the application event

edit:

Well, I was able to make the container running, but I´m not able to reach the web front end. No matter network config I choose. Any suggestions for best practise?

impestrator · Feb 18, 2023

impestrator said:

Thanks for the advices. I´ve checked k3s.daemon.log and searched for paperless. Maybe this interesting:

Code:

43039:Feb  3 22:20:47 truenas k3s[670340]: E0203 22:20:47.491007  670340 configmap.go:197] Couldn't get configMap ix-paperless/paperless-redis-health: failed to sync configmap cache: timed out waiting for the condition
43040:Feb  3 22:20:47 truenas k3s[670340]: E0203 22:20:47.491472  670340 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/698faf9f-f37f-46b2-8a4a-7cefe73f0bc5-redis-health podName:698faf9f-f37f-46b2-8a4a-7cefe73f0bc5 nodeName:}" failed. No retries permitted until 2023-02-03 22:20:47.991088974 +0100 CET m=+94688.264817852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "redis-health" (UniqueName: "kubernetes.io/configmap/698faf9f-f37f-46b2-8a4a-7cefe73f0bc5-redis-health") pod "paperless-redis-0" (UID: "698faf9f-f37f-46b2-8a4a-7cefe73f0bc5") : failed to sync configmap cache: timed out waiting for the condition
43041:Feb  3 22:20:47 truenas k3s[670340]: I0203 22:20:47.495445  670340 request.go:682] Waited for 1.004513548s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6443/api/v1/namespaces/ix-paperless/configmaps?fieldSelector=metadata.name%3Dpaperless-redis-health&resourceVersion=16915577
43042:Feb  3 22:20:47 truenas k3s[670340]: time="2023-02-03T22:20:47+01:00" level=info msg="Will attempt to re-write config file /mnt/HDD/ix-applications/docker/containers/e43a0b53b7bef5e2343efb1a298351cc8e0445079cb85c950b14a80eb03da3fe/resolv.conf as [nameserver 172.17.0.10 search ix-paperless.svc.cluster.local svc.cluster.local cluster.local options ndots:1]"
43044:Feb  3 22:20:47 truenas k3s[670340]: I0203 22:20:47.926061  845696 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-paperless", Name:"paperless-postgresql-0", UID:"50ad9e81-d94a-4ad2-85e7-01e33f880ffe", APIVersion:"v1", ResourceVersion:"16916625", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.3.55/16] from ix-net
43046:Feb  3 22:20:48 truenas k3s[670340]: time="2023-02-03T22:20:48+01:00" level=info msg="Will attempt to re-write config file /mnt/HDD/ix-applications/docker/containers/db8f9f4521df507f7b859d5d6c927fc2b6c498e8faf9e82bc6c7a055b88f12cc/resolv.conf as [nameserver 172.17.0.10 search ix-paperless.svc.cluster.local svc.cluster.local cluster.local options ndots:1]"
43048:Feb  3 22:20:49 truenas k3s[670340]: I0203 22:20:49.030281  845844 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-paperless", Name:"paperless-redis-0", UID:"698faf9f-f37f-46b2-8a4a-7cefe73f0bc5", APIVersion:"v1", ResourceVersion:"16916635", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.3.56/16] from ix-net
43050:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.055102  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43051:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.107456  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43052:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.107702  670340 event.go:294] "Event occurred" object="ix-paperless/paperless-paperless-ng-manifests" fieldPath="" kind="Job" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: paperless-paperless-ng-manifests-xcft6"
43053:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.121842  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43055:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.132076  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43056:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.135706  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43057:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.178225  670340 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cllv\" (UniqueName: \"kubernetes.io/projected/7dc87929-92e1-4bce-8de4-4f1dfc2b8968-kube-api-access-9cllv\") pod \"paperless-paperless-ng-manifests-xcft6\" (UID: \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\") " pod="ix-paperless/paperless-paperless-ng-manifests-xcft6"
43058:Feb  3 22:20:57 truenas k3s[670340]: I0203 22:20:57.178316  670340 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"paperless-paperless-ng-manifests-temp\" (UniqueName: \"kubernetes.io/empty-dir/7dc87929-92e1-4bce-8de4-4f1dfc2b8968-paperless-paperless-ng-manifests-temp\") pod \"paperless-paperless-ng-manifests-xcft6\" (UID: \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\") " pod="ix-paperless/paperless-paperless-ng-manifests-xcft6"
43059:Feb  3 22:20:58 truenas k3s[670340]: time="2023-02-03T22:20:58+01:00" level=info msg="Will attempt to re-write config file /mnt/HDD/ix-applications/docker/containers/260338f86c2e2ce85025756911ff5d7f1ba317ead307d3c6e5772172fc393845/resolv.conf as [nameserver 172.17.0.10 search ix-paperless.svc.cluster.local svc.cluster.local cluster.local options ndots:5]"
43060:Feb  3 22:20:58 truenas k3s[670340]: I0203 22:20:58.276311  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43061:Feb  3 22:20:58 truenas k3s[670340]: I0203 22:20:58.263704  847840 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ix-paperless", Name:"paperless-paperless-ng-manifests-xcft6", UID:"7dc87929-92e1-4bce-8de4-4f1dfc2b8968", APIVersion:"v1", ResourceVersion:"16916686", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [172.16.3.57/16] from ix-net
43063:Feb  3 22:20:59 truenas k3s[670340]: I0203 22:20:59.282775  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43064:Feb  3 22:21:00 truenas k3s[670340]: I0203 22:21:00.298318  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43068:Feb  3 22:21:05 truenas k3s[670340]: I0203 22:21:05.382604  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43069:Feb  3 22:21:06 truenas k3s[670340]: I0203 22:21:06.392643  670340 job_controller.go:510] enqueueing job ix-paperless/paperless-paperless-ng-manifests
43070:Feb  3 22:21:07 truenas k3s[670340]: I0203 22:21:07.257697  670340 reconciler.go:211] "operationExecutor.UnmountVolume started for volume \"paperless-paperless-ng-manifests-temp\" (UniqueName: \"kubernetes.io/empty-dir/7dc87929-92e1-4bce-8de4-4f1dfc2b8968-paperless-paperless-ng-manifests-temp\") pod \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\" (UID: \"7dc87929-92e1-4bce-8de4-4f1dfc2b8968\") "
43072:Feb  3 22:21:07 truenas k3s[670340]: W0203 22:21:07.258273  670340 empty_dir.go:523] Warning: Failed to clear quota on /var/lib/kubelet/pods/7dc87929-92e1-4bce-8de4-4f1dfc2b8968/volumes/kubernetes.io~empty-dir/paperless-paperless-ng-manifests-temp: clearQuota called, but quotas disabled

I´ve also checked /var/log/pods/ix-XXXX. I´ve got two paperless folders starting with ix-paperless, but both are empty.

I´ve created a support ticket over a week ago, but it seems that support team cannot help. The webUI of paperless is still not reachable and telnet to IP Port does not give a response. I get replies for ping request and paperless itself consumes files.

All other applications work, even when I create a new paperless instance. So I assume that there is somthing wrong with the webserver.

I really appriciate your assistance.

Important Announcement for the TrueNAS Community.

paperless-ngx issues after Bluefin update

Guru

Explorer

Guru

Explorer

Attachments

Guru

Explorer

Attachments

Guru

Explorer

Guru

Explorer

Guru

Explorer

Dabbler

Guru

Explorer

Dabbler

Dabbler

Dabbler

Similar threads