SOLVED Multiple sync problems at the same time

[ieronymous]

Hi

I have some days now that I was informed via email that one of the boot drives is going bad due to one of the mirrored ssd drives had some write errors (11 in number). Scrub in boot-pool didn t fix anything and after a restart or two the boot-pool wasn t at a degrade state.
That was not my main problem though, since in parallel with the drive problem I also had the following errors / warnings ...

Under Alerts I have
<< Failed to sync OFFICIAL catalog: [EFAULT] Failed to clone 'https://github.com/truenas/charts.git' repository at '/mnt/creativepool/ix-applications/catalogs/github_com_truenas_charts_git_master' destination: Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/truenas/charts.git /mnt/creativepool/ix-applications/catalogs/github_com_truenas_charts_git_master stderr: 'Cloning into '/mnt/creativepool/ix-applications/catalogs/github_com_truenas_charts_git_master'... fatal: unable to access 'https://github.com/truenas/charts.git/': server certificate verification failed. CAfile: none CRLfile: none '
07/08/2022 16:25:51 >>

When I try to update by selecting Apply Pending Updates I have that cert error below

<< ClientConnectorCertificateError
TrueNAS was unable to reach update servers.

Error: Cannot connect to host update.ixsystems.com:443 ssl:True
[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: certificate is not yet valid (_ssl.c:1123)')] >>

In current train it shows nothing. If i can recall I was on TrueNAS-SCALE-Bluefin 12.2
Check for updates button is disabled

On dashport -> Network I have the messsage: The reporting database is broken
and the button FIX DATABASE which gives me this message by pressing it
<< RRD file interface-enp2s0/if_octets.rrd has update time in the future. Data collection will be paused for 6 days, 19 hours and 15 minutes.
You can clear reporting database and start data collection immediately. >>


Any thoughts?

PS Permissions are as follows at /mnt/creativepool/ix-applications
drwxr-xr-x 8 root root 10 Feb 24 2023 ix-applications

Also noticed that on directory /..../ix-applications/catalogs there are no files
github_com_truenas_charts_git_master and github_com_truenas_charts_catalog_main

 

[sretalla]

Any thoughts?

Those SSL verification errors are often related to time sync being out by too much.

Although with some potential corruption on your boot pool, I'd be more inclined to think it was this message that's pointing to a missing or corrupt file in the boot pool:

server certificate verification failed. CAfile: none CRLfile: none '

Based on that, I'd be taking a backup of the config, installing fresh to a new drive and then restore the config.

 

[ieronymous]

Those SSL verification errors are often related to time sync being out by too much.

Although with some potential corruption on your boot pool, I'd be more inclined to think it was this message that's pointing to a missing or corrupt file in the boot pool:


Based on that, I'd be taking a backup of the config, installing fresh to a new drive and then restore the config.

At system settings->Localization I can t see a problem there. It could be though, if it was syncing from bios (something to the onboard weird battery type makes it loose time after system being out of power source even for a while. But it syncs on net so I don t believe this has something to do).

If there was corruption scrub should have reported it. But no errors found or attempted for correction.

So is that so catastrophic that I can t save it without re setting everything ?

PS Maybe manual update?

 

[sretalla]

I don't know that it's worth the time to recover it another way.

You could continue digging into what's preventing that certificate from validating.

 

[sretalla]

Looking at it again.. more evidence that it's a clock problem:

<< RRD file interface-enp2s0/if_octets.rrd has update time in the future.

 

[ieronymous]

Looking at it again.. more evidence that it's a clock problem:

.... and a clock problem it was. After shutting down the system, went to bios and fixed time (was off by some years hahah) and after restarting the server, everything was ok. Also the folders github_com_truenas_charts_git_master and github_com_truenas_charts_catalog_main
have been created on /..../ix-applications/catalogs directory.

Still seems weird why this happened since the whole system syncs time with time servers and not locally.

 

[sretalla]

Usually a problem with the BIOS battery involved somewhere in the story when you're out by that much.