SOLVED Upgrade to 22.12.4 takes a long time and causes web GUI to freeze, then fails

[fayelund]

When I'm trying to upgrade to 22.12.4(from 22.12.3) the update takes a very log time, anbd the web gui stops working.
Im doing the upgrade via manual file, since online update does not work through my proxy.
The update experience is the same with all options for where to host the file(Memory, SSD storage, Nearline storage)

What I observe is:
- Messages in console of random processes being blocked for more than 120 seconds(prcesses oberverd blocked: nfsd, kworker, unsquasfs, asyncio_loop, rrdcached, txg_sync)
- Web UI responding from time to time, but often not at all
- Login windows being locked at the "set up TrueNAS authentication method", not contuing to the actual login
- extraction of the upgrade file is sluggish and goes in waves(2gb extraction takes 20-30 minutes)
- There is no high CPU or memory usage reported
- There is no errors detected on disks
- /var/log/error has a lot of this message: "collectd[9538]: plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics"

The upgrade fails with the folowing error:

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 428, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 463, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1378, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 428, in file
await self.middleware.call('update.install', job, destfile, update_options)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1395, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1355, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install_linux.py", line 32, in install
self.middleware.call_sync("update.install_scale", mounted, progress_callback, options)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in call_sync
return methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install.py", line 52, in install_scale
self._execute_truenas_install(mounted, command, progress_callback)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install.py", line 83, in _execute_truenas_install
raise CallError(result)
middlewared.service_exception.CallError: [EFAULT] Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/tmp/tmp49tv90gw/truenas_install/__main__.py", line 560, in <module>
main()
File "/tmp/tmp49tv90gw/truenas_install/__main__.py", line 379, in main
enable_system_user_services(root, old_root)
File "/tmp/tmp49tv90gw/truenas_install/__main__.py", line 124, in enable_system_user_services
configure_serial_port(root, database_path(old_root))
File "/tmp/tmp49tv90gw/truenas_install/__main__.py", line 112, in configure_serial_port
advanced = query_config_table("system_advanced", db_path, prefix="adv_")
File "/tmp/tmp49tv90gw/truenas_install/__main__.py", line 104, in query_config_table
return query_row(f"SELECT * FROM {table}", database_path, prefix)
File "/tmp/tmp49tv90gw/truenas_install/__main__.py", line 92, in query_row
c.execute(query)
sqlite3.OperationalError: database is locked

 

[fayelund]

Problem solved, one of the disks in the boot pool got slow IO errors after a few days. After taking that disk out everything worked well, and the udate went through.