FINAL EDIT - it was absolutely one of my hard drives slowly dying. The fun part was that it was passing smartctl / SMART tests. As soon as I pulled it from the array and started running in degraded, everything started working smoothly once again. Replaced and rebuilt the array last week and all is well. Everything below.... is immaterial now. :)
Edit update: Rolling back to RC1 manually to see if that resolves things.
Edit update 2: Nope. No difference. Damn.
Edit update 3: Yeah, the problem is very likely a dying hard drive in my array causing things to run really slowly, which throws everything off. I'll know more as soon as my replacement (and second spare) arrive.
Background -
Was running 11.2 RC1 (and previously the beta) with no issues at all. Overall setup seems pretty stable still after upgrading to RC2.
The problem is with my hosted VMs in Bhyve. I have 4 -
Windows 7 pro (runs all sorts of fun utilities like Radarr etc)
Ubuntu 18.04 server running pihole
Ubuntu 18.04 server running docker / hosted whatever / testing system
Ubuntu 18.04 server running plex
Prior to the upgrade, I was happy with the performance. Everything was great. :) Saw no issues reported at all around VMs in the first few days around the RC2 release, took the plunge... and now none of the VMs are booting fully.
This is the console output of one of the VMs at boot, and it's fairly typical of all 3 Ubuntu VMs.
Am currently trying to install a fresh VM (again ubuntu 18.04 since that's onhand and relatively current), and it's just hanging after booting the ISO.
SO! Any ideas where to start troubleshooting? I'm a bit out of my depth here. :)
Edit to add - output of tail /var/log/messages:
Nov 26 09:57:32 freenas uwsgi: line 544, in _open
'_open', req)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/site-packages/raven/utils/http.py", line 46, in https_open
return self.do_open(ValidHTTPSConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error timed out>
Nov 26 09:57:32 freenas uwsgi: [sentry.errors.uncaught:702] ['timeout: timed out', ' File "django/core/handlers/exception.py", line 42, in inner', ' File "django/core/handlers/base.py", line 244, in _legacy_get_response', ' File "freenasUI/freeadmin/middleware.py", line 296, in process_request', ' File "freenasUI/middleware/auth.py", line 8, in authenticate', ' File "freenasUI/middleware/client.py", line 20, in __enter__', ' File "middlewared/client/client.py", line 320, in __init__', ' File "middlewared/client/client.py", line 313, in __init__', ' File "middlewared/client/client.py", line 170, in connect', ' File "ws4py/client/__init__.py", line 215, in connect']
Edit update: Rolling back to RC1 manually to see if that resolves things.
Edit update 2: Nope. No difference. Damn.
Edit update 3: Yeah, the problem is very likely a dying hard drive in my array causing things to run really slowly, which throws everything off. I'll know more as soon as my replacement (and second spare) arrive.
Background -
Was running 11.2 RC1 (and previously the beta) with no issues at all. Overall setup seems pretty stable still after upgrading to RC2.
The problem is with my hosted VMs in Bhyve. I have 4 -
Windows 7 pro (runs all sorts of fun utilities like Radarr etc)
Ubuntu 18.04 server running pihole
Ubuntu 18.04 server running docker / hosted whatever / testing system
Ubuntu 18.04 server running plex
Prior to the upgrade, I was happy with the performance. Everything was great. :) Saw no issues reported at all around VMs in the first few days around the RC2 release, took the plunge... and now none of the VMs are booting fully.
This is the console output of one of the VMs at boot, and it's fairly typical of all 3 Ubuntu VMs.
Am currently trying to install a fresh VM (again ubuntu 18.04 since that's onhand and relatively current), and it's just hanging after booting the ISO.
SO! Any ideas where to start troubleshooting? I'm a bit out of my depth here. :)
Edit to add - output of tail /var/log/messages:
Nov 26 09:57:32 freenas uwsgi: line 544, in _open
'_open', req)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/site-packages/raven/utils/http.py", line 46, in https_open
return self.do_open(ValidHTTPSConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error timed out>
Nov 26 09:57:32 freenas uwsgi: [sentry.errors.uncaught:702] ['timeout: timed out', ' File "django/core/handlers/exception.py", line 42, in inner', ' File "django/core/handlers/base.py", line 244, in _legacy_get_response', ' File "freenasUI/freeadmin/middleware.py", line 296, in process_request', ' File "freenasUI/middleware/auth.py", line 8, in authenticate', ' File "freenasUI/middleware/client.py", line 20, in __enter__', ' File "middlewared/client/client.py", line 320, in __init__', ' File "middlewared/client/client.py", line 313, in __init__', ' File "middlewared/client/client.py", line 170, in connect', ' File "ws4py/client/__init__.py", line 215, in connect']
Last edited: