Lock Order Reversals Bringing Down System?

taig-er

Cadet
Joined
Jul 7, 2020
Messages
4
Hey Everyone!

Current Build - AMD Ryzen 5 2400G on a GA-AB350M-DS3H MoBo running FreeNAS 11.3-U3.2 with an Intel NIC. 8 gb memory. 4 jails installed, running Plex w/Plexpass, qBittorrent, radarr, and sonarr.

I've been having some issues - my machine goes down, and is totally unreachable. No SSHing in, if I put a monitor on it, I can't do anything with the keyboard. Seems like a kernel panic, but I'm having some serious problems troubleshooting. Thus far, I've gotten this picture of a Lock Order Reversal
(which I've never heard of, and it seems like there's limited information on). I've tried my best to make my logs persistent, but I can't seem to save the /var/log/messages. I was able to pull the below off of the Debug file.

Any advice is appreciated. If anybody has another forum they recommend posting on, that's appreciated too, as well as any advice on how to troubleshoot better or provide better information. Hope everyone enjoys their day!


Code:
Jul  7 00:30:00 freenas /middlewared: [EtcService:275] No new changes for /etc/krb5.conf
Jul  7 00:30:00 freenas /middlewared: [EtcService:275] No new changes for /etc/krb5.conf
Jul  7 01:14:00 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update-master.ixsystems.com:443
Jul  7 01:14:00 freenas /middlewared: [urllib3.connectionpool:437] https://update-master.ixsystems.com:443 "GET /FreeNAS/trains_redir.json HTTP/1.1" 200 235
Jul  7 01:14:00 freenas /middlewared: [freenasOS.Configuration:606] TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/trains.txt', 'https://update-master.ixsystems.com/FreeNAS/trains.txt'])
Jul  7 01:14:00 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update.ixsystems.com:443
Jul  7 01:14:00 freenas /middlewared: [urllib3.connectionpool:437] https://update.ixsystems.com:443 "GET /FreeNAS/trains.txt HTTP/1.1" 301 169
Jul  7 01:14:00 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update.freenas.org:443
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:437] https://update.freenas.org:443 "GET /FreeNAS/trains.txt HTTP/1.1" 200 473
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:745] TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/trains.txt', 'https://update-master.ixsystems.com/FreeNAS/trains.txt']):  Read 473 bytes total
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:606] TryGetNetworkFile(['https://update-master.ixsystems.com/FreeNAS/FreeNAS-11.3-STABLE/LATEST'])
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update-master.ixsystems.com:443
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:437] https://update-master.ixsystems.com:443 "GET /FreeNAS/FreeNAS-11.3-STABLE/LATEST HTTP/1.1" 200 1967
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:745] TryGetNetworkFile(['https://update-master.ixsystems.com/FreeNAS/FreeNAS-11.3-STABLE/LATEST']):  Read 1967 bytes total
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:606] TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem'])
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update-master.ixsystems.com:443
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:437] https://update-master.ixsystems.com:443 "GET /updates/ix_crl.pem HTTP/1.1" 200 1028
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:77] CheckFreeSpace(path=/tmp/tmpfzjl7aaz.pem, pool=None, required=1028)
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:745] TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem']):  Read 1028 bytes total
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Update:1014] Going to try checking cached manifest /var/db/system/update/MANIFEST
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Update:1038] Got this exception: [Errno 2] No such file or directory: '/var/db/system/update/MANIFEST'
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:606] TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt', 'https://update-master.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt'])
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update.ixsystems.com:443
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:437] https://update.ixsystems.com:443 "GET /FreeNAS/Validators/ValidateUpdate-0frmasef.txt HTTP/1.1" 301 169
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:959] Starting new HTTPS connection (1): update.freenas.org:443
Jul  7 01:14:01 freenas /middlewared: [urllib3.connectionpool:437] https://update.freenas.org:443 "GET /FreeNAS/Validators/ValidateUpdate-0frmasef.txt HTTP/1.1" 200 185
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:77] CheckFreeSpace(path=/var/db/system/update/ValidateUpdate, pool=None, required=185)
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Configuration:745] TryGetNetworkFile(['https://update.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt', 'https://update-master.ixsystems.com/FreeNAS/Validators/ValidateUpdate-0frmasef.txt']):  Read 185 bytes total
Jul  7 01:14:01 freenas /middlewared: [freenasOS.Update:1071] DownloadUpdate:  No update available
Jul  7 03:45:00 freenas /middlewared: [middlewared.plugins.pool:3532] Pool 'freenas-boot' last scrub datetime.datetime(2020, 7, 6, 3, 46, 7)
Jul  7 04:15:02 freenas /autosnap.py: [urllib3.connectionpool:959] Starting new HTTPS connection (1): 127.0.0.1:443
Jul  7 04:15:02 freenas /autosnap.py: [urllib3.connectionpool:437] https://127.0.0.1:443 "GET /api/v1.0/storage/task/ HTTP/1.1" 200 None
Jul  7 04:15:02 freenas /autosnap.py: [urllib3.connectionpool:959] Starting new HTTPS connection (1): 127.0.0.1:443
Jul  7 04:15:02 freenas /autosnap.py: [urllib3.connectionpool:437] https://127.0.0.1:443 "GET /api/v1.0/storage/replication/ HTTP/1.1" 200 None
Jul  7 05:30:00 freenas /middlewared: [EtcService:275] No new changes for /etc/krb5.conf
Jul  7 05:30:00 freenas /middlewared: [EtcService:275] No new changes for /etc/krb5.conf
Jul  7 10:30:00 freenas /middlewared: [EtcService:275] No new changes for /etc/krb5.conf
Jul  7 10:30:00 freenas /middlewared: [EtcService:275] No new changes for /etc/krb5.conf
Jul  7 10:43:05 freenas /middlewared: [UsageService:45] Scheduled next run in 52249 seconds
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Have you disabled Cool 'n Quiet and/or C-state 6? Those are the biggest culprits affecting Ryzen stability on FreeNAS.
 

taig-er

Cadet
Joined
Jul 7, 2020
Messages
4
To anyone who finds this later with similar issues, it appears that disabling Cool n' Quiet / C-State worked. I had to really dig around in the BIOS settings to find it- it actually took me searching the owner's manual online, but I finally found it. Unrelated, but I previously also upgraded to an Intel NIC which seems to also cause issues.

Additionally, I also got this awesome explanation of a Lock Order Reversal, courtesy of /u/dnebdal over at r/FreeNAS:

Just to explain what a Lock Order Reversal message means in FreeBSD: The report you see is from witness(4), which is an optional kernel component. By default, it's compiled into the under-development branches but omitted from release versions, I don't know the TrueNAS policy on it. What it does is look for cases where something grabs two locks in the wrong order: Say you need to grab input and output from a device separately to work with it, and you have two different kernel threads that both want that. If they don't grab the locks in the same order, you can end up with a deadlock where they have one lock each and sit there waiting for each other.
I don't know if that exact case happens, but it's at least the kind of problem that witness tries to detect. (See also the FAQ). It is just a warning, though: The intention is for the kernel to be written defensively enough that this kind of deadlock eventually resolves itself (e.g. through giving up and waiting a random amount of time before trying again). Besides, witness is known to have false positives - and that's probably why it's disabled in the release kernels. It's mostly a "somebody should probably take a look at this" tool for the kernel developers.
In your case, this means that the LOR is probably not related to your problems.

Thanks everyone for your help!
 
Top