How to fix permanent error .system:<0x105>

[Craig White]

Hello,
I have a long running system that had the MB die last week. I brought all the drives (2x SSD boot mirror + 4x HDD raidz1 pool) to another system to get back up and running.
There were about 10 files listed as having permanent errors (in big pool, not boot) which I deleted and then did a scrub. Those file errors went away as expected.

However, "zpool status -v" now lists the following permanent error (which wasn't there originally) and I'm not sure how to proceed to clean this up.
I ran a second scrub with no change.

Any help is appreciated.

Code:

  pool: ocean
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 05:34:01 with 1 errors on Sun Jan 29 07:52:19 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        ocean                                           DEGRADED     0     0     0
          raidz1-0                                      DEGRADED     0     0     0
            gptid/60030a59-5dad-11eb-b51a-001cc0bf19b4  DEGRADED     0     0    10  too many errors
            gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4  DEGRADED     0     0     9  too many errors
            gptid/0a5f8dea-5ceb-11eb-9413-001cc0bf19b4  DEGRADED     0     0     9  too many errors
            gptid/ed155b22-5b96-11eb-8655-001cc0bf19b4  DEGRADED     0     0     9  too many errors

errors: Permanent errors have been detected in the following files:

        ocean/.system:<0x105>

Running TrueNAS-12.0-U2
Pool ocean is 4x 6TB WD Red drives (CMR)
New System: MB: gigabyte h370 hd3, Intel Core i5-9600k, 16GB non-ECC ram

 

[Arwen]

In general, ZFS won't allow corruption in a pool, even on motherboard failure. Nor on un-expected power loss. However, something else obviously corrupted your pool.

It appears that the corruption is in your system folder, used by the OS. And potentially in a snapshot. I don't remember all the details, but it may be possible to move your system folder to your boot-pool since you are using SSDs. Then, if the corruption is still present in the ocean pool, remove the system folder. Re-run scrub, if good, clear the error counts.

Let us wait for someone else to verify my suggestion. Their does not appear to be any hurry now, your ocean pool's data files seem to be fine now.

 

[Craig White]

Thanks for the initial response and info. Now I understand a little better what that .system entry is, which led me to some further reading.
I suppose the system dataset was just on ocean from before I had a boot pool. (It's been 15 yrs since this NAS was born, so there's been a couple mutations along the way).

I'll wait for any other gurus to confirm...

I see in the web gui the "Configure System Dataset" section. I suppose it's as simple as changing that from ocean to freenas-boot, and all the magic will happen to move the settings over to the other pool?
Any particular backups I should do first? (I've done the System/General/SaveConfig)
After, how would I remove the system folder from the ocean pool? I don't see it as a folder.

A couple other command line captures if this helps people... (ocean/.system and the freenas-boot pool status)

Code:

nas1% zfs list ocean/.system
NAME            USED  AVAIL     REFER  MOUNTPOINT
ocean/.system  1.61G  8.34T     1.53G  legacy

nas1% zpool status -v freenas-boot
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:11 with 0 errors on Sat Jan 28 03:45:11 2023
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            ada5p2    ONLINE       0     0     0
            ada4p2    ONLINE       0     0     0

errors: No known data errors