Help removing two permanent ZFS errors

[hungarianhc]

Hi there,

On my ZFS box, I'm getting the following errors when I do zpool status -v:

errors: Permanent errors have been detected in the following files:

<0x144>:<0x0>
Tank/.system/syslog-f1ae6c68bbe041c7bb38cadeec088781:<0x0>

For the first one, I'm not sure how to go about removing it. For the second, if I ssh into the box, I can't get into /mnt/Tank/.system ~ I'm not seeing that. Can I just delete the syslog file?

Thanks very much in advance for the help!

 

[SweetAndLow]

What does the rest of your zpool status output say? Did you have a disk failure or bit rot of some kind? You should be able to run a scrub and fix the issues unless you are doing something you shouldn't be doing.

 

[danb35]

The .system dataset is mounted on /var/db/system. I wouldn't recommend deleting the syslog-bignum directory, though--it has all the system log files in it.

 

[hungarianhc]

What does the rest of your zpool status output say?

[root@freenas ~]# zpool status -v Tank
pool: Tank
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 6h58m with 0 errors on Wed Aug 5 00:52:32 2015
config:

NAME STATE READ WRITE CKSUM
Tank ONLINE 0 0 0
gptid/fd55177b-78ef-11e4-ae8b-d050991929b1 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

<0x144>:<0x0>
Tank/.system/syslog-f1ae6c68bbe041c7bb38cadeec088781:<0x0>

Did you have a disk failure or bit rot of some kind? You should be able to run a scrub and fix the issues unless you are doing something you shouldn't be doing.

I haven't, to my knowledge, had any bit rot or disk failure. I have had one instance of what I would consider to be abnormal behavior... I was on FreeNAS 9.2. I upgraded to 9.3. My system wouldn't boot. Turns out it was something about being able to boot w/ GPT / MBR, and I got frustrated, and I downloaded Nas4Free. It worked. I started to set it up, didn't like it, and decided to go back to troubleshooting my FreeNAS boot issue. I discovered it was a BIOS problem. I upgraded my BIOS. I then flashed a new drive with 9.3, and it booted up again. I click the upgrade button to upgrade the ZFS flags. So now I'm here... So yeah.... Lots of weirdness, but I hope I'm in a steady state now. All of these files are backed up so I COULD wipe everything out and start clean, but I'd prefer to try and fix the issue.

I ran a zfs scrub, and it didn't appear to fix the issue. Now to be totally transparent, this installation is not running with ECC memory. Before I get my head bitten off for not using a recommended configuration, it's worth noting that my personal FreeNAS absolutely runs with 32GB of ECC RAM in a RaidZ2 configuration. I care about my data! The zpool in question here is at my parents' house. I have syncthing set up to back all of my media up to their house on a lower cost server, which is without ECC RAM, and then they run a plex server off of it for their house.

So that's the long story of how I got here. I'm totally fine with removing the data that is problematic. Would just prefer to not wipe everything out, as transferring 2TB over the internet via syncthing from scratch again will take some time...

Thanks again for the help!

 

[fta]

You should be able to run a scrub and fix the issues unless you are doing something you shouldn't be doing.

A scrub can't fix a permanent error.

 

[SirMaster]

Well it doesn't "fix" the error per say. But the scrub will clear the error from the log if you have first removed the corrupted file, all snapshots, and any other handles to it.

 

[hungarianhc]

The scrub didn't clear the error. Is that because I didn't have ECC RAM? In any case, I decided to wipe the disk and start from scratch. Syncthing is now copying about 3TB of data from San Francisco to Los Angeles... Haha it might take a month to finish!

 

[Robert Trevellyan]

The scrub didn't clear the error. Is that because I didn't have ECC RAM?

It's because a scrub can't fix a permanent error, as was already stated. As for what led to a permanent error, there's a chance that not having ECC RAM was a factor.