Non-existent file with "Permanent errors" in Pool Preventing Resilver

[stuible]

Hi everyone,
I've been struggling with ZFS for the past couple days and could really use some guidance. It all started when one of the drives in my raidz2 pool started to fail SMART tests. I took this warning and replaced the drive. Resilvering completed but the pool remained "Degraded" citing one error. When I run zpool status -v it says that because there is one unrecoverable file it cannot complete the resilver. If I reboot it simply starts the resilver again but hangs at 100%. I have since disconnected the new drive so the drive is OFFLINE, trying to solve this issue. If I run zpool status -v the file in questions is:

Code:

errors: Permanent errors have been detected in the following files:

        /mnt/NAS/Media/TV/Sharp Objects/Season 1/Sharp.Objects.S01E02.1080p.BluRay.x264-CiNEFiLE.mkv

This file doesn't even exist in my pool anymore (it was renamed). I'm not positive but I believe this error was present before the drive failed and before I replaced it. I have since tried scrubbing the pool (which didn't help), and even putting the file back (which also didn't help). I've also searched this and many other forums but haven't found a solution that has worked. How can I get ZFS to stop caring about this supposedly corrupted non-existent file and complete the resilvering process?

Any help at all wold be greatly appreciated, going down to one disk redundancy is a little too spooky for me, even around Halloween.

FreeNAS version: FreeNAS-11.2-U6

 

[PhiloEpisteme]

Hi @stuible. I'm surprised the scrub didn't fix the issue. I doubt the issue is with your new disk though and you're probably better off putting the new disk back in and letting the resilver go. You could try zpool clear <pool as suggested by this forum post over at freebsd.org. I'm not 100% sure it will work; but it could solve the issue. I assume you have backups in the event something goes catastrophically wrong?

 

[stuible]

Hi @stuible. I'm surprised the scrub didn't fix the issue. I doubt the issue is with your new disk though and you're probably better off putting the new disk back in and letting the resilver go. You could try zpool clear <pool as suggested by this forum post over at freebsd.org. I'm not 100% sure it will work; but it could solve the issue. I assume you have backups in the event something goes catastrophically wrong?

Thank you for the response, I've actually tried that a few times to no avail. I am currently moving the data in the dataset with the "currupt" file to a new dataset so I can delete that entire dataset. Hopefully that will clear it up.

 

[stuible]

Thank you for the response, I've actually tried that a few times to no avail. I am currently moving the data in the dataset with the "currupt" file to a new dataset so I can delete that entire dataset. Hopefully that will clear it up.

This did not work :(

 

[garm]

Interesting issue, I’m not sure how ZFS deals with renaming but I don’t think it rewrites the whole file. Did you delete the dataset?

 

[stuible]

Interesting issue, I’m not sure how ZFS deals with renaming but I don’t think it rewrites the whole file. Did you delete the dataset?

So I think I've discovered the issue, and it's absolutely wild. I created a Dataset named TV2 and moved all my files from the TV Dataset there before deleting the TV Dataset. I then notice that there is STILL a folder called TV in my pool, except there are DIFFERENT files in it, including the file that ZFS was complaining about. It looks like the TV Dataset was covering up some hidden folder that had a corrupted file in it. I have now deleted the perviously hidden TV Folder and the corrupted file is finally being reported as "NAS/Media:<0x11eb>". I am now scrubbing the pool again and with any luck I've solved the issue and can continue with resilvering.

 

[garm]

ok, this happens when a dataset is mounted in an existing folder with content. At some point you created a folder called “tv” and after that you created the dataset “tv” and it got mounted in that folder. The content of the folder don’t disappear, the mounted dataset “hides” it.