SOLVED pycache aka __pycache__ corrupting on boot drive

DarkSideMilk

Dabbler
Joined
Feb 22, 2019
Messages
13
So I keep getting this error...

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ArrowBck-Pool 21.8T 7.95T 13.8T - - 21% 36% 1.00x ONLINE /mnt
freenas-boot 111G 11.2G 99.8G - - - 10% 1.00x ONLINE -

pool: freenas-boot
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.

I run zpool status -v and it's always the same folder/file saying is corrupted

/usr/local/lib/python3.6/site-packages/__pycache__

Seeing that it is just a cache I figure, well I'll just delete that folder and let the cache get recreated.
So I do. But The next morning the same error is in my email again, so I run zpool status -v again and find that the cache is recreated and corrupt again.

I did recently update to the latest version of freenas, this started happening almost 2 weeks ago now. 2 of the data drives in the zfs pool failed (one at a time, they were both rather old) around the same time. I replaced them and the updated to freeNas 11.2 after the second replacement finished resilvering hoping that would fix the cache corruption problem but no dice.

I ran a smart test on the boot drive and it passed. There is one property in the smart output that says pre-fail, so it may be that the boot drive is just failing, but elsewhere in the forum I'm seeing that there's usually a larger number of failed files. So I'm wondering if maybe there's something else that could be causing this cache to be corrupted that I could fix by something other than reinstalling freenas on a new boot drive. That is my next step and I have already exported my config, but want to see if anyone has any different insights.

Please and thank you =)

Just for gits and shiggles here's the smartctl output of the boot drive

1550859947366.png

1550859981445.png

1550860025363.png

1550860048496.png

1550860078391.png
 

DarkSideMilk

Dabbler
Joined
Feb 22, 2019
Messages
13
I also don't know that I trust the smartctl readings. I did a read on one I just replaced and it has a couple pre-fail attributes already. Unless that's a sign of the boot pool drive indeed dying.
 

DarkSideMilk

Dabbler
Joined
Feb 22, 2019
Messages
13
I gave re-installing FreeNas 11.2 U1 a go on a new ssd. Backed up the config from system-general beforehand and uploaded it after the install completed.
So far so good, but the error usually shows up overnight. I'll report back tomorrow.
Sadly if it is fixed it won't truly answer if the problem was with the drive or just a general installation issue.

p.s.
FreeNas info
Super micro server mobo
8GB DDR 2 RAM
Intel(R) Xeon(R) CPU X3220 @ 2.40GHz (4 cores)
FreeNAS-11.2-RELEASE-U1
3ware array with 12 2TB drives (all formatted at 1.5 TB because they all replaced what was initially 1.5 TB drives)
 

DarkSideMilk

Dabbler
Joined
Feb 22, 2019
Messages
13
After reinstalling on a new drive the error has gone away. Not 100% sure still if I just needed to reinstall or if the drive was in fact dying. Seems a bit odd that the SSD would die after only 2 years (just shy of 25 months based on the power on hours). Maybe that sandisk ssd was just cheap. We put in a samsung 860 Pro which is supposed to have a higher fault tolerance/longer life. I guess we will see.
 
Top