Data corruption, help?

[Anda]

Hey,

I just got those mails from my server.

Subject: Cron <root@freenas> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 15 NAS
Content: starting first scrub (since reboot) of pool 'NAS'

And 12 minutes later this one.

Subject: freenas.local: Kritischer Alarm (Critical alarm)
Content: The volume NAS (ZFS) state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

I still have access to my data. And it seems that all operate normal.

And thats in the WebGUI: http://puu.sh/qj6um/e1e256a94a.png

What do I need to do now? Have really no clue..

 

[anodos]

Hey,

I just got those mails from my server.

Subject: Cron <root@freenas> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 15 NAS
Content: starting first scrub (since reboot) of pool 'NAS'

And 12 minutes later this one.

Subject: freenas.local: Kritischer Alarm (Critical alarm)
Content: The volume NAS (ZFS) state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

I still have access to my data. And it seems that all operate normal.

And thats in the WebGUI: http://puu.sh/qj6um/e1e256a94a.png

What do I need to do now? Have really no clue..

Type "zpool status" in the CLI and post the full output here enclosed in code tags.

Sounds like a great time to make sure your backups are in good order until you get further instruction from forum members.

 

[Anda]

Here. My backups are safe, luckily.

Code:

[root@freenas ~]# zpool status                                                                                                     
  pool: NAS                                                                                                                         
state: ONLINE                                                                                                                     
status: One or more devices has experienced an error resulting in data                                                             
        corruption.  Applications may be affected.                                                                                 
action: Restore the file in question if possible.  Otherwise restore the                                                           
        entire pool from backup.                                                                                                   
   see: http://illumos.org/msg/ZFS-8000-8A                                                                                         
  scan: scrub in progress since Sat Jul 30 00:00:02 2016                                                                           
        115G scanned out of 620G at 40.4M/s, 3h33m to go                                                                           
        0 repaired, 18.60% done                                                                                                     
config:                                                                                                                             
                                                                                                                                   
        NAME                                          STATE     READ WRITE CKSUM                                                   
        NAS                                           ONLINE      42     0     0                                                   
          gptid/63a3c656-4699-11e6-90d9-38eaa7ff0954  ONLINE       0     0     0                                                   
          gptid/646b214b-4699-11e6-90d9-38eaa7ff0954  ONLINE      42     0     0                                                   
        cache                                                                                                                       
          gptid/0a89f0b3-48f6-11e6-9ab9-38eaa7ff0954  ONLINE       0     0     0                                                   
                                                                                                                                   
errors: 1 data errors, use '-v' for a list                                                                                         
                                                                                                                                   
  pool: freenas-boot                                                                                                               
state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0h22m with 0 errors on Fri Jul 29 04:07:18 2016                                                         
config:                                                                                                                             
                                                                                                                                   
        NAME                                          STATE     READ WRITE CKSUM                                                   
        freenas-boot                                  ONLINE       0     0     0                                                   
          gptid/b8a9d7d7-8c6c-11e4-bf16-38eaa7ff0954  ONLINE       0     0     0                                                   
                                                                                                                                   
errors: No known data errors                                                                                                       
[root@freenas ~]# ^C                                                                                                               
[root@freenas ~]#  

 

[anodos]

Here. My backups are safe, luckily.

Code:

[root@freenas ~]# zpool status                                                                                                    
  pool: NAS                                                                                                                        
state: ONLINE                                                                                                                    
status: One or more devices has experienced an error resulting in data                                                            
        corruption.  Applications may be affected.                                                                                
action: Restore the file in question if possible.  Otherwise restore the                                                          
        entire pool from backup.                                                                                                  
   see: http://illumos.org/msg/ZFS-8000-8A                                                                                        
  scan: scrub in progress since Sat Jul 30 00:00:02 2016                                                                          
        115G scanned out of 620G at 40.4M/s, 3h33m to go                                                                          
        0 repaired, 18.60% done                                                                                                    
config:                                                                                                                            
                                                                                                                                  
        NAME                                          STATE     READ WRITE CKSUM                                                  
        NAS                                           ONLINE      42     0     0                                                  
          gptid/63a3c656-4699-11e6-90d9-38eaa7ff0954  ONLINE       0     0     0                                                  
          gptid/646b214b-4699-11e6-90d9-38eaa7ff0954  ONLINE      42     0     0                                                  
        cache                                                                                                                      
          gptid/0a89f0b3-48f6-11e6-9ab9-38eaa7ff0954  ONLINE       0     0     0                                                  
                                                                                                                                  
errors: 1 data errors, use '-v' for a list                                                                                        
                                                                                                                                  
  pool: freenas-boot                                                                                                              
state: ONLINE                                                                                                                    
  scan: scrub repaired 0 in 0h22m with 0 errors on Fri Jul 29 04:07:18 2016                                                        
config:                                                                                                                            
                                                                                                                                  
        NAME                                          STATE     READ WRITE CKSUM                                                  
        freenas-boot                                  ONLINE       0     0     0                                                  
          gptid/b8a9d7d7-8c6c-11e4-bf16-38eaa7ff0954  ONLINE       0     0     0                                                  
                                                                                                                                  
errors: No known data errors                                                                                                      
[root@freenas ~]# ^C                                                                                                              
[root@freenas ~]#  

Looks like you have three striped drives and one of them is ready to start pining for the fjords. Once it does, it will kill your entire pool. You should back up your data, destroy the pool, do further diagnostics on the problematic drive, and rebuild with known-good drives and some redundancy.

 

[maglin]

It looks like you might either have a stripped pool or a mirrored pool, and either of them with a cache drive. Why did you install a cache drive? Your scrub is still in progress when you did your zpool status. I also think it might not be the best idea to be doing anything as root right now. You run the risk of doing something you will regret. You can check your pools as a regular user. I see you can view your errors with a "zpool status -v" have you done that?

I'm not sure how hard it is to remove a cache drive but I would remove it. It's probably hurting your performance a LOT based on your hardware. I wouldn't try to change anything until your scrub has finished. Which is about 10PM ish CST by my calcs.

 

[anodos]

It looks like you might either have a stripped pool or a mirrored pool, and either of them with a cache drive. Why did you install a cache drive? Your scrub is still in progress when you did your zpool status. I also think it might not be the best idea to be doing anything as root right now. You run the risk of doing something you will regret. You can check your pools as a regular user. I see you can view your errors with a "zpool status -v" have you done that?

I'm not sure how hard it is to remove a cache drive but I would remove it. It's probably hurting your performance a LOT based on your hardware. I wouldn't try to change anything until your scrub has finished. Which is about 10PM ish CST by my calcs.

Hah. Don't know how I missed the cache drive. That's what I get for posting from a mobile phone.