SOLVED Disk Going Bad?

[nello]

My security logs seem to show problems with one of my disks:

Code:

OctoberGroupNAS.local kernel log messages:
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 d8 1d 79 40 67 00 00 01 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 e0 c8 85 40 38 01 00 01 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 c0 e0 d2 30 40 39 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 e8 07 fa 40 39 01 00 01 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 28 35 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 68 35 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 a8 35 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 e8 35 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 28 36 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 68 36 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 a8 36 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 e8 36 ba 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 e8 f0 b9 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
> (ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 68 f1 b9 40 3a 01 00 00 00 00
> (ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
> (ada2:ahcich2:0:0:0): Retrying command
…

I've had messages like this in logs for the past couple of weeks, but not every day.

I presume this means that the disk is going bad.

At what point should I call it quits and replace it?

- nello

 

[DrKK]

Looks like the retries might be succeeding.

The seriousness of this could range from "minor" to "critical"

Can I see the output, please, of

Code:

smartctl -qnoserial -x /dev/ada2

 

[nello]

@DrKK

Code:

[root@OctoberGroupNAS ~]# smartctl -qnoserial -x /dev/ada2                    
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)  
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org  
                                                                             
=== START OF INFORMATION SECTION ===                                          
Model Family:     Western Digital Red                                        
Device Model:     WDC WD30EFRX-68AX9N0                                        
Firmware Version: 80.00A80                                                    
User Capacity:    3,000,592,982,016 bytes [3.00 TB]                          
Sector Sizes:     512 bytes logical, 4096 bytes physical                      
Device is:        In smartctl database [for details use: -P show]            
ATA Version is:   ACS-2 (minor revision not indicated)                        
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                      
Local Time is:    Sun Jan  3 11:54:25 2016 CST                                
SMART support is: Available - device has SMART capability.                    
SMART support is: Enabled                                                    
AAM feature is:   Unavailable                                                
APM feature is:   Unavailable                                                
Rd look-ahead is: Enabled                                                    
Write cache is:   Enabled                                                    
ATA Security is:  Disabled, frozen [SEC2]                                    
Wt Cache Reorder: Enabled                                                    
                                                                             
=== START OF READ SMART DATA SECTION ===                                      
SMART overall-health self-assessment test result: PASSED                      
                                                                             
General SMART Values:                                                        
Offline data collection status:  (0x00) Offline data collection activity      
                                        was never started.                    
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.                            
Total time to complete Offline                                                
data collection:                (39120) seconds.                              
Offline data collection                                                      
capabilities:                    (0x7b) SMART execute Offline immediate.      
                                        Auto Offline data collection on/off supp
ort.                                                                          
                                        Suspend Offline collection upon new  
                                        command.                              
                                        Offline surface scan supported.      
                                        Self-test supported.                  
                                        Conveyance Self-test supported.      
                                        Selective Self-test supported.        
SMART capabilities:            (0x0003) Saves SMART data before entering      
                                        power-saving mode.                    
                                        Supports SMART auto save timer.      
Error logging capability:        (0x01) Error logging supported.              
                                        General Purpose Logging supported.    
Short self-test routine                                                      
recommended polling time:        (   2) minutes.                              
Extended self-test routine                                                    
recommended polling time:        ( 393) minutes.                              
Conveyance self-test routine                                                  
recommended polling time:        (   5) minutes.                              
SCT capabilities:              (0x70bd) SCT Status supported.                
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.        
                                        SCT Data Table supported.            
                                                                             
SMART Attributes Data Structure revision number: 16                          
Vendor Specific SMART Attributes with Thresholds:                            
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE        
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0                
  3 Spin_Up_Time            POS--K   177   176   021    -    6150            
  4 Start_Stop_Count        -O--CK   095   095   000    -    5887            
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0                
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0                
  9 Power_On_Hours          -O--CK   068   068   000    -    23499            
10 Spin_Retry_Count        -O--CK   100   100   000    -    0                
11 Calibration_Retry_Count -O--CK   100   253   000    -    0                
12 Power_Cycle_Count       -O--CK   100   100   000    -    51              
192 Power-Off_Retract_Count -O--CK   200   200   000    -    28              
193 Load_Cycle_Count        -O--CK   199   199   000    -    5858            
194 Temperature_Celsius     -O---K   123   111   000    -    27              
196 Reallocated_Event_Count -O--CK   200   200   000    -    0                
197 Current_Pending_Sector  -O--CK   200   200   000    -    0                
198 Offline_Uncorrectable   ----CK   100   253   000    -    0                
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0                
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0                
                            ||||||_ K auto-keep                              
                            |||||__ C event count                            
                            ||||___ R error rate                              
                            |||____ S speed/performance                      
                            ||_____ O updated online                          
                            |______ P prefailure warning                      
                                                                             
General Purpose Log Directory Version 1                                      
SMART           Log Directory Version 1 [multi-sector log support]            
Address    Access  R/W   Size  Description                                    
0x00       GPL,SL  R/O      1  Log Directory                                  
0x01           SL  R/O      1  Summary SMART error log                        
0x02           SL  R/O      5  Comprehensive SMART error log                  
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log            
0x06           SL  R/O      1  SMART self-test log                            
0x07       GPL     R/O      1  Extended self-test log                        
0x09           SL  R/W      1  Selective self-test log                        
0x10       GPL     R/O      1  NCQ Command Error log                          
0x11       GPL     R/O      1  SATA Phy Event Counters                        
0x21       GPL     R/O      1  Write stream error log                        
0x22       GPL     R/O      1  Read stream error log                          
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log                      
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log                    
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log                    
0xbd       GPL,SL  VS       1  Device vendor specific log                    
0xc0       GPL,SL  VS       1  Device vendor specific log                    
0xc1       GPL     VS      93  Device vendor specific log                    
0xe0       GPL,SL  R/W      1  SCT Command/Status                            
0xe1       GPL,SL  R/W      1  SCT Data Transfer                              
                                                                             
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)                
No Errors Logged                                                              
                                                                             
SMART Extended Self-test Log Version: 1 (1 sectors)                          
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                                              
# 1  Extended offline    Completed without error       00%     23497         -
# 2  Short offline       Completed without error       00%     23488         -
# 3  Short offline       Completed without error       00%     23466         -
# 4  Short offline       Completed without error       00%     23464         -
# 5  Short offline       Interrupted (host reset)      10%     23453         -
# 6  Short offline       Completed without error       00%     23418         -
# 7  Short offline       Completed without error       00%     23416         -
# 8  Short offline       Completed without error       00%     23394         -
# 9  Short offline       Completed without error       00%     23392         -
#10  Short offline       Completed without error       00%     23370         -
#11  Short offline       Completed without error       00%     23368         -
#12  Short offline       Completed without error       00%     23353         -
#13  Extended offline    Completed without error       00%     23338         -
#14  Short offline       Completed without error       00%     23320         -
#15  Short offline       Completed without error       00%     23298         -
#16  Short offline       Completed without error       00%     23296         -
#17  Short offline       Completed without error       00%     23274         -
#18  Short offline       Completed without error       00%     23272         -
                                                                             
SMART Selective self-test log data structure revision number 1                
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                  
    1        0        0  Not_testing                                          
    2        0        0  Not_testing                                          
    3        0        0  Not_testing                                          
    4        0        0  Not_testing                                          
    5        0        0  Not_testing                                          
Selective self-test flags (0x0):                                              
  After scanning selected spans, do NOT read-scan remainder of disk.          
If Selective self-test is pending on power-up, resume after 0 minute delay.  
                                                                             
SCT Status Version:                  3                                        
SCT Version (vendor specific):       258 (0x0102)                            
SCT Support Level:                   1                                        
Device State:                        Active (0)                              
Current Temperature:                    27 Celsius                            
Power Cycle Min/Max Temperature:     25/31 Celsius                            
Lifetime    Min/Max Temperature:     12/39 Celsius                            
Under/Over Temperature Limit Count:   0/0                                    
                                                                             
SCT Temperature History Version:     2                                        
Temperature Sampling Period:         1 minute                                
Temperature Logging Interval:        1 minute                                
Min/Max recommended Temperature:      0/60 Celsius                            
Min/Max Temperature Limit:           -41/85 Celsius                          
Temperature History Size (Index):    478 (395)                                
                                                                             
Index    Estimated Time   Temperature Celsius                                
396    2016-01-03 03:58    28  *********                                    
...    ..( 53 skipped).    ..  *********                                    
450    2016-01-03 04:52    28  *********                                    
451    2016-01-03 04:53    27  ********                                      
...    ..( 37 skipped).    ..  ********                                      
  11    2016-01-03 05:31    27  ********                                      
  12    2016-01-03 05:32    28  *********                                    
...    ..(382 skipped).    ..  *********                                    
395    2016-01-03 11:55    28  *********                                    
                                                                             
SCT Error Recovery Control:                                                  
           Read:     70 (7.0 seconds)                                        
          Write:     70 (7.0 seconds)                                        
                                                                             
Device Statistics (GP Log 0x04) not supported                                
                                                                             
SATA Phy Event Counters (GP Log 0x11)                                        
ID      Size     Value  Description                                          
0x0001  2            0  Command failed due to ICRC error                      
0x0002  2          101  R_ERR response for data FIS                          
0x0003  2            0  R_ERR response for device-to-host data FIS            
SCT Error Recovery Control:                                                  
           Read:     70 (7.0 seconds)                                        
          Write:     70 (7.0 seconds)                                        
                                                                             
Device Statistics (GP Log 0x04) not supported                                
                                                                             
SATA Phy Event Counters (GP Log 0x11)                                        
ID      Size     Value  Description                                          
0x0001  2            0  Command failed due to ICRC error                      
0x0002  2          101  R_ERR response for data FIS                          
0x0003  2            0  R_ERR response for device-to-host data FIS            
0x0004  2          101  R_ERR response for host-to-device data FIS            
0x0005  2            0  R_ERR response for non-data FIS                      
0x0006  2            0  R_ERR response for device-to-host non-data FIS        
0x0007  2            0  R_ERR response for host-to-device non-data FIS        
0x0008  2            0  Device-to-host non-data FIS retries                  
0x0009  2           91  Transition from drive PhyRdy to drive PhyNRdy        
0x000a  2           87  Device-to-host register FISes sent due to a COMRESET  
0x000b  2           45  CRC errors within host-to-device FIS                  
0x000f  2           45  R_ERR response for host-to-device data FIS, CRC      
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC  
0x8000  4      1982390  Vendor specific                                      
                                                                             
[root@OctoberGroupNAS ~]#

 

[DrKK]

I mean, this looks pretty good.

Nothing is jumping out at me, except for a higher-than-normal-but-still-OK "load cycle count" (you must be attempting to use "spin down" or other power saving features).

Even the SMART regimen looks good---in fact, you just came off of a "long" test which looks like it went fine. Still, I hate having any sort of errors showing up in the logs. It doesn't appear that anything is imminently dying at all, in fact, the drive looks well-cared for, and about 2.5 years old. Probably has 2.5 more years of life in it with the way you're taking care of it.

Maybe one of the other guys will have seen something I didn't, but my instinct, if you are noticing no performance issues, is to let it ride.

 

[nello]

I mean, this looks pretty good.

Nothing is jumping out at me, except for a higher-than-normal-but-still-OK "load cycle count" (you must be attempting to use "spin down" or other power saving features).

Yes, I am trying to invoke power-saving options on all the drives because the server is often idle for hours at a time:

Even the SMART regimen looks good---in fact, you just came off of a "long" test which looks like it went fine. Still, I hate having any sort of errors showing up in the logs. It doesn't appear that anything is imminently dying at all, in fact, the drive looks well-cared for, and about 2.5 years old. Probably has 2.5 more years of life in it with the way you're taking care of it.

That's good news!

Maybe one of the other guys will have seen something I didn't, but my instinct, if you are noticing no performance issues, is to let it ride.

I'm having quite a few irregularities, including:

• MythTV utility for flagging commercials (mythcommflag) reports problems with encoding; perhaps some of the writes during recording are bad?
• MythTV web interface won't list all my recordings (the entries of mythconverg.recorded) even though MySQL repair shows no problems with any table and this table seems to respond quickly and without error to local (python and perl) scripts and a remote SQL client. So, it seems to be a MythTV problem, not a disk problem.
• MythTV/Plex playback pauses regularly seeming to indicate that the server can't stream the video file quickly enough (despite wired connection to server) or Plex can't transcode fast enough. Or there are read errors; but read errors aren't being logged, so probably not due to the disk.

How can I tell whether any given "performance issue" is related to this particular disk?

 

[Ericloewe]

That's quite a different diagnosis that I was given last time I posted similar log errors: https://forums.freenas.org/index.php?threads/dying-drive.39378/

Well, SMART looks good. It could still be a device-to-host transmission issue, so I'd try to replace the cable.

Other than that, the spinup time is very slightly on the high side of what three-platter 5400RPM WD typically do. I doubt the extra 100ms are an issue, though. However, the drives (plural) spinning down can explain your performance issues. If Plex streams a section of the file from the pool that is large enough to last beyond the spin-down timer, it'll have to wait for the drives to spin back up, causing hiccups.

 

[Jailer]

Well since you're using all 6 SATA ports on that motherboard you can try new a cable on ada2. Check all the SATA connections as well to ensure that they are secure.

 

[nello]

… so I'd try to replace the cable.

You mean the SATA data cable, right?

If Plex streams a section of the file from the pool that is large enough to last beyond the spin-down timer, it'll have to wait for the drives to spin back up, causing hiccups.

Can you point me to where I can tweak the time a disk should wait before spinning down?

Thank you.

 

[Jailer]

Can you point me to where I can tweak the time a disk should wait before spinning down?

Yeah, save the wear and tear on the drives and don't spin them down at all.

 

[Ericloewe]

You mean the SATA data cable, right?

Yeah

 

[joeschmuck]

There is a theme here, I concur that replacing the SATA Data cable is the next thing to do in troubleshooting the problem. If that doesn't fix it, move the SATA data cable (drive end or motherboard end, which ever is easier for you) to another drive/port in order to see if the problem moves or remains the same. When dealing with troubleshooting of a drive or it's interface, always use drive serial numbers vice the adaX designations as they can move around when you are identifying a drive, always note the SATA port number on the motherboard as well. It's best to gather all the info one time vice doing something over a few times.

Good Luck.