FAILED SMART self-check. BACK UP DATA NOW! HELP!!!

[Ljunkan]

Hi

I build my first Freenas a couple of weeks ago. Now im seing this error in my log.


an 27 18:03:58 freenas1 smartd[2091]: Device: /dev/ada2, FAILED SMART self-check. BACK UP DATA NOW!
Jan 27 18:03:58 freenas1 smartd[2091]: Device: /dev/ada2, Failed SMART usage Attribute: 7 Seek_Error_Rate.

I have zfs in a raidz1 with 5 disk. All the disk are new.

Here are some more outputs:
[root@freenas1] ~# smartctl -a /dev/ada2 -t
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=======> ARGUMENT REQUIRED FOR OPTION: t
=======> VALID ARGUMENTS ARE: offline, short, long, conveyance, force, vendor,N, select,M-N, pending,N, afterselect,[on|off] <=======

Use smartctl -h to get a usage summary

[root@freenas1] ~# smartctl -a /dev/ada2 -t short
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda (SATA 3Gb/s, 4K Sectors)
Device Model: ST2000DM001-1CH164
Serial Number: W2416ZFF
LU WWN Device Id: 5 000c50 05b70f664
Firmware Version: CC24
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Mon Jan 28 22:08:50 2013 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 600) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 246) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 119 099 006 Pre-fail Always - 208725536
3 Spin_Up_Time 0x0003 099 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 154
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 031 030 030 Pre-fail Always In_the_past 5226976574251
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 453
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 14
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 1
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 046 035 045 Old_age Always In_the_past 54 (1 24 54 40 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 8
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 189
194 Temperature_Celsius 0x0022 054 065 000 Old_age Always - 54 (0 20 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 2
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 245062243975620
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 1622328236
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 245363061

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Mon Jan 28 22:09:50 2013

am I in danger?

 

[Ljunkan]

More info:

Code:

martctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

dev/ada2: Unable to detect device type
Smartctl: please specify device type with the -d option.

Use smartctl -h to get a usage summary

[root@freenas1] ~# smartctl -a /dev/ada2
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda (SATA 3Gb/s, 4K Sectors)
Device Model:     ST2000DM001-1CH164
Serial Number:    W2416ZFF
LU WWN Device Id: 5 000c50 05b70f664
Firmware Version: CC24
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Mon Jan 28 22:35:44 2013 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  600) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 246) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       208729648
  3 Spin_Up_Time            0x0003   099   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       154
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   031   030   030    Pre-fail  Always   In_the_past 5265631280300
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       454
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       14
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   046   035   045    Old_age   Always   In_the_past 54 (1 24 54 40 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       189
194 Temperature_Celsius     0x0022   054   065   000    Old_age   Always       -       54 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       2
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       160734856085957
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1622334644
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       245363077

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       453         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Stephens]

Jan 27 18:03:58 freenas1 smartd[2091]: Device: /dev/ada2, FAILED SMART self-check. BACK UP DATA NOW!
Jan 27 18:03:58 freenas1 smartd[2091]: Device: /dev/ada2, Failed SMART usage Attribute: 7 Seek_Error_Rate.

Seems pretty clear to me. Back up your data now. Yes, you're in danger.

Did you do smart long tests on your drives before using them? No matter how new they are, it needs to be done to make sure you don't have DOA drives. I have had them, and many others have too.

What's the output of "zpool status" from the console? Hopefully your errors are confined to one drive and you can replace it and resilver before any other drives fail. Once you've backed up your data, turn off the NAS if it's not mission critical until you get a replacement drive installed. Please read and understand the documentations instructions for replacing a failed drive.

 

[Ljunkan]

Zpool status:

Code:

pool: Mirror
 state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        Mirror                                          ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/b4b6d8e9-5c23-11e2-a554-50465db58da7  ONLINE       0     0     0
            gptid/b5373a38-5c23-11e2-a554-50465db58da7  ONLINE       0     0     0

errors: No known data errors

  pool: Storage1
 state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        Storage1                                        ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/366f1ccc-5a86-11e2-a7d0-50465db58da7  ONLINE       0     0     0
            gptid/36f6e4e0-5a86-11e2-a7d0-50465db58da7  ONLINE       0     0     0
            gptid/37832b8c-5a86-11e2-a7d0-50465db58da7  ONLINE       0     0     0
            gptid/380a1413-5a86-11e2-a7d0-50465db58da7  ONLINE       0     0     0
            gptid/3896f3d4-5a86-11e2-a7d0-50465db58da7  ONLINE       0     0     0

errors: No known data errors

 

[Stephens]

Great. You haven't lost any data. I wouldn't trust the drive in question, though.

 

[paleoN]

I would replace ada2 and fix your temperature problem. The disk exceeded its maximum operating temperature and it still far too hot. I would guess the heat also caused 7 Seek_Error_Rate to rise above it's liberal threshold. You should also look at the temperature of all your other disks.

 

[Ljunkan]

I would replace ada2 and fix your temperature problem. The disk exceeded its maximum operating temperature and it still far too hot. I would guess the heat also caused 7 Seek_Error_Rate to rise above it's liberal threshold. You should also look at the temperature of all your other disks.

Hi thanks for your replay.

Thats my conclusion as well, and i have already fixed the temps. So now the are under 30 degres.

But if the temp was causing the problem, then the problem is fixet and i should not have more high seek rate.

Could some one please correct me if im doing this calculation wrong. If i fallow this article ----> http://forums.seagate.com/t5/Barracuda-XT-Barracuda-Barracuda/Seagate-s-Seek-Error-Rate-Raw-Read-Error-Rate-and-Hardware-ECC/td-p/122382

The seek rate is just now 5583459171527 = 5140019bcc7 in hex.
(-10) log(0x5140 / 0x019bcc7) = 19.0898146

My calculation is that i have around 10 error per thousand counts.


And i guess that is really BAD. But it shuold improve over time, right? If the temp was the problem.

 

[cyberjock]

Personal experience with hard drives that got too hot is "No, they won't improve but they will definitely become more unreliable as time goes on". Also, you may find that if you RMA the drive it will be rejected for not using the drive within its assigned operating envelope. I've seen 3 overheated drives, 2 were rejected from RMA, one was accepted.

 

[sana]

Need help with this error - checked all the above solutions nothing works for me : Device: /dev/ada2, FAILED SMART self-check. BACK UP DATA NOW

 

[Glorious1]

I'm not sure what help you are looking for. Clearly the drive is failing. Clearly you should back up your data now if you can. Then, or if you can't back up, you need to follow the guide directions for replacing a failed drive. You don't indicate what your pool structure is, but if it is raidz1 and no backup, hope and pray that another drive doesn't fail while going through the resilver process.

 

[sana]

Hello
Thank you for your reply! We have Raid Z1 configured and also can a hard disk get damaged so easily?
Our NAS went live in November. What is the alternative solution to this? Will my data be lost ?

 

[Yorick]

raidz1 means "I merely like my data. I have a backup and restoring from backup is no big deal"

Start by taking a backup, if these data aren't trivially replaced.

Then replace the failing drive, wait for resilver and hope you don't have a second failure.

Then set up regular backup and, with some urgency, plan for moving to raidz2 and restoring data to raidz2 from backup.

raidz2 means "I love my data. I have a backup but restoring from it would be a big deal, I'd really rather not have to do that"

Having no backup at all always means "I don't care about these data, so what if they get lost". That can be scratch space for example. Or in my case it's my Plex library - no backup but meh, I can always re-create the library from the physical DVDs and Blurays, should my raidz2 truly fail. It's a case of wagering that it's not that likely two drives will fail; that in the case my house burns down I have other problems than my movie library; and that if raidz2 does fail, I can always recreate the library.

Even with something as inconsequential as an entertainment library, I'd never do raidz1. I care at least a litte about my data. Not enough to take backups, but enough to reduce the likelihood I'll have to do the transfer all over again.

 

[Glorious1]

Hello
Thank you for your reply! We have Raid Z1 configured and also can a hard disk get damaged so easily?
Our NAS went live in November. What is the alternative solution to this? Will my data be lost ?

Hard drives can fail at any time. Assuming the drives were new when you went live, this is sooner than expected. However, most careful people do a "burn-in" with new drives, before making a pool with it, writing and reading dummy data to it repeatedly for days with tests before and after. This way, drives that may fail prematurely are likely caught early so they can be returned and replaced with no problem.

Your data will be lost if you replace the drive and a second drive fails while the first is getting synced to the pool (called resilvering). Resilvering puts a lot of demand on drives, so this is not as unlikely as you might think. Make sure you follow the guide procedure carefully for offlining and replacing the drive.

If you want more info about the drive, type "sudo smartctl -a /dev/ada2", copy the output and paste it here in code tags. Maybe some knowledgeable person can tell you more about the failure.