Device: /dev/ada2, 65 Offline uncorrectable sectors

[siteswap]

Good evening to all, as the title I received this warning message ,
the disc is a red wd 3 tb ( stripe )
This morning I gave the command
smartctl -t long /dev/ada2 ,
and this is the result :
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 891
3 Spin_Up_Time 0x0027 194 179 021 Pre-fail Always - 5258
4 Start_Stop_Count 0x0032 061 061 000 Old_age Always - 39452
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 16753
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 75
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 51
193 Load_Cycle_Count 0x0032 187 187 000 Old_age Always - 39459
194 Temperature_Celsius 0x0022 125 094 000 Old_age Always - 25
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 65
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 199 000 Old_age Offline - 4
My question is , I have to replace the hard disk?
or you can fix the problem ??
Thanks to everyone in advance

 

[anodos]

Sounds like it's pining for the fjords. BTW, that's a high load cycle count for a WD red. Are you spinning them down or something?

 

[siteswap]

the disk was part of an old Nas, and I use it mainly for transmission, the first month has been working fine, I have no idea what happened

 

[danb35]

What happened appears to be that the disk is dying (it's not pining, it's passed on!). Please post the full output of smartctl -a /dev/ada2 in code tags.

Edit: and are you saying that this disk is part of a striped pool? Do you not care about your data?

 

[siteswap]

Yes I've care about the my data ,let me explain..
The first pool is 2x3tb in mirror ( BackupDataPool), Music Movies etc etc..
The second pool (stripe is the "download place for torrent, and nothing more" ) and unfortunately it seems to be dead ,

Code:

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD30EFRX-68EUZN0
Serial Number:	WD-WMC4N1952362
LU WWN Device Id: 5 0014ee 6aedf256d
Firmware Version: 80.00A80
User Capacity:	3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5400 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Thu Nov  3 19:35:53 2016 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  ( 121)	The previous self-test completed having
					the read element of the test failed.
Total time to complete Offline
data collection:		 (39900) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 400) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   891
  3 Spin_Up_Time			0x0027   194   179   021	Pre-fail  Always	   -	   5258
  4 Start_Stop_Count		0x0032   061   061   000	Old_age   Always	   -	   39452
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   078   078   000	Old_age   Always	   -	   16755
10 Spin_Retry_Count		0x0032   100   100   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   75
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   51
193 Load_Cycle_Count		0x0032   187   187   000	Old_age   Always	   -	   39460
194 Temperature_Celsius	 0x0022   125   094   000	Old_age   Always	   -	   25
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   200   200   000	Old_age   Offline	  -	   65
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   199   000	Old_age   Offline	  -	   4

SMART Error Log Version: 1
ATA Error Count: 167 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 167 occurred at disk power-on lifetime: 16729 hours (697 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 7f 00 40 40  Error: UNC 1 sectors at LBA = 0x0040007f = 4194431

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 01 7f 00 40 40 08	  00:01:12.909  READ DMA
  c8 00 01 7f 00 40 40 08	  00:01:09.545  READ DMA
  c8 00 01 7f 00 40 40 08	  00:01:06.182  READ DMA
  c8 00 01 7f 00 40 40 08	  00:01:02.819  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:59.456  READ DMA

Error 166 occurred at disk power-on lifetime: 16729 hours (697 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 7f 00 40 40  Error: UNC 1 sectors at LBA = 0x0040007f = 4194431

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 01 7f 00 40 40 08	  00:01:09.545  READ DMA
  c8 00 01 7f 00 40 40 08	  00:01:06.182  READ DMA
  c8 00 01 7f 00 40 40 08	  00:01:02.819  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:59.456  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:56.092  READ DMA

Error 165 occurred at disk power-on lifetime: 16729 hours (697 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 7f 00 40 40  Error: UNC 1 sectors at LBA = 0x0040007f = 4194431

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 01 7f 00 40 40 08	  00:01:06.182  READ DMA
  c8 00 01 7f 00 40 40 08	  00:01:02.819  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:59.456  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:56.092  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:52.729  READ DMA

Error 164 occurred at disk power-on lifetime: 16729 hours (697 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 7f 00 40 40  Error: UNC 1 sectors at LBA = 0x0040007f = 4194431

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 01 7f 00 40 40 08	  00:01:02.819  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:59.456  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:56.092  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:52.729  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:49.366  READ DMA

Error 163 occurred at disk power-on lifetime: 16729 hours (697 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 7f 00 40 40  Error: UNC 1 sectors at LBA = 0x0040007f = 4194431

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 01 7f 00 40 40 08	  00:00:59.456  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:56.092  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:52.729  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:49.366  READ DMA
  c8 00 01 7f 00 40 40 08	  00:00:46.003  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed: read failure	   90%	 16753		 2189888
# 2  Extended offline	Completed: read failure	   90%	 16752		 2189888
# 3  Extended offline	Completed: read failure	   90%	 16751		 2189888
# 4  Extended offline	Completed: read failure	   90%	 16750		 2189888
# 5  Extended offline	Completed: read failure	   90%	 16749		 2189888
# 6  Extended offline	Completed: read failure	   90%	 16748		 2189888
# 7  Extended offline	Completed: read failure	   90%	 16747		 2189888
# 8  Extended offline	Completed: read failure	   90%	 16746		 2189888
# 9  Extended offline	Completed: read failure	   90%	 16745		 2189888
#10  Extended offline	Completed: read failure	   90%	 16744		 2189888
#11  Extended offline	Completed: read failure	   90%	 16744		 2189888
#12  Extended offline	Completed: read failure	   90%	 16417		 326598
#13  Extended offline	Completed: read failure	   90%	 16417		 326592
#14  Short offline	   Completed without error	   00%	 12038		 -
#15  Short offline	   Completed without error	   00%		 0		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Here is the output

 

[danb35]

Well, yes, the drive is dying. Dozens of bad sectors, and failing every long SMART test for a while (though it looks like those were running every hour, which is way too often). Time to replace it. Even in a striped pool, you can replace drives without data loss, as long as the drive hasn't totally died (which it doesn't yet appear to have done). It's possible that the bad sectors would result in some corrupt data, but even then, at least ZFS will know about it.

 

[DrKK]

Can I get you guys, @danb35, @anodos, etc, can I get you guys to request "smartctl -x" from people, instead of smartctl -a? That gives a lot more information, and more clues about how the drive is being used and stored. 16755 hours is a really strange number of hours for a WD RED to die. It normally would have failed long before then, or long after. And while the load cycle count is surely high, that cannot be the responsible entity for this early death.

I'm guessing in the old NAS, this drive was way, way, way over temperature spec, which we would have seen with smartctl -x.

 

[Robert Trevellyan]

failing every long SMART test for a while (though it looks like those were running every hour, which is way too often)

To the point of absurdity, since a 3TB drive takes several hours to complete an extended test.

What SMART testing schedule do you have on your other drives? If your short tests are running more than daily, or your extended tests are more than weekly, you're running them too often to be useful. My shorts run weekly and extended run monthly, but some members consider that too infrequent.