SOLVED Do I need to replace my disk?

[Kapurai]

Hi:

I built a FreeNAS machine ~3.5 years ago with the following spec. It has been running continuously since it was built, except for a few times when it has been taken offline for removing dust and other maintenance.

Show

Intel Core i3-4130T
Crucial 16GB DDR3 ECC Unbuffered RAM (2x8GB).
Supermicro X10SLL-F
Seasonic SSR-450RM
Fractal Design Arc Midi R2
Seagate NAS drive 4TB ST4000VN4000 (x4) in one RAID-Z2 vdev

Testing with badblocks told me that one of the drives was bad, so I RMA'd it and got a replacement, which tested fine.
Back then I tested them with:

Code:

badblocks -ws /dev/ada#

Now this same drive which I got back from RMA shows 8 pending sectors. This appeared recently (I think this week) and the previous long test was fine.
Unfortunately, the drives are no longer under warranty (they had a 3 year warranty when I got them)
Here is the output of smartctl -a /dev/ada3 (this is the failing disk)

Show : smartctl -a /dev/ada3

Code:

smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Seagate NAS HDD
Device Model:	 ST4000VN000-1H4168
Serial Number:	Z301VQWR
LU WWN Device Id: 5 000c50 067391861
Firmware Version: SC44
User Capacity:	4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5900 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Wed Mar 28 17:52:21 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (  117) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 505) minutes.
Conveyance self-test routine
recommended polling time:	 (   2) minutes.
SCT capabilities:			(0x10bd)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   106   099   006	Pre-fail  Always	   -	   12074208
  3 Spin_Up_Time			0x0003   092   091   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   87
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   059   050   030	Pre-fail  Always	   -	   661527188283
  9 Power_On_Hours		  0x0032   066   066   000	Old_age   Always	   -	   30318
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   87
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0022   060   053   045	Old_age   Always	   -	   40 (Min/Max 18/43)
191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   0
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   29
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   87
194 Temperature_Celsius	 0x0022   040   047   000	Old_age   Always	   -	   40 (0 18 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   8
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   8
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 30256		 -
# 2  Extended offline	Completed: read failure	   60%	 30163		 3953045720
# 3  Short offline	   Completed without error	   00%	 30088		 -
# 4  Short offline	   Completed without error	   00%	 29920		 -
# 5  Extended offline	Completed without error	   00%	 29833		 -
# 6  Short offline	   Completed without error	   00%	 29753		 -
# 7  Short offline	   Completed without error	   00%	 29585		 -
# 8  Extended offline	Aborted by host			   90%	   244		 -
# 9  Extended offline	Completed without error	   00%		 7		 -
#10  Short offline	   Completed without error	   00%		 0		 -
#11  Conveyance offline  Completed without error	   00%		 0		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The WebUI console is also spammed with messages saying that this drive has 8 offline unreadable/pending sectors.

zpool status shows that the pool is fine, but I haven't run a scrub.

Show : zpool status

Code:

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h20m with 0 errors on Fri Mar 16 04:05:17 2018
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  da0p2	 ONLINE	   0	 0	 0

errors: No known data errors

  pool: mypool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 6h58m with 0 errors on Thu Mar 15 10:58:28 2018
config:

	NAME											STATE	 READ WRITE CKSUM
	mypool										   ONLINE	   0	 0	 0
	  raidz2-0									  ONLINE	   0	 0	 0
		gptid/8efb686e-21ad-11e4-97b3-002590d2c231  ONLINE	   0	 0	 0
		gptid/8f513d76-21ad-11e4-97b3-002590d2c231  ONLINE	   0	 0	 0
		gptid/8fafccd2-21ad-11e4-97b3-002590d2c231  ONLINE	   0	 0	 0
		gptid/900495f3-21ad-11e4-97b3-002590d2c231  ONLINE	   0	 0	 0

errors: No known data errors

SMART long tests are scheduled twice a month and short tests four times a month. Scrubs run twice a month.

What is the recommended course of action? Should I replace this disk? Should I scrub the zpool now? Is it likely that other such errors are hiding in the rest of the 60% of the disk that the SMART test wasn't able to get to?
If I am to replace it, is it recommended to first offline the failing disk and then put in the new one, or is it possible to copy the data from the failing disk to the new one or replace in-place somehow?
Should I expect other disks to start dying as well?

Thanks for any help that you could provide.

 

[JohnDigital]

Yes this drive is failing. The last long smart test did not complete. I would order a replacement for it ASAP. Hold off on any scrubbing until you get a new drive and get it resilvered. There is plenty of documentation to reference this process.

Basically, you figure out what the failing drive is physically in the machine by serial number, remove it, replace with a new one, go into the FreeNAS GUI and replace the now missing drive with the new one (it should be the only one to choose from). Wait for resilver, then do a scrub.

 

[Kapurai]

Great, thanks for the quick answer.

Are there any recommendations for a good replacement drive?
Would the WD Red 4TB (WD40EFRX) be a good choice for the replacement?
I am also seeing Toshiba N300/Seagate IronWolf 4TB disks for cheaper. Are these also ok? I've heard to stay away from Seagate before, but is it still true?


EDIT: do I need to make sure that the drive's capacity is at least that of the current one? Is it possible that the exact number of bytes in the drive will be different depending on the vendor/model? (could it end up like the "drivemaker's kilobyte" ?)

 

[JohnDigital]

Also I feel like 40c is a bit hot for your drives. But this is debatable. I like mine around 30c.

I might get bashed for it, but I've been liking Toshiba x300 drives. I bought 12 once when they were far cheaper than red drives. That was years ago. I've replaced One in that time 24.7.365 usage. Still rocking as of right now no issues whatever. I feel like all major hard drive makers are about the same. I'd stay away from anything "green".

 

[JohnDigital]

Also it's not my experience that mix matched drives differ whereas FN wouldn't allow you to use it in your pool. Any advertised 4tb *should work.

 

[Kapurai]

Great, thanks a lot.