Greetings everyone,
Long time lurker but firs time posting.
We have a FREENAS-MINI-3.0-XL+ with 8 drives running on FreeNAS-11.2-U8, We recently has a DEGRADED pool and we replaced the disk with issues with a fresh one. Pool is now HEALTHY but running smartctl on the new disk resulted with the following:
One that strikes me is the DEVICE ERROR COUNT. Checked the remaining disks and they don't have this error. This is the first time we had to replace a disk and not sure how to troubleshoot the cause.
Looking for any guidance from the community. Very much appreciate any help.
Long time lurker but firs time posting.
We have a FREENAS-MINI-3.0-XL+ with 8 drives running on FreeNAS-11.2-U8, We recently has a DEGRADED pool and we replaced the disk with issues with a fresh one. Pool is now HEALTHY but running smartctl on the new disk resulted with the following:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFAX-68JH4N1
Serial Number: WD-WXN2AA2KWCDF
LU WWN Device Id: 5 0014ee 26ae38497
Firmware Version: 83.00A83
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Apr 25 08:01:25 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 464) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 776) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3039) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 100 253 021 - 0
4 Start_Stop_Count -O--CK 100 100 000 - 2
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 100 100 000 - 277
10 Spin_Retry_Count -O--CK 100 253 000 - 0
11 Calibration_Retry_Count -O--CK 100 253 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 2
192 Power-Off_Retract_Count -O--CK 200 200 000 - 0
193 Load_Cycle_Count -O--CK 200 200 000 - 1
194 Temperature_Celsius -O---K 114 110 000 - 36
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x04 GPL R/O 256 Device Statistics log
0x04 SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 2048 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x24 GPL R/O 294 Current Device Internal Status Data log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb6 GPL,SL VS 1 Device vendor specific log
0xb7 GPL,SL VS 78 Device vendor specific log
0xb9 GPL,SL VS 4 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 38908 (device log contains only the most recent 24 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 38908 [3] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 02 ba a0 f4 18 40 00 Error: IDNF at LBA = 0x2baa0f418 = 11721045016
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 d0 00 02 ba a0 f4 18 40 08 7d+04:34:57.925 WRITE FPDMA QUEUED
61 00 08 00 c8 00 02 ba a0 f2 18 40 08 7d+04:34:57.925 WRITE FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 00 08 7d+04:34:57.922 READ LOG EXT
61 00 08 00 b8 00 02 ba a0 f4 18 40 08 7d+04:34:50.843 WRITE FPDMA QUEUED
61 00 08 00 b0 00 02 ba a0 f2 18 40 08 7d+04:34:50.843 WRITE FPDMA QUEUED
Error 38907 [2] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 02 ba a0 f2 18 40 00 Error: IDNF at LBA = 0x2baa0f218 = 11721044504
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 b8 00 02 ba a0 f4 18 40 08 7d+04:34:50.843 WRITE FPDMA QUEUED
61 00 08 00 b0 00 02 ba a0 f2 18 40 08 7d+04:34:50.843 WRITE FPDMA QUEUED
61 00 08 00 a8 00 00 00 40 04 18 40 08 7d+04:34:50.843 WRITE FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 00 08 7d+04:34:50.841 READ LOG EXT
61 00 08 00 98 00 02 ba a0 f4 18 40 08 7d+04:34:43.546 WRITE FPDMA QUEUED
Error 38906 [1] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 00 00 40 04 18 40 00 Error: IDNF at LBA = 0x00400418 = 4195352
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 98 00 02 ba a0 f4 18 40 08 7d+04:34:43.546 WRITE FPDMA QUEUED
61 00 08 00 90 00 02 ba a0 f2 18 40 08 7d+04:34:43.546 WRITE FPDMA QUEUED
61 00 08 00 88 00 00 00 40 04 18 40 08 7d+04:34:43.546 WRITE FPDMA QUEUED
61 00 08 00 80 00 00 00 40 02 18 40 08 7d+04:34:43.546 WRITE FPDMA QUEUED
ea 00 00 00 00 00 00 00 00 00 00 40 08 7d+04:34:41.706 FLUSH CACHE EXT
Error 38905 [0] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 02 32 89 5e 28 40 00 Error: IDNF at LBA = 0x232895e28 = 9437797928
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 38 00 a0 00 02 32 89 5f 28 40 08 7d+04:34:25.398 WRITE FPDMA QUEUED
61 01 00 00 98 00 02 32 89 5e 28 40 08 7d+04:34:25.398 WRITE FPDMA QUEUED
61 00 58 00 90 00 02 32 89 5d c8 40 08 7d+04:34:25.397 WRITE FPDMA QUEUED
61 00 58 00 88 00 02 32 89 65 50 40 08 7d+04:34:25.397 WRITE FPDMA QUEUED
61 00 30 00 80 00 02 32 89 65 20 40 08 7d+04:34:25.396 WRITE FPDMA QUEUED
Error 38904 [23] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 02 ba a0 f4 08 40 00 Error: IDNF at LBA = 0x2baa0f408 = 11721045000
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 00 00 02 ba a0 f4 08 40 08 7d+04:34:13.570 WRITE FPDMA QUEUED
61 00 08 00 f8 00 02 ba a0 f2 08 40 08 7d+04:34:13.570 WRITE FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 00 08 7d+04:34:13.568 READ LOG EXT
61 00 08 00 e8 00 02 ba a0 f4 08 40 08 7d+04:34:06.412 WRITE FPDMA QUEUED
61 00 08 00 e0 00 02 ba a0 f2 08 40 08 7d+04:34:06.412 WRITE FPDMA QUEUED
Error 38903 [22] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 02 ba a0 f2 08 40 00 Error: IDNF at LBA = 0x2baa0f208 = 11721044488
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 e8 00 02 ba a0 f4 08 40 08 7d+04:34:06.412 WRITE FPDMA QUEUED
61 00 08 00 e0 00 02 ba a0 f2 08 40 08 7d+04:34:06.412 WRITE FPDMA QUEUED
61 00 08 00 d8 00 00 00 40 04 08 40 08 7d+04:34:06.412 WRITE FPDMA QUEUED
2f 00 00 00 01 00 00 00 00 00 10 00 08 7d+04:34:06.409 READ LOG EXT
61 00 08 00 c8 00 02 ba a0 f4 08 40 08 7d+04:33:58.302 WRITE FPDMA QUEUED
Error 38902 [21] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 00 00 00 00 00 40 04 08 40 00 Error: IDNF at LBA = 0x00400408 = 4195336
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 c8 00 02 ba a0 f4 08 40 08 7d+04:33:58.302 WRITE FPDMA QUEUED
61 00 08 00 c0 00 02 ba a0 f2 08 40 08 7d+04:33:58.302 WRITE FPDMA QUEUED
61 00 08 00 b8 00 00 00 40 04 08 40 08 7d+04:33:58.302 WRITE FPDMA QUEUED
61 00 08 00 b0 00 00 00 40 02 08 40 08 7d+04:33:58.302 WRITE FPDMA QUEUED
ea 00 00 00 00 00 00 00 00 00 00 40 08 7d+04:33:56.091 FLUSH CACHE EXT
Error 38901 [20] occurred at disk power-on lifetime: 172 hours (7 days + 4 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
10 -- 51 01 00 00 01 1b 9d 8f 30 40 00 Error: IDNF at LBA = 0x11b9d8f30 = 4758277936
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
61 00 08 00 38 00 01 1b 9d 8f 30 40 08 7d+04:33:40.271 WRITE FPDMA QUEUED
35 00 00 01 00 00 01 1b 9d 8e 30 40 08 7d+04:33:40.101 WRITE DMA EXT
35 00 00 01 00 00 01 1b 9d 8e 30 40 08 7d+04:33:33.151 WRITE DMA EXT
06 00 01 00 01 00 00 00 00 00 00 40 08 7d+04:33:33.151 DATA SET MANAGEMENT
61 00 10 00 18 00 01 14 9e b8 08 40 08 7d+04:33:33.055 WRITE FPDMA QUEUED
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 266 -
# 2 Short offline Completed without error 00% 190 -
# 3 Short offline Completed without error 00% 13 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 36 Celsius
Power Cycle Min/Max Temperature: 34/39 Celsius
Lifetime Min/Max Temperature: 26/40 Celsius
Under/Over Temperature Limit Count: 0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/65 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (294)
Index Estimated Time Temperature Celsius
295 2023-04-25 00:04 35 ****************
... ..(263 skipped). .. ****************
81 2023-04-25 04:28 35 ****************
82 2023-04-25 04:29 34 ***************
... ..( 11 skipped). .. ***************
94 2023-04-25 04:41 34 ***************
95 2023-04-25 04:42 35 ****************
... ..( 12 skipped). .. ****************
108 2023-04-25 04:55 35 ****************
109 2023-04-25 04:56 34 ***************
110 2023-04-25 04:57 35 ****************
... ..( 2 skipped). .. ****************
113 2023-04-25 05:00 35 ****************
114 2023-04-25 05:01 34 ***************
... ..( 3 skipped). .. ***************
118 2023-04-25 05:05 34 ***************
119 2023-04-25 05:06 35 ****************
... ..(166 skipped). .. ****************
286 2023-04-25 07:53 35 ****************
287 2023-04-25 07:54 36 *****************
... ..( 6 skipped). .. *****************
294 2023-04-25 08:01 36 *****************
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) supported [please try: '-l defects']
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 2 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 89766 Vendor specific
One that strikes me is the DEVICE ERROR COUNT. Checked the remaining disks and they don't have this error. This is the first time we had to replace a disk and not sure how to troubleshoot the cause.
Looking for any guidance from the community. Very much appreciate any help.