rungekutta
Contributor
- Joined
- May 11, 2016
- Messages
- 146
Setup: TrueNAS-12.0-U1.1. SuperMicro X11-SSM-F. 6x12TB in RaidZ2, SuperDOM boot, S3700 SLOG, NVMe L2ARC (on PCIe adapter card). The disks are on the m/board SATA ports. The spinning disks are Seagate Exos X16, only a few months old (bought them staggered over the last 4 months).
I got an alert from TrueNAS overnight:
I kicked off a long SMART test on the offending device and meanwhile SMART looks like this:
As far as I can tell the SMART readings look ok, but there are still ATA errors registered. Any suggestions how I should troubleshoot this (including cabling?), or should I just return the disk under warranty?
Thankful for any advise or pointers.
I got an alert from TrueNAS overnight:
Code:
New alerts: * Device: /dev/ada6, ATA error count increased from 0 to 12.
I kicked off a long SMART test on the offending device and meanwhile SMART looks like this:
Code:
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST12000NM001G-2MV103
Serial Number: ZLW1KZ1C
LU WWN Device Id: 5 000c50 0c6d52145
Firmware Version: SN02
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Feb 7 13:20:23 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 247) Self-test routine in progress...
70% of test remaining.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: (1188) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 069 064 044 Pre-fail Always - 199069432
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 12
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 079 060 045 Pre-fail Always - 73664844
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1324
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 12
18 Unknown_Attribute 0x000b 100 100 050 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 088 088 000 Old_age Always - 12
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 060 049 040 Old_age Always - 40 (Min/Max 35/43)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 2
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 64
194 Temperature_Celsius 0x0022 040 048 000 Old_age Always - 40 (0 24 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0023 100 100 001 Pre-fail Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1321 (22 208 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 31650196978
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 57994025711
SMART Error Log Version: 1
ATA Error Count: 12 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 12 occurred at disk power-on lifetime: 1313 hours (54 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 25d+06:49:36.508 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:36.508 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 25d+06:49:36.475 READ LOG EXT
60 00 00 ff ff ff 4f 00 25d+06:49:32.425 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:32.425 READ FPDMA QUEUED
Error 11 occurred at disk power-on lifetime: 1313 hours (54 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 25d+06:49:32.425 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:32.425 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:32.425 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 25d+06:49:32.326 READ LOG EXT
60 00 00 ff ff ff 4f 00 25d+06:49:28.247 READ FPDMA QUEUED
Error 10 occurred at disk power-on lifetime: 1313 hours (54 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 25d+06:49:28.247 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:28.247 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:28.247 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 25d+06:49:28.244 READ LOG EXT
60 00 00 ff ff ff 4f 00 25d+06:49:24.145 READ FPDMA QUEUED
Error 9 occurred at disk power-on lifetime: 1313 hours (54 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 25d+06:49:24.145 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:24.144 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 25d+06:49:24.144 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 25d+06:49:24.111 READ LOG EXT
61 00 20 ff ff ff 4f 00 25d+06:49:19.968 WRITE FPDMA QUEUED
Error 8 occurred at disk power-on lifetime: 1313 hours (54 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 20 ff ff ff 4f 00 25d+06:49:19.968 WRITE FPDMA QUEUED
61 00 18 ff ff ff 4f 00 25d+06:49:19.968 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 25d+06:49:19.967 WRITE FPDMA QUEUED
61 00 10 ff ff ff 4f 00 25d+06:49:19.967 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 25d+06:49:19.967 WRITE FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Self-test routine in progress 70% 1324 -
# 2 Short offline Completed without error 00% 1309 -
# 3 Short offline Completed without error 00% 1285 -
# 4 Short offline Completed without error 00% 1261 -
# 5 Short offline Completed without error 00% 1237 -
# 6 Short offline Completed without error 00% 1213 -
# 7 Short offline Completed without error 00% 1189 -
# 8 Short offline Completed without error 00% 1165 -
# 9 Short offline Completed without error 00% 1141 -
#10 Short offline Completed without error 00% 1117 -
#11 Short offline Completed without error 00% 1093 -
#12 Short offline Completed without error 00% 1069 -
#13 Short offline Completed without error 00% 1045 -
#14 Short offline Completed without error 00% 1021 -
#15 Short offline Completed without error 00% 997 -
#16 Short offline Completed without error 00% 973 -
#17 Short offline Completed without error 00% 949 -
#18 Short offline Completed without error 00% 925 -
#19 Short offline Completed without error 00% 901 -
#20 Short offline Completed without error 00% 877 -
#21 Short offline Completed without error 00% 853 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
Error 8 occurred at disk power-on lifetime: 1313 hours (54 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 20 ff ff ff 4f 00 25d+06:49:19.968 WRITE FPDMA QUEUED
61 00 18 ff ff ff 4f 00 25d+06:49:19.968 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 25d+06:49:19.967 WRITE FPDMA QUEUED
61 00 10 ff ff ff 4f 00 25d+06:49:19.967 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 25d+06:49:19.967 WRITE FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Self-test routine in progress 70% 1324 -
# 2 Short offline Completed without error 00% 1309 -
# 3 Short offline Completed without error 00% 1285 -
# 4 Short offline Completed without error 00% 1261 -
# 5 Short offline Completed without error 00% 1237 -
# 6 Short offline Completed without error 00% 1213 -
# 7 Short offline Completed without error 00% 1189 -
# 8 Short offline Completed without error 00% 1165 -
# 9 Short offline Completed without error 00% 1141 -
#10 Short offline Completed without error 00% 1117 -
#11 Short offline Completed without error 00% 1093 -
#12 Short offline Completed without error 00% 1069 -
#13 Short offline Completed without error 00% 1045 -
#14 Short offline Completed without error 00% 1021 -
#15 Short offline Completed without error 00% 997 -
#16 Short offline Completed without error 00% 973 -
#17 Short offline Completed without error 00% 949 -
#18 Short offline Completed without error 00% 925 -
#19 Short offline Completed without error 00% 901 -
#20 Short offline Completed without error 00% 877 -
#21 Short offline Completed without error 00% 853 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
As far as I can tell the SMART readings look ok, but there are still ATA errors registered. Any suggestions how I should troubleshoot this (including cabling?), or should I just return the disk under warranty?
Thankful for any advise or pointers.
