Can I just quickly confirm -- From what I have read, since I'm not changing my boot pool, I should be able to just shut down the server, disconnect and reconnect all drives direct to the mb, then boot back up? And should be able to do that without exporting/ disconnecting and going through the import process?
-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Pool keeps degrading
- Thread starter buswedg
- Start date
I literally just beat you to it. I ended up doing the above luckily, bar backing-up/re-importing the TrueNAS-config.
Looks like its all back up and running. There's a scrub task scheduled in about 10 hours. So I guess that'll be the first real test.
I'll report back here if I see any issues.
- Joined
- Jan 11, 2021
- Messages
- 415
A German idiom says: "Two fools have to share one thought ..."
- Joined
- Jan 11, 2021
- Messages
- 415
Not necessarily so, it seems ... not when it comes to AHCI/ALPM ...
Serial ATA AHCI: Specification, Rev. 1.3.1
Defines functional behavior and Advanced Host Controller Interface software for moving data between system memory and serial ATA devices.
Not necessarily so, it seems ... not when it comes to AHCI/ALPM ...
Serial ATA AHCI: Specification, Rev. 1.3.1
Defines functional behavior and Advanced Host Controller Interface software for moving data between system memory and serial ATA devices.www.intel.com
Well that's frustrating. Means I've made two changes at the same time. So harder to know what the root cause is.
- Joined
- Jan 11, 2021
- Messages
- 415
I would rather say You went back to "Start here". If Your MoBo/BIOS is the problem, You'll see it. If its the additional HBA, You'll see it. As long as You switch enable/disable in the BIOS with the LSI card installed, You'll never know, what part of the hardware causes the problems/is "incompatible".
I'm leaning towards the view, that the drives are to rule out, since the errors aren't tied to certain drives and vary by type, while smartctl gives no "indisputable facts".
I'm leaning towards the view, that the drives are to rule out, since the errors aren't tied to certain drives and vary by type, while smartctl gives no "indisputable facts".
/dev/sdc faulted, and dmesg is showing something I've not seen before.
Code:
root@truenas[~]# zpool status storage-pool
pool: storage-pool
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0B in 19:50:05 with 0 errors on Tue Sep 13 15:15:36 2022
config:
NAME STATE READ WRITE CKSUM
storage-pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
1766c859-9231-4127-ad35-882937845b76 ONLINE 0 0 0
89b557d9-cc55-463c-9124-3765cff9aaac ONLINE 0 0 0
4a07f2eb-442c-430d-9db6-0e5fd80302a9 FAULTED 15 0 0 too many errors
c6287e21-49e0-4ffe-86a1-78a33bbbec70 ONLINE 0 0 0
1c817e68-35de-4b8c-be59-4e98fc1fc9fe ONLINE 0 0 0
28dba822-3f63-4e2c-82ad-8e4467bc81a1 ONLINE 0 0 0
cache
f5456732-9153-44fb-977b-fe5f1021b959 ONLINE 0 0 0Code:
[181797.236312] ata3.00: exception Emask 0x0 SAct 0x80000618 SErr 0x0 action 0x0
[181797.244136] ata3.00: irq_stat 0x40000008
[181797.248703] ata3.00: failed command: READ FPDMA QUEUED
[181797.254319] ata3.00: cmd 60/80:48:30:b0:6f/03:00:70:03:00/40 tag 9 ncq dma 458752 in
res 43/40:80:30:b0:6f/00:03:70:03:00/40 Emask 0x409 (media error) <F>
[181797.271356] ata3.00: status: { DRDY SENSE ERR }
[181797.276383] ata3.00: error: { UNC }
[181797.369805] ata3.00: configured for UDMA/133
[181797.374978] sd 3:0:0:0: [sdc] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[181797.385095] sd 3:0:0:0: [sdc] tag#9 Sense Key : Medium Error [current]
[181797.392636] sd 3:0:0:0: [sdc] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
[181797.402269] sd 3:0:0:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 03 70 6f b0 30 00 00 03 80 00 00
[181797.412073] blk_update_request: I/O error, dev sdc, sector 14771269680 op 0x0:(READ) flags 0x700 phys_seg 14 prio class 0
[181797.423901] zio pool=storage-pool vdev=/dev/disk/by-partuuid/4a07f2eb-442c-430d-9db6-0e5fd80302a9 error=5 type=1 offset=7560742526976 size=458752 flags=40080c80
[181797.439668] ata3: EH complete
Last edited:
Code:
smartctl -a /dev/sdc
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD161KRYZ-01AGBB0
Serial Number: 3FKEJZZT
LU WWN Device Id: 5 000cca 2c2f06f26
Firmware Version: 01.01H01
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Sep 13 16:41:31 2022 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 101) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1746) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
2 Throughput_Performance 0x0005 148 148 054 Pre-fail Offline - 48
3 Spin_Up_Time 0x0007 085 085 001 Pre-fail Always - 320 (Average 301)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 100 100 001 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 1674
10 Spin_Retry_Count 0x0013 100 100 001 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36
22 Helium_Level 0x0023 100 100 025 Pre-fail Always - 100
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 117
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 117
194 Temperature_Celsius 0x0002 063 063 000 Old_age Always - 32 (Min/Max 24/47)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 1 occurred at disk power-on lifetime: 1674 hours (69 days + 18 hours)
When the command that caused the error occurred, the device was doing SMART Offline or Self-test.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 43 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 80 48 30 b0 6f 40 08 2d+02:31:20.023 READ FPDMA QUEUED
61 40 20 90 51 c9 40 08 2d+02:31:15.388 WRITE FPDMA QUEUED
61 40 50 d0 51 c9 40 08 2d+02:31:15.388 WRITE FPDMA QUEUED
60 40 18 f8 ea 6b 40 08 2d+02:31:13.089 READ FPDMA QUEUED
60 00 f8 f8 e6 6b 40 08 2d+02:31:13.087 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 1673 -
# 2 Extended offline Interrupted (host reset) 80% 1619 -
# 3 Short offline Completed without error 00% 1586 -
# 4 Short offline Completed without error 00% 1538 -
# 5 Short offline Completed without error 00% 1490 -
# 6 Extended offline Interrupted (host reset) 60% 1469 -
# 7 Short offline Completed without error 00% 1418 -
# 8 Short offline Completed without error 00% 1370 -
# 9 Extended offline Completed without error 00% 1347 -
#10 Short offline Completed without error 00% 1250 -
#11 Short offline Completed without error 00% 1223 -
#12 Short offline Completed without error 00% 1207 -
#13 Extended offline Interrupted (host reset) 70% 1180 -
#14 Short offline Completed without error 00% 1135 -
#15 Short offline Completed without error 00% 1087 -
#16 Short offline Completed without error 00% 1041 -
#17 Short offline Completed without error 00% 972 -
#18 Short offline Completed without error 00% 923 -
#19 Short offline Completed without error 00% 875 -
#20 Short offline Completed without error 00% 863 -
#21 Extended offline Completed without error 00% 796 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay./dev/sdd showing errors this morning. These dmesg errors look a little more violent, since plugging each drive direct to the mb sata headers.
Code:
pool: storage-pool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 22.1G in 00:04:10 with 0 errors on Tue Sep 13 16:32:17 2022
config:
NAME STATE READ WRITE CKSUM
storage-pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
1766c859-9231-4127-ad35-882937845b76 ONLINE 0 0 0
89b557d9-cc55-463c-9124-3765cff9aaac ONLINE 0 0 0
4a07f2eb-442c-430d-9db6-0e5fd80302a9 ONLINE 0 0 0
c6287e21-49e0-4ffe-86a1-78a33bbbec70 ONLINE 6 0 0
1c817e68-35de-4b8c-be59-4e98fc1fc9fe ONLINE 0 0 0
28dba822-3f63-4e2c-82ad-8e4467bc81a1 ONLINE 0 0 0
cache
f5456732-9153-44fb-977b-fe5f1021b959 ONLINE 0 0 0Code:
[243679.797552] ata5.00: exception Emask 0x0 SAct 0x7800009 SErr 0x0 action 0x0
[243679.805048] ata5.00: irq_stat 0x40000008
[243679.809412] ata5.00: failed command: READ FPDMA QUEUED
[243679.815008] ata5.00: cmd 60/70:c0:68:af:c0/01:00:8d:04:00/40 tag 24 ncq dma 188416 in
res 43/40:70:68:af:c0/00:01:8d:04:00/40 Emask 0x409 (media error) <F>
[243679.832100] ata5.00: status: { DRDY SENSE ERR }
[243679.837202] ata5.00: error: { UNC }
[243679.864114] ata5.00: configured for UDMA/133
[243679.869095] sd 5:0:0:0: [sdd] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[243679.879020] sd 5:0:0:0: [sdd] tag#24 Sense Key : Medium Error [current]
[243679.886407] sd 5:0:0:0: [sdd] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
[243679.895813] sd 5:0:0:0: [sdd] tag#24 CDB: Read(16) 88 00 00 00 00 04 8d c0 af 68 00 00 01 70 00 00
[243679.905568] blk_update_request: I/O error, dev sdd, sector 19558084456 op 0x0:(READ) flags 0x700 phys_seg 7 prio class 0
[243679.917114] zio pool=storage-pool vdev=/dev/disk/by-partuuid/c6287e21-49e0-4ffe-86a1-78a33bbbec70 error=5 type=1 offset=10011591692288 size=188416 flags=40080c80
[243679.932591] ata5: EH completeCode:
root@truenas[~]# smartctl -a /dev/sdd
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD161KRYZ-01AGBB0
Serial Number: 4ZG86LJV
LU WWN Device Id: 5 000cca 2a6c3bbbc
Firmware Version: 01.01H01
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Sep 14 09:46:17 2022 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 247) Self-test routine in progress...
70% of test remaining.
Total time to complete Offline
data collection: ( 101) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1685) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
2 Throughput_Performance 0x0005 149 149 054 Pre-fail Offline - 44
3 Spin_Up_Time 0x0007 084 084 001 Pre-fail Always - 340 (Average 327)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 100 100 001 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 1691
10 Spin_Retry_Count 0x0013 100 100 001 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36
22 Helium_Level 0x0023 100 100 025 Pre-fail Always - 100
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 116
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 116
194 Temperature_Celsius 0x0002 064 064 000 Old_age Always - 31 (Min/Max 23/41)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 1 occurred at disk power-on lifetime: 1691 hours (70 days + 11 hours)
When the command that caused the error occurred, the device was doing SMART Offline or Self-test.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 43 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 70 c0 68 af c0 40 08 2d+19:42:41.926 READ FPDMA QUEUED
61 40 18 30 b4 de 40 08 2d+19:42:38.668 WRITE FPDMA QUEUED
61 40 d0 f0 b3 de 40 08 2d+19:42:38.668 WRITE FPDMA QUEUED
60 08 00 d8 e8 70 40 08 2d+19:42:35.131 READ FPDMA QUEUED
60 c0 c8 d0 ef c0 40 08 2d+19:42:35.026 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 1673 -
# 2 Extended offline Interrupted (host reset) 80% 1619 -
# 3 Short offline Completed without error 00% 1586 -
# 4 Short offline Completed without error 00% 1538 -
# 5 Extended offline Completed without error 00% 1512 -
# 6 Short offline Completed without error 00% 1418 -
# 7 Short offline Completed without error 00% 1370 -
# 8 Extended offline Completed without error 00% 1343 -
# 9 Short offline Completed without error 00% 1250 -
#10 Short offline Completed without error 00% 1223 -
#11 Short offline Completed without error 00% 1207 -
#12 Extended offline Interrupted (host reset) 70% 1180 -
#13 Short offline Completed without error 00% 1135 -
#14 Short offline Completed without error 00% 1087 -
#15 Short offline Completed without error 00% 1041 -
#16 Short offline Completed without error 00% 972 -
#17 Short offline Completed without error 00% 923 -
#18 Short offline Completed without error 00% 875 -
#19 Short offline Completed without error 00% 838 -
#20 Extended offline Completed without error 00% 795 -
#21 Short offline Completed without error 00% 772 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.And now /dev/sdg/
Code:
pool: storage-pool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 22.1G in 00:04:10 with 0 errors on Tue Sep 13 16:32:17 2022
config:
NAME STATE READ WRITE CKSUM
storage-pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
1766c859-9231-4127-ad35-882937845b76 ONLINE 0 0 0
89b557d9-cc55-463c-9124-3765cff9aaac ONLINE 0 0 0
4a07f2eb-442c-430d-9db6-0e5fd80302a9 ONLINE 0 0 0
c6287e21-49e0-4ffe-86a1-78a33bbbec70 ONLINE 0 0 0
1c817e68-35de-4b8c-be59-4e98fc1fc9fe ONLINE 0 0 0
28dba822-3f63-4e2c-82ad-8e4467bc81a1 ONLINE 1 0 0
cache
f5456732-9153-44fb-977b-fe5f1021b959 ONLINE 0 0 0Code:
[373382.523345] ata4.00: exception Emask 0x0 SAct 0x3002c SErr 0x0 action 0x0
[373382.531181] ata4.00: irq_stat 0x40000008
[373382.536224] ata4.00: failed command: READ FPDMA QUEUED
[373382.542127] ata4.00: cmd 60/40:80:a0:e5:2f/00:00:44:01:00/40 tag 16 ncq dma 32768 in
res 43/40:40:a0:e5:2f/00:00:44:01:00/40 Emask 0x409 (media error) <F>
[373382.559193] ata4.00: status: { DRDY SENSE ERR }
[373382.564295] ata4.00: error: { UNC }
[373382.599830] ata4.00: configured for UDMA/133
[373382.604722] sd 4:0:0:0: [sdg] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[373382.614658] sd 4:0:0:0: [sdg] tag#16 Sense Key : Medium Error [current]
[373382.621965] sd 4:0:0:0: [sdg] tag#16 Add. Sense: Unrecovered read error - auto reallocate failed
[373382.631377] sd 4:0:0:0: [sdg] tag#16 CDB: Read(16) 88 00 00 00 00 01 44 2f e5 a0 00 00 00 40 00 00
[373382.640944] blk_update_request: I/O error, dev sdg, sector 5438956960 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[373382.652300] zio pool=storage-pool vdev=/dev/disk/by-partuuid/28dba822-3f63-4e2c-82ad-8e4467bc81a1 error=5 type=1 offset=2782598414336 size=32768 flags=180980
[373382.667415] ata4: EH completeCode:
root@truenas[~]# smartctl -a /dev/sdg
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD161KRYZ-01AGBB0
Serial Number: 3XGVATSU
LU WWN Device Id: 5 000cca 285cbfa2f
Firmware Version: 01.01H01
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Sep 15 22:05:48 2022 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 243) Self-test routine in progress...
30% of test remaining.
Total time to complete Offline
data collection: ( 101) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1736) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
2 Throughput_Performance 0x0005 148 148 054 Pre-fail Offline - 48
3 Spin_Up_Time 0x0007 083 083 001 Pre-fail Always - 348 (Average 335)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 100 100 001 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 140 140 020 Pre-fail Offline - 15
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 1728
10 Spin_Retry_Count 0x0013 100 100 001 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36
22 Helium_Level 0x0023 100 100 025 Pre-fail Always - 100
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 119
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 119
194 Temperature_Celsius 0x0002 063 063 000 Old_age Always - 32 (Min/Max 24/45)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 1 occurred at disk power-on lifetime: 1727 hours (71 days + 23 hours)
When the command that caused the error occurred, the device was doing SMART Offline or Self-test.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 43 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 40 80 a0 e5 2f 40 08 4d+07:44:24.627 READ FPDMA QUEUED
61 40 88 08 05 72 40 08 4d+07:44:20.013 WRITE FPDMA QUEUED
61 40 28 88 04 72 40 08 4d+07:44:20.013 WRITE FPDMA QUEUED
60 80 10 c8 23 30 40 08 4d+07:44:17.725 READ FPDMA QUEUED
60 00 18 80 f9 fd 40 08 4d+07:44:17.725 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 1673 -
# 2 Extended offline Interrupted (host reset) 80% 1619 -
# 3 Short offline Completed without error 00% 1586 -
# 4 Short offline Completed without error 00% 1538 -
# 5 Extended offline Completed without error 00% 1516 -
# 6 Short offline Completed without error 00% 1418 -
# 7 Short offline Completed without error 00% 1370 -
# 8 Extended offline Completed without error 00% 1346 -
# 9 Short offline Completed without error 00% 1250 -
#10 Short offline Completed without error 00% 1223 -
#11 Short offline Completed without error 00% 1207 -
#12 Extended offline Interrupted (host reset) 70% 1180 -
#13 Short offline Completed without error 00% 1135 -
#14 Short offline Completed without error 00% 1087 -
#15 Short offline Completed without error 00% 1041 -
#16 Short offline Completed without error 00% 972 -
#17 Short offline Completed without error 00% 923 -
#18 Short offline Completed without error 00% 875 -
#19 Short offline Completed without error 00% 840 -
#20 Extended offline Completed without error 00% 796 -
#21 Short offline Completed without error 00% 772 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
I'm just throwing things at this problem now. I read that you can force disable the drives from going into standby/spin down via the following. So, I've set this option for all drives now.
Code:
hdparm -S 0 /dev/sdc hdparm -S 0 /dev/sdd hdparm -S 0 /dev/sde hdparm -S 0 /dev/sdf hdparm -S 0 /dev/sdg hdparm -S 0 /dev/sdh
Last edited:
Ran the above hdparam cmds and changed the sata cables for all drives a couple of days ago. Got another error on /dev/sde this morning.
Code:
root@truenas[~]# zpool status storage-pool
pool: storage-pool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 0B in 19:18:38 with 0 errors on Mon Sep 19 19:18:39 2022
config:
NAME STATE READ WRITE CKSUM
storage-pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
1766c859-9231-4127-ad35-882937845b76 ONLINE 0 0 0
89b557d9-cc55-463c-9124-3765cff9aaac ONLINE 0 0 0
4a07f2eb-442c-430d-9db6-0e5fd80302a9 ONLINE 0 0 0
c6287e21-49e0-4ffe-86a1-78a33bbbec70 ONLINE 1 0 0
1c817e68-35de-4b8c-be59-4e98fc1fc9fe ONLINE 0 0 0
28dba822-3f63-4e2c-82ad-8e4467bc81a1 ONLINE 0 0 0
cache
f5456732-9153-44fb-977b-fe5f1021b959 ONLINE 0 0 0
errors: No known data errorsCode:
[141751.493277] ata5.00: exception Emask 0x0 SAct 0x80c03001 SErr 0x0 action 0x0
[141751.501098] ata5.00: irq_stat 0x40000008
[141751.505692] ata5.00: failed command: READ FPDMA QUEUED
[141751.511267] ata5.00: cmd 60/08:60:40:28:a5/00:00:2a:03:00/40 tag 12 ncq dma 4096 in
res 43/40:08:40:28:a5/00:00:2a:03:00/40 Emask 0x409 (media error) <F>
[141751.527947] ata5.00: status: { DRDY SENSE ERR }
[141751.532879] ata5.00: error: { UNC }
[141751.564724] ata5.00: configured for UDMA/133
[141751.569530] sd 5:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[141751.579403] sd 5:0:0:0: [sde] tag#12 Sense Key : Medium Error [current]
[141751.586653] sd 5:0:0:0: [sde] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed
[141751.595994] sd 5:0:0:0: [sde] tag#12 CDB: Read(16) 88 00 00 00 00 03 2a a5 28 40 00 00 00 08 00 00
[141751.605513] blk_update_request: I/O error, dev sde, sector 13600368704 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[141751.616892] zio pool=storage-pool vdev=/dev/disk/by-partuuid/c6287e21-49e0-4ffe-86a1-78a33bbbec70 error=5 type=1 offset=6961241227264 size=4096 flags=180880
[141751.631745] ata5: EH completeCode:
root@truenas[~]# smartctl -a /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Gold
Device Model: WDC WD161KRYZ-01AGBB0
Serial Number: 4ZG86LJV
LU WWN Device Id: 5 000cca 2a6c3bbbc
Firmware Version: 01.01H01
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Sep 20 14:08:42 2022 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 101) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1685) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
2 Throughput_Performance 0x0005 149 149 054 Pre-fail Offline - 44
3 Spin_Up_Time 0x0007 084 084 001 Pre-fail Always - 340 (Average 327)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 37
5 Reallocated_Sector_Ct 0x0033 100 100 001 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 001 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 1840
10 Spin_Retry_Count 0x0013 100 100 001 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36
22 Helium_Level 0x0023 100 100 025 Pre-fail Always - 100
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 123
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 123
194 Temperature_Celsius 0x0002 064 064 000 Old_age Always - 31 (Min/Max 23/41)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 2
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 2 occurred at disk power-on lifetime: 1839 hours (76 days + 15 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 43 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 60 40 28 a5 40 08 8d+23:33:16.185 READ FPDMA QUEUED
61 08 b0 30 09 a1 40 08 8d+23:33:10.521 WRITE FPDMA QUEUED
61 08 68 40 8b 69 40 08 8d+23:33:10.521 WRITE FPDMA QUEUED
60 80 b8 70 d8 cb 40 08 8d+23:33:09.234 READ FPDMA QUEUED
60 40 f8 30 d8 cb 40 08 8d+23:33:09.232 READ FPDMA QUEUED
Error 1 occurred at disk power-on lifetime: 1691 hours (70 days + 11 hours)
When the command that caused the error occurred, the device was doing SMART Offline or Self-test.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 43 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 70 c0 68 af c0 40 08 2d+19:42:41.926 READ FPDMA QUEUED
61 40 18 30 b4 de 40 08 2d+19:42:38.668 WRITE FPDMA QUEUED
61 40 d0 f0 b3 de 40 08 2d+19:42:38.668 WRITE FPDMA QUEUED
60 08 00 d8 e8 70 40 08 2d+19:42:35.131 READ FPDMA QUEUED
60 c0 c8 d0 ef c0 40 08 2d+19:42:35.026 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 1778 -
# 2 Extended offline Completed without error 00% 1751 -
# 3 Short offline Completed without error 00% 1673 -
# 4 Extended offline Interrupted (host reset) 80% 1619 -
# 5 Short offline Completed without error 00% 1586 -
# 6 Short offline Completed without error 00% 1538 -
# 7 Extended offline Completed without error 00% 1512 -
# 8 Short offline Completed without error 00% 1418 -
# 9 Short offline Completed without error 00% 1370 -
#10 Extended offline Completed without error 00% 1343 -
#11 Short offline Completed without error 00% 1250 -
#12 Short offline Completed without error 00% 1223 -
#13 Short offline Completed without error 00% 1207 -
#14 Extended offline Interrupted (host reset) 70% 1180 -
#15 Short offline Completed without error 00% 1135 -
#16 Short offline Completed without error 00% 1087 -
#17 Short offline Completed without error 00% 1041 -
#18 Short offline Completed without error 00% 972 -
#19 Short offline Completed without error 00% 923 -
#20 Short offline Completed without error 00% 875 -
#21 Short offline Completed without error 00% 838 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Actually think the problem might be related to this: https://github.com/openzfs/zfs/issues/4713
Also, someone on here mentioned that setting hdparm -B to 255 was letting their WD gold 16tb drives still idle down. Seems that setting to 254 corrected the issue. So, I'm going to give that a shot. https://bbs.archlinux.org/viewtopic.php?id=264279
Similar threads
