Hi All,
My machine has been off for a while because I was out of town. I turned it on and CAM Status: ATA Status Errors started showing up.
This repeats a while. I tried a smart test on all drives. ADA4 had issues:
And then..
Is the drive dead? Should I turn off the whole machine until I can order a replacement? It didn't have issues before as far as I know. Thanks. Very worried
My machine has been off for a while because I was out of town. I turned it on and CAM Status: ATA Status Errors started showing up.
Code:
Jul 4 21:58:18 freenas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 80 03 40 40 00 00 00 01 00 00 Jul 4 21:58:18 freenas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error Jul 4 21:58:18 freenas (ada4:ahcich13:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jul 4 21:58:18 freenas (ada4:ahcich13:0:0:0): RES: 41 40 b0 03 40 40 00 00 00 00 00 Jul 4 21:58:18 freenas (ada4:ahcich13:0:0:0): Retrying command
This repeats a while. I tried a smart test on all drives. ADA4 had issues:
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68EUZN0 Firmware Version: 82.00A82 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Jul 4 22:30:34 2020 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (39060) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 392) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 40 3 Spin_Up_Time 0x0027 181 181 021 Pre-fail Always - 5908 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 43 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 1 9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 22869 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 43 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 16 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 259 194 Temperature_Celsius 0x0022 119 111 000 Old_age Always - 31 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 ATA Error Count: 6 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 6 occurred at disk power-on lifetime: 22869 hours (952 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 00 00 00 00 40 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- e0 00 00 00 00 00 40 08 1d+17:33:23.352 STANDBY IMMEDIATE b0 d5 01 01 4f c2 40 08 1d+17:29:55.339 SMART READ LOG b0 d5 01 06 4f c2 40 08 1d+17:29:55.338 SMART READ LOG Error 5 occurred at disk power-on lifetime: 22850 hours (952 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 10 90 02 40 40 Device Fault; Error: ABRT 16 sectors at LBA = 0x00400290 = 4194960 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 10 90 02 40 40 08 23:04:33.130 READ DMA c8 00 10 90 02 40 40 08 23:04:33.130 READ DMA c8 00 10 90 02 40 40 08 23:04:33.130 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA Error 4 occurred at disk power-on lifetime: 22850 hours (952 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 10 90 02 40 40 Device Fault; Error: ABRT 16 sectors at LBA = 0x00400290 = 4194960 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 10 90 02 40 40 08 23:04:33.130 READ DMA c8 00 10 90 02 40 40 08 23:04:33.130 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA Error 3 occurred at disk power-on lifetime: 22850 hours (952 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 10 90 02 40 40 Device Fault; Error: ABRT 16 sectors at LBA = 0x00400290 = 4194960 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 10 90 02 40 40 08 23:04:33.130 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA Error 2 occurred at disk power-on lifetime: 22850 hours (952 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 10 90 02 40 40 Device Fault; Error: ABRT 16 sectors at LBA = 0x00400290 = 4194960 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 22823 - # 2 Short offline Completed without error 00% 22744 - # 3 Short offline Completed without error 00% 22576 - # 4 Extended offline Completed without error 00% 22487 - # 5 Short offline Completed without error 00% 22408 - # 6 Short offline Completed without error 00% 22168 - # 7 Extended offline Completed without error 00% 22080 - # 8 Short offline Completed without error 00% 22001 - # 9 Short offline Completed without error 00% 21833 - #10 Extended offline Completed without error 00% 21744 - #11 Short offline Completed without error 00% 21666 - #12 Short offline Completed without error 00% 21474 - #13 Extended offline Completed without error 00% 21385 - #14 Short offline Completed without error 00% 21306 - #15 Short offline Completed without error 00% 21138 - #16 Extended offline Completed without error 00% 21050 - #17 Short offline Completed without error 00% 20971 - #18 Short offline Completed without error 00% 20731 - #19 Extended offline Completed without error 00% 20642 - #20 Short offline Completed without error 00% 20563 - #21 Short offline Completed without error 00% 20395 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing Error 2 occurred at disk power-on lifetime: 22850 hours (952 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 10 90 02 40 40 Device Fault; Error: ABRT 16 sectors at LBA = 0x00400290 = 4194960 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA c8 00 10 90 02 40 40 08 23:04:33.129 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 22823 - # 2 Short offline Completed without error 00% 22744 - # 3 Short offline Completed without error 00% 22576 - # 4 Extended offline Completed without error 00% 22487 - # 5 Short offline Completed without error 00% 22408 - # 6 Short offline Completed without error 00% 22168 - # 7 Extended offline Completed without error 00% 22080 - # 8 Short offline Completed without error 00% 22001 - # 9 Short offline Completed without error 00% 21833 - #10 Extended offline Completed without error 00% 21744 - #11 Short offline Completed without error 00% 21666 - #12 Short offline Completed without error 00% 21474 - #13 Extended offline Completed without error 00% 21385 - #14 Short offline Completed without error 00% 21306 - #15 Short offline Completed without error 00% 21138 - #16 Extended offline Completed without error 00% 21050 - #17 Short offline Completed without error 00% 20971 - #18 Short offline Completed without error 00% 20731 - #19 Extended offline Completed without error 00% 20642 - #20 Short offline Completed without error 00% 20563 - #21 Short offline Completed without error 00% 20395 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
And then..
Code:
CRITICAL: July 4, 2020, 9:58 p.m. - Device: /dev/ada4, 4 Currently unreadable (pending) sectors CRITICAL: July 4, 2020, 10:04 p.m. - The volume jvol state is DEGRADED: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
Is the drive dead? Should I turn off the whole machine until I can order a replacement? It didn't have issues before as far as I know. Thanks. Very worried