blueether
Patron
- Joined
- Aug 6, 2018
- Messages
- 259
I started doing a disk replace with a old-ish HDD that I had spare as one was degraded with ~1.5k CKSUM
Now that the resilver has started the error count is climbing and the old-ish drive is throwing unreadable (pending) sectors
I have backups of important data on another box, just not the jails
Now that the resilver has started the error count is climbing and the old-ish drive is throwing unreadable (pending) sectors
I have backups of important data on another box, just not the jails
Code:
Oct 12 07:46:55 freenas ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 Oct 12 07:46:55 freenas ada2: <ST2000DM001-1E6164 CC43> ATA8-ACS SATA 3.x device Oct 12 07:46:55 freenas ada2: Serial Number Z1E2K9TC Oct 12 07:46:55 freenas ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) Oct 12 07:46:55 freenas ada2: 1907729MB (3907029168 512 byte sectors) Oct 12 07:46:55 freenas ada2: quirks=0x1<4K> Oct 12 07:49:33 freenas ZFS: vdev state changed, pool_guid=14900235701281331664 vdev_guid=15589422846635699418 Oct 12 09:29:47 freenas smartd[36233]: Device: /dev/ada2, 8 Currently unreadable (pending) sectors Oct 12 09:29:47 freenas smartd[36233]: Device: /dev/ada2, 8 Offline uncorrectable sectors undefinedOct 12 09:34:45 freenas smartd[36233]: Device: /dev/ada2, 8 Currently unreadable (pending) sectors Oct 12 09:34:45 freenas smartd[36233]: Device: /dev/ada2, 8 Offline uncorrectable sectors Oct 12 09:39:46 freenas smartd[36233]: Device: /dev/ada2, 8 Currently unreadable (pending) sectors Oct 12 09:39:46 freenas smartd[36233]: Device: /dev/ada2, 8 Offline uncorrectable sectors Oct 12 09:44:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors (changed +8) Oct 12 09:44:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors (changed +8) Oct 12 09:49:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 09:49:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 09:54:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 09:54:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 09:59:47 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 09:59:47 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:04:47 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:04:47 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:09:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:09:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:14:45 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:14:45 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:19:47 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:19:47 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:24:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:24:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:29:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:29:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:34:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:34:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors Oct 12 10:39:46 freenas smartd[36233]: Device: /dev/ada2, 16 Currently unreadable (pending) sectors Oct 12 10:39:46 freenas smartd[36233]: Device: /dev/ada2, 16 Offline uncorrectable sectors
Code:
root@freenas:~ # zpool status -v IronWolf pool: IronWolf state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sat Oct 12 07:49:33 2019 586G scanned at 60.6M/s, 133G issued at 13.8M/s, 1.56T total 133G resilvered, 8.34% done, 1 days 06:14:24 to go config: NAME STATE READ WRITE CKSUM IronWolf DEGRADED 0 0 1.67K mirror-0 DEGRADED 0 0 6.66K gptid/79d283da-979d-11e8-a256-94de80b75a94 ONLINE 0 0 6.66K replacing-1 DEGRADED 0 0 4.82K gptid/7a8be22e-979d-11e8-a256-94de80b75a94 DEGRADED 0 0 6.66K too many errors gptid/db372b8d-ec57-11e9-9162-001b21981f60 ONLINE 0 0 4.82K errors: Permanent errors have been detected in the following files: IronWolf/Media:<0x0>
Code:
root@freenas:~ # smartctl -a /dev/ada1 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST2000VN004-2E4164 Serial Number: Z5248MM7 LU WWN Device Id: 5 000c50 0afef4c46 Firmware Version: SC60 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5900 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sat Oct 12 11:07:06 2019 NZDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 128) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 277) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 42188216 3 Spin_Up_Time 0x0003 097 095 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1464 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 080 060 030 Pre-fail Always - 107899659 9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 10109 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 120 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 112 190 Airflow_Temperature_Cel 0x0022 069 048 045 Old_age Always - 31 (Min/Max 27/37) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 53 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1474 194 Temperature_Celsius 0x0022 031 052 000 Old_age Always - 31 (0 15 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 10109 - # 2 Short offline Completed without error 00% 10108 - # 3 Short offline Completed without error 00% 10107 - # 4 Short offline Completed without error 00% 10106 - # 5 Short offline Completed without error 00% 10106 - # 6 Short offline Completed without error 00% 10105 - # 7 Short offline Completed without error 00% 10104 - # 8 Short offline Completed without error 00% 10103 - # 9 Short offline Completed without error 00% 10102 - #10 Short offline Completed without error 00% 10101 - #11 Short offline Completed without error 00% 10100 - #12 Short offline Completed without error 00% 10099 - #13 Short offline Completed without error 00% 10098 - #14 Short offline Completed without error 00% 10097 - #15 Short offline Completed without error 00% 10096 - #16 Short offline Completed without error 00% 10095 - #17 Short offline Completed without error 00% 10095 - #18 Short offline Completed without error 00% 10094 - #19 Short offline Completed without error 00% 10094 - #20 Short offline Completed without error 00% 10093 - #21 Short offline Completed without error 00% 10092 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Code:
root@freenas:~ # smartctl -a /dev/ada2 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST2000DM001-1E6164 Serial Number: Z1E2K9TC LU WWN Device Id: 5 000c50 04f5afad3 Firmware Version: CC43 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sat Oct 12 11:08:24 2019 NZDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: ( 592) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3081) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 132726480 3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 083 083 020 Old_age Always - 17982 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 067 060 030 Pre-fail Always - 21502710673 9 Power_On_Hours 0x0032 086 086 000 Old_age Always - 12688 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 083 083 020 Old_age Always - 17898 183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 2 3 23 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1 190 Airflow_Temperature_Cel 0x0022 061 033 045 Old_age Always In_the_past 39 (5 49 39 20 0) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 092 092 000 Old_age Always - 16211 193 Load_Cycle_Count 0x0032 083 083 000 Old_age Always - 34183 194 Temperature_Celsius 0x0022 039 067 000 Old_age Always - 39 (0 9 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 40 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 40 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1520h+44m+14.992s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 7568979139 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 10795411086 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Self-test routine in progress 90% 12688 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.