Hi Folks,
Looking for some assistance here as I'm at my wits end. I recently upgraded my pool, but notably failed to do an extensive burn in on the drives. I replaced 2 of them after detecting heat issues leading to actual failures upon S.M.A.R.T. tests. After resilvering and replacing, everything seemed to be ok. Later, upon scrubbing, ALL drives showed up in a degraded status. I went through the following motions after spending god-only-knows how many hours reading through posts here for similar issues before posting.
1. Smart tests: short, conveyance, and long tests on each of the drives with no errors, and no failures. (More specifically I paid attention to Reallocated_Sector_Ct or Pending which are zero across all drives.)
Results:
2. Checked cabling, routing, etc. reseated all the drives and quadruple checked. Ensured airflow / temp were ok. This time moved cabling from the mobo to the SAS controller for simplicity.
3. Zpool clear + another scrub, resulting in degraded status for all disks again
4. Re-run smart tests (just in case) - still no errors.
5. Ran another zpool status-v revealed a handful of media files where permanent errors have been detected. I haven't gone as far as deleting them yet.
5. My next steps (once the current long re-test is complete) are as follows:
- Bring the old pool back online and attempt to replace the corrupted files
- Delete them and re-scrub
- Replace the controller & / or the cables
- Yeet the server out a window and build something from scratch
Any feedback here would be greatly appreciated. Cheers.
Looking for some assistance here as I'm at my wits end. I recently upgraded my pool, but notably failed to do an extensive burn in on the drives. I replaced 2 of them after detecting heat issues leading to actual failures upon S.M.A.R.T. tests. After resilvering and replacing, everything seemed to be ok. Later, upon scrubbing, ALL drives showed up in a degraded status. I went through the following motions after spending god-only-knows how many hours reading through posts here for similar issues before posting.
1. Smart tests: short, conveyance, and long tests on each of the drives with no errors, and no failures. (More specifically I paid attention to Reallocated_Sector_Ct or Pending which are zero across all drives.)
Results:
Code:
=== START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST12000VN0008-2YS101 Serial Number: ZR906K4V LU WWN Device Id: 5 000c50 0e664bb2a Firmware Version: SC60 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Mar 17 08:05:47 2024 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 559) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1048) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 083 064 044 Pre-fail Always - 217656664 3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 29 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 077 060 045 Pre-fail Always - 48013411 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 522 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 31 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 061 034 000 Old_age Always - 39 (Min/Max 38/52) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 22 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 634 194 Temperature_Celsius 0x0022 039 066 000 Old_age Always - 39 (0 18 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 1 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 100 000 Old_age Offline - 242h+13m+04.236s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 10840822454 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 83557889527 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Conveyance offline Completed without error 00% 522 - # 2 Extended offline Completed without error 00% 521 - # 3 Short offline Completed without error 00% 498 - ---- root@truenas[~]# smartctl -a /dev/ada1 smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST12000VN0008-2YS101 Serial Number: ZRT0SK89 LU WWN Device Id: 5 000c50 0e6e34d38 Firmware Version: SC60 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Mar 17 08:09:55 2024 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1041) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 084 064 044 Pre-fail Always - 233195960 3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 28 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 077 060 045 Pre-fail Always - 48458018 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 522 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 31 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 062 034 000 Old_age Always - 38 (Min/Max 37/51) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 21 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 635 194 Temperature_Celsius 0x0022 038 066 000 Old_age Always - 38 (0 18 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 100 000 Old_age Offline - 242h+20m+46.898s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 10841563747 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 83555086222 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Conveyance offline Completed without error 00% 522 - # 2 Extended offline Completed without error 00% 521 - # 3 Short offline Completed without error 00% 498 - ---- root@truenas[~]# smartctl -a /dev/ada2 smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST12000VN0008-2YS101 Serial Number: ZV70GQVJ LU WWN Device Id: 5 000c50 0e7fa33f7 Firmware Version: SC60 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sun Mar 17 08:10:27 2024 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1036) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 084 064 044 Pre-fail Always - 237145616 3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 29 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 077 060 045 Pre-fail Always - 48020276 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 522 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 31 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 059 042 000 Old_age Always - 41 (Min/Max 41/55) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 22 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 634 194 Temperature_Celsius 0x0022 041 058 000 Old_age Always - 41 (0 19 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 100 000 Old_age Offline - 242h+26m+20.094s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 10839095938 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 83549278837 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Conveyance offline Completed without error 00% 522 - # 2 Extended offline Completed without error 00% 520 - # 3 Short offline Completed without error 00% 498 - ---- === START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST12000VN0008-2YS101 Serial Number: ZRT17J56 LU WWN Device Id: 5 000c50 0e838eae7 Firmware Version: SC60 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sun Mar 17 08:10:54 2024 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1065) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 074 065 044 Pre-fail Always - 25827912 3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 7 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 074 060 045 Pre-fail Always - 24312920 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 106 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 7 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 062 049 000 Old_age Always - 38 (Min/Max 38/51) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 6 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 10 194 Temperature_Celsius 0x0022 038 051 000 Old_age Always - 38 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 100 000 Old_age Offline - 105h+35m+06.696s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 24179512580 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 16914344880 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Conveyance offline Completed without error 00% 105 - # 2 Extended offline Completed without error 00% 104 - # 3 Short offline Completed without error 00% 82 - ----- === START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST12000VN0008-2YS101 Serial Number: ZRT18M7M LU WWN Device Id: 5 000c50 0e8499305 Firmware Version: SC60 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Mar 17 08:11:28 2024 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1060) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 073 064 044 Pre-fail Always - 19635272 3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 7 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 074 060 045 Pre-fail Always - 23665851 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 106 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 7 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 059 045 000 Old_age Always - 41 (Min/Max 40/55) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 7 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 9 194 Temperature_Celsius 0x0022 041 055 000 Old_age Always - 41 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 100 000 Old_age Offline - 105h+28m+29.563s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 10545364484 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 30564777026 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Conveyance offline Completed without error 00% 105 - # 2 Extended offline Completed without error 00% 104 - # 3 Short offline Completed without error 00% 82 -
2. Checked cabling, routing, etc. reseated all the drives and quadruple checked. Ensured airflow / temp were ok. This time moved cabling from the mobo to the SAS controller for simplicity.
3. Zpool clear + another scrub, resulting in degraded status for all disks again
4. Re-run smart tests (just in case) - still no errors.
5. Ran another zpool status-v revealed a handful of media files where permanent errors have been detected. I haven't gone as far as deleting them yet.
Code:
pool: NewPool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 09:43:31 with 10 errors on Mon Mar 18 02:24:48 2024 config: NAME STATE READ WRITE CKSUM MyPool DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/7c875ab1-db1a-11ee-abaf-94de806254ce DEGRADED 0 0 20 too many errors gptid/013ed923-e0df-11ee-ad17-94de806254ce DEGRADED 0 0 20 too many errors gptid/7cb37fd5-db1a-11ee-abaf-94de806254ce DEGRADED 0 0 20 too many errors gptid/7cbedb7c-db1a-11ee-abaf-94de806254ce DEGRADED 0 0 20 too many errors gptid/ed064fd0-e0de-11ee-ad17-94de806254ce DEGRADED 0 0 20 too many errors errors: Permanent errors have been detected in the following files: MyPool/Media@manual-2024-03-15_18-25:/TV Shows/XYZ.whatever MyPool/Media@manual-2024-03-15_18-25:/TV Shows/XYZ.whatever MyPool/Media@manual-2024-03-15_18-25:/Movies/XYZ.whatever MyPool/Media@manual-2024-03-15_18-25:/TV Shows/XYZ.whatever MyPool/Media@manual-2024-03-15_18-25:/TV Shows/XYZ.whatever pool: boot-pool state: ONLINE status: One or more devices are configured to use a non-native block size. Expect reduced performance. action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool. scan: scrub repaired 0B in 00:00:03 with 0 errors on Fri Mar 15 03:45:03 2024 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da3p2 ONLINE 0 0 0 da4p2 ONLINE 0 0 0 block size: 512B configured, 4096B native errors: No known data errors
5. My next steps (once the current long re-test is complete) are as follows:
- Bring the old pool back online and attempt to replace the corrupted files
- Delete them and re-scrub
- Replace the controller & / or the cables
- Yeet the server out a window and build something from scratch
Any feedback here would be greatly appreciated. Cheers.
- Gigabyte GA-Z77N-WIFI
- Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
- 2x Corsair XMS3 8GB DDR3
- 5x 12TB Seagate IronWolf 12TB NAS HDD (ST12000VNZ008) with a RAIDZ2 config
- 1x Samsung SSD 850 + 1x ATA WD Blue SA510 mirrored for the boot pool (1TB each)
- LSI 9207-8i RAID Controller Card
- Onboard NIC: Realtek GbE LAN chip (10/100/1000 Mbit)