Hello, am new to TrueNas Scale and encountered an issue that I am not sure how to deal with. I've done some research but have not found a solution/what to do.
I have a pool with 2 physical drives running in raid 1.
I noticed that there were a lot of notifications stating that increasing amount of sectors cannot be read. One drive was marked as "Degraded" and the second one marked as "Faulty". When running
I do not understand what can be going on and implications of this behaviour.
What does Faulted and Degraded actually mean? If there is an Online drive why did it not correct the data on the degraded drive? Why did it claim that 3 files are unrecoverable but I could later access them? The server spends most of the time idle, with no reads/writes from me, yet it started to randomly fail 10 days ago. Is this likely a hardware issue or I have not configured the pool correctly to store data safely?
TrueNas Scale version: TrueNAS-SCALE-22.02.4
Errors in the dahsboard:
Result of
Results of
My apologies if this a bit of a waffle, I am just very confused and concerned. Do I just need to replace the drive or is this fixable?
Thanks in advance,
Eugene
I have a pool with 2 physical drives running in raid 1.
I noticed that there were a lot of notifications stating that increasing amount of sectors cannot be read. One drive was marked as "Degraded" and the second one marked as "Faulty". When running
zpool status -v
, 3 files were listed that were corrupted. Just to try to check that they are corrupted I tried navigating to them using windows file explorer, and it was horrifically slow. I decided to re-boot the system and after the reboot, one of drive's state changed to "Online" and one of the drive's states changed to "degraded". zpool status -v
suggested that silvering has happened and has fixed ~125mb of data with 0 errors and that no files were corrupted anymore. Just to make sure everything was ok, I ran a scrub task and now the previously "degraded" drive is "faulted" again.I do not understand what can be going on and implications of this behaviour.
What does Faulted and Degraded actually mean? If there is an Online drive why did it not correct the data on the degraded drive? Why did it claim that 3 files are unrecoverable but I could later access them? The server spends most of the time idle, with no reads/writes from me, yet it started to randomly fail 10 days ago. Is this likely a hardware issue or I have not configured the pool correctly to store data safely?
TrueNas Scale version: TrueNAS-SCALE-22.02.4
Errors in the dahsboard:
Code:
Pool MasterYoda state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. The following devices are not healthy: Disk ST4000VN008-2DR166 ZGYA106E is FAULTED 2023-12-17 07:28:26 (America/Los_Angeles)
Code:
CRITICAL Device: /dev/sdc [SAT], 88 Currently unreadable (pending) sectors. 2023-07-15 19:59:56 (America/Los_Angeles) Dismiss
Result of
zpool status -v
:Code:
root@truenas:~# zpool status -v pool: MasterKenobi state: ONLINE scan: scrub repaired 0B in 00:27:01 with 0 errors on Sun Dec 3 00:27:03 2023 config: NAME STATE READ WRITE CKSUM MasterKenobi ONLINE 0 0 0 f8b22328-72b1-45e6-8cf8-ed4cb8007e5e ONLINE 0 0 0 errors: No known data errors pool: MasterYoda state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 4M in 01:54:13 with 0 errors on Sun Dec 17 07:49:51 2023 config: NAME STATE READ WRITE CKSUM MasterYoda DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 b72f7dc9-d952-4878-ba38-822a597684ef ONLINE 0 0 3 509cfa77-39f7-46ac-8610-1cbefea6b8ac FAULTED 22 0 16 too many errors errors: No known data errors pool: boot-pool state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:00:46 with 0 errors on Thu Dec 14 03:45:48 2023 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 sdb3 ONLINE 0 0 0 errors: No known data errors
Results of
smartctl -a /dev/sdc
:Code:
=== START OF INFORMATION SECTION === Model Family: Seagate IronWolf Device Model: ST4000VN008-2DR166 Serial Number: ZGYA106E LU WWN Device Id: 5 000c50 0e36a82be Firmware Version: SC60 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5980 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Dec 17 08:50:21 2023 PST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 581) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 612) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 075 052 044 Pre-fail Always - 157924656 3 Spin_Up_Time 0x0003 094 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 46 5 Reallocated_Sector_Ct 0x0033 098 098 010 Pre-fail Always - 1256 7 Seek_Error_Rate 0x000f 079 060 045 Pre-fail Always - 39398134161 9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 13621 (47 88 0) 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 46 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 281 188 Command_Timeout 0x0032 100 083 000 Old_age Always - 618484859026 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 086 053 040 Old_age Always - 14 (Min/Max 5/33) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 54 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 113 194 Temperature_Celsius 0x0022 014 047 000 Old_age Always - 14 (0 3 0 0 0) 197 Current_Pending_Sector 0x0012 070 069 000 Old_age Always - 2512 198 Offline_Uncorrectable 0x0010 070 069 000 Old_age Offline - 2512 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 13618h+39m+44.408s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 21977262824 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 20110909383 SMART Error Log Version: 1 ATA Error Count: 281 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 281 occurred at disk power-on lifetime: 13619 hours (567 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 53 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 21d+09:14:58.669 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:55.430 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:55.347 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:55.315 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:55.312 READ FPDMA QUEUED Error 280 occurred at disk power-on lifetime: 13619 hours (567 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 53 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 21d+09:14:41.556 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:38.315 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:38.236 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:38.227 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:38.227 READ FPDMA QUEUED Error 279 occurred at disk power-on lifetime: 13619 hours (567 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 53 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 21d+09:14:25.420 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:22.176 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:22.169 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:22.162 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:14:22.155 READ FPDMA QUEUED Error 278 occurred at disk power-on lifetime: 13619 hours (567 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 53 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 21d+09:01:45.718 READ FPDMA QUEUED 60 00 60 ff ff ff 4f 00 21d+09:01:42.227 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:01:42.227 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:01:42.176 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 21d+09:01:42.049 READ FPDMA QUEUED Error 277 occurred at disk power-on lifetime: 13617 hours (567 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 53 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 00 08 ff ff ff 4f 00 21d+07:25:33.718 WRITE FPDMA QUEUED 61 00 20 ff ff ff 4f 00 21d+07:25:33.718 WRITE FPDMA QUEUED 61 00 08 ff ff ff 4f 00 21d+07:25:33.717 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 21d+07:25:33.717 WRITE FPDMA QUEUED 61 00 18 ff ff ff 4f 00 21d+07:25:33.717 WRITE FPDMA QUEUED SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 13620 - # 2 Short offline Completed without error 00% 7609 - # 3 Extended offline Completed without error 00% 6953 - # 4 Short offline Completed without error 00% 6932 - # 5 Extended offline Completed without error 00% 6769 -
My apologies if this a bit of a waffle, I am just very confused and concerned. Do I just need to replace the drive or is this fixable?
Thanks in advance,
Eugene