Hi,
I've been running this cheap home server for a more than 6 years:
- Intel i5-650
- ZOTAC Motherboard H55ITX-A-E
- 8GB non-ECC RAM
I recently added a SAS 9207-8I controller.
The server has a pool with two RAID-Z2 vdevs. It has been running fine until a couple of weeks ago, when this happened:
S.M.A.R.T. was (and still is) fine for the drives, I've attached the output in the footer. I run scrubs every month, and never has a single issue.
When I rebooted the server, the affected drives were not visible to the server, so I tried to connect them to the new controller (which had a couple of spare parts), which detected the drives, and I got this:
The drives with the checksum errors are the affected drives.
I read the linked article and I decided to clear the errors, but after a few minutes the number of CKSUM errors went up again:
After a while, as expected, ZFS decided to detach the drives.
Note how the number is the same for both drives: in each subsequent test I’ve done the number of checksum errors *always* increased at the same rate, at the same time for both drives.
Then I’ve ran 4 days of memtest on the server and it was fine.
Today I connected all drives to a new PC, also connecting the power of the affected drives to the new PSU, and I still get the same results, even if apparently the number of checksum errors was increasing more slowly.
Whenever I see the checksum errors, I also see this kind of errors in the logs:
and
The last errors always in pair.
I think I’ve taken out of the equation:
- CPU / motherboard / memory
- controller (I’ve connected the affected drives to both the MB and the controller ports)
- SATA cables
- PSU and power cables
So my question is: are both drives failing at the same time, am I another example of why you should use ECC RAM, or am I missing something else? I’d be ok if it turns out the pool is corrupted, I have backups of what matters, but I’d like to know how to proceed.
Thanks a lot!
Marco
S.M.A.R.T for the affected drives:
I've been running this cheap home server for a more than 6 years:
- Intel i5-650
- ZOTAC Motherboard H55ITX-A-E
- 8GB non-ECC RAM
I recently added a SAS 9207-8I controller.
The server has a pool with two RAID-Z2 vdevs. It has been running fine until a couple of weeks ago, when this happened:
Code:
Feb 18 18:45:25 PigZilla (ada1:ata2:0:1:0): CAM status: Command timeout Feb 18 18:45:25 PigZilla (ada1:ata2:0:1:0): Retrying command Feb 18 18:53:13 PigZilla (ada1:ata2:0:1:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 Feb 18 18:53:13 PigZilla (ada1:ata2:0:1:0): CAM status: Command timeout Feb 18 18:53:13 PigZilla (ada1:ata2:0:1:0): Retrying command Feb 18 18:53:13 PigZilla ada0 at ata2 bus 0 scbus2 target 0 lun 0 Feb 18 18:53:13 PigZilla ada0: <ST4000DM000-1F2168 CC52> s/n W300C91L detached Feb 18 18:53:13 PigZilla ada1 at ata2 bus 0 scbus2 target 1 lun 0 Feb 18 18:53:13 PigZilla ada1: <ST4000DX000-1C5160 CC42> s/n Z1Z0084Z detached Feb 18 18:53:14 PigZilla swap_pager: I/O error - pagein failed; blkno 62,size 8192, error 6 Feb 18 18:53:14 PigZilla GEOM_ELI: vm_fault: pager read error, pid 1200 (devd) Feb 18 18:53:14 PigZilla Device ada0p1.eli destroyed. Feb 18 18:53:14 PigZilla GEOM_ELI: Detached ada0p1.eli on last close. Feb 18 18:53:14 PigZilla GEOM_ELI: Device ada1p1.eli destroyed. Feb 18 18:53:14 PigZilla GEOM_ELI: Detached ada1p1.eli on last close. Feb 18 18:53:14 PigZilla swap_pager: I/O error - pagein failed; blkno 524321,size 16384, error 6 Feb 18 18:53:14 PigZilla vm_fault: pager read error, pid 1200 (devd) Feb 18 18:53:14 PigZilla kernel: Failed to write core file for process devd (error 14) Feb 18 18:53:14 PigZilla kernel: Failed to write core file for process devd (error 14) Feb 18 18:53:14 PigZilla kernel: pid 1200 (devd), uid 0: exited on signal 11 Feb 18 18:53:14 PigZilla zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_14071799796770780259_vdev_8531010438066722682.case. Feb 18 18:53:14 PigZilla zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_14071799796770780259_vdev_15017151504146195539.case. Feb 18 18:53:14 PigZilla (ada0:ata2:0:0:0): Periph destroyed Feb 18 18:53:14 PigZilla (ada1:ata2:0:1:0): Periph destroyed
S.M.A.R.T. was (and still is) fine for the drives, I've attached the output in the footer. I run scrubs every month, and never has a single issue.
When I rebooted the server, the affected drives were not visible to the server, so I tried to connect them to the new controller (which had a couple of spare parts), which detected the drives, and I got this:
Code:
pool: tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: resilvered 12.9M in 0h0m with 0 errors on Mon Feb 20 19:55:24 2017 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/c4205d4d-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 25 gptid/c4a2f333-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 23 gptid/c50380dc-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 0 gptid/c5720042-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 0 gptid/64867e5c-9c51-11e6-8898-00012e2ccfc4 ONLINE 0 0 0 gptid/c64f25ac-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 gptid/0264cad6-d0fd-11e6-a095-00012e2ccfc4 ONLINE 0 0 0 gptid/7ec466a4-d0fd-11e6-a095-00012e2ccfc4 ONLINE 0 0 0 gptid/7fba0667-d0fd-11e6-a095-00012e2ccfc4 ONLINE 0 0 0 gptid/d29093db-f391-11e6-8dca-00012e2ccfc4 ONLINE 0 0 0 gptid/8150e980-d0fd-11e6-a095-00012e2ccfc4 ONLINE 0 0 0 gptid/822ae3a5-d0fd-11e6-a095-00012e2ccfc4 ONLINE 0 0 0 errors: No known data errors
The drives with the checksum errors are the affected drives.
I read the linked article and I decided to clear the errors, but after a few minutes the number of CKSUM errors went up again:
Code:
gptid/c4205d4d-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 20 gptid/c4a2f333-14e3-11e5-930a-00012e2ccfc4 ONLINE 0 0 20
After a while, as expected, ZFS decided to detach the drives.
Note how the number is the same for both drives: in each subsequent test I’ve done the number of checksum errors *always* increased at the same rate, at the same time for both drives.
Then I’ve ran 4 days of memtest on the server and it was fine.
Today I connected all drives to a new PC, also connecting the power of the affected drives to the new PSU, and I still get the same results, even if apparently the number of checksum errors was increasing more slowly.
Whenever I see the checksum errors, I also see this kind of errors in the logs:
Code:
zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_14071799796770780259_vdev_15017151504146195539.case.
and
Code:
daemon.log:Mar 3 11:10:47 PigZilla zfsd: ZFS: Notify class=ereport.fs.zfs.checksum ena=2331296708105873409 parent_guid=2206385843928437561 parent_type=raidz pool=tank pool_context=0 pool_failmode=continue pool_guid=14071799796770780259 subsystem=ZFS timestamp=1488532247 type=ereport.fs.zfs.checksum vdev_guid=15017151504146195539 vdev_path=/dev/gptid/c4a2f333-14e3-11e5-930a-00012e2ccfc4 vdev_type=disk zio_err=0 zio_objset=193 zio_offset=1892544516096 zio_size=4096 daemon.log:Mar 3 11:10:47 PigZilla zfsd: ZFS: Notify class=ereport.fs.zfs.checksum ena=2331296708105873409 parent_guid=2206385843928437561 parent_type=raidz pool=tank pool_context=0 pool_failmode=continue pool_guid=14071799796770780259 subsystem=ZFS timestamp=1488532247 type=ereport.fs.zfs.checksum vdev_guid=8531010438066722682 vdev_path=/dev/gptid/c4205d4d-14e3-11e5-930a-00012e2ccfc4 vdev_type=disk zio_err=0 zio_objset=193 zio_offset=1892544516096 zio_size=4096
The last errors always in pair.
I think I’ve taken out of the equation:
- CPU / motherboard / memory
- controller (I’ve connected the affected drives to both the MB and the controller ports)
- SATA cables
- PSU and power cables
So my question is: are both drives failing at the same time, am I another example of why you should use ECC RAM, or am I missing something else? I’d be ok if it turns out the pool is corrupted, I have backups of what matters, but I’d like to know how to proceed.
Thanks a lot!
Marco
S.M.A.R.T for the affected drives:
Code:
[root@PigZilla] /var/log# smartctl -a /dev/da6 smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda XT Device Model: ST4000DX000-1C5160 Serial Number: Z1Z0084Z LU WWN Device Id: 5 000c50 035a0bd05 Firmware Version: CC42 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Mar 3 12:20:18 2017 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 516) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x1085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 117 094 006 Pre-fail Always - 165030344 3 Spin_Up_Time 0x0003 089 089 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 139 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 084 060 030 Pre-fail Always - 284547522 9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 16959 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 137 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 087 087 000 Old_age Always - 13 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 42950328335 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 069 006 045 Old_age Always In_the_past 31 (13 48 31 27 0) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 96 193 Load_Cycle_Count 0x0032 097 097 000 Old_age Always - 7230 194 Temperature_Celsius 0x0022 031 094 000 Old_age Always - 31 (0 12 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 16836 (126 169 0) 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 496698347 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3420019828 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 16936 - # 2 Short offline Completed without error 00% 16925 - # 3 Short offline Completed without error 00% 16901 - # 4 Short offline Completed without error 00% 16877 - # 5 Short offline Completed without error 00% 16853 - # 6 Short offline Completed without error 00% 16829 - # 7 Short offline Completed without error 00% 16805 - # 8 Short offline Completed without error 00% 16781 - # 9 Extended offline Completed without error 00% 16768 - #10 Short offline Completed without error 00% 16757 - #11 Short offline Completed without error 00% 16733 - #12 Short offline Completed without error 00% 16709 - #13 Short offline Completed without error 00% 16685 - #14 Short offline Completed without error 00% 16661 - #15 Short offline Completed without error 00% 16637 - #16 Short offline Completed without error 00% 16613 - #17 Extended offline Completed without error 00% 16600 - #18 Short offline Completed without error 00% 16589 - #19 Short offline Completed without error 00% 16565 - #20 Short offline Completed without error 00% 16541 - #21 Short offline Completed without error 00% 16517 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. [root@PigZilla] /var/log# smartctl -a /dev/da7 smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Desktop HDD.15 Device Model: ST4000DM000-1F2168 Serial Number: W300C91L LU WWN Device Id: 5 000c50 06977b8b5 Firmware Version: CC52 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5900 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Mar 3 12:20:50 2017 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 612) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 538) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x1085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 45917704 3 Spin_Up_Time 0x0003 092 091 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 73 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 202195012 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15214 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 73 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 096 096 000 Old_age Always - 4 190 Airflow_Temperature_Cel 0x0022 067 060 045 Old_age Always - 33 (Min/Max 30/33) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 55 193 Load_Cycle_Count 0x0032 097 097 000 Old_age Always - 6508 194 Temperature_Celsius 0x0022 033 040 000 Old_age Always - 33 (0 15 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 15058h+44m+26.796s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 31633084667 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 109535584255 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 15191 - # 2 Short offline Completed without error 00% 15180 - # 3 Short offline Completed without error 00% 15156 - # 4 Short offline Completed without error 00% 15132 - # 5 Short offline Completed without error 00% 15109 - # 6 Short offline Completed without error 00% 15085 - # 7 Short offline Completed without error 00% 15061 - # 8 Short offline Completed without error 00% 15037 - # 9 Extended offline Completed without error 00% 15024 - #10 Short offline Completed without error 00% 15013 - #11 Short offline Completed without error 00% 14989 - #12 Short offline Completed without error 00% 14965 - #13 Short offline Completed without error 00% 14941 - #14 Short offline Completed without error 00% 14917 - #15 Short offline Completed without error 00% 14893 - #16 Short offline Completed without error 00% 14869 - #17 Extended offline Completed without error 00% 14856 - #18 Short offline Completed without error 00% 14845 - #19 Short offline Completed without error 00% 14821 - #20 Short offline Completed without error 00% 14797 - #21 Short offline Completed without error 00% 14773 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
