tomf84
Dabbler
- Joined
- Apr 4, 2013
- Messages
- 10
Hi
Trying to work out if I am just the victim of bad luck here - and actions I should take to dig myself back out.
I have a RAIDZ1 pool of 4x 1TB SATA drives. A few days ago, I got some READ errors on ada1 (6659063e-3dd8-11e1-b2b4-6cf049d3b305), so I was just getting ready to offline it ready to power down the box to replace the disk.
Powered it down this morning, booted it up this evening (untouched) to find "zfs status" now giving me this:
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: scrub canceled on Thu Apr 4 20:15:05 2013
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
gptid/65e94683-3dd8-11e1-b2b4-6cf049d3b305 ONLINE 0 0 0
gptid/6659063e-3dd8-11e1-b2b4-6cf049d3b305 ONLINE 0 0 0
gptid/66cac68e-3dd8-11e1-b2b4-6cf049d3b305 ONLINE 0 0 0
1765056975075706550 UNAVAIL 0 0 0 was /dev/gptid/6759ec40-3dd8-11e1-b2b4-6cf049d3b305
(I started a scrub then cancelled it straight away.)
Before all of this, glabel status showed the following. Now the line with ada3 is missing.
gptid/65e94683-3dd8-11e1-b2b4-6cf049d3b305 N/A ada0p2
gptid/6659063e-3dd8-11e1-b2b4-6cf049d3b305 N/A ada1p2
gptid/66cac68e-3dd8-11e1-b2b4-6cf049d3b305 N/A ada2p2
gptid/6759ec40-3dd8-11e1-b2b4-6cf049d3b305 N/A ada3p2
ada3 appears in /dev but ada3p2 does not - unlike all of the others.
[root@bunker] ~# ls /dev/ada*
/dev/ada0 /dev/ada0p1 /dev/ada0p2 /dev/ada1 /dev/ada1p1 /dev/ada1p2 /dev/ada2 /dev/ada2p1 /dev/ada2p2 /dev/ada3
I can run smartctl -a against ada3, and to my dismay I see errors:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 067 067 011 Pre-fail Always - 10680
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 533
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10061
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 23230
10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 322
13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 025 025 000 Pre-fail Always - 75
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 083 037 000 Old_age Always - 17 (Min/Max 10/17)
194 Temperature_Celsius 0x0022 074 036 000 Old_age Always - 26 (4 227 26 10 0)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 55577
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 5
200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 1
201 Soft_Read_Error_Rate 0x000a 253 253 000 Old_age Always - 0
A couple of days ago this read as follows (for the same drive):
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 069 069 011 Pre-fail Always - 9950
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 532
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10061
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 23229
10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 321
13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 000 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 069 037 000 Old_age Always - 31 (Min/Max 30/43)
194 Temperature_Celsius 0x0022 067 036 000 Old_age Always - 33 (4 227 43 30 0)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 238100568
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 1
200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0
201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0
Have I just encountered the worst enemy of RAIDZ1 - failure of two disks?
The whole pool is backed up to an external 3TB drive (about 2.5TB of data exists), and the backup was verified on the external drive with a zfs scrub. So all is not absolutely lost if my pool is toast.
However I'd really like to recover this pool - partly as a learning exercise, partly because I can't be THAT unlucky and I'm not seeing something obvious. Right?
Just really don't know what my next move should be.
Cheers....
Trying to work out if I am just the victim of bad luck here - and actions I should take to dig myself back out.
I have a RAIDZ1 pool of 4x 1TB SATA drives. A few days ago, I got some READ errors on ada1 (6659063e-3dd8-11e1-b2b4-6cf049d3b305), so I was just getting ready to offline it ready to power down the box to replace the disk.
Powered it down this morning, booted it up this evening (untouched) to find "zfs status" now giving me this:
pool: tank0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: scrub canceled on Thu Apr 4 20:15:05 2013
config:
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
gptid/65e94683-3dd8-11e1-b2b4-6cf049d3b305 ONLINE 0 0 0
gptid/6659063e-3dd8-11e1-b2b4-6cf049d3b305 ONLINE 0 0 0
gptid/66cac68e-3dd8-11e1-b2b4-6cf049d3b305 ONLINE 0 0 0
1765056975075706550 UNAVAIL 0 0 0 was /dev/gptid/6759ec40-3dd8-11e1-b2b4-6cf049d3b305
(I started a scrub then cancelled it straight away.)
Before all of this, glabel status showed the following. Now the line with ada3 is missing.
gptid/65e94683-3dd8-11e1-b2b4-6cf049d3b305 N/A ada0p2
gptid/6659063e-3dd8-11e1-b2b4-6cf049d3b305 N/A ada1p2
gptid/66cac68e-3dd8-11e1-b2b4-6cf049d3b305 N/A ada2p2
gptid/6759ec40-3dd8-11e1-b2b4-6cf049d3b305 N/A ada3p2
ada3 appears in /dev but ada3p2 does not - unlike all of the others.
[root@bunker] ~# ls /dev/ada*
/dev/ada0 /dev/ada0p1 /dev/ada0p2 /dev/ada1 /dev/ada1p1 /dev/ada1p2 /dev/ada2 /dev/ada2p1 /dev/ada2p2 /dev/ada3
I can run smartctl -a against ada3, and to my dismay I see errors:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 067 067 011 Pre-fail Always - 10680
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 533
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10061
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 23230
10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 322
13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 025 025 000 Pre-fail Always - 75
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 083 037 000 Old_age Always - 17 (Min/Max 10/17)
194 Temperature_Celsius 0x0022 074 036 000 Old_age Always - 26 (4 227 26 10 0)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 55577
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 5
200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 1
201 Soft_Read_Error_Rate 0x000a 253 253 000 Old_age Always - 0
A couple of days ago this read as follows (for the same drive):
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 069 069 011 Pre-fail Always - 9950
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 532
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10061
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 23229
10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 321
13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 000 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 069 037 000 Old_age Always - 31 (Min/Max 30/43)
194 Temperature_Celsius 0x0022 067 036 000 Old_age Always - 33 (4 227 43 30 0)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 238100568
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 1
200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0
201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0
Have I just encountered the worst enemy of RAIDZ1 - failure of two disks?
The whole pool is backed up to an external 3TB drive (about 2.5TB of data exists), and the backup was verified on the external drive with a zfs scrub. So all is not absolutely lost if my pool is toast.
However I'd really like to recover this pool - partly as a learning exercise, partly because I can't be THAT unlucky and I'm not seeing something obvious. Right?
Just really don't know what my next move should be.
Cheers....