two bad drives, unable to offline

seanellis

Dabbler
Joined
Oct 14, 2018
Messages
10
Hello,
this follows up a post I made earlier, reflecting where things are now. I believe that I'm in a situation where I will need to rebuild the pool entirely with two replacement disks. Can anyone comment or confirm this based on what I will post below.

The box was originally inaccessible. After a reboot it will begin to resilver but never completes, as far as I can see, because it will either lock up or reboot after a few hours. The zpool status details vary from invocation to invocation, always having data errors; many now. Two drives, ada1 and ada3 show errors in the log files. The same drives show bad results with a smartctl check. I thought that I might to try to offline what I felt was the worse of the two drives, attempting to take the drive offline isn't allowed due to no valid replicas; because it's resilvering I can go further with that.

TIA

Code:
Sep 12 09:11:17 freenas smartd[24138]: Device: /dev/ada3, 4832 Offline uncorrectable sectors
Sep 12 09:11:17 freenas smartd[24138]: Device: /dev/ada1, 41784 Currently unreadable (pending) sectors

Code:
Sep 12 01:41:47 freenas siisch3: Error while READ LOG EXT
Sep 12 01:41:47 freenas (ada3:siisch3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 28 80 4a 6f 40 1c 01 00 00 00 00
Sep 12 01:41:47 freenas (ada3:siisch3:0:0:0): CAM status: ATA Status Error
Sep 12 01:41:47 freenas (ada3:siisch3:0:0:0): ATA status: 00 ()
Sep 12 01:41:47 freenas (ada3:siisch3:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
Sep 12 01:41:47 freenas (ada3:siisch3:0:0:0): Error 5, Periph was invalidated

ada1
Code:
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   113   092   006    Pre-fail  Always       -       51837688
  3 Spin_Up_Time            0x0003   095   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   078   078   010    Pre-fail  Always       -       29064
  7 Seek_Error_Rate         0x000f   079   060   030    Pre-fail  Always       -       99101519
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       7352
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       22
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       621
188 Command_Timeout         0x0032   100   092   000    Old_age   Always       -       28 29 39
189 High_Fly_Writes         0x003a   084   084   000    Old_age   Always       -       16
190 Airflow_Temperature_Cel 0x0022   060   044   045    Old_age   Always   In_the_past 40 (Min/Max 39/46 #13)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       11
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       76
194 Temperature_Celsius     0x0022   040   056   000    Old_age   Always       -       40 (0 18 0 0 0)
197 Current_Pending_Sector  0x0012   001   001   000    Old_age   Always       -       41784
198 Offline_Uncorrectable   0x0010   001   001   000    Old_age   Offline      -       41784
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       5
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       7351h+39m+01.385s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       25648930986
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3578276333359

ada3
Code:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   082   006    Pre-fail  Always       -       76064911
  3 Spin_Up_Time            0x0003   095   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   079   060   030    Pre-fail  Always       -       89673874
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7996
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       23
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       9091
188 Command_Timeout         0x0032   100   092   000    Old_age   Always       -       282 282 311
189 High_Fly_Writes         0x003a   085   085   000    Old_age   Always       -       15
190 Airflow_Temperature_Cel 0x0022   062   047   045    Old_age   Always       -       38 (Min/Max 37/44)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       11
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       80
194 Temperature_Celsius     0x0022   038   053   000    Old_age   Always       -       38 (0 18 0 0 0)
197 Current_Pending_Sector  0x0012   071   071   000    Old_age   Always       -       4832
198 Offline_Uncorrectable   0x0010   071   071   000    Old_age   Offline      -       4832
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       29
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       7995h+55m+45.376s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       13181833043
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       364588048186

Code:
  pool: main_volume
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 12 09:20:30 2019
        909G scanned at 303M/s, 427G issued at 142M/s, 4.53T total
        20.0G resilvered, 9.19% done, 0 days 08:26:28 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        main_volume                                     ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/4074d0a8-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0
            gptid/41779cad-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0
            gptid/427882a1-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0
            gptid/4388574d-cc10-11e8-9942-d05099aa523b  ONLINE       0     0    23

errors: 2563253 data errors, use '-v' for a list
 
Joined
Oct 18, 2018
Messages
969
Hi @seanellis. I assume your original post is this one? If so, it may provide relevant details.

Sep 12 09:11:17 freenas smartd[24138]: Device: /dev/ada3, 4832 Offline uncorrectable sectors Sep 12 09:11:17 freenas smartd[24138]: Device: /dev/ada1, 41784 Currently unreadable (pending) sectors
These two drives are in very bad shape. As you probably already know, they need to be replaced. They may not yet be completely unavailable or offline, but they are likely on their way out.

The fact that rebooting the machine and checking the results of zpool status yields different results every time suggests a possible issue with your controller or cabling. In addition, both ada1 and ada3 are showing >0 values for 199 UDMA_CRC_Error_Count which is further indication of communication issues between the drives and controller. Based on your earlier post this seems to be at least part of the problem as reseating the cables resulted in the drives coming online.

It would be very helpful for you to post the output of zpool status -v as well as screenshots of the scrub status changing etc. If you do anything to the system or drives it would also be helpful if you report back here exactly what you did and exactly how the system responded; this way folks can track the current state of affairs.

Also, what caused the system to become temporarily unavailable?
 

seanellis

Dabbler
Joined
Oct 14, 2018
Messages
10
Hi Philo,

Hi @seanellis. I assume your original post is this one? If so, it may provide relevant details.
thanks for replying. That is my earlier post in the link you included, yes..
The fact that rebooting the machine and checking the results of zpool status yields different results every time suggests a possible issue with your controller or cabling. In addition, both ada1 and ada3 are showing >0 values for 199 UDMA_CRC_Error_Count which is further indication of communication issues between the drives and controller. Based on your earlier post this seems to be at least part of the problem as reseating the cables resulted in the drives coming online.
The zpool status differences are various. The data errors was originally a double digit number, now it's 2563253 data errors. The READ, WRITE and CKSUM have been different at different times. The STATE of the disks have been different. Now the zpool status shows no data errors.

Code:
> sudo zpool status -v
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:10 with 0 errors on Wed Sep 11 03:45:10 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: main_volume
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 12 12:04:31 2019
        909G scanned at 370M/s, 410G issued at 167M/s, 4.53T total
        16.1G resilvered, 8.84% done, 0 days 07:11:51 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        main_volume                                     ONLINE       0     0    10
          raidz1-0                                      ONLINE       0     0    20
            gptid/4074d0a8-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0
            gptid/41779cad-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0
            gptid/427882a1-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0
            gptid/4388574d-cc10-11e8-9942-d05099aa523b  ONLINE       0     0     0

errors: No known data errors


It would be very helpful for you to post the output of zpool status -v as well as screenshots of the scrub status changing etc. If you do anything to the system or drives it would also be helpful if you report back here exactly what you did and exactly how the system responded; this way folks can track the current state of affairs.

Also, what caused the system to become temporarily unavailable?

I don't know why the system became unavailable. Sometimes I have access to it's one jail via ssh, bit no ssh or GUI to the host. Sometimes neither. Nothing catastrophic in the logs.

I hadn't tried reseating the cables. I will look at that now.

--
Sean
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
I would also check the power connectors. Make certain everything is plugged in firmly.

Why are you using the Syba Raid card? You have 4 disks and the motherboard has 4 ports.. Have you tried connecting all the disks to the motherboard to see if that makes any difference?
 

seanellis

Dabbler
Joined
Oct 14, 2018
Messages
10
That's a great suggestion about the motherboard ports, that should have occurred to me. The machine was assembled by someone else. The four drives had been stacked on each other with no air between them; I can't help but wonder if that was a part of the problem. I've spread them out now, but with errors on both disks and the other difficulties I've had I'm ready to draw the line on further fiddling. On the plus side I've learned a bunch. This thread may as well be considered closed at this point, although I will try taking the card out of the situation though. Thanks!
 
Top