Disk Faulted

[adrianwi]

Just noticed this and the lack of SMART test e-mails for the last month :-(

I'll fix the later after the former, but it looks like the offending disk has removed itself as I can't see anything to detach.

Is it just a case of identifying the correct drive and replacing? Do I need to do anything to add it back into the pool and start it resilvering?

Code:

root@freenas1:~ # zpool status
  pool: APEpool1
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: scrub repaired 0 in 0 days 18:42:48 with 0 errors on Sun Jun 16 18:43:13 2019
config:

    NAME                                            STATE     READ WRITE CKSUM
    APEpool1                                        DEGRADED     0     0     0
      raidz2-0                                      DEGRADED     0     0     0
        gptid/e0e32ce5-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/57eade3e-ddae-11e6-b31a-0cc47a3359a2  ONLINE       0     0     0
        gptid/e1c2cef3-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/e247d320-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/5fc4958b-29a8-11e6-ac13-0cc47a3359a2  ONLINE       0     0     0
        gptid/e3350d15-0616-11e6-bbb6-0cc47a3359a2  FAULTED      2    28     0  too many errors
        gptid/e3a7c3e9-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/db8fac68-75d5-11e7-b631-0cc47a3359a2  ONLINE       0     0     0
        gptid/e4990a6c-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:25 with 0 errors on Thu Jun  6 03:45:25 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      ada0p2    ONLINE       0     0     0

errors: No known data errors
root@freenas1:~ #

Thanks

 

[joeschmuck]

To locate the suspect drive use

Code:

glabel status

and it will list the drives related to the gptid numbers.

Run a SMART Long Test of this drive, it may have failures. Report the failures. You can also look at the Hard Drive Troubleshooting Guide (see link in my signature) to help you out.

Good Luck!

 

[adrianwi]

The drive seems to have removed itself completely, as I'm only seeing 8 of the HDDs (pool) and the 1 SSD (boot)

Code:

root@freenas1:~ # glabel status
                                      Name  Status  Components
gptid/2fe896e5-3ce2-11e7-a1b2-0cc47a3359a2     N/A  ada0p1
gptid/e0e32ce5-0616-11e6-bbb6-0cc47a3359a2     N/A  ada1p2
gptid/5fc4958b-29a8-11e6-ac13-0cc47a3359a2     N/A  da0p2
gptid/e247d320-0616-11e6-bbb6-0cc47a3359a2     N/A  da1p2
gptid/e1c2cef3-0616-11e6-bbb6-0cc47a3359a2     N/A  da2p2
gptid/57eade3e-ddae-11e6-b31a-0cc47a3359a2     N/A  da3p2
gptid/e4990a6c-0616-11e6-bbb6-0cc47a3359a2     N/A  da4p2
gptid/db8fac68-75d5-11e7-b631-0cc47a3359a2     N/A  da5p2
gptid/e3a7c3e9-0616-11e6-bbb6-0cc47a3359a2     N/A  da6p2
root@freenas1:~ #

I should be able to work out which one to swap, as I labelled them all and have the serial numbers of drives still connected in the Storage > Disk WUI view.

 

[sretalla]

Just to be clear, gptid isn't related to the serial number. You can see the serial numbers listed in the GUI disks view.

 

[adrianwi]

I knew that, but better safe than sorry, so thanks!

 

[joeschmuck]

Yup, dropped drive. Could be a cable connection if you had done anything internal or just moved the computer.

 

[adrianwi]

Agh!

So I took the right drive out and replaced with a new one. Started back up no problems but still degraded:

Code:

root@freenas1:~ # zpool status
  pool: APEpool1
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0 days 18:42:48 with 0 errors on Sun Jun 16 18:43:13 2019
config:

    NAME                                            STATE     READ WRITE CKSUM
    APEpool1                                        DEGRADED     0     0     0
      raidz2-0                                      DEGRADED     0     0     0
        gptid/e0e32ce5-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/57eade3e-ddae-11e6-b31a-0cc47a3359a2  ONLINE       0     0     0
        gptid/e1c2cef3-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/e247d320-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/5fc4958b-29a8-11e6-ac13-0cc47a3359a2  ONLINE       0     0     0
        13291488237085976922                        UNAVAIL      0     0     0  was /dev/gptid/e3350d15-0616-11e6-bbb6-0cc47a3359a2
        gptid/e3a7c3e9-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0
        gptid/db8fac68-75d5-11e7-b631-0cc47a3359a2  ONLINE       0     0     0
        gptid/e4990a6c-0616-11e6-bbb6-0cc47a3359a2  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:25 with 0 errors on Thu Jun  6 03:45:25 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      ada0p2    ONLINE       0     0     0

errors: No known data errors
root@freenas1:~ #

Tried zpool online:

Code:

zpool online -e APEpool1 13291488237085976922
warning: device '13291488237085976922' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
root@freenas1:~ #

and then zpool replace:

Code:

root@freenas1:~ # zpool replace -f APEpool1 /dev/gptid/e3350d15-0616-11e6-bbb6-0cc47a3359a2 13291488237085976922
cannot open '13291488237085976922': no such GEOM provider
must be a full path or shorthand device name
root@freenas1:~ #

but think I'm missing something obvious? Maybe I should have removed the old drive first, but as it wasn't in the WUI I'd assumed I didn't need to do anything.

UPDATE

The bit I missed was RTFM :D

I did think it was a little strange that I couldn't find how to do this stuff in the new UI when it was there in the old one. A quick look in the manual and I realised I'd completely overlooked the cog icon and the pool settings!

I used the replace option in here and it's now happily resilvering. Only 16 days to go, although I expect that will be shorted somewhat once it's been running a little longer!

 

[joeschmuck]

The bit I missed was RTFM :D

That is funny that you said that because when I read your first sentence this is the exact thought I had. Then I finished reading your posting and laughed. Glad you are resilvering. A 4TB drive can take a few days if you have a ton of small files, but I doubt 16 days.

 

[adrianwi]

No, I was down to 5 hours this morning so should be finished later today.