recover zpool from failed drive.

[msanger]

So I have a raiz1 pool configured with 3 3TB drives, about a month ago ada1 began having issues and failing chksum

root@freenas:/mnt/zfs # zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:10:02 with 0 errors on Fri Nov 13 03:55:02 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

pool: zfs
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub repaired 42.0M in 2 days 23:43:56 with 0 errors on Tue Nov 10 23:44:49 2020
config:

NAME STATE READ WRITE CKSUM

zfs ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/3873cdde-828a-11e4-bde5-842b2bac8a2d ONLINE 0 0 0
gptid/2893e7df-82a2-11e4-858f-842b2bac8a2d ONLINE 0 0 1
gptid/24df54ca-82f4-11e4-aa0e-842b2bac8a2d ONLINE 0 0 0

errors: No known data errors

The system recovered to which I thought I have time and began trying to prepare for a drive replacement, I later found the following status

root@freenas:~ # zpool status

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:11:42 with 0 errors on Wed Jan 6 03:56:42 2021
config:

NAME STATE READ WRITE CKSUM

freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

pool: zfs
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 119M in 1 days 05:47:11 with 0 errors on Mon Dec 14 05:49:24 2020
config:

NAME STATE READ WRITE CKSUM

zfs DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
gptid/3873cdde-828a-11e4-bde5-842b2bac8a2d ONLINE 0 0 0
13634946646434885157 REMOVED 0 0 0 was /dev/gptid/2893e7df-82a2-11e4-858f-842b2bac8a2d
gptid/24df54ca-82f4-11e4-aa0e-842b2bac8a2d ONLINE 0 0 0

errors: No known data errors

To which I followed and ordered a replacement drive, once it arrived I followed

https://www.ixsystems.com/documentation/freenas/11.3-U1/storage.html#replacing-a-failed-disk

I offlined the drive, and since i do not have a hot-swappable system I powered down and replaced the drive. To only be greeted by a non functioning zpool this is what I am currently seeing

root@freenas:~ # zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:11:42 with 0 errors on Wed Jan 6 03:56:42 2021
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

root@freenas:~ # zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 14.4G 4.00G 10.4G - - - 27% 1.00x ONLINE -


root@freenas:~ # zpool import
pool: zfs
id: 15740514284552979682
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://illumos.org/msg/ZFS-8000-3C
config:

zfs UNAVAIL insufficient replicas
raidz1-0 UNAVAIL insufficient replicas
3060771620888435127 UNAVAIL cannot open
13634946646434885157 OFFLINE
gptid/24df54ca-82f4-11e4-aa0e-842b2bac8a2d ONLINE

I am aware of the noobish nature of my question but any help to dig myself out of this mess is appreciated, and I can provide any and all missing infos to help in fixing. prerequisite 5 units will be added to the tab for future enjoyment.

Thank you

 

[msanger]

Looking at the status messages closer, it seems that one of the other "good" drives is not being recognized by the system. I will reconnect everything and try again.

 

[msanger]

That was the issue, reseated all connections second good drive showed up and allowed me to replace the old drive with the new and resilve the pool.

 

[Chris Moore]

Looks like we are late to the party. I am happy you were able to resolve the issue. Good job troubleshooting. I can't tell you the number of times that reseating the connection fixed a fault. Always worth trying.

 

[Etorix]

Good to know you solved the issue.
Note thet it is not necessary to offline a drive to replace it. You can plug the new drive to a free SATA port, go to Pool>Status, and ask to replace the drive while it's still in, so you resilver with full redundancy.