I've been upgrading my 10x 4TB drives to 10x 10TB drives one at a time (raid-z2). The last 4 drives I did two replace drives at a time, while leaving all drives connected (12 drives at once), then removing the old drives. Everyone would be fine and worked very smoothly until the last drives which were getting resilvered and then looking like I had one more drive to replace. I removed the 2 last drives so I had the 10x 10tb new drives, and had a faulted drive, so I restarted and scrubbed. Tons of errors started popping up and all drives are faulted with errors. Not sure what happened or the best way to fix it.
I still have the 10x 4tb drives and don't know if it would be better to start putting those back in, or start over with just the 4tb drives. Although it shows 10 errors, it's nothing critical and I have backups. I can't remove the unavailable drives either with detach because it's giving me the error no replicas available, even though they have been replaced and resilvered.
Anyone have some ideas of what to do next to fix this?
I still have the 10x 4tb drives and don't know if it would be better to start putting those back in, or start over with just the 4tb drives. Although it shows 10 errors, it's nothing critical and I have backups. I can't remove the unavailable drives either with detach because it's giving me the error no replicas available, even though they have been replaced and resilvered.
Anyone have some ideas of what to do next to fix this?
Code:
root@freenas:~ # zpool status -v pool: JailsSSD state: ONLINE scan: scrub repaired 0 in 0 days 00:22:26 with 0 errors on Sun Oct 20 00:22:27 2019 config: NAME STATE READ WRITE CKSUM JailsSSD ONLINE 0 0 0 gptid/225c5dcc-cdbf-11e8-8739-0cc47a40699d ONLINE 0 0 0 errors: No known data errors pool: Media state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Nov 18 18:37:26 2019 1.40T scanned at 1.72G/s, 330G issued at 405M/s, 32.3T total 88.6M resilvered, 1.00% done, 0 days 22:58:25 to go config: NAME STATE READ WRITE CKSUM Media DEGRADED 0 0 86 raidz2-0 DEGRADED 0 0 172 gptid/31f8eaec-0771-11ea-8878-0cc47a40699d DEGRADED 0 0 0 too many errors gptid/e7f2af10-06a4-11ea-a7e0-0cc47a40699d DEGRADED 0 0 0 too many errors gptid/6b83f8a4-0771-11ea-8878-0cc47a40699d FAULTED 3 177 24 too many errors replacing-3 DEGRADED 0 0 0 8734188899632115243 UNAVAIL 0 0 0 was /dev/gptid/af4f2962-85a8-11e5-975c-0cc47a40699d gptid/a655397b-082f-11ea-8137-0cc47a40699d ONLINE 0 0 0 gptid/a2c7f92a-05d9-11ea-9c1e-0cc47a40699d DEGRADED 0 0 0 too many errors replacing-5 DEGRADED 0 0 0 6482462145072973473 UNAVAIL 0 0 0 was /dev/gptid/7bb4f9ec-c0e2-11e6-b93d-0cc47a40699d gptid/d433c504-08f6-11ea-9164-0cc47a40699d ONLINE 0 0 0 gptid/db5cf4e4-051b-11ea-a112-0cc47a40699d DEGRADED 0 0 0 too many errors gptid/9087a6ef-037c-11ea-a888-0cc47a40699d DEGRADED 0 0 0 too many errors gptid/ac95eee3-0284-11ea-b143-0cc47a40699d DEGRADED 0 0 0 too many errors gptid/cf3f5c1e-043a-11ea-8129-0cc47a40699d DEGRADED 0 0 0 too many errors errors: Permanent errors have been detected in the following files: Media/Media/Movies@auto-20191116.0900-1m:<0x0> /var/db/system/syslog-adb946163d914f088dc14617dbc0bec3/log/utx.log /var/db/system/syslog-adb946163d914f088dc14617dbc0bec3/log/samba4/log.winbindd /var/db/system/syslog-adb946163d914f088dc14617dbc0bec3/log/cron Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 Media/Media@auto-20191020.0900-1m:/Music/MP3/****.mp3 <0x41f7>:<0x424> <0x41f7>:<0x428> <0x41f7>:<0x42b> <0x41f7>:<0x42c> <0x41f7>:<0x42e> <0x41f7>:<0x42f> <0x41f7>:<0x430> <0x41f7>:<0x431> pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0 days 01:25:35 with 0 errors on Fri Nov 15 05:10:35 2019 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 gptid/78ecb373-aa82-11e4-b2d8-0cc47a40699d ONLINE 0 0 0 errors: No known data errors