Replaced Disk Duplicates / Not Detaching or Removing (have been researched a bunch of threads).

[Jimbob]

TrueNAS-13.0-U4 Core, AMD 3950x, 128gb ECC RAM, ASROCK - X570M Pro4 MOBO

BACK STORY - Had 2x failed disks. Physically removed them. Added new discs. Chose replace from the menu following the DISK REPLACE GUIDE. Resilvered. Scrubbed & SMART tested. All show no errors post resilvering / no email alerts. Still showing as DEGRADED with ghost drives listed in the pool status.

NOW - I now have the old drives showing in pool / status & pool saying ‘degraded’, with the drives status as offline & unavailable. Out of ideas, need a lil expert halping! Be good to clean this up & fix the degraded state. My N00B guess is a conflict after a few weeks research.

I have tried the online / take offline trick a few times. Just made more confusion as da7 ghosted around.

Detach & remove don’t work (throw errors) & the spare drive sort of moves around / phantom ghosts as well.

Found some threads on this BUT seem close BUT not relevant. Disk set all looks fine (13 in total).

Tried shell commands for 'zpool remove / detach' which didn't work.

Zpool status -v below.

root@truenas[~]# zpool status -v
pool: BigBoi
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0B in 08:54:47 with 0 errors on Sun Jun 18 08:54:47 2023
config:

NAME STATE READ WRITE CKSUM
Barrel DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/16ea0816-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/16a7f8e2-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/16c3758f-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/16cf24a0-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
spare-4 DEGRADED 0 0 0
gptid/a946fb3c-05d6-11ee-8f4a-a8a15938b457 ONLINE 0 0 0
11113997412615612199 OFFLINE 0 0 0 was /dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457
gptid/5592df15-05d9-11ee-8f4a-a8a15938b457 OFFLINE 0 0 0
gptid/1863c5af-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/1842b18e-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/18579aaa-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/18246fdb-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
gptid/18821fbd-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0
spares
gptid/d3683bab-de07-11eb-ab6d-a8a15938b457 UNAVAIL cannot open
gptid/5592df15-05d9-11ee-8f4a-a8a15938b457 AVAIL

errors: No known data errors

pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:07 with 0 errors on Sat Jun 17 03:45:08 2023

LINKS & RESOURCES

Unable to detach missing drive (2017).

Replacement Disk Unable to Detach (2020).

 

[winnielinnie]

Can you post in the correct "code" formatting? It's difficult to read the text.

 

[winnielinnie]

Tried shell commands for 'zpool remove / detach' which didn't work.

What command(s) did you use precisely?

 

[Jimbob]

Can you post in the correct "code" formatting? It's difficult to read the text.

Super N00B here.. trying.. erm.. how do I do that? It's just pasting in as plain text..

 

[Jimbob]

What command(s) did you use precisely?

From Shell =
01 zpool remove 11113997412615612199
02 zpool remove gptid/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457

 

[Jimbob]

Code:

 pool name
root@truenas[~]# zpool status -v
  pool: Barrel
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0B in 08:54:47 with 0 errors on Sun Jun 18 08:54:47 2023
config:

        NAME                                              STATE     READ WRITE CKSUM
        Barrel                                            DEGRADED     0     0   0
          raidz2-0                                        DEGRADED     0     0   0
            gptid/16ea0816-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16a7f8e2-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16c3758f-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16cf24a0-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            spare-4                                       DEGRADED     0     0   0
              gptid/a946fb3c-05d6-11ee-8f4a-a8a15938b457  ONLINE       0     0   0
              11113997412615612199                        OFFLINE      0     0   0  was /dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457
              gptid/5592df15-05d9-11ee-8f4a-a8a15938b457  OFFLINE      0     0   0
            gptid/1863c5af-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/1842b18e-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18579aaa-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18246fdb-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18821fbd-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
        spares
          gptid/d3683bab-de07-11eb-ab6d-a8a15938b457      UNAVAIL   cannot open
          gptid/5592df15-05d9-11ee-8f4a-a8a15938b457      AVAIL

errors: No known data errors

  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:07 with 0 errors on Sat Jun 17 03:45:08 2023
config:

        NAME        STATE     READ WRITE CKSUM

 

[winnielinnie]

zpool remove gptid/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457

Is this a typo? You repeated "gptid" twice.

 

[Jimbob]

Is this a typo? You repeated "gptid" twice.

Yes.. typo.. exact sting here..

 

[winnielinnie]

Can you SSH into your server so that you can copy + paste directly from the terminal?

That way you can enclose the output within [code] tags to result with something like this:

zpool status -v testpool

Code:

  pool: testpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 10:56:04 with 0 errors on Sun Jun 18 10:56:04 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        testpool                                        ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/123456-xxxxxx                         ONLINE       0     0     0
            gptid/123457-xxxxxx                         ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/123458-xxxxxx                         ONLINE       0     0     0
            gptid/123459-xxxxxx                         ONLINE       0     0     0

 

[Jimbob]

Can you SSH into your server so that you can copy + paste directly from the terminal?

That way you can enclose the output within [code] tags to result with something like this:

zpool status -v testpool

Code:

  pool: testpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 10:56:04 with 0 errors on Sun Jun 18 10:56:04 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        testpool                                        ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/123456-xxxxxx                         ONLINE       0     0     0
            gptid/123457-xxxxxx                         ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/123458-xxxxxx                         ONLINE       0     0     0
            gptid/123459-xxxxxx                         ONLINE       0     0     0

cannot open 'gptid/d3683bab-de07-11eb-ab6d-a8a15938b457': invalid character '/'in pool name root@truenas[~]# zpool status -v pool: Barrel state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: scrub repaired 0B in 08:54:47 with 0 errors on Sun Jun 18 08:54:47 2023 config: NAME STATE READ WRITE CKSUM Barrel DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/16ea0816-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/16a7f8e2-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/16c3758f-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/16cf24a0-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 spare-4 DEGRADED 0 0 0 gptid/a946fb3c-05d6-11ee-8f4a-a8a15938b457 ONLINE 0 0 0 11113997412615612199 OFFLINE 0 0 0 was /dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457 gptid/5592df15-05d9-11ee-8f4a-a8a15938b457 OFFLINE 0 0 0 gptid/1863c5af-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/1842b18e-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/18579aaa-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/18246fdb-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 gptid/18821fbd-dd57-11eb-ab6d-a8a15938b457 ONLINE 0 0 0 spares gptid/d3683bab-de07-11eb-ab6d-a8a15938b457 UNAVAIL cannot open gptid/5592df15-05d9-11ee-8f4a-a8a15938b457 AVAIL errors: No known data errors pool: boot-pool state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:00:07 with 0 errors on Sat Jun 17 03:45:08 2023

 

[winnielinnie]

I am unfamiliar with this layout. I thought spares were "pool-wide", and no longer in use once the "missing" drive is replaced and resilvered in the vdev in question.

Is this a 9-wide RAIDZ2 that you had (successfully) replaced a failed drive, in which a hot spare temporarily fulfilled its role in the meantime? After resilvering the RAIDZ2 vdev, the hot spare is still being considered "in use" and refuses to be removed?

Honestly, the layout confuses me, and I'll need to learn what the heck is going on, myself.

@HoneyBadger (or anyone else), are you familiar with a situation and layout seen here?

I'll repaste your output below for the sake of making it easier to read:

Code:

root@truenas[~]# zpool status -v
  pool: Barrel
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0B in 08:54:47 with 0 errors on Sun Jun 18 08:54:47 2023
config:

        NAME                                              STATE     READ WRITE CKSUM
        Barrel                                            DEGRADED     0     0   0
          raidz2-0                                        DEGRADED     0     0   0
            gptid/16ea0816-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16a7f8e2-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16c3758f-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16cf24a0-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            spare-4                                       DEGRADED     0     0   0
              gptid/a946fb3c-05d6-11ee-8f4a-a8a15938b457  ONLINE       0     0   0
              11113997412615612199                        OFFLINE      0     0   0  was /dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457
              gptid/5592df15-05d9-11ee-8f4a-a8a15938b457  OFFLINE      0     0   0
            gptid/1863c5af-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/1842b18e-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18579aaa-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18246fdb-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18821fbd-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
        spares
          gptid/d3683bab-de07-11eb-ab6d-a8a15938b457      UNAVAIL   cannot open
          gptid/5592df15-05d9-11ee-8f4a-a8a15938b457      AVAIL

errors: No known data errors

 

[jgreco]

I am unfamiliar with this layout. I thought spares were "pool-wide", and no longer in use once the "missing" drive is replaced and resilvered in the vdev in question.

It looks fine to me. One of the spares got pulled up to replace what I assume is a failed drive. The failed drives just need to be detached and then the pool will no longer show spare-4 and raidz2-0 to be degraded. Just detach the two OFFLINE drives.

 

[Jimbob]

I am unfamiliar with this layout. I thought spares were "pool-wide", and no longer in use once the "missing" drive is replaced and resilvered in the vdev in question.

Is this a 9-wide RAIDZ2 that you had (successfully) replaced a failed drive, in which a hot spare temporarily fulfilled its role in the meantime? After resilvering the RAIDZ2 vdev, the hot spare is still being considered "in use" and refuses to be removed?

Honestly, the layout confuses me, and I'll need to learn what the heck is going on, myself.

@HoneyBadger (or anyone else), are you familiar with a situation and layout seen here?

I'll repaste your output below for the sake of making it easier to read:

Code:

root@truenas[~]# zpool status -v
  pool: Barrel
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0B in 08:54:47 with 0 errors on Sun Jun 18 08:54:47 2023
config:

        NAME                                              STATE     READ WRITE CKSUM
        Barrel                                            DEGRADED     0     0   0
          raidz2-0                                        DEGRADED     0     0   0
            gptid/16ea0816-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16a7f8e2-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16c3758f-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/16cf24a0-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            spare-4                                       DEGRADED     0     0   0
              gptid/a946fb3c-05d6-11ee-8f4a-a8a15938b457  ONLINE       0     0   0
              11113997412615612199                        OFFLINE      0     0   0  was /dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457
              gptid/5592df15-05d9-11ee-8f4a-a8a15938b457  OFFLINE      0     0   0
            gptid/1863c5af-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/1842b18e-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18579aaa-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18246fdb-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
            gptid/18821fbd-dd57-11eb-ab6d-a8a15938b457    ONLINE       0     0   0
        spares
          gptid/d3683bab-de07-11eb-ab6d-a8a15938b457      UNAVAIL   cannot open
          gptid/5592df15-05d9-11ee-8f4a-a8a15938b457      AVAIL

errors: No known data errors

Thanks for looking at this @Winnie.. I'm a bit confused too.. hope the below helps clarify. Really appreciate a 2nd set of eyes here.

It WAS prior to the failure / replacement ALL ONE BIG POOL of 10 drives with 1x SPARE. MY best visual attempts below. My understanding is it should be pretty straight forward.

POST replacement.. it has gone a bit wonky with 2x SPARE items now shoing..

Being a N00B I thought this is maybe related to being degraded / the old ghost drives / conflict & would go back to the old ONE BIG 10x HDD pool once removed.

 

[winnielinnie]

Really appreciate a 2nd set of eyes here.

Third set of eyes.

You missed @jgreco's post above:

The failed drives just need to be detached and then the pool will no longer show spare-4 and raidz2-0 to be degraded. Just detach the two OFFLINE drives.

 

[winnielinnie]

Just detach the two OFFLINE drives.

Wait, on second reading, it appears @Jimbob already tried to detach the drive, and it spit out an error.

(It's in the first post.)

 

[Jimbob]

It looks fine to me. One of the spares got pulled up to replace what I assume is a failed drive. The failed drives just need to be detached and then the pool will no longer show spare-4 and raidz2-0 to be degraded. Just detach the two OFFLINE drives.

Ta @JG.. the issue I am having is that they won't detach / remove. Even after using the Zpool commands From 'zpool remove 11113997412615612199' OR 'zpool remove gptid/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457'.

Any thoughts here? Thanks for the halpings.

 

[jgreco]

Try a reboot?

 

[Jimbob]

Try a reboot?

Have tried a few times incl. scrub.

That's where the SPARE da7 started to duplicate & ghost around the place last week. See results of remove script below (same happens for detach command). Bit of a weird one by the sounds. Based on my reading, doesn't usually seem to be this tricky.

TIA.

Code:

root@truenas[~]# zpool remove d3683bab-de07-11eb-ab6d-a8a15938b457
cannot open 'd3683bab-de07-11eb-ab6d-a8a15938b457': no such pool
root@truenas[~]# zpool remove gptid/d3683bab-de07-11eb-ab6d-a8a15938b457
cannot open 'gptid/d3683bab-de07-11eb-ab6d-a8a15938b457': invalid character '/'in pool name
root@truenas[~]# zpool remove /dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457
cannot open '/dev/gptid/d3683bab-de07-11eb-ab6d-a8a15938b457': invalid character '/' in pool name

 

[Johnny Fartpants]

Looks like one of your hot-spares kicked in and after replacement didn't go back to being a spare (this is a bug that was only recently fixed in U5.

Have you tried zpool detach Barrel gptid/d3683bab-de07-11eb-ab6d-a8a15938b457

 

[Jimbob]

Thanks Johnny Fartpants.. I been offline a cuppa days.

That 'detach' command did the trick, thank you.. the gptid/d3683bab-de07-11eb-ab6d-a8a15938b457 drive has gone. Did a reboot.

Just need to remove the offline duplicate da7, the POOL is till showing as degraded.

I do get the error [EZFS_NOTSUP] Cannot detach root-level vdevs when trying to detach the offline da7 (SPARE). Any tips to clear that little guy?