SOLVED zpool import, permission denied

[xantra]

Hi,
I searched around for this on the forum, but I don't think this exact issue has been encountered and resolved yet.

I'm running TrueNAS-13.0-U6 (core), and I have a raidz2 pool.
2 of my HDD died recently, so I swapped those out today, but after swapping them out physicaly and rebooting to replace them in zfs, I started having issues when the pool wouldn't show up, saying it's offline.

When I run `zpool import` I get this :

Code:

   pool: tank
     id: 2160150738180114986
  state: DEGRADED
status: One or more devices are missing from the system.
 action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
 config:

        tank                                            DEGRADED
          raidz2-0                                      DEGRADED
            gptid/8969b3fe-de15-11ed-a631-a8a159abc2e3  ONLINE
            gptid/7793d0d8-917a-11e8-8ecf-000c29c07bb1  ONLINE
            gptid/78a4f2b9-917a-11e8-8ecf-000c29c07bb1  ONLINE
            12966275271037246285                        UNAVAIL  cannot open
            gptid/c6c8350a-de15-11ed-a631-a8a159abc2e3  ONLINE
            ada1p2                                      FAULTED  corrupted data

Trying to re-import the pool in the command line with `zpool import 2160150738180114986 tank` results in the following error :

Code:

cannot import 'tank' as 'tank': permission denied
        Destroy and re-create the pool from
        a backup source.

And trying to import it via the web GUI gives the following error :

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 985, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 352, in import_pool
    self.logger.error(
  File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 346, in import_pool
    zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
  File "libzfs.pyx", line 1175, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1203, in libzfs.ZFS.__import_pool
libzfs.ZFSException: permission denied
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
    await self.future
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 981, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1462, in import_pool
    await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1283, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1248, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1254, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1173, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1156, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('permission denied',)

As far as I can see, I should be able to import the pool in degraded state and then swap out the missing drives and re-build the array.
What do you suggest I do ?


Thanks

 

[Davvo]

Never seen a permission denied error importing a pool on this forum... congrats?
You are logged as root, aren't you?

What is the output of zpool import tank?

EDIT:
You are trying to import the tank pool as another pool, with the same name; the command above should solve your issue.

 

[Patrick M. Hausen]

Don't forget -o altroot=/mnt ...

 

[xantra]

Yeah I'm logged in as root via ssh.
Running `zpool import tank -o altroot=/mnt` results in the same "permission denied" error :

Code:

root@truenas[~]# zpool import tank -o altroot=/mnt
cannot import 'tank': permission denied
        Destroy and re-create the pool from
        a backup source.

I tried specifying the device of each remaining and working device as follow, and it seems one of the missing/faulty drive is clashing with the device descriptor of an existing drive for some reason :

Code:

root@truenas[~]# zpool import -d /dev/da0p2 -d /dev/da2p2 -d /dev/ada0p2 -d /dev/ada1p2
   pool: tank
     id: 2160150738180114986
  state: DEGRADED
status: One or more devices are missing from the system.
 action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
 config:

        tank                      DEGRADED
          raidz2-0                DEGRADED
            da0p2                 ONLINE
            ada1p2                ONLINE
            ada0p2                ONLINE
            12966275271037246285  UNAVAIL  cannot open
            da2p2                 ONLINE
            ada1p2                FAULTED  corrupted data

As you can see here, ada1p2 is showing twice here. That's bad right ? This might be what is causing the problem.

 

[Davvo]

Output of zpool import -f -FXn tank?
Was there any encryption?
I wonder, how did you get into this situation? Can you put back the "dead" drives?

 

[xantra]

Running `zpool import -f -FXn tank` doesn't return anything, but it doesn't import the pool either, but an error pops in dmesg saying that ada1p2 can't be mounted. I think something went wrong and it thinks that one of the missing drive is ada1p2 which is the descriptor for another drive now since I had to move one from the sata controller (ada) to the sas controller (da).
Yes my next step is to try to put back the old drives. I don't have physical access to the machine unfortunately, so those operations are always a bit tricky because I have to guide someone else through it all blindly.
I think I should probably have detached the dead drives from the pool before swapping them out for new drives.

 

[Davvo]

Running `zpool import -f -FXn tank` doesn't return anything, but it doesn't import the pool either

It could be a long wait, run it in a tmux process. It won't import the pool, it's just a dry run to see what it says to us.

 

[xantra]

So I managed to get the pool imported after putting one of the old hdd back on the sata controller. I managed to replace and repair one of the dead drive, and now I'm trying to offline the other dead drive and remove it properly before switching off the system and replacing the hdd. But it's showing weirdly again, it's referenced as ada1p2 instead of a volume id, so I'm afraid that once I remove this drive physically I'm going to have the same problem again on my next boot where it's going to think that ada1p2 is present in the pool twice.

Code:

root@truenas[~]# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: resilvered 12K in 00:00:03 with 0 errors on Thu Dec  7 19:29:32 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/8969b3fe-de15-11ed-a631-a8a159abc2e3  ONLINE       0     0     0
            gptid/7793d0d8-917a-11e8-8ecf-000c29c07bb1  ONLINE       0     0     0
            gptid/78a4f2b9-917a-11e8-8ecf-000c29c07bb1  ONLINE       0     0     0
            gptid/3500447b-9514-11ee-9329-a8a159abc2e3  ONLINE       0     0     0
            gptid/c6c8350a-de15-11ed-a631-a8a159abc2e3  ONLINE       0     0     0
            ada1p2                                      FAULTED     16     0     0  external device fault

I think there's some records here that need fixing so the drive is referenced by it's id rather than ata1p2. Any idea how I can resolve this ?
I think this issue came about because I used to be on TrueNAS-13.0-RELEASE which has disk replacement bugged and that's one of the drive I swapped while running that bugged version of truenas.

Would anyone have a recommendation ? Is there such command to change the reference truenas uses for a drive ? Or could I just remove the reference to the drive from the array ?

 

[Davvo]

Please define SATA controller.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

I don't remember 13 having such a bug.

 

[xantra]

I think something's realy wrong, look at the following, it just doesn't make sense, one command says that drive is part of the pool but the next says that it isn't.

Code:

root@truenas[~]# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: resilvered 12K in 00:00:03 with 0 errors on Thu Dec  7 19:29:32 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/8969b3fe-de15-11ed-a631-a8a159abc2e3  ONLINE       0     0     0
            gptid/7793d0d8-917a-11e8-8ecf-000c29c07bb1  ONLINE       0     0     0
            gptid/78a4f2b9-917a-11e8-8ecf-000c29c07bb1  ONLINE       0     0     0
            gptid/3500447b-9514-11ee-9329-a8a159abc2e3  ONLINE       0     0     0
            gptid/c6c8350a-de15-11ed-a631-a8a159abc2e3  ONLINE       0     0     0
            ada1p2                                      FAULTED     16     0     0  external device fault

errors: No known data errors

root@truenas[~]# glabel status
                                      Name  Status  Components
gptid/aa8c4733-f8a3-11ec-a4aa-25ad20ffa947     N/A  nvd0p1
gptid/aa94a622-f8a3-11ec-a4aa-25ad20ffa947     N/A  nvd1p1
gptid/8969b3fe-de15-11ed-a631-a8a159abc2e3     N/A  da0p2
gptid/c6c8350a-de15-11ed-a631-a8a159abc2e3     N/A  da2p2
gptid/78a4f2b9-917a-11e8-8ecf-000c29c07bb1     N/A  ada0p2
gptid/7793d0d8-917a-11e8-8ecf-000c29c07bb1     N/A  ada2p2
gptid/3500447b-9514-11ee-9329-a8a159abc2e3     N/A  da1p2
gptid/7c162b9d-917a-11e8-8ecf-000c29c07bb1     N/A  ada1p2
gptid/aa97bfbc-f8a3-11ec-a4aa-25ad20ffa947     N/A  nvd1p3
gptid/aa8f8b45-f8a3-11ec-a4aa-25ad20ffa947     N/A  nvd0p3

root@truenas[~]# zpool replace -f tank ada1p2 /dev/gptid/7c162b9d-917a-11e8-8ecf-000c29c07bb1
invalid vdev specification
the following errors must be manually repaired:
/dev/gptid/7c162b9d-917a-11e8-8ecf-000c29c07bb1 is part of active pool 'tank'

root@truenas[~]# zpool offline tank /dev/gptid/7c162b9d-917a-11e8-8ecf-000c29c07bb1
cannot offline /dev/gptid/7c162b9d-917a-11e8-8ecf-000c29c07bb1: no such device in pool

 

[xantra]

Please define SATA controller.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

I don't remember 13 having such a bug.

I have some drives attached to the motherboards's sata controller, they are the /dev/ada* devices, and then I have some devices attached to a PCIe sas controller, they are the /dev/da* devices.
/dev/ada1 is the faulted drive, and is identified as such in the pool. If I take it out, ada2 will become ada1 on the next reboot, and as the pool identifies that faulted drive by /dev/ada1 instead of it's uuid, then there's a problem because the pool now thinks that two drives are the same device.
I need to edit my pool so ada1 is identified by it's uuid instead, allowing me to reboot without issues while I took that drive out.
Otherwise I need to find a way to get another power connector so I can add the replacement drive without having to disconnect the dead drive.

 

[xantra]

I think I'm out of the woods now, I managed to plug another drive in while keeping the dead one in, and ran a replace through the gui, and now the new drive has been registered properly from it's uuid and the old one will just disappear once the resilvering is complete.
Not sure what had happened to that drive before, it was definitely a pain to sort this out.