pool UNAVAIL 'missing device' / all disks ONLINE, but not: The pool cannot be imported due to damaged devices or data.

[ant0n]

Hi, would you be able to help me please?

The old TruesNAS 13 was a VM in Proxmox with 2 4TB hardware disks passed through to the VM and used in pool with RAID1
The new TrueNAS 13 is on the the same host but not a VM anymore, i have overwritten proxmox with TrueNAS
I did not change the LSI controller nor the disks in the system.
The situation is like this after the restore of a TrueNASbackup which is 2 months old:

```
root@antruenas[~]# zpool import
pool: antruenaspool
id: 16884376106474008389
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

antruenaspool UNAVAIL missing device
mirror-0 ONLINE
gptid/b8d74017-a38d-11ec-af09-75cdfa3618e1 ONLINE
gptid/b8f26292-a38d-11ec-af09-75cdfa3618e1 ONLINE
logs
gptid/ba838e33-a38d-11ec-af09-75cdfa3618e1 UNAVAIL cannot open


root@antruenas[~]# zpool status -v
pool: boot-pool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da2p2 ONLINE 0 0 0
da3p2 ONLINE 0 0 0

errors: No known data errors


root@antruenas[~]# camcontrol devlist
<ATA WDC WD40EFAX-68J 0A83> at scbus0 target 14 lun 0 (pass0,da0)
<ATA WDC WD40EFAX-68J 0A83> at scbus0 target 15 lun 0 (pass1,da1)
<ATA Samsung SSD 840 5B0Q> at scbus0 target 16 lun 0 (pass2,da2)
<ATA Samsung SSD 840 5B0Q> at scbus0 target 17 lun 0 (pass3,da3)


root@antruenas[~]# zpool import b8d74017-a38d-11ec-af09-75cdfa3618e1
cannot import 'b8d74017-a38d-11ec-af09-75cdfa3618e1': no such pool available


root@antruenas[~]# zpool import 16884376106474008389
cannot import 'antruenaspool': pool was previously in use from another system.
Last accessed by (hostid=6d5df28d) at Sat Dec 24 19:39:20 2022
The pool can be imported, use 'zpool import -f' to import the pool.


root@antruenas[~]# zpool import -f
pool: antruenaspool
id: 16884376106474008389
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

antruenaspool UNAVAIL missing device
mirror-0 ONLINE
gptid/b8d74017-a38d-11ec-af09-75cdfa3618e1 ONLINE
gptid/b8f26292-a38d-11ec-af09-75cdfa3618e1 ONLINE
logs
gptid/ba838e33-a38d-11ec-af09-75cdfa3618e1 UNAVAIL cannot open


root@antruenas[~]# zpool import -f 16884376106474008389
The devices below are missing or corrupted, use '-m' to import the pool anyway:
gptid/ba838e33-a38d-11ec-af09-75cdfa3618e1 [log]
cannot import 'antruenaspool': one or more devices is currently unavailable

root@antruenas[~]# camcontrol devlist
<ATA WDC WD40EFAX-68J 0A83> at scbus0 target 14 lun 0 (pass0,da0)
<ATA WDC WD40EFAX-68J 0A83> at scbus0 target 15 lun 0 (pass1,da1)
<ATA Samsung SSD 840 5B0Q> at scbus0 target 16 lun 0 (pass2,da2)
<ATA Samsung SSD 840 5B0Q> at scbus0 target 17 lun 0 (pass3,da3)


root@antruenas[~]# zpool status
pool: boot-pool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da2p2 ONLINE 0 0 0
da3p2 ONLINE 0 0 0

errors: No known data errors


root@antruenas[~]# glabel status
Name Status Components
gptid/5a792bec-8405-11ed-9362-3ca82aa09300 N/A da2p1
gptid/5ad0817c-8405-11ed-9362-3ca82aa09300 N/A da3p1
gptid/b8f26292-a38d-11ec-af09-75cdfa3618e1 N/A da1p2
gptid/b8d74017-a38d-11ec-af09-75cdfa3618e1 N/A da0p2
gptid/b8893592-a38d-11ec-af09-75cdfa3618e1 N/A da0p1
gptid/b8add411-a38d-11ec-af09-75cdfa3618e1 N/A da1p1


root@antruenas[~]# glabel status | grep b8d74017-a38d-11ec-af09-75cdfa3618e1
gptid/b8d74017-a38d-11ec-af09-75cdfa3618e1 N/A da0p2


root@antruenas[~]# glabel status | grep b8f26292-a38d-11ec-af09-75cdfa3618e1
gptid/b8f26292-a38d-11ec-af09-75cdfa3618e1 N/A da1p2
```

How to get antruenaspool back up and running?


is it safe to use -m ?

`zpool import -f -m 16884376106474008389`

Thank you

 

[joeschmuck]

gptid/ba838e33-a38d-11ec-af09-75cdfa3618e1 UNAVAIL cannot open

Looks like you are missing the log drive you had setup. By chance was this a virtual drive? If it were a physical drive then I'd reconnect it and try again. If this were a virtual drive then I would tell you to investigate how to remove a log drive.

You might need to restore Proxmox and then remove the log drive, then try bare metal again, but I'm not that familiar with log drives nor Proxmox at all. Worst case is use Proxmox to backup your data from the drives, unless you have a backup already, then start all over.

Out of curiosity, when you installed TrueNAS on bare metal, did you restore a configuration file? If you did, then I'd revert back to a clean in stall and then try to import your pool. Also, did you export your pool before you got rid of Proxmox?

Good luck.

 

[ant0n]

Thank you for your answer

i am just guessing the next abstracts:

It is possible that I had a log device on a 3rd physical drive. there are two HDDs and two SSDs in the system and three disks have been physically passed through to the TrueNas VM.

I actually did not think about anything like exporting the pool. i just did a backup of the VM and thought i will be able to restore the backup.

it did not work out as i meanwhile accidentially 'destroyed' the proxmox host with an irrevesible action. (adding to a proxmox cluster and removing the host from that cluster - solution of proxmox officially is, if you remove a proxmox host from a cluster you will have to reinstall it to have a good working single host ;))) )

i think proxmox was installed on the one SSD which also contained the truenas vm, that was NOT passed through to the VM.

i have a backup of the VM and i have a backup of the /etc folder of proxmox.
I have a backup of the config or TrueNAS and i have a backup of all data of the antruenaspool just not the plex jail and the videos.

i think that the second ssd which was also passed through to the VM contained the log.

now, unluckily the truenas was installed on the two ssds in raid 1 mode, so that is why the ssd that might have been the log device was formatted and newly partitioned ;)

i do not remember why i added this log device, maybe it was mandatory to add a third disk or just a recommendation that i forgot about because i did not take notes ;)

so the -m says according to manual and forums, that i try to restore a pool that originally contained a log without the log (256 GB in my case) and might lose the latest transactions, which are most likely ALL on my backup.


thanks for reading ;)

 

[joeschmuck]

You certainly can try using the -m parameter and maybe it will work. I would find it difficult to believe it would cause any harm. And if you shutdown TrueNAS normally then you would not have any transactions that were not flushed.

Hope it all works out.

 

[ant0n]

Hi joeschmuck

thank you for supporting me yesterday,
I was trying the -m in the following order

zpool import -a
did not work

zpool import -a -f
did not work

zpool import -a -f -m
worked

The Pool was Degraded, it was missing its log device, so i removed the log device

I double checked some of the last transactions that i remembered, they are all on the storage
the plex jail is also on the storage
all VMs that are using the storage, eg photoprism and nextcloud and the SMB share worked and found the storage
after the import it was though MANDATORY TO REBOOT until the shares came back. restarting the services was not enough.

I was thinking and reading about ZIL log devices and the idea to use a single log drive like i used before is actually nothing secure.

two reasons why i removed the log device:
- i have a 1 GBit connection to the storage, the 2 HDDs in RAID1 are fast enough for reaching 120 MB/s which is the bottle neck of the network., no additional SSD needed that could improve speed
- i have only one SSD in the machine, which is better invested for the TrueNAS OS used in RAID1.

Hopefully this article could help some users in the future, I wish a good time in nintegrating and solving your incidents :)
Bye ant0nwax

 

[joeschmuck]

Glad you removed the log, too many people install a log without understanding where it actually causes a benefit. They watch a YouTube video and follow it blindly. Unfortunately that is the world we live in today. Not saying that is what you did but we run across it all the time.

Take care and Happy New Year!