SOLVED Can't import full, encrypted pool. vdev state changed and my physical pool disks are really, really hard at work

[PhiloEpisteme]

I had wondered it it would be possible to use dd to duplicate each drive to a larger drive, use the same keys to decrypt, and then import the drives anew. I'm not sure if there is geli metadata and/or zfs metadata that would need to be updated to reflect the new drives. Very much speculation and haven't had the time to research it at all yet.

 

[Redcoat]

See post #4 here - not quite the same situation, but sems likely similar https://www.ixsystems.com/community/threads/errno-28-no-space-left-on-device.39476/#post-244327

 

[pro lamer]

the SMART self test is ran by smartctl. When I ask for any running processes with that name, it turns up

They are started this way. They run in background so you will not see smartctl in the ps list.

Some other smartctl command parameter is used to check if tests in progress, I can't remember from the top of my head.

TL;DR - are you still able to "decrypt" your drives from the CLI (geli attach IIRC) ? All of them or only a few?

Sent from my phone

 

[PhiloEpisteme]

TL;DR - are you still able to "decrypt" your drives from the CLI (geli attach IIRC) ? All of them or only a few?

From this post a thread or so back it looks like they were decrypted. Only the import failed, or so it seemed. It is certainly worth retrying the decryption step.

 

[pro lamer]

From this post a thread or so back it looks like they were decrypted. Only the import failed, or so it seemed. It is certainly worth retrying the decryption step.

I think you're confusing something. I'd name this step "unlocking" instead of "decryption". The data in the HDDs stays encrypted all the time. Attach command creates a special virtual device /dev/....geli. Every time anything reads this special file blocks the blocks get decrypted in-RAM-on-the-fly.

IIRC each reboot causes the special geli devices lost. Very good - the pool gets locked and the password needs to be entered every reboot/every unlock attempt.

Sent from my phone

 

[PhiloEpisteme]

I think you're confusing something. I'd name this step "unlocking" instead of "decryption". The data in the HDDs stays encrypted all the time. Attach command creates a special virtual device /dev/....geli. Every time anything reads this special file blocks the blocks get decrypted in-RAM-on-the-fly.

IIRC each reboot causes the special geli devices lost. Very good - the pool gets locked and the password needs to be entered every reboot/every unlock attempt.

Sent from my phone

True, a bit fast and loose with terminology. The drives do unlock, but do not import. After the unlock it isnt clear to me that the encrypted disks is the issue but rather the disk being full affecting import regardless of encryption or not. Are you suggesting that perhaps it may still be an encryption issue rather than disk full issue regardless of encrypted disks?

 

[Redcoat]

The drives do unlock,

How is that unlock evidenced in this case?

 

[PhiloEpisteme]

How is that unlock evidenced in this case?

The post history is long and links other threads bit perhaps if you haven't browsed through it may be worthwhile in case you see something we missed.

The salient posts to your question I think are the one linked above where OP used geli attach -k key /dev/gptid/<device> to unlock the 8 3TB drives. Following that, the command zpool import displayed ZFS_8x_3TB_RAIDz2_pool as an importable pool whereas before it had not shown the pool as importable. That suggested to me that the system was able to successfully unlock the drives and decrypt the data as indicated by zfs' ability to read the appropriate data on the disks to determine that they were a part of a zfs pool.

I think one issue from earlier attempt was the omission of the correct import flags. Hopefully, for OP's sake, someone sees something we have missed.

 

[Redcoat]

Following that, the command zpool import displayed ZFS_8x_3TB_RAIDz2_pool as an importable pool whereas before it had not shown the pool as importable.

Yes, I recall that now. Thanks.

 

[pro lamer]

I thought the OP - after the initial decryption success - started having hardware problems and the drives became so unstable that it started being impossible to even unlock them...

Sent from my phone

 

[PhiloEpisteme]

I thought the OP - after the initial decryption success - started having hardware problems and the drives became so unstable that it started being impossible to even unlock them...

Sent from my phone

I'm not so certain that is the case. I think a fresh attempt to decrypt and import would maybe clarify.

 

[devnullius]

Oh a lot has happened while I took the weekend off - THANK YOU. I'll read what got said.

For some good news: My original problem (importing encrypted pool fails with GUI errors) got solved! Read all about it here https://www.ixsystems.com/community...-at-storage-volume-2-unlock.75032/post-523072 :)

xx

No, a full set of larger drives - the https://ixsystems.com/documentation/freenas/11.2/storage.html#disks 9.5.2 method to expand a pool. I am sure that I've seen a post here on this forum of someone doing just this in the OP's situation, with success, but I have not been able to find it.

Oh my… That does not seem to be for the faint of hearth…!

I had wondered it it would be possible to use dd to duplicate each drive to a larger drive, use the same keys to decrypt, and then import the drives anew. I'm not sure if there is geli metadata and/or zfs metadata that would need to be updated to reflect the new drives. Very much speculation and haven't had the time to research it at all yet.

Wouldn't that have you end up with disks that still have the original partitions and as a consequence are "full", despite the extra space on the underlying hardware...? So you would have to resize the partitions on each disk to the exact same new and bigger size? Is that even possible? For sure they are not NTFS partitions that can be changed at-will with plenty of software tools around? Just reading along... :)

They are started this way. They run in background so you will not see smartctl in the ps list.

Some other smartctl command parameter is used to check if tests in progress, I can't remember from the top of my head.

Ah, thank you for that. I was grasping, true :)

True, a bit fast and loose with terminology. The drives do unlock, but do not import. After the unlock it isnt clear to me that the encrypted disks is the issue but rather the disk being full affecting import regardless of encryption or not. Are you suggesting that perhaps it may still be an encryption issue rather than disk full issue regardless of encrypted disks?

I appreciate the semantics in this context. In our defense, the GUI explicitly never told me the device was unlocked (instead, we got GUI errors).

I think one issue from earlier attempt was the omission of the correct import flags.

Import flags...? Like? What should I google to learn more? :)

Yes, I recall that now. Thanks.

Thank YOU too :) I'm learning some new stuff again

I thought the OP - after the initial decryption success - started having hardware problems and the drives became so unstable that it started being impossible to even unlock them...

Sent from my phone

I still have no fracking clue as to what it was doing to my disks. It made *NO* sense and I think we kinda ruled out all the usual suspects. This one will be a mysterie for times to come... :/

 

[pro lamer]

My original problem (importing encrypted pool fails with GUI errors) got solved!

Great! I hope it's not April fool's day joke really :)

Sent from my phone

 

[devnullius]

Great! I hope it's not April fool's day joke really :)

I'd never willingly put you on an Easter Egg hunt ;p