Replication between legacy and native encrypted pools?

[Dunuin]

Hi,

I got 2 servers created with FreeNAS 11.2 that are both now running TrueNAS CORE 12.0-U6.
The main server got a GELI encrypted HDD pool and a GELI encrypted SSD pool.
The other server is just for backup and got identical hardware but only got that GELI encrypted HDD pool.
Normally I do a weekly replication of the main servers HDD + SSD pool to the backup servers HDD pool.

But because of a dying SSD I needed to maintain the SSD pool so I used that to destroy and recreate the SSD pool with more drives. GELI encryption isn'T available anymore so I created a unencrypted pool and only encrypted a dataset that all other datasets/zvols are childs of. That worked fine so far but now I got the problem that I can't backup my SSD pool anymore because it looks like it is not possible to replicate a ZFS natively encrypted dataset/zvol to a "non" (or GELI) encrypted pool.

Is there a way to somehow get the replication working without needing to destroy and recreate both of the HDD pools? Recreation would be very annoying because when I destroy my main GELI encrypted HDD pool and recreate it with ZFS native encrytion I can't use replication or "zfs send | zfs recv" to copy the data over from the backup HDD pool because I run into the same problem like with the SSD pool that replication isn't possible between GELI and native encrypted pools. So I would need to create all datasets and zvols from scratch and manually copy over the files using rsync and dd (like I did it with the SSD pool before).

Any hints on how to best do that?

 

[winnielinnie]

That worked fine so far but now I got the problem that I can't backup my SSD pool anymore because it looks like it is not possible to replicate a ZFS natively encrypted dataset/zvol to a "non" (or GELI) encrypted pool.

I believe the issue is that there's a "hard stop" for legacy pools that CORE detects as residing on top of GELI. You should be able to replicate native ZFS encrypted datasets to a GELI pool, but you cannot use native encryption on the legacy pool (which is already built atop GELI).. And so you simply decrypt the datasets on the source, and replicate them as "non-encrypted" records on the target. They will still be protected with GELI encryption if the drives are removed or sent in for warranty.

 

[Dunuin]

Thats what I tried and recursive replication fails with this:

Code:

Error​
Unable to send encrypted dataset 'SSDpool/VeryLongDatasetName/VLT/SRT' to existing unencrypted or unrelated dataset 'BACKUPHDDpool/RPL/SSDpool/VLT/SRT'.
Logs​
[2021/10/31 00:00:02] INFO     [Thread-28] [zettarepl.paramiko.replication_task__task_45] Connected (version 2.0, client OpenSSH_8.4-hpn14v15)
[2021/10/31 00:00:02] INFO     [Thread-28] [zettarepl.paramiko.replication_task__task_45] Authentication (publickey) successful!
[2021/10/31 00:00:03] INFO     [replication_task__task_45] [zettarepl.replication.run] For replication task 'task_45': doing push from 'SSDpool/VeryLongDatasetName/VLT/SRT' to 'BACKUPHDDpool/RPL/SSDpool/VLT/SRT' of snapshot='SSDpool_VLT_SRT_weekly-2021-10-31_00-00' incremental_base=None receive_resume_token=None encryption=False
[2021/10/31 00:00:04] ERROR    [replication_task__task_45] [zettarepl.replication.run] For task 'task_45' non-recoverable replication error ReplicationError("Unable to send encrypted dataset 'SSDpool/VeryLongDatasetName/VLT/SRT' to existing unencrypted or unrelated dataset 'BACKUPHDDpool/RPL/SSDpool/VLT/SRT'")

"SSDpool/VeryLongDatasetName/VLT/SRT" is natively encrypted (inherited by "SSDpool/VeryLongDatasetName/VLT" which is encrypted) and the complete other pool "BACKUPHDDpool" is only using legacy encryption and TrueNAS will also not allow me to create natively encrypted datasets ontop of a legacy encrypted pool (which would be bad in production because of double encryption but would be nice to have as a workaround for pool migrations). So "BACKUPHDDpool/RPL/SSDpool/VLT/SRT" is a unencrypted dataset I created as target for my replication.

 

[winnielinnie]

How is your Replication Task configured? Can you take screenshots of it and expand the Advanced Options?

I think you might have to let the Replication Task create the new dataset (SRT) into the BackupHDDpool, rather than have a pre-made dataset to be overwritten.

You may have to manually enter the dataset properties to exclude in the Advanced Options -> Properties Exclude. Such properties would be encryption, encryptionroot, keyformat, keylocation, keystatus, and pbkdf2iters.

I do not have a legacy pool with which to test this, so I can't speak on experience.

 

[Dunuin]

How is your Replication Task configured? Can you take screenshots of it and expand the Advanced Options?

I think you might have to let the Replication Task create the new dataset (SRT) into the BackupHDDpool, rather than have a pre-made dataset to be overwritten.

I always created that dataset first. If I tried it without creating it first and for example replicate SSDpool/VeryLongDatasetName/VLT/SRT (which also got child datasets like SSDpool/VeryLongDatasetName/VLT/SRT/Dataset1 and so on) to BACKUPHDDpool/RPL/SSDpool/VLT it will end up like BACKUPHDDpool/RPL/SSDpool/VLT/Dataset1 instead of BACKUPHDDpool/RPL/SSDpool/VLT/SRT/Dataset1.

You may have to manually enter the dataset properties to exclude in the Advanced Options -> Properties Exclude. Such properties would be encryption, encryptionroot, keyformat, keylocation, keystatus, and pbkdf2iters.

That is a good idea, thanks. Will try that.

 

[winnielinnie]

If I tried it without creating it first and for example replicate SSDpool/VeryLongDatasetName/VLT/SRT (which also got child datasets like SSDpool/VeryLongDatasetName/VLT/SRT/Dataset1 and so on) to BACKUPHDDpool/RPL/SSDpool/VLT it will end up like BACKUPHDDpool/RPL/SSDpool/VLT/Dataset1 instead of BACKUPHDDpool/RPL/SSDpool/VLT/SRT/Dataset1.

I meant as in you input the target as BACKUPHDDpool/RPL/SSDpool/VLT/SRT, even though the dataset SRT does not yet exist on BACKUPHDDpool. (Worth a shot, possibly, but the Replication Tasks GUI has always been rigid and poorly documented.)

---

That is a good idea, thanks. Will try that.

Fingers crossed!