ZFS replication to ZoL encrypted dataset?

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
Intrigued by a suggestion from @garm, I'm trying to set up an Odroid HC2 as a replication target for FreeNAS. The HC2 is a single-board computer with a SATA interface, mounted to a heatsink that'll easily hold a 3.5" HDD. It runs Linux (among other things, though not FreeBSD) and should make for a handy, low-power, low-price storage brick.

I'm writing up what I'm doing as I go on my wiki. Since the shipped kernel is compiled for 32-bit, and the ZFS packages require 64-bit, I'm needing to build ZFS from source--and since there doesn't seem to be a kernel-headers package available that matches the installed kernel, that means also rebuilding the kernel from source. Straightforward enough, though, as far as it goes.

So I've got ZFSonLinux running on the thing, I've created a pool, and I've created an encrypted dataset on that pool (since ZoL supports dataset encryption, and the idea is that this might live remotely). (the wiki also notes that I've created a non-privileged replication user, but I'm seeing issues with that--that's a separate thread if I can't figure it out)

So, thinking I should be ready to go (ignoring the byzantine process to set up a replication task in the FreeNAS GUI), I run
Code:
zfs send -Rv pool/dataset@snapshot | ssh root@odroid_ip zfs recv -F dozer/encrypted


...and it fails, complaining that it can't overwrite an encrypted dataset. Bother. If I send it to a non-encrypted dataset (creatively named dozer/cleartext), the replication goes merrily along, but of course it isn't encrypted on disk. Is it possible to do what I'm trying to do here? Or would FreeNAS's ZFS need to support encrypted datasets for this to work?
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
zfs send is nice and all, but what about copying the snapshot content using rsync and then snapshot the encrypted dataset once rsync is finished? Something close to @Arwen ’s proposed dedup substitution? That is what I’m doing to back up my Nextcloud database and vault. Given how OpenZFS native encryption is set up I would be surprised if you could zfs send an unencrypted dataset to and encrypted one, even with the same feature flags/version being used in both ends.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
It should definitely be possible, anything else is a bug in my book.

Do you have a man page for your version of ZFS? I think zfs recv might need new flags to do this, but I don't have a man page with encryption support at hand.

Silly question: Is the dataset unlocked while trying to recv?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
Code:
root@freenas2:~ # zfs send -v tank/emily-tm@auto-20181003.1200-2w | ssh root@192.168.1.202 zfs recv dozer/encrypted
full send of tank/emily-tm@auto-20181003.1200-2w estimated size is 452G
total estimated size is 452G
TIME        SENT   SNAPSHOT
cannot receive new filesystem stream: destination 'dozer/encrypted' exists
must specify -F to overwrite it
warning: cannot send 'tank/emily-tm@auto-20181003.1200-2w': signal received
root@freenas2:~ # zfs send -v tank/emily-tm@auto-20181003.1200-2w | ssh root@192.168.1.202 zfs recv -F dozer/encrypted
full send of tank/emily-tm@auto-20181003.1200-2w estimated size is 452G
total estimated size is 452G
TIME        SENT   SNAPSHOT
cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one
warning: cannot send 'tank/emily-tm@auto-20181003.1200-2w': signal received
root@freenas2:~ #
 

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
Ah, progress...

So dozer is the pool, and encrypted is the creatively-named encrypted dataset. zfs recv dozer/encrypted tries to destroy and recreate dozer/encrypted, which is causing errors. However, zfs recv dozer/encrypted/something is running merrily away, and zfs get encryption dozer/encrypted/something reports that dataset has inherited the encryption from its parent:
Code:
dan@odroid:~$ zfs list
NAME					   USED  AVAIL  REFER  MOUNTPOINT
dozer					  620G  6.42T   112K  /dozer
dozer/backup			   452G  6.42T	96K  /dozer/backup
dozer/backup/emily-tm	  452G  6.42T   452G  /dozer/backup/emily-tm
dozer/encrypted			169G  6.42T   192K  /dozer/encrypted
dozer/encrypted/emily-tm   169G  6.42T   169G  /dozer/encrypted/emily-tm
dan@odroid:~$ zfs get encryption dozer/encrypted
NAME			 PROPERTY	VALUE		SOURCE
dozer/encrypted  encryption  aes-256-ccm  -
dan@odroid:~$ zfs get encryption dozer/encrypted/emily-tm
NAME					  PROPERTY	VALUE		SOURCE
dozer/encrypted/emily-tm  encryption  aes-256-ccm  -
dan@odroid:~$


Cute. It should work, I'd think, though the connector might need to be changed
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
...
Cute. It should work, I'd think, though the connector might need to be changed
Looks like they sell a variant that includes a 12v 3amp power supply, 8 plug adapters and a pigtail cable for feeding from other 12v sources, (like a car).

http://www.fit-pc.com/web/purchasing/order-fit-uptime/

This brings back an old memory: I had wished vendors had standardized power connectors. Micro USB became one for 5v with various amperages. But we still lack a 12v standard, (except for the too large car cigarette lighter type). Plus, too many laptop power connectors and standards. Hopefully USB Type C connectors will reign supreme some day.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
But we still lack a 12v standard, (except for the too large car cigarette lighter type).
Even apart from being much too large, the lighter plug is a very poor standard. Loose physical tolerances and the spring nose combine to yield a very unreliable connection. The 5.5/2.1mm barrel connector is pretty common, but far from universal.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
USB-PD over Type C ports can do 12V and 19V and a few other options, IIRC. Realistically, it's the only standard that might gain traction, but its complexity limits it to higher-end PSUs. At a minimum, it adds a PD controller, a 5 V regulator and a cap or two to the BoM.
 

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
Hi @danb35!

How is your storage "brick" doing?

In particular is the HDD support stable or the pool keeps kicked out?

Greetings

Sent from my phone
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
In particular is the HDD support stable or the pool keeps kicked out?
HDD support is just fine, but I've taken a break on getting ZFS working on the thing. It seems it can run a 64-bit kernel, but none of the available distros use one, and all of the ZoL packages want one. That, combined with the half-assed ZeroTier support in FreeNAS, make the intended use of this thing a bit less viable than I'd hoped.
 

ovizii

Patron
Joined
Jun 30, 2014
Messages
435
Ah, progress...

So dozer is the pool, and encrypted is the creatively-named encrypted dataset. zfs recv dozer/encrypted tries to destroy and recreate dozer/encrypted, which is causing errors. However, zfs recv dozer/encrypted/something is running merrily away, and zfs get encryption dozer/encrypted/something reports that dataset has inherited the encryption from its parent:
Cute. It should work, I'd think, though the connector might need to be changed
Did you figure this out or are there any news? I'm also trying to "move" data from an unencrypted dataset to an encrypted one but from what you wrote it seems impossible except if receiving into a sub-dataset?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
Did you figure this out or are there any news?
I really haven't gone further with it. But it seems straightforward enough if you're willing to end up with your data in a sub-dataset of an encrypted dataset.
 

ovizii

Patron
Joined
Jun 30, 2014
Messages
435
I really haven't gone further with it. But it seems straightforward enough if you're willing to end up with your data in a sub-dataset of an encrypted dataset.
THX. I ended up creating new encrypted datasets and then doing a "mv unenc/* encr/"
 

pitbullb

Cadet
Joined
Mar 25, 2020
Messages
7
Good morning,

are there any updates on this? I installed a second server with the nightlies of TrueNas 12 Core and tried to send a dataset (geli on zpool) from FreeNas 11 to an encrypted dataset in a pool in TrueNas 12 but it fails.

Are there any migration possibilities or are there any planned for the future or what would be the best way to migrate to ZoL/ZoF in TrueNas?

Thank you very much.
 
Top