ZFS replication to ZoL encrypted dataset?

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
9,611
Thanks
3,351
#1
Intrigued by a suggestion from @garm, I'm trying to set up an Odroid HC2 as a replication target for FreeNAS. The HC2 is a single-board computer with a SATA interface, mounted to a heatsink that'll easily hold a 3.5" HDD. It runs Linux (among other things, though not FreeBSD) and should make for a handy, low-power, low-price storage brick.

I'm writing up what I'm doing as I go on my wiki. Since the shipped kernel is compiled for 32-bit, and the ZFS packages require 64-bit, I'm needing to build ZFS from source--and since there doesn't seem to be a kernel-headers package available that matches the installed kernel, that means also rebuilding the kernel from source. Straightforward enough, though, as far as it goes.

So I've got ZFSonLinux running on the thing, I've created a pool, and I've created an encrypted dataset on that pool (since ZoL supports dataset encryption, and the idea is that this might live remotely). (the wiki also notes that I've created a non-privileged replication user, but I'm seeing issues with that--that's a separate thread if I can't figure it out)

So, thinking I should be ready to go (ignoring the byzantine process to set up a replication task in the FreeNAS GUI), I run
Code:
zfs send -Rv pool/dataset@snapshot | ssh root@odroid_ip zfs recv -F dozer/encrypted


...and it fails, complaining that it can't overwrite an encrypted dataset. Bother. If I send it to a non-encrypted dataset (creatively named dozer/cleartext), the replication goes merrily along, but of course it isn't encrypted on disk. Is it possible to do what I'm trying to do here? Or would FreeNAS's ZFS need to support encrypted datasets for this to work?
 

garm

FreeNAS Guru
Joined
Aug 19, 2017
Messages
836
Thanks
221
#2
zfs send is nice and all, but what about copying the snapshot content using rsync and then snapshot the encrypted dataset once rsync is finished? Something close to @Arwen ’s proposed dedup substitution? That is what I’m doing to back up my Nextcloud database and vault. Given how OpenZFS native encryption is set up I would be surprised if you could zfs send an unencrypted dataset to and encrypted one, even with the same feature flags/version being used in both ends.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
15,802
Thanks
3,713
#3
It should definitely be possible, anything else is a bug in my book.

Do you have a man page for your version of ZFS? I think zfs recv might need new flags to do this, but I don't have a man page with encryption support at hand.

Silly question: Is the dataset unlocked while trying to recv?
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
9,611
Thanks
3,351
#4
Code:
root@freenas2:~ # zfs send -v tank/emily-tm@auto-20181003.1200-2w | ssh root@192.168.1.202 zfs recv dozer/encrypted
full send of tank/emily-tm@auto-20181003.1200-2w estimated size is 452G
total estimated size is 452G
TIME        SENT   SNAPSHOT
cannot receive new filesystem stream: destination 'dozer/encrypted' exists
must specify -F to overwrite it
warning: cannot send 'tank/emily-tm@auto-20181003.1200-2w': signal received
root@freenas2:~ # zfs send -v tank/emily-tm@auto-20181003.1200-2w | ssh root@192.168.1.202 zfs recv -F dozer/encrypted
full send of tank/emily-tm@auto-20181003.1200-2w estimated size is 452G
total estimated size is 452G
TIME        SENT   SNAPSHOT
cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one
warning: cannot send 'tank/emily-tm@auto-20181003.1200-2w': signal received
root@freenas2:~ #
 

pro lamer

FreeNAS Experienced
Joined
Feb 16, 2018
Messages
444
Thanks
74
#5

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
9,611
Thanks
3,351
#7
Ah, progress...

So dozer is the pool, and encrypted is the creatively-named encrypted dataset. zfs recv dozer/encrypted tries to destroy and recreate dozer/encrypted, which is causing errors. However, zfs recv dozer/encrypted/something is running merrily away, and zfs get encryption dozer/encrypted/something reports that dataset has inherited the encryption from its parent:
Code:
dan@odroid:~$ zfs list
NAME					   USED  AVAIL  REFER  MOUNTPOINT
dozer					  620G  6.42T   112K  /dozer
dozer/backup			   452G  6.42T	96K  /dozer/backup
dozer/backup/emily-tm	  452G  6.42T   452G  /dozer/backup/emily-tm
dozer/encrypted			169G  6.42T   192K  /dozer/encrypted
dozer/encrypted/emily-tm   169G  6.42T   169G  /dozer/encrypted/emily-tm
dan@odroid:~$ zfs get encryption dozer/encrypted
NAME			 PROPERTY	VALUE		SOURCE
dozer/encrypted  encryption  aes-256-ccm  -
dan@odroid:~$ zfs get encryption dozer/encrypted/emily-tm
NAME					  PROPERTY	VALUE		SOURCE
dozer/encrypted/emily-tm  encryption  aes-256-ccm  -
dan@odroid:~$


Cute. It should work, I'd think, though the connector might need to be changed
 

Arwen

FreeNAS Expert
Joined
May 17, 2014
Messages
1,064
Thanks
516
#8
...
Cute. It should work, I'd think, though the connector might need to be changed
Looks like they sell a variant that includes a 12v 3amp power supply, 8 plug adapters and a pigtail cable for feeding from other 12v sources, (like a car).

http://www.fit-pc.com/web/purchasing/order-fit-uptime/

This brings back an old memory: I had wished vendors had standardized power connectors. Micro USB became one for 5v with various amperages. But we still lack a 12v standard, (except for the too large car cigarette lighter type). Plus, too many laptop power connectors and standards. Hopefully USB Type C connectors will reign supreme some day.
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
9,611
Thanks
3,351
#9
But we still lack a 12v standard, (except for the too large car cigarette lighter type).
Even apart from being much too large, the lighter plug is a very poor standard. Loose physical tolerances and the spring nose combine to yield a very unreliable connection. The 5.5/2.1mm barrel connector is pretty common, but far from universal.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
15,802
Thanks
3,713
#10
USB-PD over Type C ports can do 12V and 19V and a few other options, IIRC. Realistically, it's the only standard that might gain traction, but its complexity limits it to higher-end PSUs. At a minimum, it adds a PD controller, a 5 V regulator and a cap or two to the BoM.
 

pro lamer

FreeNAS Experienced
Joined
Feb 16, 2018
Messages
444
Thanks
74
#11
Hi @danb35!

How is your storage "brick" doing?

In particular is the HDD support stable or the pool keeps kicked out?

Greetings

Sent from my phone
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
9,611
Thanks
3,351
#12
In particular is the HDD support stable or the pool keeps kicked out?
HDD support is just fine, but I've taken a break on getting ZFS working on the thing. It seems it can run a 64-bit kernel, but none of the available distros use one, and all of the ZoL packages want one. That, combined with the half-assed ZeroTier support in FreeNAS, make the intended use of this thing a bit less viable than I'd hoped.
 
Top