Hello
I am having trouble with my setup. I think I might be doing something unusual but not extraordinary. But I am running into issues and I can't find what I need by searching.
In short, I am trying to backup datasets from my Ubuntu Server machine to my TrueNAS system, and from this replicate to a remote backup on another TrueNAS system. This has worked, but I am running into trouble when I start to use encrypted datasets. I will describe how my replication job are made now at the end of the post. It might be a false start anyway
This is my setup, including naming conventions to be used in the rest of the post:
Machine #1
Name: Blackbox
OS: Ubuntu 20.04
ZFS pool: zfs-utils 0.8.3
User: Dedicated user "freenas" that is allowed to zfs send
Machine #2
Name: Home_Nas
OS: TrueNAS 12.0-U1
User: root
Machine #3
Name: Remote_NAS
OS: TrueNAS 12.0-U1
Users: Dedicated user "Kebba_replication". zfs permissions on
What I want to achieve:
Blackbox is going to take over the heavy lifting at home. I therefore want to have a lot of data be created on this filesystem. I have snapshots being managed by sanoid on this machine. Backups shall be replicated (pulled) to Home_NAS and selected datasets will be replicated to Remote_NAS (pushed). I want to use encrypted datasets for most datasets. Remote_NAS should not have keys to decrypt the backups (preferably). Same goes for Home_NAS if possible.
Visualization:
Blackbox -> (Pulled) Home_NAS -> (Pushed) Remote_NAS.
What works now?
I can pull snapshots from blackbox into a dedicated dataset
I get a lot of permission errors when replicating (the user freenas is not allowed to set dataset properties and can't mount filesystems) but the snapshot get pushed anyway. I can then log into blackbox, set the correct properties, load the keys and mount the datasets where they should be.
So what do I need help with?
Issue #1
It is harder than it should to replicate Blackbox -> Home_NAS. Datasets must live under an encrypted parent
Issue #2
I can't push the backed up datasets pulled from blackbox to Remote_NAS from Home_NAS. It will fail with a few different error messages depending on what I try. Examples:
"Unable to send encrypted dataset to existing unencrypted or unrelated dataset"
"dataset key must be loaded"
And a combination of different permission errors. (With "Full filesystem replication" enabled)
Are there any known solution to what I want to achieve or are there showstoppers that make it so I must reconsider some parts of the concept?
Thank you for your time
Example of pull job from Blackbox to Home_NAS (this works)
I am having trouble with my setup. I think I might be doing something unusual but not extraordinary. But I am running into issues and I can't find what I need by searching.
In short, I am trying to backup datasets from my Ubuntu Server machine to my TrueNAS system, and from this replicate to a remote backup on another TrueNAS system. This has worked, but I am running into trouble when I start to use encrypted datasets. I will describe how my replication job are made now at the end of the post. It might be a false start anyway
This is my setup, including naming conventions to be used in the rest of the post:
Machine #1
Name: Blackbox
OS: Ubuntu 20.04
ZFS pool: zfs-utils 0.8.3
User: Dedicated user "freenas" that is allowed to zfs send
MB: SuperMicro X10SLM+-LN4F
CPU: Intel Xeon E3-1220 v3
RAM: 2*8GB
Pool: 1 pool of 6 2TB disks in mirrors[
CPU: Intel Xeon E3-1220 v3
RAM: 2*8GB
Pool: 1 pool of 6 2TB disks in mirrors[
Machine #2
Name: Home_Nas
OS: TrueNAS 12.0-U1
User: root
CPU/Mobo:Asrock C2550D4I
RAM:32GB
Pool: RaidZ2 6*3TB
RAM:32GB
Pool: RaidZ2 6*3TB
Machine #3
Name: Remote_NAS
OS: TrueNAS 12.0-U1
Users: Dedicated user "Kebba_replication". zfs permissions on
MainRaidZ/Kebba_Replication
: create,destroy,diff,mount,readonly,receive,release,send,userpropNot sure fo the moment. Running ontop of Proxmox on a machine with an 8-thread Xeon and 32GB or RAM
What I want to achieve:
Blackbox is going to take over the heavy lifting at home. I therefore want to have a lot of data be created on this filesystem. I have snapshots being managed by sanoid on this machine. Backups shall be replicated (pulled) to Home_NAS and selected datasets will be replicated to Remote_NAS (pushed). I want to use encrypted datasets for most datasets. Remote_NAS should not have keys to decrypt the backups (preferably). Same goes for Home_NAS if possible.
Visualization:
Blackbox -> (Pulled) Home_NAS -> (Pushed) Remote_NAS.
What works now?
I can pull snapshots from blackbox into a dedicated dataset
vol1/Backup/blackbox-backup/blackbox-lockbox
on Home_NAS. This works recursively with child datasets as well. I have also confirmed that I can restore these to blackbox. A restore require that I push the snapshots using CLI on Home_NASCode:
zfs send -vR vol1/Backup/blackbox-backup/blackbox-lockbox/example_dataset@latest_snapshot --raw | ssh freenas@blackbox zfs receive tank/temp/testrestore
I get a lot of permission errors when replicating (the user freenas is not allowed to set dataset properties and can't mount filesystems) but the snapshot get pushed anyway. I can then log into blackbox, set the correct properties, load the keys and mount the datasets where they should be.
So what do I need help with?
Issue #1
It is harder than it should to replicate Blackbox -> Home_NAS. Datasets must live under an encrypted parent
/tank/Backup/blackbox-lockbox
or the replication job fails with a "Unable to send encrypted dataset to existing unencrypted or unrelated dataset" error message. Creating an encrypted root does solve this problem. I am pulling the dataset with the "Full filesystem replication" option enabled. I am not sure if this is an advisable way to do it. It works, but is maybe not as elegant as I wish and I am slightly worried I am shooting myself in the foot by doing this. It is possible to achieve this without a root dataset if both target and source are running TrueNAS 12. This issue is not high priority for me if the "shoot myself in the foot" part is just paranoia.Issue #2
I can't push the backed up datasets pulled from blackbox to Remote_NAS from Home_NAS. It will fail with a few different error messages depending on what I try. Examples:
"Unable to send encrypted dataset to existing unencrypted or unrelated dataset"
"dataset key must be loaded"
And a combination of different permission errors. (With "Full filesystem replication" enabled)
Are there any known solution to what I want to achieve or are there showstoppers that make it so I must reconsider some parts of the concept?
Thank you for your time
Example of pull job from Blackbox to Home_NAS (this works)
"Direction" - Pull
"Transport" - SSH
"Source" - tank/source_dataset_example
Full filesystem replication: Yes
Naming Schema: autosnap_%Y-%m-%d_%H:%M:%S_hourly/daily/monthly
Destination" - vol1/Backup/blackbox-backup/blackbox_lockbox
"Transport" - SSH
"Source" - tank/source_dataset_example
Full filesystem replication: Yes
Naming Schema: autosnap_%Y-%m-%d_%H:%M:%S_hourly/daily/monthly
Destination" - vol1/Backup/blackbox-backup/blackbox_lockbox