Replication from TrueNAS SCALE to CORE, Best Practices?

[yottabit]

[xpost from the TrueNAS SCALE subforum; hoping for more response here since this applies to more than just SCALE]

tl;dr: when replicating datasets between hosts of different ZFS versions, and wishing to preserve the entire state of the dataset, is it better to just send into a file rather than a remote dataset?

I have been trying for days now to replicate two dozen datasets from TrueNAS SCALE (with upgraded pool ZFS 2.1) to TrueNAS CORE (older pool, ZFS 2.0).

I am using the "full filesystem" option in replication, which should preserve the dataset properties. I have encountered several problems, causing me to restart the replication (maybe it's resuming, hard to tell yet):

1. On the remote system I'm not receiving as root, so the root user there has to use `zfs allow` for the properties I need into my root dataset. The problem is that I don't know which properties will be required ahead of time, so I don't know which to ask him to add until the replication fails (which seems to be at the very end of sending a dataset, as it seems to not set the properties at the remote until the end).
2. There is no allow `all` property, and just adding every property to the list also doesn't work because the command will return an error that a property is not allowed for that type of dataset, but it doesn't tell you which property caused the problem. (If I had root, I would just run a shell loop on an array of all properties to make sure all that were possible were set, but he isn't going to be writing a shell script himself. I guess I could write it for him...)
3. As a non-root user, I don't have permission to mount datasets on his system (nor do I need to), so I have been using the override property `canmount=off` to work around this; but it seems that I have to enter this into the replication form every time I edit the form as it doesn't seem to preserve what was entered last time when it loads the form for edit.
4. Some OpenZFS 2.1 properties are not supported on OpenZFS 2.0, so I have to add these to the exclude properties, and have the same form problem as #3 above.

As ZFS forked long ago, and keeps diverging even more now between Oracle v. FreeBSD (and maybe OpenSolaris?) v. OpenZFS, and even different versions of the same branch, how does everyone handle this when replicating to remote systems? Is there a best practices guide?

I have sent several TB of data so far, and these datasets still show up in a pending/resumable state on the remote, but not yet in the regular dataset list yet. Each time I fix a property exclude, and start the replication again, it seems to start uploading another dataset. I am hoping it eventually finishes all the other datasets that failed at the last instance due to lack of permission to set a property or an unsupported property.

But I am starting to wonder if I should just send these datasets into a remote file instead. I think that should preserve all properties, right? And I wouldn't have to mess with `canmount`, restoring its state later when I pull all the data back. Downsides to this approach:

1. Not supported in the TrueNAS UI
2. Would have to manually script the send of each dataset into a remote file
3. Not able to resume if there is a failed transfer (minor; connection has been 100% stable so far)
4. ... something I'm missing?

The idea is to send all my data sets away, refactor my vdevs and pool, and then pull all the datasets back. I am becoming increasingly weary that I will be able to pull back all my datasets as-is without spending considerable time fixing up all the missing and/or reset properties locally at the end.

How does everyone else handle situations like this?

 

[yottabit]

Here is what I landed on. I have finished the sends. I will do some restore tests next to make sure everything is as I expect, and then I will rsync these away to a remote server. I could pipe them directly to the remote, but I had a spare disk big enough to handle almost all of my datasets when compressed like mad.

Note about the compression: if you don't have about 1.25 GiB of RAM per CPU thread, don't use `pixz -9` ..either use `pixz` (default compression, multithreaded) or `xz -9` (maximum compression, singlethreaded) or the -T option to specify how many threads. You've been warned.

Code:

#!/usr/bin/env bash
#

set -euo pipefail

declare -a datasets=( \
  'books' \
  'cloud_push' \
  'images' \
  'multimedia' \
  'music' \
  'pictures' \
 'tv' \
 'tv_wrc' \
  'video' \
  'working' \
)

declare -a enc_datasets=( \
  'crypt' \
)

declare -a full_datasets=( \
  'ix-applications' \
  'container_states' \
  'software' \
  'virtual_machines' \
)

for dataset in ${datasets[@]}; do
  nice -n19 sh -c \
   "zfs send -Lv "vol1/${dataset}@migrate220206" | pixz -9 > "${dataset}.zfs.xz""
  done

for dataset in ${enc_datasets[@]}; do
  nice -n19 sh -c \
   "zfs send -LRvw "vol1/${dataset}@migrate220206" | pixz -1 > "${dataset}-w.zfs.xz""
  done

for dataset in ${full_datasets[@]}; do
  nice -n19 sh -c \
   "zfs send -LRv "vol1/${dataset}@migrate220206" | pixz -9 > "${dataset}-R.zfs.xz""
  done

Then plan is:

1. break my boot mirror
2. reset to defaults
3. clear the zpool
4. create a new zpool with 2x Z2 vdevs to replace the existing 2x Z1 vdevs
5. `zfs recv` the datasets from the compressed files back into the refactored pool with the original name
6. swap back to my other (original config) boot drive
7. ensure everything is working as expected
8. rebuild the boot mirror

If anything goes wrong, it will cost me some hours and a $200 cloud storage object retrieval bill. Let's hope that doesn't happen.

 

[yottabit]

tl;dr: with exception of the restoring the boot pool mirror, everything worked perfectly, and I now have my main pool vdevs refactored from 2xZ1 to 2xZ2. All apps/containers/VMs/shares/tasks were preserved and everything is running great. Added bonus: 0% fragmentation on my pool now after clearing 10 years of cruft!

Everything went well except step #8.

Before reinstalling the original version boot disk from the original mirror, while still running the temporary boot disk, I used dd to write zeros to the first 20 MB of the disk, hoping to prevent the original OS from being confused as to which of the two boot disks were the correct version and causing an incorrect resilver.

This worked, and I was able to perform the replace operation in step #8, and I saw the resilver complete successfully. One last reboot to test, and unfortunately the boot halted with a kernel panic. I then tried each of the mirrored boot drives separately, and had the same result.

I was able to boot into maintenance mode on an older release without the kernel panic, checked zpool status, and saw unrecoverable corruption reported. I'm not sure how this happened, as the original boot disk was just fine in steps #6 and #7, and the resilver completed successfully in step #8. But somehow corruption happened.

I ended up booting the TrueNAS SCALE install media, and performing a format & fresh install on the mirror boot pool. I imported the backed-up config database, and voila everything is running perfectly again.

I do wonder now, whether I should move the system dataset from the boot pool to the 2xZ2 pool. But as long as I'm keeping database backups regularly (always after major changes, and before any upgrades) I guess the system dataset isn't so important since I don't need it to fetch the backup database from. Afaict, no other important data is kept in the system dataset that cannot be regenerated after import of the config database, as my 30 containers and Kubernetes configurations were all preserved.