(SOLVED) Replication Task from TrueNAS 13.0 to a remote Debian system, cannot send snapshot

[natepichler]

I am attempting to setup an automated replication task from my TrueNAS server to a Debian machine that is offsite.

ZFS is installed on the Debian machine, and I can SSH into it. The SSH keypair and connection are properly set up in TrueNAS.

Here is the config I have for the replication task:

red is the system dataset pool on my local TrueNAS server.
debian-remote/replication is the dataset on the remote system I intend on using for the replication. I have confirmed that there are no existing snapshots or other data in that dataset.

This is the error message I get from TrueNAS:

Code:

cannot unmount '/debian-remote/replication': unmount failed
warning: cannot send 'red@auto-2022-07-08_23-55': signal received.

Logs

[2022/07/15 09:35:49] INFO     [Thread-22] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.4p1)
[2022/07/15 09:35:49] INFO     [Thread-22] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2022/07/15 09:35:50] INFO     [replication_task__task_1] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2022/07/15 09:35:50] INFO     [replication_task__task_1] [zettarepl.paramiko.replication_task__task_1.sftp] [chan 5] Opened sftp connection (server version 3)
[2022/07/15 09:35:51] INFO     [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'red' to 'debian-remote/replication' of snapshot='auto-2022-07-08_23-55' incremental_base=None receive_resume_token=None encryption=False
[2022/07/15 09:35:53] WARNING  [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 1 recoverable replication error RecoverableReplicationError("cannot unmount '/debian-remote/replication': unmount failed\nwarning: cannot send 'red@auto-2022-07-08_23-55': signal received")
[2022/07/15 09:35:53] INFO     [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 1 seconds
[2022/07/15 09:35:55] INFO     [replication_task__task_1] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2022/07/15 09:35:55] INFO     [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'red' to 'debian-remote/replication' of snapshot='auto-2022-07-08_23-55' incremental_base=None receive_resume_token=None encryption=False
[2022/07/15 09:35:57] WARNING  [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 2 recoverable replication error RecoverableReplicationError("cannot unmount '/debian-remote/replication': unmount failed\nwarning: cannot send 'red@auto-2022-07-08_23-55': signal received")
... 3 more lines ...
[2022/07/15 09:36:03] WARNING  [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 3 recoverable replication error RecoverableReplicationError("cannot unmount '/debian-remote/replication': unmount failed\nwarning: cannot send 'red@auto-2022-07-08_23-55': signal received")
[2022/07/15 09:36:03] INFO     [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 4 seconds
[2022/07/15 09:36:08] INFO     [replication_task__task_1] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2022/07/15 09:36:08] INFO     [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'red' to 'debian-remote/replication' of snapshot='auto-2022-07-08_23-55' incremental_base=None receive_resume_token=None encryption=False
[2022/07/15 09:36:10] WARNING  [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 4 recoverable replication error RecoverableReplicationError("cannot unmount '/debian-remote/replication': unmount failed\nwarning: cannot send 'red@auto-2022-07-08_23-55': signal received")
[2022/07/15 09:36:10] INFO     [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 8 seconds
[2022/07/15 09:36:19] INFO     [replication_task__task_1] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots: []
[2022/07/15 09:36:20] INFO     [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'red' to 'debian-remote/replication' of snapshot='auto-2022-07-08_23-55' incremental_base=None receive_resume_token=None encryption=False
[2022/07/15 09:36:22] WARNING  [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 5 recoverable replication error RecoverableReplicationError("cannot unmount '/debian-remote/replication': unmount failed\nwarning: cannot send 'red@auto-2022-07-08_23-55': signal received")
[2022/07/15 09:36:22] ERROR    [replication_task__task_1] [zettarepl.replication.run] Failed replication task 'task_1' after 5 retries

I have tried granting permissions for mount/unmount to the ssh user and recreating the periodic snapshot task.

I have read most of the other threads about this replication problem, and none of those solutions were able to get the task to run successfully.
I looks like there is (or used to be) an option to do a full replication if no other snapshots existed. The threads that mentioned this were a number of years old, and I can't find the same option in the 13.0 UI.
Hopefully somebody here can guide me in the right direction.

Thanks!

 

[natepichler]

As per usual, I figured out the answer to my own question.

The user that is used for the SSH connection needs certain ZFS permissions for this to work

Code:

sudo zfs allow -ld <user> clone,create,diff,hold,mount,receive,snapshot <dataset>