Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

11.3 Replication new method not working

Joined
Feb 11, 2020
Messages
2
Hello,
I tried to set replication in the new way but I can't get it to working state.

I have setup, where I use non-root user on remote machine, in legacy mode works everything fine. But with new method I end with "Command not found" errors right after replication task starts.

Can you give me some advice what I can to try for debugging this issue? I tried to look to zettarepl.log and in other logs for "command not found" on both machines but I didn't find anything.

Code:
[2020/02/11 10:30:25] INFO     [Thread-3] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/02/11 10:30:25] INFO     [Thread-3] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2020/02/11 10:30:31] WARNING  [replication_task__task_1] [zettarepl.replication.run] No incremental base for replication task 'task_1' on dataset 'sasbox01', destroying all destination snapshots
[2020/02/11 10:30:31] INFO     [replication_task__task_1] [zettarepl.snapshot.destroy] On <Shell(<SSH Transport(repuser1@10.88.62.167)>)> for dataset 'TH-Backup01/REPL-sasbox01' destroying snapshots ['auto-20200202.1023-3d', 'auto-20200202.1123-3d', 'auto-20200202.1223-3d', 'auto-20200202.1323-3d', 'auto-20200202.1423-3d', 'auto-20200202.1523-3d', 'auto-20200202.1623-3d', 'auto-20200202.1723-3d', 'auto-20200202.1823-3d', 'auto-20200202.1923-3d', 'auto-20200202.2023-3d', 'auto-20200202.2123-3d', 'auto-20200202.2223-3d', 'auto-20200202.2323-3d', 'auto-20200203.0023-3d', 'auto-20200203.0123-3d', 'auto-20200203.0223-3d', 'auto-20200203.0323-3d', 'auto-20200203.0423-3d', 'auto-20200203.0523-3d', 'auto-20200203.0623-3d', 'auto-20200203.0723-3d', 'auto-20200203.0823-3d', 'auto-20200203.0923-3d', 'auto-20200203.1023-3d', 'auto-20200203.1123-3d', 'auto-20200203.1223-3d', 'auto-20200203.1323-3d', 'auto-20200203.1423-3d', 'auto-20200203.1523-3d', 'auto-20200203.1623-3d', 'auto-20200203.1723-3d', 'auto-20200203.1823-3d', 'auto-20200203.1923-3d', 'auto-20200203.2023-3d', 'auto-20200203.2123-3d', 'auto-20200203.2223-3d', 'auto-20200203.2323-3d', 'auto-20200204.0023-3d', 'auto-20200204.0123-3d', 'auto-20200204.0223-3d', 'auto-20200204.0323-3d', 'auto-20200204.0423-3d', 'auto-20200204.0523-3d', 'auto-20200204.0623-3d', 'auto-20200204.0723-3d', 'auto-20200204.0823-3d', 'auto-20200204.0923-3d', 'auto-20200204.1023-3d', 'auto-20200204.1123-3d', 'auto-20200204.1223-3d', 'auto-20200204.1323-3d', 'auto-20200204.1423-3d', 'auto-20200204.1523-3d', 'auto-20200204.1623-3d', 'auto-20200204.1723-3d', 'auto-20200204.1823-3d', 'auto-20200204.1923-3d', 'auto-20200204.2023-3d', 'auto-20200204.2123-3d', 'auto-20200204.2223-3d', 'auto-20200204.2323-3d', 'auto-20200205.0023-3d', 'auto-20200205.0123-3d', 'auto-20200205.0223-3d', 'auto-20200205.0323-3d', 'auto-20200205.0423-3d', 'auto-20200205.0523-3d', 'auto-20200205.0623-3d', 'auto-20200205.0723-3d', 'auto-20200205.0823-3d', 'auto-20200205.0923-3d', 'auto-20200205.1023-3d']
[2020/02/11 10:30:44] INFO     [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'sasbox01' to 'TH-Backup01/REPL-sasbox01' of snapshot='auto-20200208.1023-3d' incremental_base=None receive_resume_token=None
[2020/02/11 10:30:44] ERROR    [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(1, "full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K\nfull send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K\nUnmatched '''.\nIllegal variable name.\nAmbiguous output redirect.\n{: Command not found.\n}: Command not found.\n{: Command not found.\n}: Command not found.\nToo many )'s.\npipestatus0: Undefined variable.\npipestatus1: Undefined variable.\nUnmatched '''.\n")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 119, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 180, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 352, in run_replication_steps
    replicate_snapshots(step_template, incremental_base, snapshots, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 404, in replicate_snapshots
    run_replication_step(step_template.instantiate(incremental_base=incremental_base, snapshot=snapshot), observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 449, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh.py", line 146, in wait
    self.async_exec.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/async_exec_tee.py", line 100, in wait
    raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
Unmatched '''.
Illegal variable name.
Ambiguous output redirect.
{: Command not found.
}: Command not found.
{: Command not found.
}: Command not found.
Too many )'s.
pipestatus0: Undefined variable.
pipestatus1: Undefined variable.
Unmatched '''.
 
Joined
Feb 11, 2020
Messages
2
I tried to investigate what happens and I end with "root" user replication to remote server, with "root" works everything fine. That is fact, that chapter from documentation for how to setup nonroot replication disappeared and looks it like only root setup is supported now?

There is some problem with permission on remote server if you want use non-root user. It is somehow dedicated to this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232021 . Owner of pool directories is root and then nonroot user cannot mount that pools. I tried to change owner permission to nonroot user, but after unmount/mount was there root again.


There is log what I tried:

I tried to run manual replication acording to manual
Code:
root@TH-Freenas[~]# zfs send -V -p sasbox01@auto-20200208.1023-3d | ssh -i /data/ssh/replication repuser1@10.88.62.167 zfs recv -s -F TH-Backup01/REPL-sasbox01
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
cannot receive aclinherit property on TH-Backup01/REPL-sasbox01: permission denied
cannot receive compression property on TH-Backup01/REPL-sasbox01: permission denied
cannot receive aclmode property on TH-Backup01/REPL-sasbox01: permission denied
cannot mount 'TH-Backup01/REPL-sasbox01': Insufficient privileges
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/VMDS': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/sasbox01': failed to create mountpoint



I tried to fix missing permissions on pool
Code:
root@freenas-backup[~]# zfs allow TH-Backup01/REPL-sasbox01
---- Permissions on TH-Backup01/REPL-sasbox01 ------------------------
Local+Descendent permissions:
    user repuser1 create,destroy,diff,mount,readonly,receive,release,send,userprop

root@freenas-backup[~]# zfs allow -ldu repuser1 aclinherit,compression,aclmode TH-Backup01/REPL-sasbox01

root@freenas-backup[~]# zfs allow TH-Backup01/REPL-sasbox01
---- Permissions on TH-Backup01/REPL-sasbox01 ------------------------
Local+Descendent permissions:
    user repuser1 aclinherit,aclmode,compression,create,destroy,diff,mount,readonly,receive,release,send,userprop


I tried manual replication again:

Code:
root@TH-Freenas[~]# zfs send -V -p sasbox01@auto-20200208.1023-3d | ssh -i /data/ssh/replication repuser1@10.88.62.167 zfs recv -s -F TH-Backup01/REPL-sasbox01
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
cannot receive new filesystem stream: destination has snapshots (eg. TH-Backup01/REPL-sasbox01@auto-20200208.1023-3d)
must destroy them to overwrite it
root@TH-Freenas[~]#

root@freenas-backup[~]# zfs destroy TH-Backup01/REPL-sasbox01@auto-20200208.1023-3d

root@TH-Freenas[~]# zfs send -V -p sasbox01@auto-20200208.1023-3d | ssh -i /data/ssh/replication repuser1@10.88.62.167 zfs recv -s -F TH-Backup01/REPL-sasbox01
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
full send of sasbox01@auto-20200208.1023-3d estimated size is 18.6K
cannot mount 'TH-Backup01/REPL-sasbox01': Insufficient privileges
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/Datastorage_synology': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/Library': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/VMDS': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/home': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage/download': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage/images': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage/jails': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage/log': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage/releases': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/iocage/templates': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/sasbox01': failed to create mountpoint
cannot mount '/mnt/TH-Backup01/REPL-sasbox01/timemachine': failed to create mountpoint
root@TH-Freenas[~]#


I tried to change owner permission on pool and do it traverse for all childs.
Only permissions of parent are changed.

I tried to change owner permission on child pool
Error: [EFAULT] [path] null not allowed

From browser console I found
Code:
arguments: [ "TH-Backup01/REPL-sasbox01/Datastorage_synology”, { acl: [], options: { recursive: false, stripacl: false, traverse: false }, user: "repuser1" } ]

Code:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
    await self.future
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 3125, in permission
    raise CallError(pjob.error)
middlewared.service_exception.CallError: [EFAULT] [path] null not allowed


And this is where I end. I tried to switch ssh user to "root" and with it works it like usually.
 
Joined
Jul 7, 2018
Messages
6
Hi,
same error here, even with the root user.
I have no idea how to fix this problem.
First I thought, the reason is that I want to replicate a dataset called M2TB to a dataset called Backup recursivly.
But even a M2TB/Allgemein --> Backup/Allgemein didn't work. (Like it is necessary for legacy mode)
Any hints?
 
Top