replication issues

[hescominsoon]

I have two 11.3u4 mahcin3es. I have setup ssh and been able to have each get the other machines ssh key. When i try to link the two machines via the replication task dialog I get the following:
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 403, in list_datasets
datasets = await self.middleware.run_in_thread(list_datasets, shell)
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/zettarepl/dataset/list.py", line 12, in list_datasets
return [dataset["name"] for dataset in list_datasets_with_properties(shell, dataset, recursive)]
File "/usr/local/lib/python3.7/site-packages/zettarepl/dataset/list.py", line 28, in list_datasets_with_properties
return [dict(zip(properties, line.split("\t"))) for line in filter(None, shell.exec(args).split("\n"))]
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/interface.py", line 83, in exec
return self.exec_async(args, encoding, stdout).wait()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/interface.py", line 87, in exec_async
async_exec.run()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/base_ssh.py", line 25, in run
client = self.shell.get_client()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/base_ssh.py", line 103, in get_client
auth_timeout=self.transport.connect_timeout,
File "/usr/local/lib/python3.7/site-packages/paramiko/client.py", line 446, in connect
passphrase,
File "/usr/local/lib/python3.7/site-packages/paramiko/client.py", line 764, in _auth
raise saved_exception
File "/usr/local/lib/python3.7/site-packages/paramiko/client.py", line 664, in _auth
self._transport.auth_publickey(username, pkey)
File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 1580, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "/usr/local/lib/python3.7/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1084, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/replication.py", line 702, in list_datasets
return await self.middleware.call("zettarepl.list_datasets", transport, ssh_credentials)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1141, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1084, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 405, in list_datasets
raise CallError(repr(e))
middlewared.service_exception.CallError: [EFAULT] AuthenticationException('Authentication failed.')

What am i doing wrong?

 

[stewartLG]

I don't think you are doing anything wrong.

I can't create replication tasks, at least if I follow each nudge from the relevant chain of wizards. I get stuck with a seeming crash when trying to create an SSH connection.

I've logged a bug:

https://jira.ixsystems.com/browse/NAS-106682?filter=-2

This seems to have prompted some development, although nobody has responded to me directly. (I'm fine that nobody has responded to me, I just can't tell you what exactly is happening, if there are any work arounds, etc.)

 

[hescominsoon]

nods it is set to be fixed in 11.3U5 and 12.0beta2.

 

[hescominsoon]

I don't think you are doing anything wrong.

I can't create replication tasks, at least if I follow each nudge from the relevant chain of wizards. I get stuck with a seeming crash when trying to create an SSH connection.

I've logged a bug:

https://jira.ixsystems.com/browse/NAS-106682?filter=-2

This seems to have prompted some development, although nobody has responded to me directly. (I'm fine that nobody has responded to me, I just can't tell you what exactly is happening, if there are any work arounds, etc.)

your reply makes it sound like you are able to creat3e a replication task...how are you getting it to work?

 

[stewartLG]

I have not been able to get any kind of replication task working at all, manual or otherwise.

I have an elaborate 100+ TB, five server migration completely stalled by the lack of working replication configuration in FreeNAS. I'm looking forward to the next release keenly.

 

[hescominsoon]

yeah i have tried both types in the gui to no avail via the webgui at all.

 

[Samuel Tai]

Are you relying on the keys in /root/.ssh on both sides of the replication, or have you defined SSH keypairs and SSH connections in the GUI on both sides? Replication only uses the latter.

 

[hescominsoon]

Are you relying on the keys in /root/.ssh on both sides of the replication, or have you defined SSH keypairs and SSH connections in the GUI on both sides? Replication only uses the latter.

the latter not the former

 

[Samuel Tai]

Have you enabled root login in the SSH service?

 

[hescominsoon]

yes on both ends.

 

[Samuel Tai]

What does /var/log/middlewared.log show on both ends during the failed replication?

 

[NasKar]

I'm having weird things happen as well. The initial replication thru the GUI doesn't replicate all of the datasets on the remote system. Setting periodic snapshot to hourly and then checking later, the missing dataset appears. Are the snapshots supposed to be replicated to the backup target machine? If so mine are not.

 

[hescominsoon]

What does /var/log/middlewared.log show on both ends during the failed replication?

look at the bug..development has been worked and set for rlease for 11.3U5...:)

 

[edisondotme]

I was having this issue as well and I finally figured out my solution, maybe this will help anyone viewing this thread in the future.

In sum, the ssh key pair on the client system must correspond to root's public key on the remote system.
Make sure the SSH Connection you setup uses that SSH Keypair.

I mistakenly thought the key pairs needed to be the same on both systems in the System > SSH Kepairs section in the GUI. TrueNAS-12.0-U8