Replication problems after upgrade from 11.2 to 11.3

[vafk]

After upgrading from 11.2 to 11.3 replication does not work as it used before.

Periodic Snapshot Tasks are working (I see Snapshots in Storage) but Replication Tasks give me trouble:

Out of 8 tasks, only 2 show that a last snapshot was replicated, 5 show finished albeit next to it says "No snapshots sent yet" and snapshot of home reports error (see below).

Anyone got this working and has an idea how to find my fault? I already created both snapshot and replication tasks from scratch...


This is the error log for dataset which says "Finished"
-----------------------------------------------------------------
[2020/02/16 16:44:51] INFO [Thread-236] [zettarepl.paramiko.replication_task__task_54] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/02/16 16:44:51] INFO [Thread-236] [zettarepl.paramiko.replication_task__task_54] Authentication (publickey) successful!
[2020/02/16 16:44:51] INFO [replication_task__task_54] [zettarepl.replication.run] No snapshots to send for replication task 'task_54' on dataset 'nas1'
[2020/02/16 16:44:52] INFO [replication_task__task_54] [zettarepl.replication.run] No snapshots to send for replication task 'task_54' on dataset 'nas1/Camera'
-----------------------------------------------------------------


This is the error log of replication of home-dataset
-----------------------------------------------------------------
full send of nas1/home@auto-2020-02-16_04-00 estimated size is 17.6M
full send of nas1/home@auto-2020-02-16_04-00 estimated size is 17.6M
Unmatched '''.
Illegal variable name.
Ambiguous output redirect.
{: Command not found.
}: Command not found.
{: Command not found.
}: Command not found.
Too many )'s.
pipestatus0: Undefined variable.
pipestatus1: Undefined variable.
Unmatched '''.
warning: cannot send 'nas1/home@auto-2020-02-16_04-00': signal received.
Logs
[2020/02/16 16:00:00] INFO [Thread-182] [zettarepl.paramiko.replication_task__task_53] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/02/16 16:00:00] INFO [Thread-182] [zettarepl.paramiko.replication_task__task_53] Authentication (publickey) successful!
[2020/02/16 16:00:01] INFO [replication_task__task_53] [zettarepl.replication.run] For replication task 'task_53': doing push from 'nas1/home' to 'backup1' of snapshot='auto-2020-02-16_04-00' incremental_base=None receive_resume_token=None
[2020/02/16 16:00:01] ERROR [replication_task__task_53] [zettarepl.replication.run] For task 'task_53' unhandled replication error ExecException(141, "full send of nas1/home@auto-2020-02-16_04-00 estimated size is 17.6M\nfull send of nas1/home@auto-2020-02-16_04-00 estimated size is 17.6M\nUnmatched '''.\nIllegal variable name.\nAmbiguous output redirect.\n{: Command not found.\n}: Command not found.\n{: Command not found.\n}: Command not found.\nToo many )'s.\npipestatus0: Undefined variable.\npipestatus1: Undefined variable.\nUnmatched '''.\nwarning: cannot send 'nas1/home@auto-2020-02-16_04-00': signal received\n")
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 119, in run_replication_tasks
run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 180, in run_replication_task_part
run_replication_steps(step_templates, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 352, in run_replication_steps
... 17 more lines ...
Ambiguous output redirect.
{: Command not found.
}: Command not found.
{: Command not found.
}: Command not found.
Too many )'s.
pipestatus0: Undefined variable.
pipestatus1: Undefined variable.
Unmatched '''.
warning: cannot send 'nas1/home@auto-2020-02-16_04-00': signal received

 

[zamana]

Same here:

Code:

full send of storage/backup@auto-2020-02-19_22-21 estimated size is 41.1G
full send of storage/backup@auto-2020-02-19_22-21 estimated size is 41.1G
Unmatched '''.
Illegal variable name.
Ambiguous output redirect.
{: Command not found.
}: Command not found.
{: Command not found.
}: Command not found.
Too many )'s.
pipestatus0: Undefined variable.
pipestatus1: Undefined variable.
Unmatched '''.
warning: cannot send 'storage/backup@auto-2020-02-19_22-21': signal received.
Logs
[2020/02/21 09:53:23] INFO     [Thread-35] [zettarepl.paramiko.replication_task__task_37] Connected (version 2.0, client OpenSSH_7.8)
[2020/02/21 09:53:23] INFO     [Thread-35] [zettarepl.paramiko.replication_task__task_37] Authentication (publickey) successful!
[2020/02/21 09:53:23] INFO     [replication_task__task_37] [zettarepl.replication.run] For replication task 'task_37': doing push from 'storage/backup' to 'storage' of snapshot='auto-2020-02-19_22-21' incremental_base=None receive_resume_token=None
[2020/02/21 09:53:25] ERROR    [replication_task__task_37] [zettarepl.replication.run] For task 'task_37' unhandled replication error ExecException(141, "full send of storage/backup@auto-2020-02-19_22-21 estimated size is 41.1G\nfull send of storage/backup@auto-2020-02-19_22-21 estimated size is 41.1G\nUnmatched '''.\nIllegal variable name.\nAmbiguous output redirect.\n{: Command not found.\n}: Command not found.\n{: Command not found.\n}: Command not found.\nToo many )'s.\npipestatus0: Undefined variable.\npipestatus1: Undefined variable.\nUnmatched '''.\nwarning: cannot send 'storage/backup@auto-2020-02-19_22-21': signal received\n")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 119, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 180, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 352, in run_replication_steps
... 17 more lines ...
Ambiguous output redirect.
{: Command not found.
}: Command not found.
{: Command not found.
}: Command not found.
Too many )'s.
pipestatus0: Undefined variable.
pipestatus1: Undefined variable.
Unmatched '''.
warning: cannot send 'storage/backup@auto-2020-02-19_22-21': signal received

 

[Samuel Mutel]

Same problem ...

 

[vafk]

Shame only Newbies like us report/have the problem and none of the programmers cares to post if this is a bug, if there is a solution or if we are to blame :-(

 

[zamana]

Well... I "solved" the problem by deleting and recreating all my replication tasks again. Again.

 

[vafk]

@zamana
I have also found another problem (bug?) which I reported here

FreeNAS 11.3 will not start replication task to rebooted target (11.3)

I have two FreeNAS running11.3 with replication tasks. These get executed from source to target but when target is switched off and source senses that target was off it says "Host is down" and remains in this state even when target gets online again. Only after manually starting the task with...

www.ixsystems.com

You may run into this as well if you reboot your target.

 

[Adrian]

I have whinged about being unable to get replication working. I hope for improvements in 11.3-U1 and that it will become usable.

 

[Kir]

Same problem. First time user and was struggling trying to figure out what I was doing wrong.

 

[lonnie776]

I was having this problem as well after using the new Replication system. At first, I couldn't get any of my previous replications to work but after I created them from scratch some of them began to work.

After some trial and error, I found that my error was caused by "Stream Compression". When Stream Compression was pigs or lz4 (didn't try plzip), it would throw the errors mentioned above (Unmatched '"'). When Stream Compression was set to Disable, it would work as expected.

Clearly there are still some bugs to be worked out, and hopefully, this fixes your problems too.

 

[echelon5]

I had the same issue but for some tasks it started working by changing the replication schedule. So, for example for a "legacy" snapshot schedule of 4 weeks, it would error with a daily replication schedule. Switching the replication schedule to 4 weeks would "fix" the task.