Replication Task Error

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
Hello,
Here is another noob question, this time on replication tasks. I set up daily, weekly, and monthly snapshot for my main data pool, with appropriate keep times. I want to replicate the snapshots to a backup drive. I set up a local replication task and select for "Periodic Snapshot Tasks" the daily snapshot and all worked fine for the last week. Today I added the weekly and monthly snapshot by clicking the checkmarks for the snapshots in "Periodic Snapshot Tasks"; so there are 3 checkmarks total now. When I run the task, I get the error below. If someone could tell me what I am doing wrong, I would appreciate it. Thank you!

Code:
[2020/07/01 08:40:17] INFO     [replication_task__task_3] [zettarepl.replication.run] For replication task 'task_3': doing push from 'tank1/users' to 'backup1/backup/users' of snapshot='monthly-auto-2020-07-01_00-00' incremental_base='daily-auto-2020-07-01_00-00' receive_resume_token=None
[2020/07/01 08:40:17] ERROR    [replication_task__task_3] [zettarepl.replication.run] For task 'task_3' unhandled replication error ExecException(1, 'cannot receive incremental stream: most recent snapshot of backup1/backup/users does not\nmatch incremental source\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 143, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 204, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 389, in run_replication_steps
    replicate_snapshots(step_template, incremental_base, snapshots, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 444, in replicate_snapshots
    run_replication_step(step_template.instantiate(incremental_base=incremental_base, snapshot=snapshot), observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 500, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/local.py", line 128, in wait
    self.async_exec.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/local.py", line 50, in wait
    raise ExecException(self.process.returncode, stdout)
zettarepl.transport.interface.ExecException: cannot receive incremental stream: most recent snapshot of backup1/backup/users does not
match incremental source
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
You'll need separate replication tasks each for the weekly and monthly snapshots. It's a 1:1 correspondence between snapshot tasks and replication tasks.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
I tried setting up a separate task for the weekly snapshots. Now I get the error message below. There is something I am not yet understanding.

Code:
[2020/07/01 09:03:34] ERROR    [replication_task__task_7] [zettarepl.replication.run] For task 'task_7' non-recoverable replication error NoIncrementalBaseReplicationError("No incremental base on dataset 'tank1/users' and replication from scratch is not allowed")
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
How many weekly snapshots do you have? You'll need at least 2.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
There are exactly 2 weekly snapshot, one dated 6/21/20 the other 6/28/20.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
OK, the replication base is in the dailies. You could try checking the checkbox for Replicate from scratch if incremental is not possible in the replication task to get around this error, as detailed in the Guide, Section 8.6.2, for advanced replication.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
I saw that option but was scared off by the "can cause data loss" warning.

EDIT: If I use this option, will this double the size of the data set on the backup drive becuase I will have one base for the dailies and another for the weeklies?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Yes, because you'll have 2 bases.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
Thanks, that's what I thought. Do you know what the reason for the data loss warning is?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Thanks, that's what I thought. Do you know what the reason for the data loss warning is?

The warning is if you already had a successful replication on the receiver. Checking this could overwrite the replicas on the target.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
Got it! Thank you for all the help again.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
I am sorry to have to reopen this one. I royally messed up the replication task, starting to get errors on the nightly task now. So I decided to start over and deleted all snapshots on the backup target. I now want to remove the user files but cannot delete them with "rm -rf *" because the file system is read-only. Mount shows
Code:
backup1/backup/users on /mnt/backup1/backup/users (zfs, local, read-only, nfsv4acls)


I tried "mount -o remount,rw /mnt/backup1/backup/users" but then I get
Code:
mount: /mnt/backup1/backup/users: unknown special file or file system


Please let me know how to delete these files.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Try zfs destroy backup1/backup/users.
 

revengineer

Contributor
Joined
Oct 27, 2019
Messages
193
Thank you once again. Based on your hint to destroy the dataset, I was able to do so from the GUI. I am just surprised that this is required and that there is no means, even as root, to simply delete the directories at the top level. In any case, the problem is solved for now and I am starting over the replication from scratch.
 
Top