Replication-"refusing to overwrite existing data"

[Lyndon]

Hi,

I have used FreeNAS for quite awhile- but new to snapshots and replication. I am using version 11.3 and the replication wizard. I have created the pool/dataset on the destination from scratch. Pool is two hard disks striped into single vdev on second freenas server. The dataset gets created successfully- but isnt completely empty. The process writes something into the dataset.

When I use replication wizard I identify source dataset and target dataset. They are both seen by wizard on source machine. I accept the defaults pretty much and create task. I then try the run now option. After a few seconds it comes up with red error box with the following error:

Target dataset 'FREENASBKUP' does not have snapshots but has data (9936896 bytes used) and replication from scratch is not allowed. Refusing to overwrite existing data.

Seems to be complaining about the 9 meg of information written at time of pool/dataset creation. How do I prevent anything written upon creation of target dataset so its completely empty- or force an overwrite? Or what I am not understanding? The wizard seems happy with rep task creation and I have snapshots to copy over...

Any thoughts?

Thanks

 

[feistyviking]

I am still doing some digging myself, but I just got hit with the same error.

Existing pool trying to push data to newly created local pool

 

[feistyviking]

So after looking more at the error, I see that it didn't have permission to start from scratch (delete old snapshots on target dataset)

Click on "Edit" for your task and then check the box for this option.

 

[Lyndon]

will try that -thanks!

 

[gconstan99]

I am having the same problem. I also tried the "Replicate from scratch if incremental is not possible", but I am still getting the same error.

 

[feistyviking]

Could you post your error log?

 

[gconstan99]

I had an entry for the Dataset in the "Periodic Snapshot Tasks" on the originating FreeNAS, but apparently that was not enough. As soon as I created an entry for the Dataset in the "Periodic Snapshot Tasks" on the receiving FreeNAS (even thought this was a totally empty Dataset), the process in the "Replication Tasks" finished successfully! It could be that Periodic Snapshot Tasks need to be setup on both sites (originating/Push as well as receiving/Pull) for the respective Datasets, before a Replication Task can be created and subsequently executed.

 

[bsodmike]

So after looking more at the error, I see that it didn't have permission to start from scratch (delete old snapshots on target dataset)

Click on "Edit" for your task and then check the box for this option.

View attachment 35462

This worked for me, thanks!

 

[markmuehlbauer]

11.3, Replication task fails to work. Maybe U1 might have this resolved.

[2020/02/12 09:32:52] INFO [Thread-19] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/02/12 09:32:52] INFO [Thread-19] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2020/02/12 09:32:52] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'Array01/edgewoodpc' to 'faster/backups/edgewood' of snapshot='auto-2020-02-12_09-27' incremental_base=None receive_resume_token=None
[2020/02/12 09:32:53] ERROR [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(141, "full send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G\nfull send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G\ncannot unmount '/mnt/faster/backups/edgewood': Device busy\nwarning: cannot send 'Array01/edgewoodpc@auto-2020-02-12_09-27': signal received\n")
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 119, in run_replication_tasks
run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 180, in run_replication_task_part
run_replication_steps(step_templates, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 352, in run_replication_steps
replicate_snapshots(step_template, incremental_base, snapshots, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 404, in replicate_snapshots
run_replication_step(step_template.instantiate(incremental_base=incremental_base, snapshot=snapshot), observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 449, in run_replication_step
ReplicationProcessRunner(process, monitor).run()
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
raise self.process_exception
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
self.replication_process.wait()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh.py", line 146, in wait
self.async_exec.wait()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/async_exec_tee.py", line 100, in wait
raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: full send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G
full send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G
cannot unmount '/mnt/faster/backups/edgewood': Device busy
warning: cannot send 'Array01/edgewoodpc@auto-2020-02-12_09-27': signal received

 

[feistyviking]

11.3, Replication task fails to work. Maybe U1 might have this resolved.

[2020/02/12 09:32:52] INFO [Thread-19] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/02/12 09:32:52] INFO [Thread-19] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2020/02/12 09:32:52] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'Array01/edgewoodpc' to 'faster/backups/edgewood' of snapshot='auto-2020-02-12_09-27' incremental_base=None receive_resume_token=None
[2020/02/12 09:32:53] ERROR [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(141, "full send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G\nfull send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G\ncannot unmount '/mnt/faster/backups/edgewood': Device busy\nwarning: cannot send 'Array01/edgewoodpc@auto-2020-02-12_09-27': signal received\n")
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 119, in run_replication_tasks
run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 180, in run_replication_task_part
run_replication_steps(step_templates, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 352, in run_replication_steps
replicate_snapshots(step_template, incremental_base, snapshots, observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 404, in replicate_snapshots
run_replication_step(step_template.instantiate(incremental_base=incremental_base, snapshot=snapshot), observer)
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 449, in run_replication_step
ReplicationProcessRunner(process, monitor).run()
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
raise self.process_exception
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
self.replication_process.wait()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh.py", line 146, in wait
self.async_exec.wait()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/async_exec_tee.py", line 100, in wait
raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: full send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G
full send of Array01/edgewoodpc@auto-2020-02-12_09-27 estimated size is 338G
cannot unmount '/mnt/faster/backups/edgewood': Device busy
warning: cannot send 'Array01/edgewoodpc@auto-2020-02-12_09-27': signal received

Based off your logs, I would say that this looks like a separate issue than the one we were experiencing. Was this an existing replication task that was working before upgrading?

 

[markmuehlbauer]

It's a new rep task created with the GUI. I know my SSH token works fine as I can browse to the destination and select the dataset.
Latest update: it looks like it negotiated the connection fine, but failed to transfer.

[2020/02/12 09:54:02] INFO [Thread-11] [zettarepl.paramiko.replication_task__task_4] Connected (version 2.0, client OpenSSH_8.0-hpn14v15)
[2020/02/12 09:54:03] INFO [Thread-11] [zettarepl.paramiko.replication_task__task_4] Authentication (publickey) successful!
[2020/02/12 09:54:03] INFO [replication_task__task_4] [zettarepl.replication.run] For replication task 'task_4': doing push from 'faster/backups/edgewood' to 'Array01/backup/Edgewoood Backup' of snapshot='auto-2020-02-12_09-43' incremental_base=None receive_resume_token=None
[2020/02/12 09:54:03] INFO [replication_task__task_4] [zettarepl.paramiko.replication_task__task_4.sftp] [chan 4] Opened sftp connection (server version 3)
[2020/02/12 09:54:03] INFO [replication_task__task_4] [zettarepl.transport.ssh_netcat] Automatically chose connect address '10.30.8.14'
[2020/02/12 09:54:13] WARNING [replication_task__task_4.stdout_copy] [zettarepl.transport.base_ssh.root@10.30.8.14.shell.6.async_exec.34] Copying stdout from <paramiko.ChannelFile from <paramiko.Channel 5 (open) window=65536 -> <paramiko.Transport at 0xee608d0 (cipher aes128-ctr, 128 bits) (active; 2 open channel(s))>>> failed: timeout()

 

[Matt_G]

Hi,

I have used FreeNAS for quite awhile- but new to snapshots and replication. I am using version 11.3 and the replication wizard. I have created the pool/dataset on the destination from scratch. Pool is two hard disks striped into single vdev on second freenas server. The dataset gets created successfully- but isnt completely empty. The process writes something into the dataset.

When I use replication wizard I identify source dataset and target dataset. They are both seen by wizard on source machine. I accept the defaults pretty much and create task. I then try the run now option. After a few seconds it comes up with red error box with the following error:

Target dataset 'FREENASBKUP' does not have snapshots but has data (9936896 bytes used) and replication from scratch is not allowed. Refusing to overwrite existing data.

Seems to be complaining about the 9 meg of information written at time of pool/dataset creation. How do I prevent anything written upon creation of target dataset so its completely empty- or force an overwrite? Or what I am not understanding? The wizard seems happy with rep task creation and I have snapshots to copy over...

Any thoughts?

Thanks

In the past, when creating a replication task, the creation wizard would create the destination dataset on the destination pool. That way it would have the proper permissions and attributes for the job. I see nothing in the newer 11.3 manual to suggest this has changed.

Can you try wiping the dataset off the destination pool and letting the system create it and see if that resolves your issue?

 

[markmuehlbauer]

In the past, when creating a replication task, the creation wizard would create the destination dataset on the destination pool. That way it would have the proper permissions and attributes for the job. I see nothing in the newer 11.3 manual to suggest this has changed.

Can you try wiping the dataset off the destination pool and letting the system create it and see if that resolves your issue?

Trying to use the Wizard to accomplish the migration. Being the wizard asks us to define the destination dataset, I don't see how I can do what you ask. If your point is to do this manually, fair enough, and no, won't bother with it.

 

[tiberiusQ]

same issue here with u1....not cool

 

[ChrisHH]

Same issue here. Deleted all snapshots, snapshot tasks and replication tasks. Started over and successfully sent the first set. On the remote server, I'm not seeing any snapshots, just data. The next day, I got the error.

 

[Basil Hendroff]

The fix for me was to change the replication task transport to LEGACY.

 

[ChrisHH]

What I found is that the corresponding snapshot task has to run on it's schedule. If you kick of the replication task first, it appears to copy the data, but not the actual snapshot. To fix, I deleted the remote datasets, verified that the snapshots existed on the main server and started the replication tasks. It had to copy the data over, but everything is working.

 

[sgtpepperaut]

this has been renamed, the option is now called
Synchronize Destination Snapshots With Source