Unable to replicate in latest freenas.

[Dariusz1989]

Hey

I've installed latest freenas to use the replication wizard to create the job but when ever I try it I get error

```
[2020/02/15 17:36:25] ERROR [replication_task__task_2] [zettarepl.replication.run] For task 'task_2' non-recoverable replication error ReplicationError("Target dataset 'generalPool/backup_CGI_dataset' does not have snapshots but has data (192192 bytes used) and replication from scratch is not allowed. Refusing to overwrite existing data.")
```

The replication task is being set to go from poolA to poolB, in poolB I made new dataset SMB type. It has nothinbg on it, yet the error above kick off the second I try to replicate.

What do I do?

 

[Dariusz1989]

Hey

Ok so I had some success. Mainly by creating 5 wizzards and one of them working... but after copying 4tb of data +/- I now get error :

Code:

cannot receive resume stream: dataset is busy warning: cannot send 'vol_CGI/d_CGI@auto-2020-02-11_00-00': signal received.

What do I do ?

Its a local copy from 1 pool to another, from 1 data set + snapshots to another empty new dataset.

 

[vafk]

Are you talking about 11.3 (please be specific) and did you install FreeNAS from scratch or did you upgrade (for instance from 11.2 to 11.3)?

I had similar problem after upgrade from 11.2 to 11.3 and solved the following way:
1. Deleted all my previous snapshots (I did not need them anyway but consider the risk).
2. Deleted all periodic snapshot tasks.
3. Deleted all replication tasks.
4. Created new periodic snapshot tasks.
5. Created new replication tasks. I did not use the wizard because anther user reported problem but used advanced replication creation button.

I still had problems because it took time before the periodic snapshot was created so I went to bed. In the morning FreeNAS1 created all necessary snapshots and after creating (4) it immediately started. I also had a error because FreeNAS2 contained old data so I had to tick "Replicate from scratch if incremental is not possible".

Hope this helps.

 

[Dariusz1989]

Hey

Yes its 11.3 latest. I updated lately, made new dataset and started using for 1st time replicaitons.

Your points 1-5 sound to me like total wipe and restart... is that really a solution? I need these snapshots.

 

[Dariusz1989]

Some more logs...

Code:

Error
cannot receive resume stream: dataset is busy
warning: cannot send 'vol_CGI/d_CGI@auto-20190720.2000-2w': signal received.
Logs
[2020/02/16 11:42:31] INFO     [replication_task__task_4] [zettarepl.replication.run] Resuming replication for destination dataset 'generalPool/backup_CGI_dataset/d_CGI'
[2020/02/16 11:42:31] INFO     [replication_task__task_4] [zettarepl.replication.run] For replication task 'task_4': doing push from 'vol_CGI/d_CGI' to 'generalPool/backup_CGI_dataset/d_CGI' of snapshot=None incremental_base=None receive_resume_token='1-11aef9a8d6-f0-789c6360640003....82a2d4e2e2fc6c063800004014215d'
[2020/02/16 11:42:36] WARNING  [replication_task__task_4] [zettarepl.replication.run] receive_resume_token for dataset 'generalPool/backup_CGI_dataset/d_CGI' references snapshot that no longer exists, discarding it
[2020/02/16 11:42:37] INFO     [replication_task__task_4] [zettarepl.replication.run] For replication task 'task_4': doing push from 'vol_CGI/d_CGI' to 'generalPool/backup_CGI_dataset/d_CGI' of snapshot='auto-20190720.2000-2w' incremental_base=None receive_resume_token=None
[2020/02/16 12:00:32] INFO     [replication_task__task_4] [zettarepl.replication.run] Resuming replication for destination dataset 'generalPool/backup_CGI_dataset/d_CGI'
[2020/02/16 12:00:32] INFO     [replication_task__task_4] [zettarepl.replication.run] For replication task 'task_4': doing push from 'vol_CGI/d_CGI' to 'generalPool/backup_CGI_dataset/d_CGI' of snapshot=None incremental_base=None receive_resume_token='1-ebac38418-f0-789c63606400031....5b50945a5c9c9fcd000700356b214f'
[2020/02/16 12:00:32] ERROR    [replication_task__task_4] [zettarepl.replication.run] For task 'task_4' unhandled replication error ExecException(255, "cannot receive resume stream: dataset is busy\nwarning: cannot send 'vol_CGI/d_CGI@auto-20190720.2000-2w': signal received\n")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 119, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
... 6 more lines ...
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/local.py", line 128, in wait
    self.async_exec.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/local.py", line 50, in wait
    raise ExecException(self.process.returncode, stdout)
zettarepl.transport.interface.ExecException: cannot receive resume stream: dataset is busy
warning: cannot send 'vol_CGI/d_CGI@auto-20190720.2000-2w': signal received

 

[MikeyG]

@Dariusz1989 Did you ever find a solution for this? I'm getting a similar error in 11.3-U2. In my case if I have two replication tasks scheduled for the same time, they don't seem to queue. One of them just fails with "dataset is busy."

 

[Dariusz1989]

@mgittelman I've updated to latest freenas and tried again from scratch. It somehow "worked"... overall it feels very unstable for simple pc users.

 

[jsb]

Hy guys,

I have a similar problem. We have two Storage the StorageA and StorageB. A replicates to StorageB. In StorageB, we set up a periodic snapshoot task once a month, and it will be maintained for 2 months, after this period it will be excluded. We also set up replication tasks in StorageB for two external hard drives that are connected directly to StorageB via USB cable. However, on the day of taking a snapshoot of the StorageB dataset and replicating this periodic snapshot to the external hard drives, it presents the following error messages as I checked in the logs:

[2020/07/01 00:00:17] INFO [retention] [zettarepl.zettarepl] Retention destroying local snapshots: []
[2020/07/01 00:00:17] WARNING [retention] [zettarepl.zettarepl] Remote retention failed on <LocalTransport ()>: error listing snapshots: ExecException (1, "cannot open 'BackupA_1': dataset does not exist \ n ")

"BackupA_1" is the external hard drive to receive data from the replication job. The Freenas

Version: FreeNAS-11.3-U3.2

Has anyone had this problem and have any idea how to solve it?

Thanks.

 

[develissimo]

pump

Same issue here - Replication 'State=ERROR':
2020/08/22 14:07:06] INFO [replication_task__task_22] [zettarepl.replication.run] For replication task 'task_22': doing push from 'hdd_pool03/dataset_ovirt_defaultDC' to 'hdd_pool03/dataset_ovirt_defaultDC' of snapshot='auto-2020-08-17_01-15' incremental_base='auto-2020-08-16_01-45' receive_resume_token=None
[2020/08/22 14:07:06] INFO [replication_task__task_22] [zettarepl.paramiko.replication_task__task_22.sftp] [chan 19] Opened sftp connection (server version 3)
[2020/08/22 14:07:06] INFO [replication_task__task_22] [zettarepl.transport.ssh_netcat] Automatically chose connect address 'example.develissimo.com'
... 17 more lines ...
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 500, in run_replication_step
ReplicationProcessRunner(process, monitor).run()
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
raise self.process_exception
File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
self.replication_process.wait()
File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh_netcat.py", line 183, in wait
raise SshNetcatExecException(connect_exec_error, self.listen_exec_error) from None
zettarepl.transport.ssh_netcat.SshNetcatExecException: I/O error
dataset is busy

Info Replication of same Data to another Server succeeded yesterday, so right now I guess the receiving side could be the issue.

FreeNAS-Versions:
Sending-Server: FreeNAS 11.3-U3.2
Receiving-Server: FreeNAS 11.3-U3.2

Already tried:
Tried Server Restart of Receiving Side. ⇾ Same issue after reboot.
Tried multiple times. ⇾ Always failing with same issue.

TODO:
I will check if next replication to the other server will still succeed. But I guess it will, so right now with all the information I have, the issue seems on the receiving server side.
Will post as soon as i have new information about that issue.

 

[Basil Hendroff]

I suggest you update to U4.1 as it included replication updates and then repeat your tests.

 

[develissimo]

@Basil Hendroff, thanks for your advice.
Right now we are not upgrading our three servers, as the main-server is in 24/7 production.

But it seems sometimes ERROR messages should be interpreted just like that: "busy" means "busy"
We waited a bit... the process performed once more again and now:

Task is running
90.00%
Sending 10 of 10: hdd_pool03/dataset_ovirt_defaultDC@auto-2020-08-23_01-45


so i will wait some more minutes and report back, if the issue got solved by its own.


Servers still running on: FreeNAS 11.3-U3.2

Important Info:
this Receiving-Server is just serving for backup purposes and was started short time before the Main-Server started the replication for energy saving reasons (climate change and so...). It seems that the dataset was just busy after reboot. We did try multiple times, but always with a reboot upfront and i guess not enough time for the server to settle.

Issue solved. FINISHED => GREEN

Kind regards.