Replication between two FreeNAS-11.2-U4.1 suddenly stopped

[vafk]

I am running FreeNAS-11.2-U4.1 on main server and a backup server with FreeNAS-11.2-U4.1. Created replication tasks of my snapshots. That job worked well but today I realized that the replication stopped 4 days ago and no way to get it running.

I found on main server that although the snapshots have a life time of 2 weeks about 2000 snapshots older than 2 weeks. I deleted them with the exception of "nas1/iocage/releases/11.2-RELEASE/root@jail-vorlage" which cannot be deleted.

After I deleted the snapshots nothing happened. I deleted and recreated all snapshot tasks - no replication. After that I deleted and recreated all replication tasks, no replication, system says waiting...

 

[artlessknave]

you need to check "Delete stale snapshots" for the replication to delete expired snapshots, otherwise everything stays on the destination. as for your replications not working, I don't see enough info here to even speculate.

 

[vafk]

For the snapshot mentioned in my post there is no replication task existing. Now there is only this snapshot, when remains even after tried to be deleted, without FreeNAS giving a message why not deleted

Anyway my problem was the replication having suddenly stopped. It was working for a few days after it was set up, then not beeing touched stopped and shows no error, just "waiting".

The replication tasks are set up in the manual mode. However I have it back working and I suspect that there is a bug so please if someone who is responsible reads this post should have a look into it.

Yesterday when I was looking into the problem I created a new snapshot and a new replication task. Then I received the error, that the remote hostname was not reachable. Unfortunately I did not make a screenshot. Today, I tried this step a second time and in the moment when I pasted the replication token obtained from backup, the error was gone and the first of my old replication tasks was showing "Sending" (without the snapshot-name). I verified if the files within the replicated folder on backup was accessible (windows explorer) and it was there so I knew that the replication has not started yet. It took some time before the status extended to "Sending nas1/Backups@auto-20190520.1000-2w" and then I could see that real data was flowing.

I suspect that for some reason FreeNAS on server "lost" the credentials to connect to backup, without giving a error. Only after the new token was pasted in the new replication task, FreeNAS got it right and picked up its work.

Any thoughts to this welcome because it is my nightmare to find this out after days just by chance...

 

[dlavigne]

Notes that the bug ticket is at https://jira.ixsystems.com/browse/NAS-101826.