Replication broken

[fullspeed]

I just upgraded two of my Freenas file servers FS07/FS08 to the latest version, FS07 replicates all datasets to FS08.

When they came back up now FS07 shows "Waiting" for replication tasks (although it still looks like data is being transmitted in network graph)

The log show it trying to delete this snapshot over and over:
ep 28 14:43:02 fs07 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "storage01/ops@auto-20150928.1026-4h"
Sep 28 14:43:02 fs07 autosnap.py: [tools.autosnap:323] Failed to destroy snapshot 'storage01/ops@auto-20150928.1026-4h': cannot destroy snapshot storage01/ops@auto-20150928.1026-4h: dataset is busy

I really hope I don't have to re-initiate replication.. t

 

[depasseg]

I think the underlying issue is snapshots. There are a couple bugs opened and I have a thread here about it. No good work around so far, other than to try to delete all the snapshots and restart replication. Stay tuned.

https://bugs.freenas.org/issues/11647
https://bugs.freenas.org/issues/11601

 

[depasseg]

https://forums.freenas.org/index.ph...us-out-of-whack-after-upgrade-20150902.38275/

 

[fullspeed]

https://forums.freenas.org/index.ph...us-out-of-whack-after-upgrade-20150902.38275/

Ugh I just got done replicating 40TB of data, that hurts.

I'm just going to stop upgrading after multiple instances of broken replication.

Thanks for info, much appreciated.

 

[fullspeed]

Alright so I blew away all my snapshots on both sides and re-enabled replication.. it doesn't say that it's sending data via "replication tasks" in the GUI but I can see it's started the process as the volume got created on my target server and data is flowing.

The speed is a bit odd, before I upgraded I was at a constant gig, now speed is all over the place.

Is this because I'm doing an initial replication or could it be another issue related to the new repl changes?

 

[depasseg]

I wish I knew.

I'm curious where the 1Gbps link in your setup is. It looks like everything in your Sig is 10GbE.

 

[fullspeed]

I wish I knew.

I'm curious where the 1Gbps link in your setup is. It looks like everything in your Sig is 10GbE.

Yup everything is 10GbE, You can see that it actually goes over 1Gbps (although barely).

That's the speed on both replication streams for my Freenas boxes, with encryption disabled.

 

[depasseg]

My 1G link also shows speeds just a hair over 1Gbps (close to 1.1). It's odd that the speed is that low for your link.

As for the sporadic nature, has the recent update (today) or a reboot helped?

 

[fullspeed]

My 1G link also shows speeds just a hair over 1Gbps (close to 1.1). It's odd that the speed is that low for your link.

As for the sporadic nature, has the recent update (today) or a reboot helped?

Yeah I don't even have copper attached, all 10 gig fibre.

Replication speeds are still erratic in comparison to my other environment with same hardware, also the gui doesn't show that its sending anything it just shows the last failed snapshot send.

The server isn't production so nothing is happening on it (aside from this replication)