Replication Task Keeps Restarting after only 150GB to 500GB of Data

[TXAG26]

I have two FreeNAS servers (Push = "A" & Pull = "B") and have a replication task setup to replicate snapshots from "A" to "B".

The replication task starts running and will transfer anywhere from 150GB to almost 500GB of data, and then just stop for about 10-12 minutes and restart anew. When it restarts, all of the data previously transferred to "B" is deleted and the replication task starts copying everything from the beginning.

Any ideas? I've tried completely deleting the replication task, deleted the replication volume on "B" and completely rebooted both machines.

EDIT: Running 9.10.1-U1 -- It appears 9.10.1-U1 killed snapshot replication...

 

[nojohnny101]

Hmmm that is a bit strange. I heard of lots of unpredictable and odd behavior with snapshots and replication but your situation is a first.

Anything in your logs? Does the web GUI show any error message on the replication tab before it restarts? Maybe it's a corrupt snapshot?

 

[TXAG26]

I did all of the initial above-described actions again, in addition to blowing away the entire Raid-Z2 volume and dataset, and rebuilt the whole thing on the Pull "B" system. I also took a new manual snap-shot. When I re-setup the replication task this time, it started up almost immediately. Previously it was taking a good 10-15 minutes before data would start to transfer. The replication is now just about maxing out a 1 gigabit connection, with 505GB+ transferred so far. Fingers crossed!

UPDATE: Spoke too soon. It failed around 1TB of data transferred.

 

[nojohnny101]

Great! Let us know and report back!

 

[TXAG26]

Spoke too soon, it failed again earlier this morning after transferring approximately 1TB of data. Can someone point me to the log file location for Replication Tasks? Or anywhere else that might provide some information on what's happening under the hood? Besides manually running scrubs and smart tests through the shell, I haven't really done much under the hood.

BTW, I don't think it's a hardware issue as I just completed heavy scrubs on both machines/pools and they are in cool locations (not overheating as IPMI on both shows excellent temps) and both cases have great airflow.

 

[depasseg]

Sounds somewhat similar to a bug I was experiencing. Try increasing the time period between replications and are you sure the destination isn't any where near running out of room?
[edit: here's the bug I opened a while ago. I think I finally destroyed the target, and extended the time between snapshots, to give the new initial replication a chance to finish. https://bugs.freenas.org/issues/12787]

 

[Stux]

I'm getting this issue too. And there is a bug report about a similar issue.

https://bugs.freenas.org/issues/17826

 

[TXAG26]

Sounds somewhat similar to a bug I was experiencing. Try increasing the time period between replications and are you sure the destination isn't any where near running out of room?

It's actually a freshly created Raidz2 pool/volume and on top of the snapshot that I'm trying to replicate, the pool/volume actually has an extra 6TB of space, above & beyond that needed by the snapshot.

Checking out the bug report now as I actually had all of this running like clock work with pretty much the same hardware/setup a couple months back...now with U1, the proverbial wheels seem to have come off the bus!

 

[Stux]

Yep, I have a fresh pool, and I'm trying to sync a samba dataset. Just fails after chugging away for a bit.

Fairly unimpressed with u1. Core functionality is broken.

 

[TXAG26]

Yep, I have a fresh pool, and I'm trying to sync a samba dataset. Just fails after chugging away for a bit.

Fairly unimpressed with u1. Core functionality is broken.

Agreed. Screwed around all weekend with this pulling my hair out...

 

[Stux]

Agreed. Screwed around all weekend with this pulling my hair out...

Unfortunately for me 9.10.1-U1 fixes a critical issue (for me) in Samba in 9.10.1, or I'd rollback.

 

[depasseg]

Unfortunately for me 9.10.1-U1 fixes a critical issue (for me) in Samba in 9.10.1, or I'd rollback.

In all the issues I've heard caused by U1, this is the first fix provided by it.

 

[Ericloewe]

In all the issues I've heard caused by U1, this is the first fix provided by it. :)

It also fixes the "UEFI installations have no local conmsole" bug, for new installs. And for existing ones if you bother to run the GRUB config generator script manually.

 

[Stux]

In all the issues I've heard caused by U1, this is the first fix provided by it. :)

More info
https://forums.freenas.org/index.ph...source-forks-beyond-128kib.46368/#post-316576

 

[Stux]

# grep "Replication result: Error" /var/log/debug.log

Code:

Oct  4 00:06:37 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 00:21:05 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 00:35:31 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 00:50:00 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:04:25 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:18:56 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:32:34 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:46:57 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:01:26 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:16:55 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:30:51 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:45:38 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 03:00:12 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 04:40:15 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:07:59 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:16:37 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:25:25 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:33:55 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:42:24 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:51:00 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:59:01 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:08:04 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:16:39 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:25:11 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:33:16 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:42:26 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:51:36 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:00:11 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:08:12 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:16:49 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:24:51 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:34:00 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:42:28 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:51:32 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:00:03 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:09:15 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:18:26 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:29:43 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:54:48 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block

I'm getting an awful lot of emails about failed replications. There is 6TB available on the desitination.

Of course, its a crapshoot, as it does eventually succeed. Replication is to a another system which is connected via gbit, and otherwise idle.

Show : successes and failures

# grep "Replication result:" /var/log/debug.log

Code:

Oct  4 00:06:37 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 00:06:43 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:06:44 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:06:51 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:06:51 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:06:52 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:06:52 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:06:53 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 00:21:05 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 00:35:31 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 00:50:00 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:04:25 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:04:32 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:04:33 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:04:40 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:04:41 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:04:41 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:04:42 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:04:42 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 01:18:56 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:32:34 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 01:46:57 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:01:26 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:01:34 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:01:34 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:01:41 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:01:42 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:01:42 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:01:43 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:01:44 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 02:16:55 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:30:51 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 02:45:38 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 03:00:12 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 03:00:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:00:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:00:26 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:00:26 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:00:27 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:00:27 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:00:28 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:24:56 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 03:51:12 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:17:33 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:35:10 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:15 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 04:40:23 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:23 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:32 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:33 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:33 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:34 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 04:40:34 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:02:50 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:07:59 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:08:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:08:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:08:15 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:08:15 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:08:16 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:08:16 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:08:17 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 05:16:37 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:25:25 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:33:55 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:42:24 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:51:00 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 05:59:01 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:08:04 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:08:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:08:10 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:08:17 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:08:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:08:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:08:19 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:08:19 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 06:16:39 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:25:11 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:33:16 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:42:26 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 06:51:36 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:00:11 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:08:12 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:08:17 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:08:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:08:25 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:08:26 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:08:27 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:08:27 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:08:28 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 07:16:49 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:24:51 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:34:00 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:42:28 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 07:51:32 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:00:03 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:09:15 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:09:21 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:09:21 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:09:29 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:09:30 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:09:30 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:09:31 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:09:31 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:18:26 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:24:33 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:29:43 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:44:41 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:49:40 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 08:54:48 titan autorepl.py: [tools.autorepl:157] Replication result: Error 35 : Write error : cannot write compressed block
Oct  4 08:57:28 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:34 titan autorepl.py: [tools.autorepl:157] Replication result: 
Oct  4 09:02:41 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:42 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:43 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:43 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:51 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:51 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:52 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:52 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:53 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:54 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:54 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:55 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:55 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:56 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:58 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:58 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:59 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:02:59 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:00 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:00 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:01 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:01 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:02 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:02 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:03 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:03 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:05 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:10 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:10 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:11 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:11 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:12 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:13 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:13 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:14 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:14 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:15 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:15 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:16 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:16 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:17 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:17 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:18 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:19 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:20 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:20 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:21 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:21 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:22 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:22 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:23 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:23 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:24 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:25 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:25 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:26 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:26 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:27 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:27 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:28 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:28 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:29 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:29 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:30 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:31 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:31 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 09:03:32 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 10:01:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:03 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 11:01:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 12:01:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:03 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:04 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:06 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:07 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:08 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded
Oct  4 13:01:09 titan autorepl.py: [tools.autorepl:157] Replication result: Succeeded

 

[TXAG26]

Update: 9.10.1-U2 I received e-mails from both of my FreeNAS boxes that a new update dropped early this morning. It appears to fix a number of the replication issues. The latest update was applied to both boxes, the main volume on Pull "B" was destroyed and re-created, a new snapshot was taken on Push "A", and finally, I re-enabled the replication task. We'll see how far it gets today. Fingers crossed!

Version: FreeNAS-9.10.1-U2
Changelog:
#17908 Bug Ready For Release Expected EFI install fails with complaint about more than one boot device
#17904 Bug Ready For Release Expected Failed to report when snapshot failed to create
#17869 Bug Ready For Release Critical Virtualbox cannot run any VMachine after upgrade to 9.10.1-U1
#17826 Bug Ready For Release Critical Errors reported during replication
#17819 Bug Ready For Release Critical iSCSI LUN ID's all set to 0
#17809 Bug Ready For Release Important GRUB needs modules specified for improved reliability
#17789 Bug Ready For Release Critical Bug from latest updates for 9.10.1 on 09/27/2016
#17787 Bug Ready For Release Expected OpenSSL Fix Use After Free for large message sizes (CVE-2016-6309)

 

[Malakym]

I just want to add on that I am having this problem too, completely new datasets on the destination and it fails over and over, every so often one works, smaller ones are more likely to work, though I did notice it took a few tries.

Annoyingly the logs shows that it was successful, but the destination size goes back to empty, 92K.

This is on -U1 too, which I had only just upgraded to prior to -U2 coming out.

-U1 also fixes some problems for me, so going to see my luck with -U2 instead of going back.

 

[TXAG26]

I just want to add on that I am having this problem too, completely new datasets on the destination and it fails over and over, every so often one works, smaller ones are more likely to work, though I did notice it took a few tries.

Annoyingly the logs shows that it was successful, but the destination size goes back to empty, 92K.

This is on -U1 too, which I had only just upgraded to prior to -U2 coming out.

-U1 also fixes some problems for me, so going to see my luck with -U2 instead of going back.

U2 appears to have resolved the issues I was seeing. My replication task is still running though as I had to throttle it back quite a bit due to network congestion.

 

[Malakym]

I've just done the update now so will test.

Luckily I didn't have to worry about network as this box is replicating to a local disk (localhost...), it was worrying though as I have been trying to push 5TB and the one dataset was 1TB and it must have tried a few times :/

EDIT: Just finished at 200GB and its gone through fine on -U2 first time, and the follow up snapshots being pushed are much faster (even before, the speed was good when data was passing but there was so much dead air).

Thanks for updating the first post, more threads need status/summaries.

 

[Stux]

I was getting emails every 20 minutes about failed replications with U1. Since updating to U2 I haven't recieved an email in 12 hours.

So lets hope that issue is fixed :)