Replication problem between 2 FreeNAS 8.2.0-Release boxes

[electricd7]

Hello,

I am trying to get snapshot replication setup between my production FreeNAS and my DR FreeNAS. I followed the wiki and am able to SSH from within Prod to DR without entering a password, so I am sure my keys are setup correctly, but it looks like some other sort of network error or something. I have periodic snapshot job setup on Prod beginning at 8:30 and ending at 10:45 and interval set to 1 day. Then on the replication task on prod, I have it pointing to pool1/backups on DR (this is a valid dataset), I have both Recursive and Initialize checked, hostname is correct, port 22 and I did the scan to derive hostkey. Should be pretty simple, but here is the output of the console during the snapshot/replication stage on Prod. Please help if you can:

Sep 19 08:30:02 freenas autosnap.py: [tools.autosnap:42] Popen()ing: /sbin/zfs snapshot -o freenas:state=NEW pool1/backups@auto-20120919.0830-2w
Sep 19 08:30:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:30:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:30:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 08:30:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:30:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:30:10 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10361 2>&1
Sep 19 08:38:02 freenas kernel: pid 10394 (zfs), uid 0: exited on signal 6
Sep 19 08:38:02 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 08:38:02 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 08:38:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:38:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:38:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 08:38:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:38:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:38:03 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10465 2>&1
Sep 19 08:38:04 freenas kernel: pid 10487 (zfs), uid 0: exited on signal 6
Sep 19 08:38:04 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 08:38:04 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 08:39:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:39:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:39:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 08:39:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:39:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:39:03 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10492 2>&1
Sep 19 08:39:04 freenas kernel: pid 10512 (zfs), uid 0: exited on signal 6
Sep 19 08:39:04 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 08:39:04 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 08:40:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:40:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:40:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 08:40:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:40:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:40:10 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10520 2>&1
Sep 19 08:48:54 freenas kernel: pid 10552 (zfs), uid 0: exited on signal 6
Sep 19 08:48:54 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 08:48:56 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 08:49:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:49:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:49:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 08:49:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:49:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:49:03 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10632 2>&1
Sep 19 08:49:10 freenas kernel: pid 10652 (zfs), uid 0: exited on signal 6
Sep 19 08:49:10 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 08:49:10 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 08:50:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:50:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:50:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 08:50:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:50:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:50:03 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10660 2>&1
Sep 19 08:50:04 freenas kernel: pid 10692 (zfs), uid 0: exited on signal 6
Sep 19 08:50:04 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 08:50:04 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 08:51:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 08:51:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 08:51:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 08:51:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 08:51:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 08:51:12 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10697 2>&1
Sep 19 09:06:10 freenas kernel: pid 10717 (zfs), uid 0: exited on signal 6
Sep 19 09:06:10 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 09:06:11 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 09:07:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 09:07:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 09:07:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 09:07:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 09:07:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 09:07:14 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-10896 2>&1
Sep 19 09:30:00 freenas kernel: pid 10916 (zfs), uid 0: exited on signal 6
Sep 19 09:30:00 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 09:30:14 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 09:31:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 09:31:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 09:31:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 09:31:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 09:31:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 09:31:03 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11165 2>&1
Sep 19 09:31:05 freenas kernel: pid 11185 (zfs), uid 0: exited on signal 6
Sep 19 09:31:05 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 09:31:05 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 09:32:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 09:32:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 09:32:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 09:32:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 09:32:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 09:32:12 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11190 2>&1
Sep 19 09:43:47 freenas kernel: pid 11210 (zfs), uid 0: exited on signal 6
Sep 19 09:43:47 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 09:44:11 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 09:45:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 09:45:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 09:45:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 09:45:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 09:45:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 09:45:10 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11340 2>&1
Sep 19 09:54:25 freenas kernel: pid 11372 (zfs), uid 0: exited on signal 6
Sep 19 09:54:25 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 09:54:29 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 09:55:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 09:55:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 09:55:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 09:55:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 09:55:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 09:55:03 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11459 2>&1
Sep 19 09:55:03 freenas kernel: pid 11494 (zfs), uid 0: exited on signal 6
Sep 19 09:55:03 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 09:55:04 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 09:56:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 09:56:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 09:56:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 09:56:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 09:56:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 09:56:13 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11499 2>&1
Sep 19 10:08:06 freenas kernel: pid 11519 (zfs), uid 0: exited on signal 6
Sep 19 10:08:06 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 10:08:10 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 10:09:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 10:09:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 10:09:02 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 10:09:02 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 10:09:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 10:09:10 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11658 2>&1
Sep 19 10:18:34 freenas kernel: pid 11678 (zfs), uid 0: exited on signal 6
Sep 19 10:18:34 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 10:18:37 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 10:19:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 10:19:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 10:19:03 freenas common.pipesubr: cannot destroy 'pool1/backups/backups': dataset is busy
Sep 19 10:19:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 10:19:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 10:19:04 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11780 2>&1
Sep 19 10:19:07 freenas kernel: pid 11800 (zfs), uid 0: exited on signal 6
Sep 19 10:19:07 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 10:19:08 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with cannot receive new filesystem stream: dataset is busy warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 10:20:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 10:20:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 10:20:02 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 10:20:02 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 10:20:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 10:20:12 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11808 2>&1
Sep 19 10:20:22 freenas ntpd[1509]: kernel time sync status change 6001
Sep 19 10:30:08 freenas kernel: pid 11840 (zfs), uid 0: exited on signal 6
Sep 19 10:30:08 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 10:30:20 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 10:31:03 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 10:31:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 10:31:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 10:31:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 10:31:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 10:31:12 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-11949 2>&1
Sep 19 10:40:04 freenas kernel: pid 11969 (zfs), uid 0: exited on signal 6
Sep 19 10:40:04 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 10:40:05 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap
Sep 19 10:41:02 freenas autorepl.py: [tools.autorepl:241] Destroying remote pool1/backups/backups
Sep 19 10:41:02 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs destroy -rRf pool1/backups/backups
Sep 19 10:41:03 freenas common.pipesubr: cannot open 'pool1/backups/backups': dataset does not exist
Sep 19 10:41:03 freenas autorepl.py: [tools.autorepl:247] Creating pool1/backups/backups on remote system
Sep 19 10:41:03 freenas autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal /sbin/zfs create -o readonly=on -p pool1/backups/backups
Sep 19 10:41:13 freenas autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R pool1/backups@auto-20120912.0830-2w | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "/sbin/zfs receive -F -d pool1/backups && echo Succeeded.") > /tmp/repl-12071 2>&1
Sep 19 10:54:30 freenas ntpd[1509]: kernel time sync status change 2001
Sep 19 11:11:05 freenas kernel: pid 12091 (zfs), uid 0: exited on signal 6
Sep 19 11:11:05 freenas autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 freenas2.boi.internal "zfs list -Hr -o name -S creation -t snapshot -d 1 pool1/backups/backups | head -n 1 | cut -d@ -f2"
Sep 19 11:11:20 freenas autorepl.py: [tools.autorepl:307] Replication of pool1/backups@auto-20120912.0830-2w failed with warning: cannot send 'pool1/backups@auto-20120912.0830-2w': Broken pipe internal error: Broken pipe Abort trap

 

[electricd7]

Bump. Please help, seems so close.

 

[electricd7]

Bump? Anyone?

 

[maglaubig]

I'm having an almost identical issue on 8.3 BETA 3. What I'm attempting is not a replicadtion of the entire ZFS volume but individual ZFS datasets. So I setup a periodic snapshot on the dataset in question e.g. "vol/replication". Auto snapshot generation works just fine.

When I setup replication my dataset name doesn't exactly match on the remote side and is also a dataset, not a ZFS volume e.g. "remote/repl". Setup works fine, I can see data is being transferred per the logs and also setup the replication to destroy data on the remote end. What I notice on the remote side is that there appears to be another dataset created which I thought first might just be a temporary place to uncompress data, however after reading on how to manually handle ZFS with "zfs send" and "zfs receive" I'm not so sure this is the case. I suspect this may be an issue of improper handling of the dataset name by the scripts handling replication?

The dataset that replication creates given my examples above is "remote/repl/replication", aside that I want this dataset to be replicated to "remote/repl" replication isn't completing so I'm not 100% sure that will be the final destination. I show in the FreeNAS web GUI that this dataset does in fact exist, but wasn't created by me and takes a mount point inside the mount point for "remote/repl" as in the path to this automatically created dataset is "/mnt/remote/repl/replication". There is a separate entry for it under "zfs list" as well.

I'm going to attempt to manually create a gzip file using "zfs send" and FTP the result over the WAN an then manually import using "zfs receive" and see if I get anything different. Let me know if you're still monitoring the thread, this is the only one I've been able to find that seems similar enough.