ZFS Replication "Sending" but actually isn't.

[cuvy]

Hi there,

I have ZFS replication setup between 2 FreeNAS servers. The status of the replication is always "Sending" even when it isn't. The replication occurs but not at the interval I have set it which is 5 minutes.

For example, I was caught up yesterday (I had run the replication manually via command line) then I let FreeNAS do its thing. Today, only a few snapshot were replicated and the last snapshot replicated was at 3pm yesterday. I'm running it manually again and it is working just fine. Something is definitely off with FreeNAS replication because this is not the first time this happened. I have 2 other replication configured on an other server and those are replicating correctly with its proper status (Sending with %, Succeeded, etc).

All my FreeNAS server are up to date to the latest STABLE version running 9.10-2 (a476f16). I'm about to setup a very simple bash script that will replicate the snapshots but I'd much rather use FreeNAS...

I tried re-creating the replication entry but it doesn't work. I'm out of ideas.

Thanks for the help guys! :)

 

[dlavigne]

The replication occurs but not at the interval I have set it which is 5 minutes.

Does the replication take longer than 5 minutes to complete? If not, it would make sense that it would not try again until both of these statements are true: the last replication has completed and there is a new snapshot to replicate.

 

[cuvy]

No it doesn't. When I ran the replication manually the snapshot was less than 100kb and took less than 5 seconds.

Thanks :)

 

[wblock]

Could you show your replication settings, both the snapshot schedule and the replication task? I know I sometimes get bitten by the default times for replication tasks being business hours.

Are the snapshots being created?

 

[cuvy]

The snapshot are being created correctly.

Snapshot: http://d.uberfive.com/yRrDzxuxHd.png
Replication: http://d.uberfive.com/YVsFVggjWB.png

Thank you! :)

 

[wblock]

That replication task is not enabled (about two-thirds of the way down).

 

[cuvy]

That replication task is not enabled (about two-thirds of the way down).

I'm currently running it manually so to get it caught up (couldn't wait)

It was enabled before though.

 

[cuvy]

I've restarted the replication and same issue. Last snapshot is at 11:52am today.

http://d.uberfive.com/HQRYIrUfgH.png

 

[cuvy]

Running the replication manually outputs:

send from @auto-20170103.1157-7d to *******/K@auto-20170103.1202-7d estimated size is 14.6M
send from @auto-20170103.1202-7d to *******/K@auto-20170103.1207-7d estimated size is 4.08M
send from @auto-20170103.1207-7d to *******/K@auto-20170103.1212-7d estimated size is 162K
send from @auto-20170103.1212-7d to *******/K@auto-20170103.1217-7d estimated size is 50K
send from @auto-20170103.1217-7d to *******/K@auto-20170103.1222-7d estimated size is 547K
send from @auto-20170103.1222-7d to *******/K@auto-20170103.1227-7d estimated size is 246K
send from @auto-20170103.1227-7d to *******/K@auto-20170103.1232-7d estimated size is 222K
send from @auto-20170103.1232-7d to *******/K@auto-20170103.1237-7d estimated size is 4.36M
send from @auto-20170103.1237-7d to *******/K@auto-20170103.1242-7d estimated size is 1.88M
send from @auto-20170103.1242-7d to *******/K@auto-20170103.1247-7d estimated size is 365K
send from @auto-20170103.1247-7d to *******/K@auto-20170103.1252-7d estimated size is 4.28M
send from @auto-20170103.1252-7d to *******/K@auto-20170103.1257-7d estimated size is 214K
send from @auto-20170103.1257-7d to *******/K@auto-20170103.1302-7d estimated size is 12.2M
send from @auto-20170103.1302-7d to *******/K@auto-20170103.1307-7d estimated size is 98K
send from @auto-20170103.1307-7d to *******/K@auto-20170103.1312-7d estimated size is 138K
send from @auto-20170103.1312-7d to *******/K@auto-20170103.1317-7d estimated size is 178K
send from @auto-20170103.1317-7d to *******/K@auto-20170103.1322-7d estimated size is 3.82M
send from @auto-20170103.1322-7d to *******/K@auto-20170103.1327-7d estimated size is 178K
send from @auto-20170103.1327-7d to *******/K@auto-20170103.1332-7d estimated size is 214K
send from @auto-20170103.1332-7d to *******/K@auto-20170103.1337-7d estimated size is 571K
send from @auto-20170103.1337-7d to *******/K@auto-20170103.1342-7d estimated size is 1.17M
send from @auto-20170103.1342-7d to *******/K@auto-20170103.1347-7d estimated size is 534K
send from @auto-20170103.1347-7d to *******/K@auto-20170103.1352-7d estimated size is 3.73M
send from @auto-20170103.1352-7d to *******/K@auto-20170103.1357-7d estimated size is 326K
send from @auto-20170103.1357-7d to *******/K@auto-20170103.1402-7d estimated size is 407K
send from @auto-20170103.1402-7d to *******/K@auto-20170103.1407-7d estimated size is 19.3M
send from @auto-20170103.1407-7d to *******/K@auto-20170103.1412-7d estimated size is 4.96M
send from @auto-20170103.1412-7d to *******/K@auto-20170103.1417-7d estimated size is 851K
send from @auto-20170103.1417-7d to *******/K@auto-20170103.1422-7d estimated size is 764K
send from @auto-20170103.1422-7d to *******/K@auto-20170103.1427-7d estimated size is 23.9M
send from @auto-20170103.1427-7d to *******/K@auto-20170103.1432-7d estimated size is 358K
send from @auto-20170103.1432-7d to *******/K@auto-20170103.1437-7d estimated size is 326K
send from @auto-20170103.1437-7d to *******/K@auto-20170103.1442-7d estimated size is 1.59M
send from @auto-20170103.1442-7d to *******/K@auto-20170103.1447-7d estimated size is 4.00M
send from @auto-20170103.1447-7d to *******/K@auto-20170103.1452-7d estimated size is 1.41M
send from @auto-20170103.1452-7d to *******/K@auto-20170103.1457-7d estimated size is 3.87M
send from @auto-20170103.1457-7d to *******/K@auto-20170103.1502-7d estimated size is 17.3M
send from @auto-20170103.1502-7d to *******/K@auto-20170103.1507-7d estimated size is 1.70M
send from @auto-20170103.1507-7d to *******/K@auto-20170103.1512-7d estimated size is 2.47M
send from @auto-20170103.1512-7d to *******/K@auto-20170103.1517-7d estimated size is 471K
send from @auto-20170103.1517-7d to *******/K@auto-20170103.1522-7d estimated size is 459K
send from @auto-20170103.1522-7d to *******/K@auto-20170103.1527-7d estimated size is 487K
send from @auto-20170103.1527-7d to *******/K@auto-20170103.1532-7d estimated size is 1.34M
send from @auto-20170103.1532-7d to *******/K@auto-20170103.1537-7d estimated size is 663K
send from @auto-20170103.1537-7d to *******/K@auto-20170103.1542-7d estimated size is 479K
send from @auto-20170103.1542-7d to *******/K@auto-20170103.1547-7d estimated size is 775K
send from @auto-20170103.1547-7d to *******/K@auto-20170103.1552-7d estimated size is 499K
send from @auto-20170103.1552-7d to *******/K@auto-20170103.1557-7d estimated size is 551K
send from @auto-20170103.1557-7d to *******/K@auto-20170103.1602-7d estimated size is 463K
send from @auto-20170103.1602-7d to *******/K@auto-20170103.1607-7d estimated size is 495K
send from @auto-20170103.1607-7d to *******/K@auto-20170103.1612-7d estimated size is 699K
send from @auto-20170103.1612-7d to *******/K@auto-20170103.1617-7d estimated size is 2.49M
send from @auto-20170103.1617-7d to *******/K@auto-20170103.1622-7d estimated size is 623K
send from @auto-20170103.1622-7d to *******/K@auto-20170103.1627-7d estimated size is 451K
send from @auto-20170103.1627-7d to *******/K@auto-20170103.1632-7d estimated size is 779K
send from @auto-20170103.1632-7d to *******/K@auto-20170103.1637-7d estimated size is 551K
send from @auto-20170103.1637-7d to *******/K@auto-20170103.1642-7d estimated size is 499K
send from @auto-20170103.1642-7d to *******/K@auto-20170103.1647-7d estimated size is 1.82M
send from @auto-20170103.1647-7d to *******/K@auto-20170103.1652-7d estimated size is 1.80M
send from @auto-20170103.1652-7d to *******/K@auto-20170103.1657-7d estimated size is 483K
send from @auto-20170103.1657-7d to *******/K@auto-20170103.1702-7d estimated size is 745K
send from @auto-20170103.1702-7d to *******/K@auto-20170103.1707-7d estimated size is 507K
send from @auto-20170103.1707-7d to *******/K@auto-20170103.1712-7d estimated size is 607K
send from @auto-20170103.1712-7d to *******/K@auto-20170103.1717-7d estimated size is 647K
send from @auto-20170103.1717-7d to *******/K@auto-20170103.1722-7d estimated size is 503K
send from @auto-20170103.1722-7d to *******/K@auto-20170103.1727-7d estimated size is 479K
send from @auto-20170103.1727-7d to *******/K@auto-20170103.1732-7d estimated size is 38.5M
send from @auto-20170103.1732-7d to *******/K@auto-20170103.1737-7d estimated size is 3.96M
send from @auto-20170103.1737-7d to *******/K@auto-20170103.1742-7d estimated size is 479K
send from @auto-20170103.1742-7d to *******/K@auto-20170103.1747-7d estimated size is 559K
send from @auto-20170103.1747-7d to *******/K@auto-20170103.1752-7d estimated size is 2.16M
send from @auto-20170103.1752-7d to *******/K@auto-20170103.1757-7d estimated size is 615K
send from @auto-20170103.1757-7d to *******/K@auto-20170103.1802-7d estimated size is 487K
send from @auto-20170103.1802-7d to *******/K@auto-20170103.1807-7d estimated size is 495K
total estimated size is 204M

 

[cuvy]

I ended up running it manually via a bash script :)