Replication lags behind schedule

[tomspost]

Hello.

I have a server which replicates to another server a ZFS volume. The problem is when there is high volume of data to be replicated (snapshot is set to hourly) and the replication takes for example 3 hours, it does not replicate the snapshots which happened during the replicationtime. It just replicates the one after the big one, staying behind several hours in replication. Instead of replicating the snapshots in between in fast succession.

Anyone experiencing such behavior? Is is possible to launch the replication process manually?

Thanks for any advice.
Thomas

Edit:
It seems that the replication is very slow. That's why he does not stay behind schedule. Im watching a replication which has the snapshot before of aprox. 800 MBytes and the one which he is replication now of 220 MByte. It's running now more than half an hour and still not finished. Shouldn't take so long over a 10 GBit link.....

 

[cyberjock]

It sounds like you have a bottleneck somewhere. You provided no hardware specs per the forum rules, but you should start trying to find your bottleneck. If the data is scattered all over the drive and not linear and contiguous, you shouldn't expect to see 800MB/sec regularly. You're probably being limited to the rate you can pull data off the zpool.

 

[tomspost]

Hi.

Oh, forgot the Hardware, here are the specs......

On both machines we have the following configuration:
20 2TB Discs in RAID 10.
2 Intel Xeon 6-Core E5-2630 2,30GHz.
128 MByte of RAM .
Replication uses a dual port Intel 10 Gigabit Intel Card, directly connect to the other machine (no switch in between) lagg in LoadBalance Mode.

The pool has aprox. 800 GB of Data on it.

 

[cyberjock]

Please, please for the love of God tell me you meant 128GB of RAM and not MB. :P

I'd do some benchmarks of your pool and see what kind of speeds you can get on both ends. I'd bet your limitations are your zpools ability to read(on the source) and write data(on the destination). I have an 18 drive RAIDZ3 and I peak out at about 1.1GB/sec. I can't imagine you'd be able to beat that by much in a "best case scenario" with just 20 disks in a RAID10. Also keep in mind that there is no defrag for ZFS so you can expect that extremely busy arrays will only get slower over time.

I'm not sure how familiar you are with LACP but quite a large number of people are not familiar with LACP and do not understand it enough to actually use it properly. You may want to try getting rid of LACP and seeing what happens with your performance.