Slow Replication on 9.2.1.3-RELEASE

[ser_rhaegar]

I'm having an issue with replicating to my backup server on a fresh pool since upgrading to 9.2.1.3. The speed caps out at 7-8Mbps. I am hoping that I am doing something stupid and someone can point me to this mistake. Otherwise I will post this in the bug tracker.

PUSH: Main system in my sig, 3cores, 12GB RAM, M1015 passthrough
PULL: Second system in my sig, 2cores, 12GB RAM, M1015 passthrough

Link: 1Gbps

Each test starts with clearing all snapshots on PUSH. All datasets on PULL are new.

Test 1:
Snapshot of pool alpha on PUSH, recursive, every 2 hours.
Replication of pool alpha on PUSH to beta/alpha on PULL, recursive.
Runs at 8Mbps. Python is listed at 100% CPU in top. Presumably this is autorepl.py.

Test 2:
Snapshot of alpha/Downloads on PUSH, NOT recursive, every 2 hours.
Replication of alpha/Downloads on PUSH to beta/alpha/Downloads on PULL, NOT recursive.
Runs at 8Mbps. Python is listed at 100% CPU in top. Presumably this is autorepl.py.

Test 3:
Snapshot of alpha/Downloads on PUSH, NOT recursive, every 2 hours.
Replication of alpha/Downloads on PUSH to beta/alpha/Downloads on PULL, NOT recursive, high speed ciphers checked.
Runs at 8Mbps. Python is listed at 100% CPU in top. Presumably this is autorepl.py.

Test 4:
SCP 5GB file from alpha/Downloads on PUSH to beta/alpha/Downloads on PULL.
Runs at 20-25MB/s (200Mbps) through the end. SSH pegs at 100% on top.

Test 5:
Snapshot of alpha/Downloads on PUSH, NOT recursive, every 2 hours.
Manual send of first alpha/Downloads snapshot on PUSH to beta/alpha/Downloads on PULL, NOT recursive.

Code:

zfs send alpha/Downloads@auto-20140322.1530-2m | ssh -i /data/ssh/replication 192.168.100.2 zfs recv beta/alpha/Downloads

Runs at 25MB/s (200Mbps) through the end. SSH pegs at 100% on top.

Test 6:
Snapshot of pool alpha on PUSH, recursive, every 2 hours.
Manual send of first alpha snapshot on PUSH to beta/alpha on PULL, recursive.

Code:

zfs send -r alpha@auto-20140322.1830-2m | ssh -i /data/ssh/replication 192.168.100.2 zfs recv beta/alpha

Runs at 25MB/s (200Mbps) through the end (well so far, it is 2.4TB so it is still running). SSH pegs at 100% on top.

Is there something I am missing in configuring the replication through FreeNAS that is capping my speed compared to a manual send?

 

[survive]

Hi ser_rhaegar,

I can't offer a solution but I believe I'm seeing some of the same things in 9.2.1.3.

I replicate locally, from an 8 drive raidz2 pool to a single 3TB drive and when replication kicks off I see python spike up to using 98-100% of one core and the replication task runs for far longer than it seems it should. I had a replication job running overnight with Python maxing out a core on a pool that had little to no changes made since before I updated.

-Will

 

[ser_rhaegar]

Can you check your network usage on the reports page during a long replication and post it? Try to get it when there is no other traffic on the box.

 

[survive]

Hi ser_rhaegar,

I could, but it's not relevent because I'm replicating locally to a single drive attached to the filer via a sata port. I'm still sending out via ssh but it's looped right back to the host so there's no network involved.

-Will

 

[ser_rhaegar]

Ah I totally misread your original post, thinking local network. :) Thanks for the info.

 

[ser_rhaegar]

Bug 4620