ZFS Replication performance

[Rand]

Hi,

got a second zfs system to play around with and am trying zfs replication atm.

System1:
i3 4310, 16gb, 8x3TB mirror'ed main pool, 10GBE (PULL)

System2:
Avoton 2750, 32GB, 6x2TB Z2, 10GBE (PUSH)

I have tested both systems inidvidually that they can send and receive data >300MB/s (thats not close to max but fair enough for me) (Cifs test from desktop).
Now when i run a replication from S1 to S2 its limited to 600mbit and i wonder whether thats an architectural problem?

I noted that i have a single core on PULL fully utilized by sshd, total load is way within limits (8 core system after all).

So is zfs replication indeed limited by the sshd and in that by single core speed ? I have selected fast ciphers to speed things up but its not performing as well as expected.

Any way to have sshd multi threaded? Would that work at all in this condition?

 

[cyberjock]

Short answer: Can't fix it, sshd is your limitation.

Long answer: This is well known and iX has a few tricks up their sleeve to deal with this coming up in 9.3. They are exploring a few options, one of which is to let you choose not to encrypt the stream at all. After all, you are quite often staying on your LAN and security of the stream en-route isn't always a concern. Another is to let sshd use the AES-NI functions of the CPU to speed up encryption and decryption.

 

[Rand]

Ah i see. Ok, thanks for the info:)

 

[Rand]

Is this fixed now? Whats the expected performance?

I just set up 9.3 on the backup machine for testing (main is still on 9.2.1.9) but performance between is still limited to 400 MBit... I assume i will need to upgrade to 9.3 on the main rig as well ?

 

[cyberjock]

Not necessarily. What you need to do is look at your replication settings and see what you are doing. You should disable the SSH encryption when doing replication (this is the primary limitation for ZFS replication to a remote server) and check your compression setting. LZ4 is likely the best choice.

Fixed is relative because it depends on what your limitation is. I can tell you that on 9.2.1.7 (which is the first version that had replication without encryption as an option) I could do almost 500MB/sec. Yes, 500 Megabytes per second.