SSH ciphers - Improving the speed of zfs send/recv

[Elliot Dierksen]

I was experimienting over the weekend with doing a zfs send of a snapshot from my primary FreeNAS to the secondary. I was a little dismayed at the speed, but I noticed that the secondary unit (Dual Intel(R) Xeon(R) CPU E5-2637 0 @ 3.00GHz - 128G RAM) was just pounded pretty hard. The primary unit (Dual Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz - 256G RAM) was taking a bit of a hit, but much smaller than the secondary. I know the CPU in the primary is a lot stronger. The thing that surprised me is that the encrypt/decrypt of SSH seemed to be what was chewing up all the CPU. I wanted to enabled the blowfish-cbc cipher, but I didn't see how I could that without hacking on system files that might get overwritten after a reboot. aes128-cbc seemed to be the lighted weight one, and that has given me some noticeably better throughput. Security isn't a concern in the environment because the 10G network is a disconnected network. Only the FreeNAS boxes and the ESXi hosts have interfaces there. I would be fine doing the zfs send/recv without encryption, but I didn't see a way to do that. I saw some forum references to using something wide open like nc but maybe I am not willing go that far. Any suggestions for optimal speed for zfs send/recv over SSH with as little CPU/encryption impact as possible?

 

[Johnny Fartpants]

I found the same replication very slow (about 500Mbps) when using 'Replication Stream Compression' = lz4 however when I turned it off I now get 2/3Gbps.

 

[Chris Moore]

If you are sending over a secure (trusted) network, why not use netcat?

Here is a post about it:

https://www.ixsystems.com/community...nstead-of-ssh-monitor-zfs-send-receive.13405/

 

[Mlovelace]

I use netcat regularly to push data between freeNAS servers on a secure 10Gbe network, and see full line-rate speeds with it. Here is a link with the commands I use and their functions, as I pipe the replication stream through mbuffer then through pipe viewer. ;)

 

[Elliot Dierksen]

Thanks all! I am going to give netcat a whirl, and see what happens.

 

[Elliot Dierksen]

This is much better and more consistent throughput than I was getting with SSH.

The circled area is with netcat and using the commands from @Mlovelace . I did noticed that I am not using the SLOG on my receiving system. I think that would kick up my receive rate if I could bring the Optane 900P SLOG into play. Are there options on zfs recv to make it do synch writes? All of the ARC us certainly being brought into play on the receiving system.

 

[Mlovelace]

This is much better and more consistent throughput than I was getting with SSH.
View attachment 29370
The circled area is with netcat and using the commands from @Mlovelace . I did noticed that I am not using the SLOG on my receiving system. I think that would kick up my receive rate if I could bring the Optane 900P SLOG into play. Are there options on zfs recv to make it do synch writes? All of the ARC us certainly being brought into play on the receiving system.
View attachment 29371

The incoming dataset would have to have sync=always flagged on it's creation; so I believe if you change the sync flag on the pool to always, the newly created dataset incoming from the replication stream will inherit the setting and utilize the slog. Just a theory to test as I don't use a slog on my systems.

 

[Arwen]

I miss the None Cipher in OpenSSH. You would still get authentication, accountability and packet verification, (meaning it would be hard for man in the middle attacks). But, no overhead for encryption.

When I extracted my DVD collection, (about 2,000 videos, movies, T.V. series and special features), I needed to copy them from my desktop to my media server. I was not able to saturate a 1Gbps network link until I used the None Cipher in OpenSSH.

Now I'll have to keep Netcat in mind.