A tale of two SSH's

[Kyle Tuck]

We recently acquired a new server to replace an aging FreeNAS box at a satellite location. This machine is a replication target for offsite backup, and the sites are connected at 200Mbps.

We installed FreeNAS (11.3-U4.1) and started the replication. No matter what settings we tried in the replication task, we could never get transfers above 10Mbps. I won't go through everything we tried, but we turned just about every switch and turned every knob. We also tried various speed tests from various machines, and even some straight scp tests that confirmed that transferring at 200Mbps was possible. I finally did a manual replication from the command line, and... 200Mbps! I checked the logs and began manually typing in the zfs send/receive commands, stripping out the various options specified in the tasks generated from the GUI, but now kept getting 10Mbps.

It boiled down to this. In my manual replication from command line, I was piping to ssh. The replication task commands were piping to /usr/local/bin/ssh. I ran `which ssh`, and discovered it uses /usr/bin/ssh. Yup. There are two completely different ssh's.

/usr/local/bin/ssh -V -> 8.0p1 (the slow one)
/usr/bin/ssh -V -> 7.5p1 (the fast one)

Is this on purpose?

 

[rvassar]

It would appear to be so, as my 11.3-U4.1 has the same. I checked the changelog for OpenSSH and most of the differences between those versions are either key-revocation, or Sprectre/Meltdown style attack protections. However, the defaults compiled in may be quite different. The older /etc/ssh/ssh_config on mine still mentioned horribly insecure ciphers and MACs that really should have been stripped out several years ago (ala: 3des-cbc & hmac-md5)... They're not mentioned in the corresponding sshd_config, but... I'm sitting here pondering what to do about this. Trying to figure out how to tell what the compiled in defaults are...

But that's likely the place to start. Compare the actual cipher negotiation steps between versions. I'll guess the 8.0p1 is using a much more CPU intensive cipher, or re-keying more often, etc...

 

[Kyle Tuck]

Re-keying is a possibility, and I will investigate a little more later this week. I know I didn't provide details on the system used, but CPU utilization is very low during replication tasks (whether it is the manual one or ones generated from GUI).

 

[rvassar]

Re-keying is a possibility, and I will investigate a little more later this week. I know I didn't provide details on the system used, but CPU utilization is very low during replication tasks (whether it is the manual one or ones generated from GUI).

Is the CPU on the other end low? Another thing to check, you may be depleting the rng entropy.