SSH Push Replication Fails against SSHD 8.8

[Richard Durso]

I have a PUSH replication that is manually executed just once a month. I've been using it Since Jan 2021 when I upgrade to 12.0, worked fine then, been working without issue. Worked fine last month on 12.0-U5. This is my first attempt with 12.0-U6. I enabled debug logging and get the following (it says 'userauth is OK'):

[2021/11/09 12:18:48] DEBUG [replication_task__task_18] [zettarepl.transport.local.shell.1.async_exec.72994] Running ['zfs', 'get', '-H', '-p', '-t', 'filesystem,volume', 'type', 'main/apps']
[2021/11/09 12:18:48] DEBUG [replication_task__task_18] [zettarepl.transport.local.shell.1.async_exec.72994] Success: 'main/apps\ttype\tfilesystem\t-\n'
[2021/11/09 12:18:48] DEBUG [replication_task__task_18] [zettarepl.transport.base_ssh.rich@dldsk01.shell.1820] Connecting...
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] starting thread (client mode): 0x10989d60
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Local version/idstring: SSH-2.0-paramiko_2.7.1
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Remote version/idstring: SSH-2.0-OpenSSH_8.8
[2021/11/09 12:18:48] INFO [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Connected (version 2.0, client OpenSSH_8.8)
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] kex algos:['curve25519-sha256', 'curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256'] server key:['rsa-sha2-512', 'rsa-sha2-256', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes256-ctr', 'aes192-ctr'] server encrypt:['aes256-ctr', 'aes192-ctr'] client mac:['hmac-sha2-512', 'hmac-sha2-256'] server mac:['hmac-sha2-512', 'hmac-sha2-256'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Kex agreed: curve25519-sha256@libssh.org
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] HostKey agreed: ecdsa-sha2-nistp256
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Cipher agreed: aes192-ctr
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] MAC agreed: hmac-sha2-256
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Compression agreed: none
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Switch to new keys ...
[2021/11/09 12:18:48] DEBUG [replication_task__task_18] [zettarepl.paramiko.replication_task__task_18] Trying SSH key b'cd57d64c73497a92ec04fbf861ea0394'
[2021/11/09 12:18:48] DEBUG [Thread-3548] [zettarepl.paramiko.replication_task__task_18] userauth is OK
[2021/11/09 12:18:48] INFO [Thread-3548] [zettarepl.paramiko.replication_task__task_18] Authentication (publickey) failed.
[2021/11/09 12:18:48] ERROR [replication_task__task_18] [zettarepl.replication.run] For task 'task_18' non-recoverable replication error ReplicationError('Authentication failed.')

To test key pair being used, I went to GUI "System > SSH Keypairs > dldsk01" and pasted contents of Private key to "dldsk01" and contents of public key to "dldsk01.pub" (TrueNAS shell):

root@truenas[~/.ssh]#

-rw------- 1 root wheel 1832 Nov 9 12:16 dldsk01
-rw------- 1 root wheel 409 Nov 9 12:15 dldsk01.pub

I then tested SSH connection to the remote, with specific user and port, connected fine (no password prompt). I enabled debug mode output as follows (if it helps):

root@truenas[~/.ssh]# ssh -v -i dldsk01 rich@dldsk01 -p 46359

OpenSSH_7.9p1, OpenSSL 1.1.1h-freebsd 24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to dldsk01 [192.168.0.246] port 46359.
debug1: Connection established.
debug1: identity file dldsk01 type 0
debug1: identity file dldsk01-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9 FreeBSD-20200214
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.8
debug1: match: OpenSSH_8.8 pat OpenSSH* compat 0x04000000
debug1: Authenticating to dldsk01:46359 as 'rich'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes192-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes192-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:XgrWnCLdBOJfcZUKki4ksprX3ZNmUFKhHU7Up2Ujrkg
DNS lookup error: general failure
debug1: Host '[dldsk01]:46359' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: Will attempt key: dldsk01 RSA SHA256:34IT9L5uW880E7cDrumfphnBPebokCb3GV08qz7/w0U explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: dldsk01 RSA SHA256:34IT9L5uW880E7cDrumfphnBPebokCb3GV08qz7/w0U explicit
debug1: Server accepts key: dldsk01 RSA SHA256:34IT9L5uW880E7cDrumfphnBPebokCb3GV08qz7/w0U explicit
debug1: Authentication succeeded (publickey).
Authenticated to dldsk01 ([192.168.0.246]:46359).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

Last login: Tue Nov 9 12:26:28 2021 from 192.168.0.250

I also tried to delete "Remote Host Key" and "Discovery Remote Host Key" on the SSH Connection Screen, didn't help. The Host, Port and User Name are correct (same as tested on command line above).

 

[Richard Durso]

Seems to be a problem with newer servers using openssh 8.8p1-1, they will log message "userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]" when TrueNAS tries to make an SSH connection.

https://dev.to/cloudx/why-openssh-8-8-cannot-find-a-host-key-type-if-ssh-rsa-is-provided-49i

Starting to see other reports of issues:
https://bbs.archlinux.org/viewtopic.php?id=270005