SSH Configuration Woes

[chelby]

So I used to have SSH configured with key pair authentication and it was working for a long time. I had configured it through the GUI and I had verified the appropriate files (sshd_config and authorized_keys) had been modified properly through the CLI.

• Root user login was disabled
• Password authentication was disabled
• Key pair authentication was enabled
• Public key was input in the users account in GUI and was reflected in the authorized_keys file

I have not needed to use SSH for quite a while and several updates have passed since. Apparently one of the updates bricked my SSH configuration and it is no longer secured with key pair authentication. I can only use it with the default sshd_config settings (password authentication enabled). Now when changes are made to the SSH configuration they do not reflect in the appropriate files (sshd_config and authorized_keys). Nor does turning off SSH in GUI actually turn off the sshd service.

I can obviously change the files manually without the GUI, and that works, but these changes reset when the server reboots. I am not aware of another way to permanently change these files without using the malfunctioning GUI.

So my two questions are:

• What would have caused this to stop working all of a sudden?
• How can I either fix the GUI or make the changes manually and permanently through the CLI?

Currently Running: TrueNAS-12.0-RC1

Thank you!

 

[chelby]

I found a CLI solution by doing the following:

Code:

mount -uw /
nano /conf/base/etc/ssh/sshd_config

By editing the config file here it seems to be semi-permanent, even after reboot. Unfortunately it will wipe after an OS update. I wish the GUI would just work correctly, and I have no idea why it isn't. Until then, this is my solution. If you have better suggestions, please let me know, thank you.

 

[rvassar]

Just a suggestion... Logging in as root directly is one of those "generally frowned upon" / "to be avoided" kind of deals. Yes, you may find yourself wanting to do so, for some automated task, script, etc... But you're much safer and more secure logging in as a regular user, and elevating their privileges as needed using "sudo" or "su". The "~/.ssh/authorized_keys" for your local user should never be overwritten by the GUI, or OS updates.

Ala:

Code:

% ssh -i myrsa.key rvassar@mynas
Last login: Sun Sep 20 08:30:39 2020 from 192.168.4.12
FreeBSD 11.3-RELEASE-p11 (FreeNAS.amd64) #0 r325575+fb17f3e15b8(HEAD): Tue Jul 28 11:09:10 EDT 2020

    FreeNAS (c) 2009-2020, The FreeNAS Development Team
    All rights reserved.
    FreeNAS is released under the modified BSD license.

    For more information, documentation, help or support, go here:
    http://freenas.org
Welcome to FreeNAS

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.
rvassar@mynas% sudo -s
Password:
root@mynas:/mnt/tank/homedir/rvassar#

 

[chelby]

I'm not sure what you mean. I wasn't logging in as root directly, root user login is disabled (as I stated) and I use ssh key pairs to login with my user account that has sudo privileges and elevate as needed. I assume you inferred I was using root because of the "#" before the code, it's just a habit of mine in my notations, I was not copying it from the CLI. Sorry for the confusion.

I was able to fix my problem (at least until an update wipes it) with the solution above. I am still looking for a more permanent solution to this issue.

My keys stored in "~/.ssh/authorized_keys" for some reason stopped working after an OS update. I tried re-initializing them through the GUI and CLI with no success. For some reason the OS sees them (they show up in the GUI after being input in the CLI), but it can't use them, it just says there are no keys when restarting the sshd service. The permissions are correct for the file and directory, unless the OS update changed the permissions requirements. I have 755 on the directory and 644 on the "authorized_keys" file, and they are owned by the user.

As a workaround I had to place the "authorized_keys" file in "/conf/base/etc/ssh/[user]/authorized_keys", with the same permissions listed above. The OS can use the keys that are stored in this location. I also had to modify the location of the "AuthorizedKeysFile" in the "sshd_config" to reflect the change to the location of this file.

 

[rvassar]

Interesting... And I apologize for assuming you were trying to ssh as root.

Disclaimer: I have not made the hop to TrueNAS yet. From what I remember, a previous version of FreeNAS had a "local homedir" default setting or initial config option, that I am now unable to quickly locate. Opening the "Accounts" -> "Users" tab, and clicking the the details for my user I see an option for a homedir there, and it is in it's own dataset on my main storage pool. I'd probably try setting up a test user, create a default homedir dataset on a storage pool, and set it in this location. Make sure the dataset is on a data pool, not on the boot pool. Anything you put on the boot-pool is likely going to be at-risk in an upgrade scenario.

 

[chelby]

No worries lol, I wasn't entirely clear on that point but I corrected the original post.

Opening the "Accounts" -> "Users" tab, and clicking the the details for my user I see an option for a homedir there

This version is relatively the same, not much has changed in that part.

it is in it's own dataset on my main storage pool. I'd probably try setting up a test user, create a default homedir dataset on a storage pool, and set it in this location. Make sure the dataset is on a data pool, not on the boot pool

Yeah I found this solution in a different troubleshooting thread. I thought it had to be a permissions problem, so I did exactly this, but it still didn't work. I will try creating a test user to see if existing migrated accounts somehow got corrupted. Perhaps a fresh user account might solve the problem, we will see. I will report back with the results. Thank you!

 

[chelby]

So here's what I did step by step:

1. I created a new user in the GUI
   1. I filled in all the normal requisite fields
   2. I made the home directory: /mnt/[main storage pool]/Users/[test user]
   3. Default home directory permissions
   4. Granted "Permit Sudo"
   5. Input the SSH Public Key
   6. Primary Group [test user]
2. This successfully created a user, created the home directory, and the requisite ~/.ssh/authorized_keys with the key auto populated.
3. I added the new file location to: /conf/base/etc/ssh/sshd_config (because the OS does not auto populate it for some reason)
4. I restarted the sshd service - SSH did not work
5. I ran ssh-keygen -A (because the OS has issues with the host keys frequently) - SSH did not work
6. I restarted the sshd service - SSH did not work
7. I rebooted the system - SSH did not work
8. I checked the following files for accuracy
   1. /conf/base/etc/ssh/sshd_config - it was correctly set up
   2. All of the authorized_keys files were correctly set up
   3. /etc/ssh/sshd_config I assume is populated by /conf/base/etc/ssh/sshd_config. This was correctly set up.
9. I ran ssh-keygen -A again - SSH did not work
10. I restarted the sshd service again - SSH magically started working again.
11. The test user with the key files in the User dataset is working. However, for the pre-existing users, this did not work.

I'm not sure exactly what this means. I assume it means the users that were migrated from the old version have some sort of issue in the database.

Then there are the other issues:

• Why do I have to run ssh-keygen -A so frequently? I assume something is wrong with the host keys.
• Why does rebooting the system not start the sshd service correctly, and why do I have to run ssh-keygen -A and then reboot the ssh service to make it work?
• Why does rebooting SSH and changing SSH service settings in the GUI do absolutely nothing?

It seems like the update not only corrupted the user database, but also crippled the SSH system's ability to maintain itself properly. It seems to also have broken the GUI's ability to run the SSH service entirely.

 

[rvassar]

I'm thinking you might have a bug to file...

 

[chelby]

Yeah I was thinking that, but I'm not sure if I have enough information to provide them with. I have no idea which update bricked everything unfortunately.

 

[chelby]

Can someone do me a favor and attach their default sshd_config file? I just need to review exactly what the stock settings are supposed to be in TRUENAS/FREENAS. Maybe there is just a syntax issue or a misplaced character, but I doubt it. I'm assuming the sshd service would output an error if something was wrong. Also, can someone verify the correct permissions for the sshd_config file, the .ssh directory and the authorized_keys file? I've been over these things a dozen times, but maybe I'm just missing something. Thank you!

 

[RiBeneke]

I have spent hours trying to get SSH working. I think I have carefully followed followed instructions in FN 11.3 user guide, deleted and recreated SSH keypairs and SSH configurations, but still cannot get it right. I must be missing something.
Setting up between 2 local network servers running TrueNAS 12.0-U3.1.
Both have root accounts for GUI login, using these for SSH without password.
Remote user <root> has public key pasted to root, disable password unchecked, options below not checked.
Local user <root> is similar but without public key.
Both servers have Services\SSH\Configure set to Port 22 and options below unchecked.

The output when I test SSH is as below, but I have no idea what I should be looking out for.
Except the final line that tells me 'Permission denied (publickey).'
Does anyone see something here to point me in the correct direction ?
Thanks.

Code:

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@localserver[~]# ssh -vv <remoteserverIP>
OpenSSH_7.9p1, OpenSSL 1.1.1h-freebsd  22 Sep 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname <remoteserverIP> is address
debug2: ssh_connect_direct
debug1: Connecting to <remoteserverIP> [<remoteserverIP>] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9 FreeBSD-20200214
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2-hpn14v15
debug1: match: OpenSSH_8.2-hpn14v15 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to <remoteserverIP>:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,none
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,none
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:cTtCb45+kb99gcpSsnSQbVvyu9Rs4oEF+6QIZb0vc/g
debug1: skipped DNS lookup for numerical hostname
debug1: Host '<remoteserverIP>' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_xmss
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_xmss
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try
root@<remoteserverIP>: Permission denied (publickey).
root@localserver[~]#

 

[RiBeneke]

I deleted all SSH connections and configurations and started again.
It now seems to be working.