SSH permissions issues and system oddities (looks like a bug)

[snax]

I'm running into a problem that's preventing me from performing Rsync or ZFS replication tasks between two systems running FreeNAS 9.3.

System "A"
- Started with FreeNAS 9.2.1 and was subsequently upgraded (with a few updates in-between) to 9.3

System "B"
- Started with FreeNAS 8.0.x and was subsequently upgraded (with a many updates in-between) to 9.3

I keep getting "Permission denied (publickey,password)" failures -- even though my authorized_keys files are correct and I first ssh into the respective systems and accept the host keys. Which brings me to the following noted oddity:

NOTE: I'm logged into both systems (i.e. SSH) as root

System "A" is not using ~/.ssh/known_hosts. It's instead using /etc/ssh/ssh_known_hosts.
System "B" is using ~/.ssh/known_hosts.

I discovered the discrepancy when I renamed ~/.ssh/known_hosts to ~/.ssh/known_hosts.old (to force its recreation) and discovered that System "A" wasn't prompting me about the host key. Investigating, I discovered that System "A" was instead using /etc/ssh/ssh_known_hosts.

At this point, I'm not even sure which is correct or if this is what is causing the permission failures.

Could someone please shed some light on this?

Thanks

 

[dlavigne]

To clarify: on a FreeBSD-based system, /etc/ssh/ contains the host's key and known_hosts file and ~/.ssh/ contains that user's key and known_hosts file, so these are 2 different things. Replication and rsync need keys for both the host and the user.

 

[snax]

Regardless, why is it that when I rename the known_hosts file under ~/.ssh/ on System "A" and reconnect to System "B" I am not prompted concerning the host key. It is only prompting if I rename ssh_known_hosts under /etc/ssh/. System "B" is behaving in the exact OPPOSITE manner to System "A."

Something is wrong.

 

[dlavigne]

Regardless, why is it that when I rename the known_hosts file under ~/.ssh/ on System "A" and reconnect to System "B" I am not prompted concerning the host key. It is only prompting if I rename ssh_known_hosts under /etc/ssh/.

That makes sense. If you remove the known host from the host's key location, it should prompt for the host key at next connect...

 

[snax]

No offense, but you're still missing my point: Both system are running the same FreeNAS version, I'm logged into both systems as root, and yet both systems are behaving completely differently with regards to their SSH file locations. I did not manually do that. Why are they different? Is this why I'm having these permission issues?

 

[dlavigne]

Both systems are acting as different endpoints. For example, from the docs on Rsync over SSH Mode, you need:

• a public/private key pair for the rsync user account (typically root) must be generated on PUSH and the public key copied to the same user account on PULL
• to mitigate the risk of man-in-the-middle attacks, the public host key of PULL must be copied to PUSH
• the SSH service must be running on PULL

It makes sense that one end needs the host key (stored in /etc/ssh) and one end needs the user key (stored in ~/.ssh). During authentication, the appropriate known_hosts is updated on the appropriate end's location.

 

[snax]

That is fine, but here is the problem:

I am logged in interactively (me -- not a software component) to each system, from my desktop computer, as root. I am typing ssh xxx.xxx.xxx.xxx (i.e. the IP address of the peer FreeNAS device) from each respective peer. If I rename ~/.ssh/know_hosts on both boxes, before initiating the SSH connection, each box should prompt me to accept the peer's key. But that is not happening on System "A", because it is not associating ~/.ssh/ with its root account -- even though its home directory is /root. Why is that?

Specifically, this is what is happening:

Step 1:
Me -->SSH --> System "A" (as root); and at the same time,
Me -->SSH --> System "B" (as root)

Step 2:
Me -->(on System "A") mv ~/.ssh/known_hosts ~/.ssh/known_hosts.old
Me -->(on System "B") mv ~/.ssh/known_hosts ~/.ssh/known_hosts.old

Step 3:
Me -->(from System "A") ssh bbb.bbb.bbb.bbb (i.e. to System "B") ===>RESULT: **NO** host key prompt !!!
Me -->(from System "B") ssh aaa.aaa.aaa.aaa (i.e. to System "A") ===>RESULT: host key prompt

System "A" will only prompt me if I rename /etc/ssh/ssh_known_hosts

That can't be right. Something is wrong.

Hopefully the above is clearer, because I have no clue how to explain it so it's clearer, without driving myself up the wall.

 

[snax]

bump

 

[saurav]

On system A, check if /etc/ssh/ssh_known_hosts has system B's host key. It might be there if there is a replication task set up from A => B.

Or run ssh with the -v option to see where it finds B's host key

ssh -v bbb.bbb.bbb.bbb

You might see something like this:

debug1: Found key in /etc/ssh/ssh_known_hosts:2

 

[snax]

Thanks for the suggestion. I haven't yet had an opportunity to look at it but will do so this weekend.