SSH Host key changed unexpectedly: How to verify host integrity

[saurav]

I am running FreeNAS-8.2.0-RELEASE-p1-x64 (r11950) from an 8 GB flash drive on an HP ProLiant Microserver N36L. I have ssh enabled on the box (sshd_config below). I am suddenly getting a "REMOTE HOST IDENTIFICATION HAS CHANGED!" warning when I try to ssh into the box from my Macbook Pro. I have certainly not done anything to change it, but it surely has changed. What I have in /etc/ssh_known_hosts doesn't match with what I get with ssh-keyscan. Its not an inadvertent client-side change, since the host is in /etc/ssh_known_hosts (not ~/.ssh/known_hosts) which I can only edit as root.

Also, I get the same warning when connecting via ssh from my phone (on Android, using an app called ConnectBot), where the ssh_known_hosts is not even accessible.

And I can understand accidentally removing the host key from my known_hosts if I'm mucking around. But changing it?

So, my question is, is there any way the ssh host keys in freenas can change or "refresh" as a matter of course? That is, can they change as part of some kind of automated security audit/scan/maintenance or something? Will it be logged anywhere? I don't see anything anywhere in /var/log/*.

If not, what do I do to verify the integrity of my system? That is, how do I go about checking if the machine is somehow compromised? I can't think of any reason why someone who gets in would change SSH host key, but that's the only possibility left.

Anything else I can add here to help folks comment on my situation?

Regards,
Saurav.

Code:

######## sshd_config #########
Protocol 2
UseDNS no
Subsystem       sftp    /usr/libexec/sftp-server
ChallengeResponseAuthentication no
ClientAliveCountMax 3
ClientAliveInterval 15
Port 22
PermitRootLogin without-password
AllowTcpForwarding yes                                                          
Compression no
PubkeyAuthentication yes

 

[tingo]

Did you upgrade FreeNAS on your box in any way? Or reinstall it perhaps?

 

[saurav]

Yes, I upgraded from 8.0.4 to 8.2, and I had saved/restored the configuration. But that was more than a month back. I have been ssh'ing to this box since then without this warning. If at all anything is wrong, it happened within the past week. I noticed the warning only today since I don't have to ssh to it every day.

FWIW, I have been getting these alerts via email, but they didn't seem like security issues:

Code:

freenas.local security run output

No /var/log/mount.today
Checking for uids of 0:
root 0

Checking for passwordless accounts:
Checking login.conf permissions:
Checking for ports with mismatched checksums:

No /var/log/pf.today
No /var/log/dmesg.today

freenas.local login failures:
freenas.local refused connections:

-- End of security output --

Code:

Cron <root@freenas> /bin/sh /root/save_rrds.sh

tar: collectd/rrd/localhost/interface/if_octets-epair0a.rrd: Cannot stat: No such file or directory
tar: collectd/rrd/localhost/interface/.if_octets-epair0a.rrd: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

 

[jgreco]

So, had you ssh'd to your FreeNAS from the Macbook Pro since you upgraded? If not, the Macbook is just telling you that the key has changed, and this would be quite expected and normal.

 

[saurav]

Yes, I had ssh'd from my MBP to my FreeNAS box many times since I upgraded. And also from my phone. Never saw the warning before, until yesterday.

Besides, don't the upgrades preserve all ssh keys? I see the ssh host keys in the sqlite config db saved from FreeNAS?

 

[jgreco]

So you probably want to consider both the MBP and the FreeNAS box compromised, until you can identify the cause of the warning. Perhaps you should start by looking at the actual keys, modification times, and related items to help isolate what's happened. I seem to recall that the OpenSSH guys have a great walkthru of possibilities to investigate, including some nonobvious ones. Since the MBP is much more likely to be what's compromised, you will want to be careful what you do on it.

 

[saurav]

I have tried to ssh from a Windows machine to FreeNAS, and got the same warning. But I rarely ssh from that Windows machine to FreeNAS, so may be it had the old ssh host keys from before upgrading FreeNAS (assuming the upgrade changes the keys at all).

But then I ssh'd from two other Linux boxes, which I'm sure have the host keys of the new (upgraded) FreeNAS host. And I got a similar warning. Now I'm sure that the host key has certainly changed on FreeNAS.

Looking for that OpenSSH walkthru...

 

[saurav]

Haven't found that OpenSSH walkthru yet, but in the meantime I thought I would _somehow_ try to verify the signature of all packages installed on FreeNAS. I tried to list all installed packages by "pkg_info --all", but nothing shows up. There's no /var/db/pkg either. I guess FreeBSD's pkg_* commands do work as usual on FreeNAS, right?