[Solved] [Possible] Security breach due to my idiocy

[zoomzoom]

I'm not sure when it occurred, but more than likely some point in the past 2 months, an intruder was able to gain access to my server... however I'm not sure which logs should be looked at to verify.

I went to log into the web gui today and it wouldn't accept the root password, and even though I was able to give the command passwd via SSH and change it to something else, it was still refused by the web gui. After a few minutes of repeatedly changing the password via SSH and having it refused by the web gui, I logged in to the server console via IPMI and changed it, which was then accepted by the web gui.

In combination with the above, the reason I'm concerned the server was breached from WAN was because around 2 months ago I was looking at setting up OpenVPN in a jail since I could get far better performance from a vpn installed on the server vs the current setup through my router. I was in the middle of configuring everything and had set up DDNS on the server when something came up for a few days and I completely forget I had enabled DDNS.

I'd like to understand what occurred and was hoping someone could point me in the right direction of what logs to read to determine if a breach did occur and if so, how to track down when it occurred and how many times the system was accessed due to the breach.

 

[Robert Trevellyan]

Start with /var/auth.log.

 

[zoomzoom]

I've already rebooted the server a few times, and I assume (though I could very well be mistaken) var is cleared upon reboot (no log files currently reside in the root of /var).

If the necessary logs aren't available due to rebooting, would I be safe to assume if I restored to a snapshot taken 3 months ago or so (prior to enabling DDNS), I'd eliminate the possibility a backdoor was left on the server?

 

[Jailer]

I'd save the config and do a fresh install.

 

[zoomzoom]

I've never had to reinstall before, so is there anything the config doesn't save that I should make note of prior to reinstalling?

Also, I assume it's possible to re-add my datasets to the new install... could you please point me in the right direction on where I could find information on how that works? <--- Nevermind, I found the applicable section in the manual =]

 

[SweetAndLow]

What ports did you open on your router? Just having ddns doesn't mean anything.

 

[BigDave]

reinstalling the os will not change your file structure or datasets, that is part of your volume or pool.
You save your config file before doing a re-install to save your smart/scrub settings, permission/share settings etc.
performing a fresh install is just the os on the boot drive, nothing more. after it's done you then use the saved config file to reinstate those settings.

A good long look at the manual might be in order.

And please make sure you re-install the same version of FreeNAS

 

[zoomzoom]

What ports did you open on your router? Just having ddns doesn't mean anything.

The only ports that would have been able to gain access would have been the ssh port for the router (public key authentication only), the vpn port for the router (for the initial connection prior to being handed off to the vpn interface, of which requires both SSL and TLS authentication), and the SSH port for the server (which isn't directly accessible from WAN as remote SSH is only allowed via VPN or via a multi-hop through the SSH of the router first).

I'm not 100% certain my server was breached, and the only reason I assumed it probably had been was because the root password had been changed. However, I don't know if it was changed by an intruder or if some glitch in the server caused it to get reset to the default password (if there is one, which I didn't think to try until after I had changed it via the server console).

 

[zoomzoom]

reinstalling the os will not change your file structure or datasets, that is part of your volume or pool.
You save your config file before doing a re-install to save your smart/scrub settings, permission/share settings etc.
performing a fresh install is just the os on the boot drive, nothing more. after it's done you then use the saved config file to reinstate those settings.

A good long look at the manual might be in order.

And please make sure you re-install the same version of FreeNAS

Yeah, I realized the manual had the information I was looking for about the datasets after I replied =]

I didn't know about the same version though, thanks for the headsup :D

 

[SweetAndLow]

The only ports that would have been able to gain access would have been the ssh port for the router (public key authentication only), the vpn port for the router (for the initial connection prior to being handed off to the vpn interface, of which requires both SSL and TLS authentication), and the SSH port for the server (which isn't directly accessible from WAN as remote SSH is only allowed via VPN or via a multi-hop through the SSH of the router first).

I'm not 100% certain my server was breached, and the only reason I assumed it probably had been was because the root password had be changed. However, I don't know if it was changed by an intruder or if some glitch in the server caused it to get reset to the default password (if there is one, which I didn't think to try until after I had changed it via the server console).

Sounds like you don't really have any way to get access to your server. It sounds like you have no way of getting access to your server remotely. I vote you typed your password wrong.

 

[zoomzoom]

Sounds like you don't really have any way to get access to your server. It sounds like you have no way of getting access to your server remotely. I vote you typed your password wrong.

I initially thought I had typed the wrong password, however after typing it twice and having it refused, I copied and pasted it and it was still refused. Even more odd, when I changed the password via the passwd command in SSH, that password was also refused and so were the next 3 I set via passwd in SSH. It wasn't until I used the server console itself via IPMI and chose the option to reset the root password that I got the web gui to accept the password.

Is there anyway a glitch could cause the root password to be reset or any other way the root password could have been changed (server is in a secure location only accessible by me and no other users have access to the server, physically or digitally)?

 

[solarisguy]

The same password is used for root SSH (if you enabled root SSH) and GUI.

So if you had enabled root SSH, your problem is not an intruder, but something in your configuration.

Merely re-installing FreeNAS might lead to you the same mistake in your configuration, so before you install FreeNAS anew try to

• recall all the configuration steps you did before over the lifetime of this installation;
  
• write down (in detail and in order) all the steps from the manual that are applicable to you (not all are!).

 

[zoomzoom]

The same password is used for root SSH (if you enabled root SSH) and GUI.

So if you had enabled root SSH, your problem is not an intruder, but something in your configuration.

Merely re-installing FreeNAS might lead to you the same mistake in your configuration, so before you install FreeNAS anew try to

• recall all the configuration steps you did before over the lifetime of this installation;
  
• write down (in detail and in order) all the steps from the manual that are applicable to you (not all are!).

SSH is done via PKI and the password on the key is different than than the root password. There is no problem with the SSH config... same configuration has been used for over a year.

Are you aware of anything within the OS that could cause an uninitiated reset of the root password (and why resetting the root password via the command passwd would have the new password not be recognized)? If there isn't, the only way it could have been changed is by an intruder; If there is, I have some troubleshooting to do.

 

[solarisguy]

If your root SSH login was a passwordless one, there is always a slim chance of a electro-mechanical problem with your keyboard.

Or you logging in to somewhere else. Password change in a root SSH shell and from the console should give the same results, intruder or no intruder.

 

[Jailer]

Sounds like you don't really have any way to get access to your server. It sounds like you have no way of getting access to your server remotely. I vote you typed your password wrong.

I agree.

Just think about this for a minute. If you managed to hack into a box wouldn't you want to keep the password so you could access it later without being detected? Changing the password would immediately alert a user that something was wrong causing them to panic and do something like start a thread on a forum saying their box has been hacked.

 

[anodos]

I agree.

Just think about this for a minute. If you managed to hack into a box wouldn't you want to keep the password so you could access it later without being detected? Changing the password would immediately alert a user that something was wrong causing them to panic and do something like start a thread on a forum saying their box has been hacked.

Or to be more specific, if you can change root's password then you already have root access making the change redundant.

 

[Robert Trevellyan]

Looks like you're in good hands here, but for future reference ...

I assume ... var is cleared upon reboot (no log files currently reside in the root of /var).

No, but I sent you to the wrong place. Take a look at the contents of /var/log. Mine looks like this:

Code:

[root@poweredge] ~# cd /var/log
[root@poweredge] /var/log# ls
./                           dmesg.today                  nginx-access.log
../                          dmesg.yesterday              nginx-access.log.0.bz2
3ware_raid_alarms.today      lpd-errs                     nginx-error.log
3ware_raid_alarms.yesterday  maillog                      pbid.log
auth.log                     maillog.0.bz2                pf.today
auth.log.0.bz2               maillog.1.bz2                ppp.log
auth.log.1.bz2               maillog.2.bz2                proftpd/
auth.log.2.bz2               maillog.3.bz2                samba4/
auth.log.3.bz2               maillog.4.bz2                security
auth.log.4.bz2               maillog.5.bz2                sssd/
auth.log.5.bz2               maillog.6.bz2                telemetry.json.bz2
auth.log.6.bz2               messages                     ups.log
cron                         messages.0.bz2               userlog
cron.0.bz2                   messages.1.bz2               utx.lastlogin
cron.1.bz2                   messages.2.bz2               utx.log
cron.2.bz2                   messages.3.bz2               wtmp
debug.log                    messages.4.bz2               xferlog
debug.log.0.bz2              mount.today
debug.log.1.bz2              mount.yesterday

You can see the current auth.log and 7 previous versions.

 

[zoomzoom]

If your root SSH login was a passwordless one, there is always a slim chance of a electro-mechanical problem with your keyboard.

Or you logging in to somewhere else. Password change in a root SSH shell and from the console should give the same results, intruder or no intruder.

I know, that's why I found it odd lol =] I do recognize this sounds like user error, however after changing the password 4 times successfully with passwd and having each refused by the web gui, I was hoping that would show it's not user error. One of the passwords I changed it to was the same password I use for the RSA SSH key, which was also refused by the web gui (this would rule out the keyboard, as I was able to SSH in with no issue)

 

[SweetAndLow]

Looks like you're in good hands here, but for future reference ...

No, but I sent you to the wrong place. Take a look at the contents of /var/log. Mine looks like this:

Code:

[root@poweredge] ~# cd /var/log
[root@poweredge] /var/log# ls
./                           dmesg.today                  nginx-access.log
../                          dmesg.yesterday              nginx-access.log.0.bz2
3ware_raid_alarms.today      lpd-errs                     nginx-error.log
3ware_raid_alarms.yesterday  maillog                      pbid.log
auth.log                     maillog.0.bz2                pf.today
auth.log.0.bz2               maillog.1.bz2                ppp.log
auth.log.1.bz2               maillog.2.bz2                proftpd/
auth.log.2.bz2               maillog.3.bz2                samba4/
auth.log.3.bz2               maillog.4.bz2                security
auth.log.4.bz2               maillog.5.bz2                sssd/
auth.log.5.bz2               maillog.6.bz2                telemetry.json.bz2
auth.log.6.bz2               messages                     ups.log
cron                         messages.0.bz2               userlog
cron.0.bz2                   messages.1.bz2               utx.lastlogin
cron.1.bz2                   messages.2.bz2               utx.log
cron.2.bz2                   messages.3.bz2               wtmp
debug.log                    messages.4.bz2               xferlog
debug.log.0.bz2              mount.today
debug.log.1.bz2              mount.yesterday

You can see the current auth.log and 7 previous versions.

I was going to mention this also. What do you mean all your logs are empty? Are you on an older version of freenas? Correct version save the logs to the pool.

 

[SweetAndLow]

I know, that's why I found it odd lol =] I do recognize this sounds like user error, however after changing the password 4 times successfully with passwd and having each refused by the web gui, I was hoping that would show it's not user error. One of the passwords I changed it to was the same password I use for the RSA SSH key, which was also refused by the web gui (this would rule out the keyboard, as I was able to SSH in with no issue)

Does changing the password via passwd actually work? Now that you have things sorted a little bit you should ssh in and change password via passwd then try using that to login to the GUI.