Bot attacks - death to the bot

James S

Explorer
Joined
Apr 14, 2014
Messages
91
In the last few days I've been seeing multiple login attemps similiar to:
Jan 6 22:19:04 freenas sshd[53755]: Disconnected from invalid user unm 51.77.140.111 port 38102 [preauth]
Jan 6 22:30:56 freenas sshd[54290]: Disconnected from invalid user zhz 34.66.28.207 port 34648 [preauth]
Jan 6 22:54:28 freenas sshd[54977]: Disconnected from invalid user nagios 218.78.54.84 port 46548 [preauth]
Jan 6 23:30:30 freenas sshd[56247]: Disconnected from invalid user pondering 59.51.65.17 port 56748 [preauth]
Jan 6 23:30:49 freenas sshd[56249]: Disconnected from invalid user pi 59.41.65.251 port 9167 [preauth]
Jan 6 23:33:04 freenas sshd[56338]: Disconnected from invalid user db2inst3 110.164.205.133 port 19259 [preauth]
Jan 6 23:45:42 freenas sshd[56687]: Disconnected from invalid user fl 51.158.104.58 port 50542 [preauth]
Jan 6 23:48:38 freenas sshd[56769]: Disconnected from invalid user odell 150.95.212.72 port 37430 [preauth]

I'm running 11.2-U7 on a local network. However, I allow users to backup to the machine over SFTP. While this puts the machine "on the net" I've followed suggestions to (1) use a non-standard port (port forward) plus each user has SSH keys (plus password). All has been quiet for well over acouple of years until these attacks. I've disabled the port-forward and all goes quiet.

What has happened here? Has a scanner managed to find the non-standard port, got lucky and then told the bot-net army to pay a visit?
As a current mitigation I've changed the external port but should I be doing more? Is Pfsense and fail2ban a good way / worthwhile way to go?

Please help me kill the bots!
Thanks.
 

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
those are...weirdly specific names for a bot. are you sure those arent the users you allow to backup, OR their friends they maybe have shared info to?
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
those are...weirdly specific names for a bot. are you sure those arent the users you allow to backup, OR their friends they maybe have shared info to?
No - luckily no recognized users here. These just seem to be a bit poking common names (admin test etc)?
 

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
poking common names
yet "Admin", probably the most common admin name ever, isn't present. so ya, i dunno. I would probably say check your security and then just block those IP's directly
 

Tigersharke

BOfH in User's clothing
Administrator
Moderator
Joined
May 18, 2016
Messages
890
I assume that your FreeNAS is NOT directly attached to the internet, ie, there is a firewall device between the NAS and the unfiltered public. ANY type of firewall will be an absolute benefit. The best being one that you can maintain and keep up-to-date, since expecting cisco/linksys not to obsolete your functioning hardware (and quit updating needed security stuff) is pointless. I use OPNsense which is actively developed and has a friendly community, devs.
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
I assume that your FreeNAS is NOT directly attached to the internet
Yes :) There is a cisco RV345 in between the FN and the outside world... so "something"
Tigersharke said:
I use OPNsense which is actively developed and has a friendly community, devs.
Interesting! Have you any recommended "how-tos" on how to implement for FN 11?

Thanks!
 

artlessknave

Wizard
Joined
Oct 29, 2016
Messages
1,506
Interesting! Have you any recommended "how-tos" on how to implement for FN 11?
uh. how to implement what for FN11? opnsense? opnsense is a router OS, a fork of pfsense; what do you mean by "implement"?
 

Tigersharke

BOfH in User's clothing
Administrator
Moderator
Joined
May 18, 2016
Messages
890
For a firewall/router that can include the greater number of features, you'll need: One low to moderately powered PC with at least 2 NICs (intel preferred), an SSD or HDD or other NOT usbstick storage to boot and run and cache as desired (squid!), requisite ethernet cables, a minimum 3GiB usbstick to use for installing OPNsense. Viola!

The community is at the OPNsense forums: 19.7 production series. They will be happy to help, plenty of info there. OPNsense is based on HardenedBSD.
IMHO, if you would need wifi access, add that to the OPNsense box as well but best to use a hardline (ethernet) for source from your provider. Ethernet is a guaranteed signal that will not suffer all of the possible frequency overlap that exists as intentional radio and unintentional noise (microwave etc).
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
a fork of pfsense; what do you mean by "implement"?
From the dictionary: "to put into effect" :)

I'd seen various posts about using pfsense on the FreeNas system and assumed this was the idea.

For a firewall/router that can include the greater number of features,
Thanks for putting me right. I need to research this a bit and think how far I want to go with this!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
yet "Admin", probably the most common admin name ever, isn't present. so ya, i dunno. I would probably say check your security and then just block those IP's directly

I've seen a number of these in our SSH honeypot recently. The large number of IP addresses available to some of these bots means that they can get away with testing at rates previously unimagined, and they have lists of previously distributed default login/passwords, common ones that have been found in a variety of ways, etc., and it doesn't really hurt *them* to try them all just in case.
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
I've seen a number of these in our SSH honeypot recently. The large number of IP addresses available to some of these bots means that they can get away with testing at rates previously unimagined, and they have lists of previously distributed default login/passwords, common ones that have been found in a variety of ways, etc., and it doesn't really hurt *them* to try them all just in case.
For some reason I missed the notification on this one. Yes, this fits. There is a lot of weird random names. The bot seems to be running down a list of possibilities. Given these are "just in case" attempts on a harden system I'll put strengthening my protection (e.g., with a firewall) on the backburner for now.
Thanks to all . . .
 

shanemikel

Dabbler
Joined
Feb 8, 2022
Messages
49
This ends up happening any time I open an SSH server to the internet.

1. Always configure SSHD to disallow root login and require ssh key authentication. That’s much more important than using a non-standard port. SSH key auth is really secure. The risk isn’t in the protocol, but it’s very difficult to secure a system shell when people get access. Not too hard to secure a shell for somewhat trusted users with filesystem permissions and sudoers, but very hard to secure a system (particularly against DOS, e.g. fork bombs) against malicious actors. If possible, it’s preferable to only allow users to execute specific commands over ssh (simple to specify in authorized_keys file).

2. The good news is that if properly configured it’s virtually impossible to DDOS an ssh server. Just make sure the authentication logs don’t completely fill up the disk (log rotation should be configured by default).

3. If the system is a high value target, prepare for the likelihood that user keys are stolen or user computers are hacked. Use disk quotas and permissions (principle of least privilege) to limit the impact of such an event. Even if attack is unlikely, it’s a good idea to prevent users from stepping on each other’s toes.

4. Fail2ban is a good measure, just know you can lock legit users out so set a timeout or be available to support them when it happens. You may want to preemptively block connections from all IPs outside of your country using a geoip database. If you have users in multiple countries you may prefer to blacklist Russia, China, and India. It should be easy enough to write a script to check where these malicious attempts are coming from to see if it will work. There are some free geoip databases and APIs such as:

5. You can prevent key leaks by requiring users to rotate keys or controlling distribution of keys. Disable user key files in $USER_HOME/.ssh/authorized_keys and read them from a file/directory without user write permissions (then you have to figure out a way for users to add their keys).

6. You can also use Yubikeys or other hardware tokens with GPG key derived SSH authentication. You can force users to use them by provisioning them yourself or ask them nicely to follow your instructions and hope they don’t deviate. Mac’s T2 (secure enclave, i.e. Touch ID) devices can be used to securely generate ssh keys as well:

7. As a policy you can ask users to use password encrypted keys, but I personally find this to be a PITA. Yubikeys can be password protected or configured to require touching the capacitive switch contact as well.

8. In corporate environments and cloud it’s popular to use what are called SSH Bastion Hosts. They’re basically reverse proxies and some of them purport to make it easy to centrally manage security policies or adapt other forms of authentication and authorization including SSO. Cloudflare has something paid, but you may be able to adapt their free services to your needs as well. If so, I’d be interested to hear the story.

9. Another alternative SSH authentication system based on SSL, which may be easier if you’re already distributing device certificates to corporate provisioned systems. This design looks similar to Kerberos but probably easier to manage:

10. Speaking of Kerberos… 2FA security tokens + Kerberos is the SSH access system I used at one of the big tech giants, behind the corporate VPN of course (as a dev user, not IT dev/admin). VPN access required device cert + user pw + 2FA token, then ssh access required device cert + user pw + 2FA token again.

On a non-critical system, I could easily stop anywhere from [2]-[4] and sleep soundly.
 
Last edited:

shanemikel

Dabbler
Joined
Feb 8, 2022
Messages
49
And, if you’re looking at open source router/firewall, I’m very fond of OpenWRT OS and PC Engines hardware.
 

James S

Explorer
Joined
Apr 14, 2014
Messages
91
And, if you’re looking at open source router/firewall, I’m very fond of OpenWRT OS and PC Engines hardware.
Thanks for the comprehensive post above with lots of sound advice.

Point 4 however is unsound and misplaced. Everyone is one the internet and faces the same threats. Though, for people running servers the risks are greater.
There is a potential myth that all unwanted traffic originates from China, India etc. (the largely "the East"). Over 90% of port scanning and SSH attempts - for me- originate from the US. Maybe these are bot-armies coordinated out of China, India etc. but I don't have data on that -- and I guess you don't either. That though is not really the security problem here. We are all trying to stop those evil bots breaking in wherever they come from.

So blocking geographic (or geo-political) regions I don't see as a sound approach. It quickly starts to smell of pretty bad things -- like politics and stereotyping and racism. A threat is identified by is behavior rather than an assumption about its -- potential -- source.

Of course, this touches on more fundamental arguments about the keeping internet open and free. That, though, is for others to chip away on!
 

shanemikel

Dabbler
Joined
Feb 8, 2022
Messages
49
Since were on an English language forum, I assumed he was running a server in North America, Europe, or the British Commonwealth, of course I could easily be wrong. If he was running a server in Russia I would suggest blocking US. The point is to shut out IPs from regions you don’t intend to serve, if you can safely make such assumptions about your users.

If it's an open online community, than this may not apply, but a server for business, family, or IRL friends, it is generally an easy policy. Just an application of PLP.

Like I said: it's easy enough to check where they are coming from. It looks like most of the IPs listed in the log snippet are from China, with a couple from France, one from Thailand, one from Japan, and one from USA.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
So blocking geographic (or geo-political) regions I don't see as a sound approach. It quickly starts to smell of pretty bad things -- like politics and stereotyping and racism.

This is alarmingly naive.

Talk to any actuarial about their profession, and you will rapidly learn that factors such as age, race, sex, education, location, etc., are all basic variables in calculating insurance risk.

Talk to anyone living in the Bay Area right now and they will tell you that leaving a new car out in the open risks catalytic converter theft, and that visible items in the car are significant risks to a window smash and grab.

The problem with the Internet is that it tends to level the playing field. Physical risks which might have been localized to a particular area (Bay Area for ex.) become more generalized over the Internet, because Russia is merely milliseconds farther away than Brazil.

Over 90% of port scanning and SSH attempts - for me- originate from the US. Maybe these are bot-armies coordinated out of China, India etc. but I don't have data on that -- and I guess you don't either.

It doesn't really matter where they are coordinated out of; there is no IP header that says "my botnet controller is at IPv4 12.34.56.78", so no real good way to do that. If you are seeing batch port scanning/SSH attempts from the US, my guess is that you're probably being probed by cloud instances (AWS, DigitalOcean, etc). Is there any chance at all that you would ever log INTO your systems FROM AWS or DO? If not, you can block them by ASN. You get an undue amount of scanning activity from Comcast, Spectrum, or AT&T residential broadband? Do you need to be able to reach your stuff from those? If not, blocking it may be prudent.

The thing that's easy to forget while considering the bullet point list above is that software is not 100%; someday, today's ssh daemon may be found to have a critical vulnerability. The more you scope your ACL's towards authorized users, the less likely it is you're going to be caught up in the next big security scare.

I don't feel bad blocking Net 103 (APNIC, mostly India) or other large swaths of IP space that I know I don't need SSH connectivity from. IP ACL's are easily changed if that turns out to be wrong, but the cheap and easy security of not exposing stuff in the first place is some of the best security practice.
 

somethingweird

Contributor
Joined
Jan 27, 2022
Messages
181
modify ssh daemon to only allow certain user@ip address with only key-based authentication. Then rotate keys every so often.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
modify ssh daemon to only allow certain user@ip address with only key-based authentication. Then rotate keys every so often.

Well, modifying the ssh daemon is probably a bad idea. There's a good chance of introducing new vulnerabilities into encryption software if you're not intimately familiar with it. You can accomplish this through configuration without modification in any case.

I'm fine with our ssh gateway implementation here at SOL.

Most bastion hosts I see tend to be UNIX boxes with a full login environment. This seems risky to me, as it provides a toehold for an intruder to "do more".

Other bastion hosts serve as a launchpad for SSH's ProxyJump feature, which has the distinct downside of relying on a ssh key held by the end user to make the jump "inside". Risky design. Nice if you totally trust your users I guess.

Ours is designed as a bastion host that holds the inside keys in escrow on behalf of the user; the user never gets the inside key. You ssh into the gateway in the normal manner, but then have to ssh to the inside target, which relies on a user key held by (and only known to) the bastion host.

The upsides include that this is all done neatly within a FreeBSD jail. It is not actually entirely bulletproof, but is one of the best implementations I've seen.
 

shanemikel

Dabbler
Joined
Feb 8, 2022
Messages
49
modify ssh daemon to only allow certain user@ip address with only key-based authentication. Then rotate keys every so often.
whitelisting IPs is very easy and solid policy but none of your users will be able to connect from the cafe or airport.

Using a bastion also has the upside that you don’t need to mess too much with the target systems (like TrueNAS) outside of supported configurations. Any ssh daemon can easily be configured to work with a bastion the way @jgreco described.

@James S lol I totally missed I was responding to OP, addressing you in the third person… if you want to be open internationally and to cloud and vpn users, that’s your business. do your users need shell access or are you only using it for sshfs or something specific?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
So blocking geographic (or geo-political) regions I don't see as a sound approach. It quickly starts to smell of pretty bad things -- like politics and stereotyping and racism.
Nonsense. If all your users are in a certain geographical area, and won't have any legitimate need to access your system from outside of that area, preemptively blocking IPs from outside that region is a legitimate strategy having nothing to do with politics, stereotypes, or racism (except in the idiotic sense in which "racism" is often used today in which literally everything is racist). You don't have to do it by any means, but that doesn't make it unsound.
 
Top