SSH, or Secure SHell, is the way in which a modern linux system is managed. Most experienced sysadmins love the direct access and power they get from being able to securely connect to a shell on their systems with relative ease. SSH replaced Telnet somewhere in the 90’s as the remote access protocol of choice, for good reason. SSH allows administrators, or users, to access a remote shell over a secure tunnel, simply by connecting to an ssh server with their ssh client. SSH can also handle file transfers, which should replace FTP, though there are a surprising number of situations which still rely on good ol clear text FTP. Why? Ugh.

In this article, I’ll be talking specifically about OpenSSH’s ssh daemon, sshd. We’ll cover some of the possible security problems you can run into, and how to mitigate or outright solve them.

Why do we need SSH?

As I mentioned, SSH is a very powerful tool. Through an SSH session, you get a virtual terminal directly into your target system. In the wrong hands, that can be very problematic. So why do we leave such power accessible? Well, the power of SSH is a delicate balance. Within minutes, an administrator can have a secure session open to their server to respond to an incident. The simplicity is elegant.

SSH is also at the core of automation tools like Ansible. Ansible can apply changes to a system using SSH without the need of an agent. All changes are carried out using SSH and python.

SSH can also be used, as I mentioned, for file transfers. RSYNC can be tunneled through SSH for a secure synchronization of data. SSH can also be used to tunnel through one system to get to another, which is great for troubleshooting, but also great for an attacker.

Why Protect SSH?

If the preceding paragraphs haven’t convinced you, I’ll just say this. SSH is a powerful tool, and like any powerful tool it can be severely abused. If an attacker is able to gain a secure shell into your system, it’s game over. And attackers know this, so they’re constantly looking for systems with SSH open to the world so they can attack them.

Common Attacks

The single most common attack to SSH is brute force. I personally have never had a brute force attack against SSH on one of my systems succeed, but before I started following some simple lock-down rules, I can tell you they certainly tried! A quick search on Shodan shows 20,984,090 systems with SSH open to the world. Do they need to be? I can almost guarantee you that each and every one of those almost 21 million systems is getting several ssh brute force attacks per second. While I was writing this article, I started up a droplet on Digital Ocean, left SSH open to the world, and started tailing /var/log/secure. Within a bout 20 minutes I saw the first SSH probe, and within an hour, the flood of brute force attacks started.

Jun 23 01:35:33 centos-s-1vcpu-1gb-sgp1-01 sshd[10926]: Did not
receive identification string from 81.22.45.137 port 61000
Jun 23 01:37:47 centos-s-1vcpu-1gb-sgp1-01 sshd[10928]: Invalid
user fake from 104.248.132.25 port 36050
Jun 23 01:37:47 centos-s-1vcpu-1gb-sgp1-01 sshd[10928]:
input_userauth_request: invalid user fake [preauth]
Jun 23 01:37:47 centos-s-1vcpu-1gb-sgp1-01 sshd[10928]: Received
disconnect from 104.248.132.25 port 36050:11: Bye Bye [preauth]

In short, a bad password on a default install of sshd could be all that stands between the bad guys and your system. Lucky for you, I’ve got some suggestions on how to make a decently secure default install of OpenSSH even better.

Hardening

Don’t allow port 22 access from the world

Starting Nmap 7.70 ( https://nmap.org ) at 2019-06-22 20:56 EDT
Nmap scan report for 167.71.200.117
Host is up (0.26s latency).
Not shown: 995 closed ports
PORT    STATE    SERVICE
22/tcp  open     ssh

This is the most basic thing you can do to secure your system. I realize that this may not work for every scenario. Maybe you’re running a public use system, or maybe the CIO travels a lot and he might be accessing a system from lots of different places (we’ll talk about VPN’s in a bit). My point is you may have a valid reason that SSH absolutely needs to be open to the general public. I’m going to say that you need to think VERY hard about not letting that be your planned configuration. I’ve done it, but mostly out of laziness. There’s almost always a better way.

If you’re in a cloud provider, look into configuring a security group to only accept ssh from the network you’ll be administering your system from. If you’re in an enterprise environment this should be pretty easy since you’ve likely got an IP range that is yours, and you can whitelist it, and block everything else. If you’re a home user, you may have to perform some tricks to get your ISP’s dynamically allocated IP block whitelisted on your system. If you’re using a security group though, you should be able to get into your cloud provider’s self service portal, and change the IP range if you get stuck in a new range for some reason.

If you’re hosting a system which has no firewall in front of it, use the host based firewall to block port 22 from anywhere but your ip range in the same way I just described. The problem there of course is, that if you get locked out because your ip changed, you’ll have a harder time changing the allowed range.

If your server is on an enterprise network, there’s a good chance there’s a network firewall you can lock down ssh on. So use that to your advantage. Defense in depth folks, it’s a thing!

VPN

Ok, so let’s talk about that traveling CIO who absolutely NEEDS access to ssh from his hotel room. This seems obvious to those of us who have been around the block, but if you’re considering opening ssh to the world… Maybe you need to hear this. A VPN, or Virtual Private Network, is a way to build an encrypted tunnel from an endpoint, like your CIO’s laptop at that hotel in Maui, back to your enterprise network in Seattle. This is protected in its own way, and encrypted. So your dozens of servers which aren’t accessible to the world, could be accessible through this single point, the VPN. Yes, in a way this just shifts the attack target to the VPN, but its one more hurdle the bad guys need to get through.

Deny root access

Now we’re getting down to actual sshd configuration! Sshd has an option within its config called “PermitRootLogin”. By default, it’s set to Yes, becaue when you install some systems, you don’t have any accounts other than root to access the system to perform initial config. I recommend adding a user during install, or as part of your golden image, and turning root logins off from the get-go! There’s no reason to be logging in as root, and there’s certainly no reason to login as root over ssh.

PermitRootLogin no

Key based auth

Initially, I used key based auth as just a convenience. I generated a private key, added the public key to my authorized_keys file on the servers I managed, and I could now securely connect to my systems without a password! It saved time, and made automation possible. WIN! It wasn’t until later that I found out how this could really be leveraged to eliminate password brute force attempts, I’ll talk about that next.

To generate a key, use ssh-keygen on a mac or linux box. Maybe you windows folks can do this natively now, but I’ve always used PuTTYGen on windows (and then used the key with PuTTY). You’ll be asked for a type of key, and a length. I’ve been making 4096 bit RSA keys lately. The more bits, the harder the key is to crack. So why not?

[gangrif@meliodas ~]$ ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/gangrif/.ssh/id_rsa): ./test-key
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in ./test-key.
Your public key has been saved in ./test-key.pub.
The key fingerprint is:
SHA256:xfHOf4t3V3CCkJ+3FbEnQusAjBwOjBXfOa9WdGXBMqc gangrif@meliodas.undrground.org
The key's randomart image is:
+---[RSA 4096]----+
|   ++o.+. ....+o.|
|  . .+o..+o+o+o..|
|      o + =o=B..o|
|         = *E.+.+|
|        S o +. * |
|         o   .. .|
|        o     . o|
|       .      .o+|
|             ...o|
+----[SHA256]-----+

What you then end up with is a new file called “test-key” (in this case), and test-key.pub. test-key is my private key, and test-key.pub is the public key. Protect the private key with your life! You can leave the public key laying around where-ever you’d like, as it’s useless without the private key.

[gangrif@meliodas ~]$ cat test-key.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDAQxmjoHO6i4Kkq8vp49KtqAsMcw+ptgvd+PGjxV
YkDPGdzRZpJhq0c3BDstFs86ENlojs61zZaP3MihLR0BlDdIyM3jh9TmVvmOPiylhu4X9Q
liAca/ODxyVp76OpTKm+QcgBi/7i/JNiJdEgcSy8izB0Oil7LZjIhS6wCs3mQFVbkPXPf
e1lmHQIcuMDOKO0RuyecY/lrKodcr/YbEE4GsM/P11QdDPZ78lq83G3H5vqLqMrxVKkLw
7Z4//+PZPFFxi/N1EBwBp/uPEo4MfD1IwSmnKromRlkYTdOYlbiCNosdvEobtMARySdqj
v7RDn0Iwb3JxioUW6jQdAXfl8d01LvX/0Yb1tq5hF44ahwvI6TyoB4z3DRs+3cFiMPWQR7
LK8/qQOvHk5I2NMBAV9KuSN2ML0d7xeB8tglw38PYjhZsXl/t2rAWT4n8PU0o6+Hrrli+W
Efvk8QWpLesmBBmzG0sLjH0mXBU/xN4UBLLJNlrsUQ/TzEsdknntMhh7leOzechlU3PiT
NuUitweT9NqLJwCP+CHbeSJn2M5qzb090CPnCGpnr2GOfm2dL+RfHQi2R1Cf04nBDq3Wuh
APJY5SoEw2llfSgA2GlOdhcY9N9Bl9rLUBPgQ6ttBd7qZJ6E3BkiPlHnU+kSSpYr8Gkw1o
DGbtd2UUjExZqvz4rQ== gangrif@meliodas.undrground.org

That public key gets copied into the .ssh/authorized_keys file of the user you’d like to connect to without a password. Now, I say without a password, you still need to unlock the private key with the password you set during generation… You set a password, right?

You can also copy this key to a server using ssh-copy-id, but this isn’t intended to be a comprehensive guide to setting up ssh key based auth.

Disable Password Based Auth

Remember when I said key based auth opens you up to the possibility of locking down ssh brute force attempts completely? Well here’s how! With your ssh private and public key in place, you’re not getting prompted by SSH for a password anymore, right? So why leave that avenue open as a vector of attack? On every Linux server I run, I add my public key as part of our base install, and I turn off ssh password authentication before the system even hits the network. This is a setting right in sshd’s configuration file. Again, the default is to allow password auth, because of course that has to work for configuration. Once you have your keys in place though, turn that crap off.

PasswordAuthentication no

Congratulations, you’re now immune to password brute force attacks.

SSH User Jail, with chroot

Chroot is a neat tool, it lets you change your root, hence the name, chroot. Usually pronounced “chi-root”, or “ch-root”. It’s great for troubleshooting a system that you have access to the disk, but it won’t boot. you just mount its disk and chroot to /mnt/whatever. It’s also a neat security tool for something like a shell server. You can chroot a user at login, so they literally can’t see the rest of the filesystem. I don’t do this often, but it is a very viable jail for users. I found what looks to be a decent write-up here.

Rotation

It’s worth considering password and ssh key rotation. I’ll try to outline the good and bad here.

Password Policy and Rotation

I’ll lump password policy in with password rotation because they’re related. Password policies that enforce complexity are a great way to make your users pick a “strong” password. However, this has some caveats, especially when coupled with arbitrary (i.e. timed) password expiration. I could write a whole article on how I feel about password policy and rotation, but I’ll try to keep it brief here.

Last year, NIST did something of an about-face on their password guidelines, changing a few of their suggestions that had been ingrained in most of our minds for most of our careers. The Infosec community (which I dabble in) has been suggesting for years, an 8 character random passwords, or 8 character minimum length with strict complexity rules, were doing more to HARM password hygiene than help it. Combine strict complexity rules with a 3-6 month password rotation policy, and you’re setting your users up for frustration. That frustration leads to password re-use and other bad habits. So think hard about whether you want to impose this on your users. The new recommendations lean toward expiry only when a breach or compromise is suspected, and policy which leads your users into strong pass phrases rather than passwords. “This 1s my p@ssw0rd 2019.” is a lot harder to guess, and crack, than “W1nt3r2019”. Which, by the way, is a common go-to for both password setters, and red teamers. Bear in mind however, if you’re asking smart admins to rotate their passwords, you may not have this problem (because they understand WHY they’re doing it, and how important it is). But for your users at large, arbitrary expiration and complex “pass words” are becoming a thing of the past.

SSH Private Key Rotation

Like any identifier that can be generated, it might be good to consider rotating your private key periodically. Your private key isn’t quite as easy to steal or guess as a password might be, but it is possible that someones lifted your key without your knowledge. This gets into the “how paranoid would you like to be” territory. Changing the password on your private key isn’t good enough however. That password unlocks that key, if I swipe your key today, and you change your password tomorrow, the copy if your key that I have still has your old password. If you’re going to do this right, you need to generate a whole new key. Now’s a good time to make the key length longer if the standard has moved up since you last generated a key. Maybe you generate a new key every time your employer issues you a new laptop, maybe you do it once a year on your work anniversary. I haven’t found a solution that will manage this for you though.

Bear in mind, that generating a new key means all of the public keys you have out there in the world are now invalid. Or at least they’re not valid with your new key. You’ll likely need to use your old key, to place your new public key.

MFA

Multi factor authentication is becoming the norm. Some people have tried to claim that SSH keys are a form of MFA, they kind of aren’t. Especially if you’ve disabled password auth. You can however integrate mfa like TOTP, or YubiKey into your systems PAM config. Central auth solutions like FreeIPA make this easy.

Conclusion

So there you have it, I hope you can take this information and apply it to your systems to help improve your security. Don’t be one of those 21 million systems with SSH open to the world, there’s just no reason for it.

Thanks for reading!