I am a sysadmin. So whether I call myself a security professional or not, security is fused with my daily tasks. I build servers, I administer servers. Web servers, database servers, identity servers. Servers that make the college I work for.. Work. If i don’t keep security in mind, at worst our private data could be stolen, or our servers could be turned into spam zombies. So its important to keep a certain level of paranoia. It’s a pain in the ass, I’ll tell you that, but in the end, its worth it. I’ve had servers breached, trust me, its no fun. You get woken up in the middle of the night because your internet feed is saturated, and you customer’s sites are down because your pipe is so clogged by floods of seemingly insignificant traffic that every 3rd request is lost. Sounds like I’ve rehearsed that? I don’t need to. I lived it for years at a little web host that I was senior admin at before I really knew what the heck I was doing.

I’ve learned a bit over the years, so I’m writing this blog to help pass along some knowledge to those who might be in the same boat I was in back at that web host. I knew linux, well enough, I had admin chops, but I lacked the experience in security. I was a security n00b. Selinux was a pain in the ass, firewalls were something I barely grasped. Attackers were light years ahead of me. Having honed their skills against much better admins than I, they found me easy sport. I did my best, but we had a number of servers cracked, luckily none of them were serious. Mostly remote code execution attacks, which resulted in spam bots.

I sit here today with the experience to identify these sort of threats, and try to prevent them before they happen, instead of waiting for the 2AM sleep-deprived panic. After I left the web host, I started working at the college, and found myself surrounded by a team of professionals. One of which had a mind for security, and the rest of which had more experience that me. It didn’t take me long to switch my mindset from 2AM firefighter to 8-5 forward thinker and defender.

Defense in depth

All of your servers need network connectivity. It’s a fact, without network, what do you have? A linux box. A secure linux box albeit. One of the few tidbits of wisdom I gained in business school was this. The only truly secure server, is one locked in a closet with no connectivity. Obviously that’s also a useless server, unless its goal is to raise your electric bill. So perfection lies in a healthy mix of locked down, and open. This is where network firewalls come into play. Identify what services you’re running, and want users to access. Then configure your network firewall to only allow access to those ports, from the target users. If you’re running a web server, that needs access to the internet, you open ports 80 and 443, to the world. If you’re only serving your employees, then open it to just the ip ranges of your employees.

Then you’re done, right? No, picture your defense in layers. You’ve covered the physical network with your firewall. But your server has a firewall too! Why would you waste the time in firewalling the same application twice? Simple. If someone manages to get past your network firewall, they may have unrestricted network access to your web server! If you firewall at the iptables level as well, they have one more hurdle to get over.

Proxies

This ties back into the defense in depth idea. So, you have a web server, and that web server more than likely talks to a database, or some other back end. If your cheap, you might have them on the same server. So what does that mean? If someone compromises your web application… GAME OVER. So what can you do about that?

Separate your web server and your back end server. Let the database live on a second server, and firewall the hell out of it. Only allow access to that server from your web server. Then to take that a step further, put a reverse web proxy, running on yet another server, in front of your web server, and firewall your web server such that it only accepts web connections from the proxy server. Now your proxy server is the public facing server. And what does it have local to it? Absolutely nothing. Except some config that points it to your actual web server.

SELinux

No two ways about it. SELinux is a pain in the ass. It was designed by the NSA, so of course it’s difficult to work with. For years, step 1 in my mental check-list for server builds was: “Disable SELinux”. Every time I did it, i thought to myself, “I should really learn how to use that thing, it sounds like something I shouldn’t be turning off.”, but I never took the time. Then, a year ago, I went to RHCE training, and the instructor showed us some of the magic which is SELinux. With enough knowledge to know how to get around common issues under my belt, i decided it was time to stop disabling SELinux, and start pushing it on the servers at the college. SELinux is one powerful security tool! It’s also a daunting one. With SELinux enabled, you can configure your web server to talk to a database, or run perl or php until your blue in the face, but until you tell SELinux to allow it, it wont work. That sounds like its counter productive, but think of it this way.

If I enable SELinux, and lock it down, on the proxy server I mentioned in the above section. That means that the proxy isnt SUPPOSED to execute any code, it just passes traffic on to its upsteam hosts. So if someone convinced it to try to run code, then SELinux would stop it. Flat out. This means that even though perl, and mybe php exist on that proxy, no web attack will ever be able to execute remote code. This really improves your security posture!

So, SELinux, it’s a pain, but live with it, it’ll save your ass.

System Analysis, and rootkit checking

One of the things attackers like to do is rootkit your system. So they can put their garbage in your system, and then hide it from you. They do things like replace /bin/ls with a modified version of the same binary, that has some code in it that hides files from the administrator. They do the same for ps, lsof, things like that. I’ve even seen one that modified sshd. Not sure what his goal was, as he built his rooted version of sshd against the wrong version of openssl, and it wouldn’t start on the system he implanted it on.

An easy way to detect this is to get a baseline of your system while its known good (like right after install) and then routinely run checks against it to verify that its still just like it was on install day. One tool that does this is called OSSEC. Ossec builds a list of checksums for important files, and then alerts you if those checksums change at any point. If they do, and you didn’t expect them to, theres a good chance there’s an issue.

Log Analysis

Along with system analysis, OSSEC also does log analysis. This does just what it says. OSSEC monitors common system logs, and any extra logs you point it at, and it reports to you when people are doing things that could be naughty. Like if a user hits a number of 404 errors in a row on your web server. Chances are, that user’s scanning your web server for common attack vectors. Block em! Which is something else OSSEC does, it blocks naughty users automatically.

PassPHRASES

We’re all hung up on the word “password”. Password implies one single word, which acts as your pass key. This is an old term. It comes from the days when systems couldn’t hold more than 8 characters for user id’s and passwords. You ended up with 8 character jumbles that were almost impossible to guess. A Pass _PHRASE_ can be a lot more secure, and a lot easier to remember. XKCD can explain it much better than I can.

Ditch the password/phrase

What’s better than a Passphrase? Keys. They are just what the sound like. Picture your house key, but in electronic form. You already use SSH to remotely manage your nix server (I Hope!), take it a step further and generate yourself an SSH key pair. Take the public key and put it on your server, and protect the private key WITH YOUR LIFE. Then whenever you want to connect to your server, the public and private keys are compared, and if there’s a match, you’re granted access.

Educate yourself

A year ago, I went to Derbycon. Ooooh man did it open my eyes. Attend conferences, join an infosec group if you can find one. Learn, and don’t be afraid to ask questions. Responsible admins benefit us all. I’m just as interested in your server being secure as I am my own. Every well secured server is one less server I’ll find flooding my network.

Conclusion

I’ve rambled on enough, and I’m out of beer. Time to close this thing up. I hope I’ve shared some of my experience, and that I may have saved you from putting on your fire gear at 2AM.