So back in the late 60’s, “The Internet” was in its infancy, several groups of technologists at various institutions were running a network called ARPANET.  ARPANET eventually evolved into what we call the internet today.  Systems connected to ARPANET, and todays Internet are each assigned an IP address, which is how other systems on the network communicate with it.  Your computer has one, your phone has one, your fridge might have one, the web server I run that you’re reading this blog on has one.  Some have several.  The problem with IP addresses, is that the human brain generally doesnt remember numbers well.  Sure, you can learn them. in fact I’ve known some network engineers that could practically recite the IP addresses of several systems on their network.  But the average person needs something more friendly to remember.  So early on the folks running ARPANET devised a way to maintain a “HOSTS” file, which mapped IP addresses to names.  Every computer had a copy of one of these hosts files.  The problem was keeping them in sync, and up to date.  Thus, the Domain Name System (DNS) was born. 

A DNS server is a system that resolves names to IP addresses.  If your computer needs to get to www.undrground.org, it asks your local DNS server, probably run by your ISP.  That DNS server asks the root servers who the authoritative server for my domain is, and directs it toward my DNS server.  My DNS server responds with the IP address of my web server.  That’s about as much detail as you’ll need to get the point.  If my DNS server is offline, there’s a secondary DNS server which also resolves undrground.org.  All domains need at least two DNS servers.  You can have several.  If you’re running a service that’s really important you might have 4 or 5 DNS servers on record.   Twitter, for example, has four.  So in twitters case, 3 of their 4 DNS servers could be unreachable, and you could still resolve twitter.com as long as one of their DNS servers is still reachable. 

I wont got into too much detail and bore you with technical details, but you should hopefully get the sense that this system was designed with a decent amount of redundancy, and resilliency.  I’ve been in IT since the 90’s, and I cant recall any point where DNS resolutions were offline due to some manner of failure, or attack.  At least not globally.  Until today.

The cloud, the cloud will save you

So, did your eyes glaze over reading that intro?  Did you skim over it because it wasn’t all that interesting.  I’m not offended, DNS is not the sort of innovative exciting work that any of us wants to do.  It is however exceedingly important.  So what do you do with an important, but mundane service?  You pay someone else to do it!  Right? We’re all getting sold on how we can’t possibly expect to do as good a job of hosting our own services as the big guys like Google and Microsoft can right?  Why should you run something as mundane as DNS?  You can pay Amazon pennies (like.. 50 of them per month) to host your domain.  If it’s low traffic, you’ll probably get away with less than a dollar a month per domain name hosted on Amazon’s Route 53.  What’s the alternative?  Run a datacenter, with servers, and power, and cooling, and host DNS yourself?  That sounds like a no-brainer right?  A dollar a month, vs maintaining infrastructure to make sure people can resolve your domain name.  Now of course there’s several degrees in-between self hosting and cloud hosting.  You could pay a smaller hosting provider to host your domain.  Maybe you have a friend who can do it, maybe you just depend on someone elses infrastructure to handle it. 

To follow our earlier example, twitter (and many many other services), apparently pays Dyn to host their DNS.  Dyn is a DNS provider who has been around for quite some time.  Back in the day I used them to provide dynamic DNS for my dial-up connected linux box.  We’re talking late 90’s.  They’ve since grown into a major player in hosted DNS.  For unknown reasons, today someone launched a DDOS attack against Dyn, bringing down many sites who use Dyn as their DNS provider.  Now, these attacks didnt actually touch the web sites themselves, they just make it impossible for users to resolve the names, www.twitter.com, to the IP address that your device needs to actually connect to.  I personally experienced this with twitter, paypal, and zmanda. 

Do you see the problem here?  We’ve traded resiliency for convenience.  We’ve taken a core function of the internet, and handed it off to these big providers that host DNS, and they do a fine job of it, until they don’t.  Today Dyn suffered what I’m sure we’ll find out was some record breaking DDOS, which took down enough of their services that resolution was affected.  This is the sort of attack that is very difficult to predict, or defend against.  The point is… It happened, and it can happen again. 

So what do we do?

I really hate when some blogger pisses into the wind and makes all sorts of claims bout how horrible things are, but then does nothing to help suggest a solution.  So what can we do about this?  Well, personally, this gives me another arrow in my quiver in my continued opposition to moving core services to “cloud” providers.  That’s not for everyone though.  I have the skills to run these things, and I enjoy it, so of course that’s what I’m going to do.  For some that’s a costly proposition.  They need to hire someone who can handle it, or learn it themselves, so paying a provider to do it makes more sense.  I’m not opposing that.  You could pay me to do it. 😉  You could consider hosting your DNS with more than one provider.  Primary at somewhere like Dyn for example, and secondary (or tertiary even) at Route 53.  DNS is designed to have that level of separation.  Now, it could be that if YOU are the target of the DDOS, you’ll still get taken down, but at least you’ve lessened your chances of becoming a casualty in someone else’s war.