Keeping Critical Systems Up 99.9% of the Time - DNS

(this article is part of a series)

If your main server fails, and you need to change over to your backup server - you'll most likely have to change the DNS records for yourdomain.com / www.yourdomain.com.

You need two DNS servers hosted at separate locations
DNS is inherently designed to be fault tolerant. Unfortunately a lot of people "work around" the fault tolerance by either not bothering with a secondary DNS server, or hosting both their primary and secondary server on the same network.

If both your DNS servers are on the same network, and your connectivity goes down - what are you going to do?

Make sure you can change your DNS quickly
At very minimum, you should have direct access to your DNS records so you can update them manually. You don't want to be dependant on your ISP handling a DNS change request for you after your server has crashed at 2am in the morning. Many ISPs only deploy DNS updates once or twice a day.

Your monitoring software should be able to update your DNS records from command-line / script
In the best of all worlds - your monitoring system will handle switching DNS over to your backup server in event of failure. This way you can leave town for the weekend - and feel somewhat secure that things will keep running (even if someone spills coffee on your main server).

To do this you need to be able to deploy DNS updates via a script that your monitoring software will trigger.

Your DNS records need to be configured with a TTL time of under 5 minutes
All DNS records are labeled with a property called "Time to Live" or TTL.

TTL tells other DNS servers how long they should "cache" information about your host before requesting new information. Common TTL times range from a few minutes up to a day or more.

See an opportunity for a problem here?

If you switch your DNS over to your backup server - and your TTL is set for one day - some users will still be pointed at your main (crashed) server for up to a day.

If you're not sure what the TTL for your DNS is currently set to - DNS Thing can help (at least if you're a Windows user). Linux users can use the "Dig" utility.

Don't worry - doing all of this isn't that hard or expensive
A company called No-IP actually provides for all of the above services for around $25 / year (no - they're not paying me to say that). There are probably similar competing services out there that may be worth checking out.

They have multiple data centers throughout the country. At last count - spambutcher.com (the main website for the SpamButcher spam blocker) has five different DNS servers - most of them on different networks in different data centers.

By querying a URL like the following:

http://dynupdate.no-ip.com/ducupdate.php?username=myusername& pass=mypassword&host=www.mywebsite.com&ip=34.43.90.226

It's possible to request that No-IP make a DNS update. This can be done from a batch file using a tool such as WGET.

The DNS change is usually made within a minute or two. Last time I checked - No-IP's TTL times were set to 60 seconds. If your monitoring software takes 3 minutes to figure out something is wrong - your total downtime in event of server failure would still only be about 5 minutes.

Uh oh! Some single points of failure...
I'd like to say that I've removed all single points of failure from my website's operation. This isn't quite true. For the benefit of these articles - I'm going to try to spell them out. Maybe you can do better. If nothing else, knowing your vulnerabilities puts you in a better situation to deal with them.

For one, I'm having a single company host all my DNS. If No-IP goes belly-up, I could have some issues to deal with. It would certainly be possible to add yet more DNS servers hosted with other providers - but there would be further costs involved. Even then, if No-IP's servers were to be online, but delivering inaccurate information - I'm not sure what I'd do.

Another issue is that any given domain only has one "registrar." Your registrar doesn't actually resolve DNS requests - but it does provide information to the "root servers" as to which DNS servers provide authoritative information for your domain. If for some reason your registrar was to start providing incorrect information - you would have problems.

Next article: Your Website

Back