Keeping Critical Systems Up 99.9% of the Time - Your Website

(this article is part of a series)

As mentioned previously - the only way to be truly confident that your website won't be taken offline by an unplugged power cable or a hurricane is to have a backup server online at all times.

It's reasonable that you might skimp on the hosting package for your backup server a little. You may not have the same bandwidth or support requirements for a server that hopefully you'll only use a few days a year. However, you do need to be sure your backup server is capable of supporting anything and everything that your website needs to run.

Your backup server should be hosted with a different ISP than your main server
If both your servers are hosted with Megahost - and their intern botches their router configuration - you're going to be offline.

Your backup server should be located at a different geographic location from your main server
If both your servers are in Seattle - and a major earthquake hits - you're going to be offline.

Setup both servers in DNS
http://megahost.yourwebsite.com might be your main server
http://cheaphost.yourwebsite.com might be your backup server

Deploy and test all aspects of your website on your backup server
Upload your site to your backup server and test anything that's of importance to you.

Don't put this off until a server crash. It may turn out your backup host doesn't support certain Apache modules or ASP functionality. There might be certain niceties that break on your backup server, but you're OK with being offline temporarily. You'll want to know the fully story before your main server crashes.

Keep your backup server up to date
Whenever you post new content to your main server, make sure it gets posted to your backup server.

Setup monitoring for both your main and backup web servers
Don't just verify your web server is responding on Port 80. Monitor the site for actual content. Pick a phrase from your homepage - and make sure it's there. This can also let you know if your site has been "defaced" by hackers.

See the article on monitoring software for more details.

If you're dependant on a database - make sure it gets monitored too. This is best done by monitoring a webpage with database-driven content.

Perform the same monitoring against both your main and your backup server. If your backup server has a drive failure - and goes offline - you need to know about it. If it's not a "live" backup - it's not doing you any good.

Configure your monitoring software to allow for the periodic "hiccup." It's reasonable that once or twice a day a request to your server might fail due to internet connectivity issues, etc. Most monitoring programs let you specify a certain number of failed requests or total downtime before considering the site, "down." I typically consider a site that's been unavailable for more than 2-3 minutes to be "down" and in need of corrective action.

Implement automated fail-over to your backup server
Configure your monitoring software to switch the DNS for your website over to your backup server's IP address in event of failure.

This requires that you have a DNS server that you can update via a batch file or script. See the article on DNS for more information about this.

Optionally - you may want to configure it so that you automatically switch back to your main server should your backup server fail.

I don't recommend automatically switching back to your main server just because it came back online. Servers that are experiencing problems sometimes "bounce" online and then back offline. You're better off just using your backup server until you have a chance to sort things out.

Tip: if you keep one file different on both servers - it can make it easier to figure out which server is currently live. For instance - server.txt might contain the name of the server it's hosted on. Calling http://www.yoursite.com/server.txt would tell you which server is currently in use.

Test a failure
This may seem hard to do - but you need to simulate a failure. Otherwise, you can't be confident you'll stay online in event of a real failure.

One trick is to remove or modify the phrase you've setup the monitoring system to scan for. In SpamButcher's case, we monitor the homepage for the word "fuzzy logic." To test our fail over - we just need to temporarily remove that phrase.

If the test works - you should now be able to sleep through the night - knowing that you monitoring system will "take care of you" in event your website crashes.

Next article: Email

Back