Planning for Failure - High Availability Hosting for Dummies

Sometime yesterday the server servicing www.spambutcher.com encountered a few problems and needed to be taken offline and rebuilt. This process took roughly a day to complete.

The good news is that I had a backup plan in place. At most, users may have noticed a few 3-5 minute periods where www.spambutcher.com was offline as the DNS flipped over to the backup server. Things generally continued to work as normal.

I have a history of doing web hosting on a professional level, and know there's no such thing as 99.9% uptime for a single system.

Several years ago I hosted with the now defunct Dell-Host (Hell-Host), and they were twice down for several days. For a lot of sites this doesn't matter, but the SpamButcher anti-spam tool is tied directly to my bank account. When it's offline I can't do anything else.

The only way to maintain really high up-time is to have redundant systems in place.

Having a backup server does cost money, but a secondary good-quality hosting account is around $40 / month. Inexpensive accounts with less functionality can be had for around $10 / month. If your website is your revenue stream, it's probably worth it.

High Availability Web Hosting Checklist:

2. Verify all services and content are deployed and running as needed on your backup server. Setup your backup server with a separate DNS entry and test it (http://backup.mysite.com).

3. Keep your backup server updated with any new content. You don't want to fail-over to your website as it looked in 1999.

4. Make sure you can switch DNS over to your backup server even if your primary server is down.

There's a number of ways to accomplish this. My DNS is hosted by a third-party - No-IP, which itself has a lot of internal redundancy. A common trap is to host your web, email and (primary and only) DNS all on the same system. When you do this, if the box crashes you are "SOL"; you are going to be offline for a while.

One note - No-IP's email forwarding service employs an anti-spam solution which is a little overzealous. I use it as part of my backup system, but I wouldn't use it for my primary email.

5. Setup some kind of monitoring system to monitor both your primary server and your backup. If your backup server's hard drive crashes, and you need it six months later - you're going to be in a world of pain.

I'd love to recommend a cheap monitoring package, but the one I'm currently using has some issues.

Extra Credit:

For truly high uptime, look into configuring your monitoring system to switch the DNS over to the backup server automatically if the main one fails. It took me a while to figure it out, but this is possible using No-IP, and is likely possible with other solutions.

I've had times where I went out on day-hikes, got notification that my server had crashed via my cell-phone, but that the backup server had automatically kicked in. Definitely worth the extra $40 / month.

There are of course further intricacies such as how to have your email route in a logical fashion if your main server is down, but I'm not going to get into them here. The important thing is to think about what will happen if (and when) your main server goes offline.

Back