Storm!

There was recently a major, major storm in the city that SpamButcher's corporate offices reside in.

The power flickered, but fortunately we never lost power or internet connectivity. While we don't have redundant connectivity to the actual office, the SpamButcher website is hosted at two separate locations with redundant DNS. This costs about another $600 a year to do, but it's worth it. It's hard to say if it literally pays for itself or not, but peace of mind is worth something.

I can't say as much for a company I do some additional work for. They never lost power, but their internet has been down since early Friday morning. Setting up redundant internet connectivity is a hassle, and more expensive than redundant hosting. Understandably, it wasn't done previously.

I made some attempts to rig up internet connectivity via neighboring companies' Wi-fi networks. I made good progress, but finishing the job would've required infrastructure changes. The company's principals evaluated the situation, and eventually decided to give everyone the day off. Other major local technology companies had also shutdown.

It's now Sunday morning, and their internet connectivity seems to still be offline. Oh, did I mention I'm their, "IT Guy?" Well, I am. That means this is my problem. I wish I could say was employed 100% by anti-spam email software, but I'm not quite there yet.

The company's ISP (DSL) is completely MIA. Their support numbers respond with, "all circuits are busy now" or similar messages. A trace-route report indicates a relatively large-scale network outage. I saw a few Usenet reports confirming this, and suggesting things might be fixed on Sunday (which I take to mean Wednesday). My assumption is they don't have power, or worse, are under several feet of water.

Fortunately, this company is not in the business of hosting websites. Unfortunately, internet connectivity is still crucial to their business.

This Sunday, I'm going to try to get them online with a wireless broadband company.

One major boo-boo on part of the ISP they apparently put all their DNS servers inside their own network. As a result, their homepage is down with no way to bring it back online until they get things fixed. If they had located at least one of their DNS servers on an external network, they could've re-pointed their webpage to an external temporary site telling their customers, "Sorry for the outage - we're working on it!" As it is, I have to assume everyone's just gone home for the weekend.

DNS was architected to be the technology that provides redundancy for the internet. It can also be used for blocking junk email. By strategic placement of DNS servers, web servers, and configuration of MX (email) DNS records, it's possible to create a network topology that can survive major regional outages. It's also possible to configure a domain's DNS records not to provide substantial redundancy. Depending on the business model of the company, either path may make sense.

Back