When Anti-Spam Goes Bad - A Few of the Hundreds of Ways Your Email Can Get Lost

Email is something that would intuitively seem to be reliable. In fact, the system was designed to be inherently robust. Like the internet itself, the major potential points of failure all have backups. There are redundant delivery routes, DNS servers and email servers.

Exactly how much email gets lost isn't clear. A quick search for, "email delivery rates" includes results from various, "legitimate email marketers" (spammers) making claims regarding how well they can insure message delivery (clog your inbox with spam).

Lyris has conducted research that indicates message delivery is averaging 89% across the internet, with the better performing mail providers such as Yahoo! closer to 97%. They accurately blame anti spam filter software for much of the blocked email. However, Lyris seems to play both sides of the fence. They provide both anti-spam products, and offer services to email marketers. This makes me suspicious of any numbers they provide.

So why does so much email never make it to its destination? Outside of the periodic IT goof-up, some kind of spam filtering is almost always at fault.

There are two fundamental types of filters.

Content filtering relies on analyzing the content of the message to see if it meets certain criteria indicating it is spam. This can range from simple plain text filtering up to complex machine learning systems.

Blacklist based filtering looks at who sent the message to determine if it's spam. The method usually involves passing the sender's IP address to a third party to see if it's on a list of known spammers. The DNS protocol is used to relay the information.

The IP addresses of servers, end-users and entire networks can be included on blacklists. Sometimes the IP addresses associated with URLs found in emails may also be analyzed. Some more advanced software will check an IP address's status across multiple blacklists, and only filter a message if it's on a certain number of them. This causes a performance hit, but can improve accuracy.

Some implementations are better than others, but both of these methods are subject to error. From my experience, simple-text filtering is perhaps the most dangerous. Including one mention of the wrong phrase can get an email blocked. Even the best anti-spam software is subject to errors no matter how advanced the filtering. Several blacklists have also been shutdown by their owners, and set to flag all IP addresses as spam in the process. This caused countless emails to be lost.

Either of these types of efforts to block junk email can take place at dozens of different points during the message's delivery process.

The sender's SMTP server can prevent the message from ever being sent if it thinks something is wrong. This is most commonly triggered by attempts to send large volumes of email, and doesn't really fall into either of the previously mentioned categories.

Any routers between the two SMTP servers may decide to drop the packets based on their content. Again, this isn't the most common kind of filtering, but it is periodically used. The packets are usually filtered based on if their originating IP's are found on a blacklists.

The destination SMTP server is probably where most spam and legitimate email is deleted. The large majority of incoming SMTP servers now employ some kind of filtering. Commonly they use both content and blacklist based software. Secondary internal groupware mail servers may also perform additional filtering.

Client anti-spam applications such as anti-spam software for Outlook also delete a large amount of spam and wanted email. The upside of client-based applications is that users can readily review blocked messages to see if they were in fact wanted. Increasingly some spam fighting tools are built-in to email client applications and sometimes are enabled without the user even being aware.

Back