In his 2002 article entitled Death by Spam, Kevin Werbach predicted that as a result of spam, email would gradually move from being an open to a closed system. Six years later, where are we now and how far have we moved in that direction?
Sadly and not surprisingly, email is still greatly hampered by spam. And now, despite advances in server side filtering, we’re even more concerned with our spam than ever before. As a result of false positive reporting and fear of losing important messages, we still often find ourselves sifting through spam proving that Kevin Werbach was right on the money when he said humans will always be better at identifying spam than automated systems.
So, clearly we cannot rely on server side filtering alone to combat spam. Even Cloudmark, which provides leading ISPs with real time, collective antispam intelligence and reports “98% accuracy and near zero false positives” is not perfect. It should be duly noted that when we added Cloudmark with SpamAssassin to our email environment, the effect was staggering — virtually no false positives and minimal spam leakage. Indeed, Coudmark’s numbers are remarkable and its performance is exceptional. However, the fact remains that in any automated filtering system, a percentage of messages are still going to get misdirected and humans must sort them out.
The idea to move to a closed system evolved naturally from the problems arising from open system email. In a typical open system, email accounts will accept incoming mail from any sending address. A closed system, on the other hand, blocks all messages and accepts only those originating from approved senders, a practice known as exclusive whitelisting. Closed system, exclusive whitelisting implementations range from simple ones that validate senders from a user’s address book to more advanced ones that require senders prove their existence and validity through a challenge and response interaction.
Unless you’re working with a fixed number of people, a closed system with exclusive whitelisting presents problems with legitimate senders not included in your whitelist. Even with the challenge-response interaction, a sender’s own spam filtering system may block sender validation emails preventing a sender from knowing their message was undelivered and that an action was required on their part to complete the delivery.
A better solution is to utilize a combination of non-exclusive whitelisting and auto purging server side filtering. Non-exclusive whitelisting accepts messages from specific or domain level addresses but does not automatically delete the rest. Messages from unlisted addresses are subject to server side spam filtering and will be delivered if they are not identified as spam. As long as spam messages are temporarily stored on the server, any false positives are recoverable. SpamAssassin actually includes an auto whitelisting feature that keeps track of sender addresses for each user.
Not only is a completely closed system impractical, it’s detrimental to modern communication. The need to receive email from new clients is fundamental to operating and growing a business. In today’s environment, we recommend using a combination of non-exclusive whitelisting and server side spam filtering. If you’re in the process of selecting an email hosting provider, look for one that uses real time detection like Cloudmark or Commtouch. Additionally, most hosting services now support auto and manual whitelisting tools. Finally, select a provider that temporarily saves spam in a webmail folder that automatically purges after a specified amount of time. A hosting configuration with these technologies can have a dramatic impact on the amount of spam you have to deal with and at the same time let you continue to communicate effectively with new users.