Ample room for improvement in spam filtering practices

A recent article from Slashdot contributor Bennett Haselton highlights the risks inherent in any automated filtering system– even one that is well intentioned. Haselton runs a mailing list through which he informs his users of web proxies, which can be used to circumvent filtering. He regularly distributes information about new proxies as existing ones become blocked. In September, Haselton emailed the list with information about ten new proxies he had created. Two weeks later, two of the proxies were placed on the domain blocklist of Spamhaus, a spam-tracking organization. This itself was not new, as spam filters have previously identified Haselton’s proxies as spam.  What was new, however, was that following Spamhaus’ action, all 10 of Haselton’s new proxies (not just the two erroneously identified as spam) were disabled.  The proxies were taken down because Haselton’s domain registrar preemptively disabled all ten domains.

Why were all 10 of Haselton’s proxies disabled? It was actually the fault of two organizations working in concert: Spamhaus and Haselton’s registrar, Afilias.  Spamhaus is one of several organizations maintaining databases to help flag spam. Spamhaus maintains blacklists– which flag, for example, IP addresses associated with spam operation.  One of those blacklists is of domains typically found in spam messages, which Spamhaus calls the DBL. Spamhaus recommends that ISPs and other entities use the DNS blacklists to reject mail from bad IP addresses before it is processed by the mail server, then use the DBL to scan the content of remaining messages for blacklisted domains. Spamhaus’s lists are publicly accessible and receive, according to the website, billions of queries every day.

The problem with Haselton’s proxies began when Spamhaus placed two of the proxy domains on their DBL. An effective domain blocklist must be careful not to identify domains that are actually legitimate.  In fact, Spamhaus crows that its DBL has a “zero-false-positive reputation.” But as Haselton discovered, there is no way to guarantee a zero-false-positive rate. In fact, it might even be possible for a malicious party to force a domain onto the DBL by repeatedly inserting it into spam messages.

Afilias, Haselton’s registrar, removed all ten of the proxies once Spamhaus placed two of them on its DBL.  As a domain registrar, Afilias uses the DBL to try to shut down spammers.  When it saw two of the domains they had registered appear on the DBL, they noticed that Haselton had registered eight others at the same time and preemptively suspended all ten. Haselton was not notified; instead, he had to wade through a circuitous series of calls to three different companies before he was told that his proxies had been placed on a blacklist. Haselton then found that he was able to instantly and automatically remove his sites from the DBL by submitting a form on Spamhaus’s site. This itself gave Haselton pause because it defeats the purpose of a blacklist if all the sites on the list can be removed so easily.

Haselton’s experience demonstrates the draconian spam-prevention policies of some domain registrars. First, Afilias should have notified Haselton with the reason that his sites were to be taken down; instead, Haselton found out only when the members of his mailing-list emailed him. Furthermore, Afilias should not have automatically suspended all of Haselton’s domains. At the very least, it should have examined the content of each of the sites to see whether they were actually connected to spam operations. Finally, Afilias should have given Haselton better resources for dealing with the suspension of his domains. Haselton received no help from Afilias and had to investigate by himself how to get his sites removed from the blacklist. Afilias’s current policy towards spam-filtering casts too wide a net and seems to offer no due process to site owners.

Haselton’s experience also underscores the need for transparency in spam-filtering practices. According to Spamhaus, the suggested implementation for its blacklists “will identify and reject approximately 85% of an average mail relay’s incoming mail traffic.” That is, 85% of messages sent to a mail server will be rejected outright– the potential recipients have no way of ever accessing or seeing those messages. This is not necessarily a problem, but the potential for abusive filtering needs to be kept in check. Both email providers and blacklist maintainers should be as transparent and public in their practices as possible and should give reasonable recourse to parties who have been wrongly marked as spammers.

About the Author: Marianna Mao