How to Filter Email with Finesse

How do you keep legitimate email messages from getting swept up by false positives?

By Simson Garfinkel

December 01, 2005 — CSO — With somewhere between 80 percent and 95 percent of all Internet messages now consisting of spam, phishing attacks and e-mail based worms, organizations have been forced to filter their incoming mail more aggressively than ever before.

E-mail filtering systems are faced with the Herculean task of separating out the bad mail from the goodâ¬and doing it fast enough so that the mail doesn't back up. Computer scientists call this a "recognition task" and say that e-mail filtering is one of the most challenging of such tasks that's ever been devised. The job just keeps getting harder and harder every day, as the bad guys adapt their messages to make them more closely resemble legitimate mail. The cost of letting through a bad message can be high: A single phishing message, successfully passed to one of your users, can compromise your internal systems. Even ordinary spam annoys your users and can cause employees to miss important messages. Spam must be stopped.

But as a CSO, you have another job as well: You need to make sure that your organization's e-mail filtering system isn't filtering out the wheat with the chaff. That's because the cost of a legitimate mail message gone missing can be equally high. More and more, I hear of business opportunities that fall through because an unexpected e-mail message was delivered to a junk mail box or silently dropped. The senders of these messages thought that the recipients weren't interested, but in fact the messages simply never arrived. Legitimate mail must not be stopped. So even as CSOs filter aggressively, the trick is to find the right combination of tools and techniques to keep the real messages from getting swept into the spam box. While many options are available today, a closer examination leads me to think that ultimately digital signatures will prove necessary if we are going to keep spam from turning e-mail into a nonviable communication medium.

Rejection Rates

There are several technical metrics that a CSO can use when trying to evaluate mail-filtering systems. Two of the most common are the false-negative and false-positive rates. A false negative might be a piece of spam that the system lets through that it shouldn't; a false positive is a message that is blocked, even though it is legitimate.

Unfortunately, there aren't a lot of reliable sources for these metrics. In part, that's because the metrics are different for every user of a given filtering system. A user who has posted her e-mail address to a popular online discussion will receive a lot of spam, and probably a greater variety of spam, than a user who has been circumspect with his online identity.

• Print

• Image Spam: By the Numbers
• The Scourge of Image Spam