Strategies to Manage the Spam Menace: Part 1

By Michael O’Neil

and David Senf, IDC

The IT industry press is often dominated for limited periods of time by “urgent” issues that quietly fade into the background as new urgencies emerge. Unlike most phenomena in the IT industry, however, the broad preoccupation with spam shows no signs of abating any time soon. The flood of messages prompted by viruses and worms like “sobig” (which is reported to have accounted for 73 percent of all message traffic at its peak), combined with a seemingly-inexhaustible supply of entreaties for reshaping/extending one’s body parts, mortgages, access to “video entertainment”, and so forth, has caused millions of users to view email in a different, less flattering light – and prompted numerous organizations to consider the real cost of unwanted, intrusive message traffic.

In order to help readers to manage the burgeoning spam menace, IDC has put together a two-part series on spam management. In this article, we look at the seven primary methods used to filter spam, while in part two, we look at methods of measuring the economic impact of spam – and the benefit of spam filtering technology – within your organization.

Approaches to Spam Management

Anyone who has been driven to investigate spam filtering technologies has no doubt encountered a wide array of possible solutions, each touting high intercept rates, and low/no instances of “false positives” – messages that are intercepted, but which should have been allowed to reach the user. This latter category is a particular headache, since users who have missed a critical email will often be outspoken in their criticism of the spam management technology. However, IS managers need to balance exposure to these criticisms with the knowledge that each spam message that reaches an end user’s desk causes some lost productivity – and can have detrimental impacts beyond the handful of seconds needed to direct the message to the trash can.

With that in mind, we have outlined the basic approaches used in today’s spam filtering systems, along with a brief description of each. The seven primary methods of managing spam are the following:

• White Lists and Black Lists
• Contextual Analysis
• Challenge and Response
• Honeypots
• Header Analysis
• Content Analysis
• Heuristics

Each of these approaches has benefits and drawbacks; as a result, many systems and corporate strategies combine multiple methods. The key attributes of each are described below.

White Lists/Black Lists: This is the simplest approach to filtering. This method involves regularly updating lists of approved (white) and disapproved (black) domain names (or user addresses). White lists reduce false positives and prevent company mail from being classified as spam, while black lists are useful for addressing mail abuse. However, this approach is relatively labor-intensive, and is becoming less appealing as spammers become more sophisticated in their ability to generate email from new domains.

Header Analysis: Most anti-spam products examine headers, looking for such items as the validity of the sender’s address, whether the same information is found in the “sender” and “from” fields of an email, and whether a specific message contains information not common to “normal” email.

Content Analysis/Keyword Searching/Text Analysis: This approach involves analyzing the text section of an email for character sequences that have already been classified as spam – often, targeting specific keywords and phrases (e.g., sex, profanities, Nigeria, Viagra, etc.) that are unlikely to appear in legitimate business correspondence.

Contextual Analysis: Contextual analysis searches scan messages for linguistic patterns that indicate whether the message is an advertisement or part of a normal conversation. These solutions can be temperamental and take time to configure and tune properly, but some vendors indicate a very high block rates (as high as 98 percent) with few or no false positives. Such solutions are becoming more common. An example of this type of solution is IM Message Inspector from Elron Software.

Heuristics: Heuristic, or self-learning, techniques are commonly applied to spam filtering. Heuristic filters sift through e-mail messages for the characteristics and behaviors that are unique to spam messages, and – as they “learn” about new approaches – get better with experience. One heuristic approach is sieve filtering. Once a spam message is identified, the antispam vendor uses an algorithm to calculate a unique string of bits, or “signature,” for the spam message (including information buried in the e-mail message header that is invisible to most e-mail recipients, such as the path the e-mail took to reach its destination); the filter uses that signature to scan new incoming messages. Another heuristic filtering approach is Bayesian analysis, in which large volumes of spam and an equal amount of legitimate e-mail undergo sophisticated statistical analysis. A comparison of the results creates a baseline threshold against which newly arriving messages are judged; proponents of this approach claim extremely high (up to 99 percent-plus) success rates for the approach. Many spam filtering programs use heuristic programming, including ActiveState’s PureMessage, Mirapoint’s MessageDirector, Lyris MailShield Server and the open source program SpamAssasin.

Challenge and Response: The challenge and response method does not deliver mail messages to an end-user mailbox until the sender is validated. This is done by having the system respond to the sender with an e-mail. To gain acceptance to a challenge and response system, the sender’s “reply-to” address has to be “live” and the sender has to actually check the account, and verify his or her “person-hood” by looking at a picture, and typing in some information from the picture. These systems function on the basis that a computer system cannot (yet) decipher simple pictures. Proponents of this approach maintain that users will tolerate the extra steps in order to be spared the inconvenience of spam; however, it must be remembered that the onus falls on the authentic sender to confirm their existence. Hotmail, AOL, and Yahoo all use challenge and response tricks to prevent spammers from signing up for accounts in the first place; spam filter vendors MailFrontier and Mailblocks also employ this approach.

Honeypots: Honeypots use decoy email accounts designed to attract spam. One vendor (Brightmail) has a spam attack analysis center staffed 24 hours a day by email experts. When a new spam attack is launched, Brightmail picks it up through its hundreds of thousands of email addresses placed at strategic domains across the Internet. All messages that land in the decoy e-mail accounts are considered by Brightmail to be spam. The company uses the network of accounts to detect developing spam attacks and to create filter rules that its customers can use to block the spam from their own messaging servers.

It is clear from this analysis that IS managers have a wide range of tools at their disposal; however, recent history also shows that there is a large and growing volume – and variety – of spam attacking corporate messaging systems. Understanding how to effectively manage this invasive traffic is an important first step to deploying a system that has the greatest benefit to your organization. In our next installment, we’ll provide insight into how to measure the economic impact of spam, and some case study results regarding the cost of spam, and the economic benefit of filtering technology.

For further information please contact Michael O’Neil, 416-673-2234 or moneil@idc.com