Pipe Cleaners: Telcos Offer Managed Security Services

From AT&T's Global Network Operations Center 40 miles west of New York City, CISO Ed Amoroso has as wide a window into the Internet as anyone. With a glance at a two-story wall covered with computer monitors and television screens, Amoroso can tell at any given moment how much e-mail, Web and voice-over-IP traffic is streaming across AT&T's data networks, buzzing its way from business to business, person to person. The amount of Internet traffic represented in the room is staggering. On the average business day, almost 10 petabytes of data pass through AT&T's networks--more information than the entire Web contained in 2000.

Too bad that almost all of it is garbage.

More than 80 percent of the e-mail coming in to AT&T is spam. About 1 million of the home computers AT&T sees each day are thought to be infected with bots, reaching out to hundreds of other IP addresses far more quickly than any Internet surfer with DSL or a cable modem ever would. Before a worm strikes, technicians see strange spikes of traffic going to normally obscure ports, as malware developers test and tweak their code. A sudden, sharp increase in the amount of Web traffic worldwide could mean breaking news--or a distributed denial-of-service (DDoS) attack being lobbed at a single company halfway around the world.

But Amoroso's window into a rapidly junkifying Internet is largely just that: a window. For the most part, he says, all he can do is sit and watch through the glass, as unwanted or malicious traffic makes its way from point A to point B.

"The standard service-level agreement is that we just push the traffic in and out," he says. "We don't touch it. We can do some upstream and downstream filtering if we see something that will affect our infrastructure, but you getting a spam, or you having some weird protocol aiming at you--I would love to filter that, but it's not that simple."

That's because a telecommunications company's job has always been to pass traffic, not pass judgment. "The starting point [for Internet carriers] is no responsibility whatsoever," says Jonathan Zittrain, professor of Internet Governance and Regulation at Oxford University. "Echoing the original spirit of Internet protocol design, the job of a router is simply to move a packet one hop closer to its destination."

This is the reason for the intense debate over whether to forgo so-called net neutrality, in which Internet carriers treat all packets the same. Even as carriers argue that they should be allowed to prioritize high-revenue content, however, AT&T has been quietly getting permission from its customers to stop certain kinds of traffic altogether. Already, some businesses have signed up to have AT&T filter out spam, viruses, DoS attacks and other malicious activity behind the scenes, before the traffic touches their enterprises. AT&T is now working on the "productization" of similar services for its home customers. In Amoroso's vision of the future, telecom companies will routinely deliver not the diseased m?lange of today's pure Internet, but a "clean pipe" of good (or at least decent) traffic. Less junk, fewer risks. Here's your bill.

It's a necessary gambit for an ocean-ship of a company ($63 billion) in an industry that faces new competition and downward pricing pressure, the result of the excess telecommunications capacity laid during the late 1990s and early 2000s. "The carriers are looking for ways to differentiate themselves so they're not just competing on who's got the cheapest bits per second, and they're also looking for ways to stop the decline in dollars per bits per second," says John Pescatore, a vice president at the IT research firm Gartner.

According to Pescatore and other observers, AT&T is farthest along in the journey of telecom companies to position themselves as security providers--although competitor Verizon took a huge leap forward in May, when it announced that it was acquiring Cybertrust, one of the country's biggest names in information security, for an undisclosed amount. Verizon said the acquisition would add 800 employees to its 300-person information security team, along with expertise in computer forensics and identity management and a solid presence in Asia.

The growing security ambitions of telecom companies could have a profound impact on how "security" is packaged and sold--by standalone security companies or by network or IT providers; to CSOs as standalone services or to CIOs within a bundle of other services; as products or in a software-as-a-service model. What's more, the outcome of what AT&T is attempting could influence the very future of the Internet as a free and unfettered, if increasingly dangerous, communications platform. The question is whether the strategy will pay off--whether AT&T's vast customer base really wants to pay extra for an Internet as safe, banal and micromanaged as a shopping mall.

Bruce Schneier, whose own security company was purchased last year by the United Kingdom's largest telecom carrier, BT, says that right now, it's not the telecom industry's role to stop bad traffic. But if a telecom company can make it profitable to do so, that role will change. And fast.

"They'll do it if it makes them money," says Schneier, chief technology officer of BT Counterpane. Until then, he believes, Internet carriers have little incentive to clean up the Internet. Why should they bother? "Bandwidth is cheap."

Telcos as Security Companies

The idea of a telecommunications company acting as a security provider is nothing new. For years, telephone companies and Internet service providers have used their existing relationships with businesses to spread security services onto the network connectivity that's their bread and butter. Gene McLean, CSO of Telus, the

$7 billion Canadian telecom company, says security services have always been part of his company's offerings; they've just never really been marketed. "When we're dealing with big clients or government contracts, then we put on our security consulting hat," McLean says simply. "We look at it as a differentiator."

Typically, telecom companies sell virtual private networks (VPNs) and take over such rote tasks as managing firewalls, intrusion detection systems or other customer premises equipment. This frees up business customers to keep lean staffs or focus on more strategic operations. While standalone managed security service providers (MSSPs) have the same capabilities and, arguably, deeper security expertise, telecom companies have one gigantic advantage: They are already on the payroll.

"Telcos don't find themselves in the position of having to market the way that pure-plays do," says Loren Rudd, an industry analyst at Frost & Sullivan, a research and consulting company. "The pure-plays have to evangelize on almost every sale that they make. It's easy for the enterprises who are migrating to managed security services to call up their telco [and add a feature] like you were adding a cable channel to your TV."

That's basically what Dan Antion of American Nuclear Insurers did when he chose to outsource security to AT&T, which had been his phone and Internet vendor for seven years. "It just seemed to make sense," says Antion, VP of information services at the Glastonbury, Conn.-based underwriter for the nuclear power industry. "We'd been through a lot of projects with them."

Given the middling need for marketing, it shouldn't be surprising that few people noticed when the telecom companies overtook most of the pure-plays in terms of market share for security services. According to Frost & Sullivan, three of the eight largest MSSPs in North America are telecom companies: AT&T, Sprint and Verizon. Three more are IT services companies--Getronics, IBM and VeriSign--that have gotten big in the MSSP space mostly by eating up smaller pure-plays (most notably IBM's purchase of Internet Security Systems earlier this year). Until recently, Cybertrust and Symantec were the last two large MSSPs with an information security focus. Symantec, though, is positioning itself more as a purveyor of "infrastructure software," and the pending purchase by Verizon of Cybertrust further narrows the field. (See chart on Page 33 for details.)

The market is still fragmented, though, with plenty of room for competition. Frost & Sullivan estimates that these large companies combined have only 40 percent of the MSSP market--a market it expects to grow about 20 percent a year through 2010. "I personally think that, if implemented correctly, telcos are a good match for the managed security market," Rudd says. "The growth trajectory of MSSPs has proven itself in recent years. It's not speculative for a telco to get involved."

Telecom companies have reach and resources in their favor, of course. But "it's not just economies of scale" that give them an advantage, Gartner's Pescatore says. "It's that the carriers have access to information that the individual enterprise doesn't."

That's the information that AT&T CSO Amoroso sees through his window on the Internet in northern New Jersey. And that's the information that he's hoping to use to move AT&T's security business from one focused on simply managing customers' security equipment, to one that's truly cleaning up the pipes and plumbing of the Internet. "It's like the blind men and the elephant," Amoroso says, referencing the folktale of the blind men who each, upon feeling a different part of an elephant, draw vastly different conclusions about the creature before them. "When you sit as one node on the network, you don't have context. The service provider sits right smack in the middle of the context and has a vantage point that nobody else can have." His favorite example is that AT&T security analysts knew about the 2003 Slammer worm before it hit, because of strange traffic going to port 1434.

"I've looked at this traffic," Amoroso continues, "and realized that there's just a gold mine of security information."

Virtual Security

The centerpiece of AT&T's strategy to build security into the network--dubbed "in-the-cloud" security services--is a concept that's gotten increasing attention over the past couple of years. Right now, as CSOs are all too aware, most companies purchase and manage (or outsource the management of) a slew of security devices, from antivirus software to firewalls to intrusion detection and prevention systems. With an in-the-cloud setup, however, many of these tasks can be handled using a virtual device administered by an MSSP. It's basically a software-as-a-service model, with monthly service fees replacing product, installation and maintenance costs. Gartner projects that as early as 2008, 30 percent of managed security service revenue could come from services delivered in the cloud.

Telecom companies aren't the only ones pushing for this model. Antispam companies such as MessageLabs and Postini have adopted it, as have pure-play MSSPs such as Perimeter eSecurity and VigilantMinds (which recently merged with another MSSP, Solutionary). "Think of us like the water utility," says Brad Miller, CEO of Perimeter

eSecurity, a $24 million, venture-capital-backed company in Milford, Conn., that used to call itself Perimeter Internetworking. "You could have one big water utility that cleans the water, or every house could have its own water filter. Which way is more efficient?" Obviously, Miller thinks the former.

A security company like Perimeter eSecurity has to either partner with telecom companies (which it does), or convince direct customers to route all their Internet traffic first to Perimeter and then back to their enterprise (which it also does). Telecom companies, on the other hand, need only to get permission from existing customers to filter the traffic that they're already handling anyway. Rather than evaluating a brand-new contract, the CIO, and perhaps CSO, are just looking at making changes to a service-level agreement and pricing for bandwidth.

Although not everything can be handled at the network level, AT&T currently offers several services in the cloud. First, there's the network-based firewall, which can be accessed and configured through a Web portal and eliminates the need for a perimeter-based firewall. Second, there's defense against DoS attacks. With this setup, when a customer's Web traffic reaches a certain threshold, AT&T diverts the traffic to scrubbers that filter out the bad traffic and direct the good to the company's website. Third, there's e-mail security, where AT&T uses third-party software to filter out viruses and spam--typically at least 80 percent of a company's inbound e-mail traffic. A similar Web security service screens incoming Web and instant-message traffic for malware. Finally, a family of services called Internet Protect notifies customers of unusual Internet activity--the junk on the screens at AT&T's network operations center--and makes recommendations. For instance, if technicians see early indications of a new worm, they may suggest that a customer temporarily block traffic to the affected port. Right now, most of AT&T's security customers still favor handling things the old-fashioned way, by turning over the management of what's known in industry lingo as customer premises equipment (CPE), such as firewalls. One customer CSO spoke with didn't even seem aware that AT&T is cheerleading the in-the-cloud model, and AT&T says that only about 10 percent of its devices are handled in the cloud. But that's changing.

For instance, the company says that the number of virtual firewalls it manages has been growing at a compounded rate of

65 percent to 75 percent annually over the past three years and has already passed the halfway point. "The shift is starting to happen pretty rapidly," says Stan Quintana, vice president of AT&T Security Services. He projects that five years from now, the ratio of in-the-cloud devices to CPE will almost have flipped, with a full 80 percent of services handled virtually.

Even before the announcement that it would acquire Cybertrust, competitor Verizon was saying that its managed security service offerings were growing at a fast clip of about 67 percent a year, with two in-the-cloud services similar to AT&T's offerings proving to be especially popular. While the Cybertrust acquisition doesn't add to Verizon's in-the-cloud offerings, a spokesperson says, it might give the company more options for adding cloud-based functions later on.

One of those already successful services is e-mail filtering, in which inbound e-mail is scrubbed by four antivirus engines and spam is deleted through a partnership with e-mail security company MessageLabs before being passed on to the customer. "It's the same service [Verizon customers] could get on their own," Verizon Business CISO Sara Santarelli says, "but they're only interacting with Verizon's customer service."

The second, faster-growing in-the-cloud service at Verizon is DoS protection, in which Web traffic is filtered for spikes of malicious activity. "Things like DoS mitigation and detection are far exceeding industry growth expectations across MSSPs," Santarelli says. "A lot of customers keep traffic running through our [DoS attack] mitigation units all the time, just as added insurance."

None of which should be much of a surprise, given that companies such as Gartner suggest that customers demand DoS protection from their connectivity provider. "That's been our recommendation," Pescatore says. "Whoever you choose for your bandwidth, tell them, 'I don't want the raw bandwidth costs. Give me your price for DoS-protected bandwidth, and I'll compare you with others on that basis--not just on who sells me the cheapest bits per second.'"

Both AT&T and Verizon declined to provide any specifics about revenue for their security operations, but Pescatore estimates that right now, telecom companies are getting about 10 percent to 20 percent additional revenue by adding security filtering to connectivity charges. The question is how long that will last. "At some point," Pescatore predicts, "one of them is going to say, 'Hey, we'll give you that DoS protection for free if you switch from them to us.'"

Indeed, much of the industry's shift to security services seems more about staying competitive than about making buckets of money. "It's not a great portion of our revenue, but it's strategic to our overall revenue," Quintana explains. "When customers are evaluating AT&T versus vendor A, B or C, our security portfolio acts as a differentiator to pull through" the sale.

But Will It Work?

Longer term, however, it remains unclear whether customers will really decide in droves to turn over their security to telecom companies--or to anyone. For one thing, not everything can happen in the cloud. Even if an Internet carrier scans incoming e-mails for viruses, for instance, the company still needs a desktop application to guard against malicious code introduced by USB drives or other portable devices. What's more, the Fortune 1000 customers that large telecom and IT companies have historically courted are likely to have contracts with multiple telecom companies for reasons of redundancy, and also tend to want security devices onsite that they can configure on a moment's notice. The outsourcing model may be better suited to small and midsize businesses that can't afford to hire round-the-clock security and IT staff--and even they may be reluctant to give up their boxes and blinking lights and move to a virtual model.

At Visions Federal Credit Union, VP and CIO Tom Hull decided to turn over 24/7 security monitoring to Perimeter eSecurity, but still keep the company's own firewalls. "I think there is a hard sell there," says Hull (whose Endicott, N.Y.-based company has just 400 employees and annual sales of

$80 million) of the in-the-cloud model. "We still retain their help in managing the firewalls, but we didn't want to rely on the schedule of a third party to institute any changes in our environment. Plus, as it relates to any outages, downtime, system maintenance or things of that matter, that was another thing we could not relinquish control of."

In London, AT&T customer Martin Joy also decided against AT&T's virtual devices. "I'm not keen to see a device on my premises. The important thing is to make sure the technology makes sense and delivers what we want," says Joy, CIO of Control Risks, a $219 million risk consultancy. Nevertheless, he felt that his business needs were best met by turning over management of firewalls and other devices to AT&T, while keeping his antispam function handled in the cloud by a separate e-mail security company. For him, it was a question of one-stop shopping versus what he perceived as best-of-breed.

On a broader scale, it's unclear whether home consumers will ever want to sign up for a "clean" Internet. AT&T is testing how it could roll out a version of its corporate security offerings to home customers, but already executives have concluded that even its target audience--parents of school-aged children--might not be content with just a Disneyfied version of going online. "Maybe Dad wants to do online gambling but keep teens away from it," Amoroso says. "We're just trying to create something people will like and that matches what people want to do." That will likely involve different versions of the Internet, perhaps delivered to homes based on who's at the computer--a far cry from really cleaning up the junk in the pipes of the Internet.

For now, and maybe for the long run, companies like AT&T will have to continue to make careful decisions about what traffic they can safely delete without violating their service-level agreements with customers or overstepping their bounds as common carriers that just pass bits from left to right. Amoroso says that AT&T can and does delete malicious traffic that will affect its infrastructure. It also deletes e-mail traffic coming from known blacklists of spammers and blocks port 25 on its DSL lines unless a customer requests otherwise. (Amoroso estimates that 75 percent of spam comes from compromised home PCs, usually on port 25, which is not the port that a typical DSL subscriber uses for outbound e-mail.) But for the most part, AT&T can do so only on behalf of a customer--not on behalf of the Internet at large.

"I don't think there's a single carrier that would do that, only because that's pretty presumptuous," Amoroso says. "If there was some general council in Geneva, some tribunal that decided all carriers must do the following, it would be easy enough to do. But I don't think that's a role that the carrier has been asked to do or would be comfortable doing."

Even deleting the most egregious traffic can raise issues. Amoroso says there have been cases where AT&T terminated a portion of an agreement with a customer who was on the blacklist of spammers--in other words, a customer whose every outgoing e-mail AT&T would normally delete.

Understandably, AT&T wants to distance its security operations from the net neutrality controversy as much as possible. After one interview with CSO, a public relations professional called to emphasize that cleaning up traffic for security reasons is entirely different from segmenting different types of traffic into high-speed lanes. But the fact remains that both activities involve value judgments about which traffic deserves to go where and when. And that further complicates Amoroso's lofty version of the "cleaner" Internet of AT&T's future.

"Filtering out traffic makes the carriers less neutral, no doubt about it," Oxford University's Zittrain says. And the more the carriers do so, he predicts, the more difficult it may become. "They are holding back not because of some ideological principle like a belief in net neutrality," Zittrain continues, "but because they see no reason to get into a customer-service nightmare of quarantining their compromised subscribers and then helping them to fix their machines."

Technically speaking, Internet carriers such as AT&T, looking out at their charts of DoS attacks and spam and unfolding worm- and bot-related activity, may indeed be in the best position to fix the Internet. But actually doing so, outside the prescribed version of the Internet that businesses want to make available, simply may not be a task that they are in a position to accomplish.

"People want simplicity, but they also want flexibility," says Andrew Odlyzko, director of the Digital Technology Center at the University of Minnesota, who worked in research at AT&T Bell Labs for 26 years. "That's the conflict. If the telecom environment were stable and predictable, then the smart Ma Bell network"--in which Internet users are carried from one clean and safe place to another--"would make a lot of sense. People don't want to worry about the complexity of spyware, viruses, corrupt files. But they want new services, like YouTube. So you have this tension. It's there, and it will continue to be there.

"I don't expect that AT&T or any other carrier can provide a foolproof solution to computer insecurity," Odlyzko continues. But he won't go so far as to say that telecom companies are just wasting their time, either. "I think they can do some [of the solution] and make money at it too."