Sevice Level Agreements: Tying the Knot

Richard Diamond is fully aware of the irony: Of all the 400 or so users on his company's nationwide network, he was the one who fatefully clicked open an e-mail attachment from a contact outside the company. And Diamond is the CIO.

"The moment it happened, I realized what I'd donebut it was too late," says Diamond, senior vice president and CIO of The Doctors Co. The physician-owned medical malpractice insurance company was infected by the Nimda virus, which busily began sending itself to everyone in Diamond's extensive companywide e-mail address book. Eradicating Nimda from The Doctors Co. took almost three days.

That unforgettable three days led Diamond's company to settle on an increasingly common solution: outsourcing its network security. The Doctors Co. contracted Symantec to keep watch over the network on a 24/7 basis, guarding against not just viruses but the full gamut of IT security threats. "On security issues, we felt we were always playing catch-up," says Diamond. No longer. "Suddenly, we find we're in the vanguard. It's a little surprising, but it seemed to be a very prudent thing to do."

There are many reasons for bringing in a managed security services provider. Some hope to lower costs. Some simply hope to hand someone else the trouble of finding and hiring information security expertise. And many CSOs say they sleep better at night with such a service in place. But the contracts that govern outsourced security relationships are tricky beasts. Most of them center around a service-level agreement (SLA), which spells out minimum performance standards that the provider must reach. In all too many cases, it's tempting to rely on the apparently tough-sounding SLA proffered by the vendor as the basis for managing and monitoring this new relationship. But that could be a mistake for a number of reasons, chief among them the fact that many of the measurements provided may prove hard to contest or simply irrelevant for what the CSO's business really needs. And negotiating a different agreement isn't as easy as it sounds.

All things considered, CSOs should scrutinize their service contracts carefully before letting a standard SLA lull them into a false sense of security.What's in an SLAIn a business such as managed security, most CSOs and CIOs figure that measuring a vendor's effectiveness shouldn't call for rocket science. Diamond, for his part, is very clear about the relationship with Symantec: "We know what it's going to cost us, and we know how we're going to measure its effectiveness," he says.

Perhaps. Poke a little deeper, though, and this seemingly straightforward view quickly becomes murkier. How is a vendor performing? Look no further than the plethora of reports that it issues to its customers. Different vendors provide different reports, but most are variations on a similar theme. How quickly were threats detected and resolved? What kind of threats were they? Were they random or deliberately targeted? Which parts of the system were under attack? Is the trend in attacks of a particular nature rising, holding or falling?

At face value, such metrics appear very sensible, and they, of course, are the measures lovingly enshrined in the standard SLA. The tricky part is determining whose purpose the metrics in service-level agreements really serve.

Fess up: Certainly CSOs and their security organization have a vested interested in them. An armful of freshly delivered statistics always comes in handy when you need to justify jobs and budgets: "We've not been hacked latelybut look what might have happened!"

As it turns out, the same logic that applies to you persuading your executive team equally applies when your vendor is selling you. That's one reason why managed security services providers appear to accept metric-laden SLAs with almost open arms. "We want our customers to see exactly how we're doing in protecting them," enthuses Pete Privateer, vice president of protection services for Atlanta-based Internet Security Systems. His customers are thus provided with a portal, protected by both password and token, that contains up-to-the-minute information about the company's performance in meeting the specified SLA standards. "If they want, customers can drill down through the data to see information on the specific threats that are pertinent to themdata on the incidence of port scans or distributed denial-of-service attacks over the last month, for example," Privateer says. "Or even firewall logs, if they want to go into that level of detail."

Again, this tidal wave of metrics contains a generous dollop of self-interest. They also indisputably serve the handy double-purpose of persuading customers that they're getting a good dealpossibly to the extent of encouraging them to upgrade to the next level of service. (Internet Security Systems, for example, offers four levels of service as standard: basic, silver, gold and platinum.)The Greater ProblemBut even taking the motive in providing (or receiving) the metrics at face value, a bigger question remains: Which metrics really matter? The answer, it seems, depends on the business.

Daniel Piggott, group IT manager with Benson Group, a British construction company, was perfectly happy to sign up to the standard service-level agreement offered by U.S.-based Via Net Works. Via provides most of the company's telecommunications and Internet access, he explains, and opting to sign up to Via's managed security services provision offered economies of scale. When structuring the contract, though, Piggott went along with the service-level agreement that Via proposed. "They said, 'Here's our standard terms,' and we felt we could live with it," he says. "You have to be reasonable. We're not a financial services company; we're a construction company and didn't feel that we couldn't survive if a security issue meant an hour's outage."

Some businesses take a different view. Multinational petrochemicals giant BP, for example, is cautious about an overreliance on simple metrics. "Both hard and soft measures are important," argues Paul Dorey, director of global security for BP. "The usual service-level metricsspeed of response, events logged and managed, and so onform the basis of regular performance meetings, but we really look for a good relationship with knowledgeable security people. We're asking them to be part of our extended team, and in security it's more important to face up to any problems and deal with them than it is to decide whose fault it was," he says.

In other words, the metric-driven approach may simply boil down to counting the number of times that the horse has bolted through the open stable door. The obvious question: Might it not be better to close it first?

That is certainly a view strongly put forward by Raleigh, N.C.-based Al Decker, director of outsourcing giant EDS's security and privacy services division. "There's a perception that managed security equates to managed intrusion detection and a managed firewall," he notes. Metrics, like technologies, need to be tied to a firm business justification. "If [a particular measurement] doesn't serve a business need, you need to query why it's there," he says.

According to Decker, managed security should really start by sitting down with your provider and analyzing the network architecture for worm holes. "If you take too narrow a focus, there's a risk that you'll leave an opening for an attack," he warns. Likewise, time spent up-front figuring out the policies and procedures that should be in place is usually a good investment. In the event of an attack, he notes, "The policy is what should guide the action that is takenand if [your actions and those of the service provider] are not in concert, then there's a chance that you may be missing the mark."

Another strike against service-level agreements comes from Bob Ayers, an information security veteran who rounded out a career in U.S. Army Intelligence and the Defense Intelligence Agency with a period as director in charge of the Department of Defense Information Systems Security Program, establishing the first Department of Defense emergency response team. Curiously, Ayers, who these days is based in London as director of business risk services at security consultancy @Stake of Cambridge, Mass., gripes that SLAs don't contain enough metrics. Or at least enough of the metrics that really matter.

It's a reflection, he says, of the imbalance that exists between managed security services providers and their customers when it comes to constructing SLAs. "Typically, you're doing it for probably the first time, while the supplier has done it many times over," he says. "The supplier uses words that make what they are going to do for you sound grand and glorious, but there's no way you can use those words to prove that they aren't doing a good job."Better Language, PleaseLook no further than the sort of phraseology used to describe the supplier's obligations regarding software updates and antivirus patches. "Remember," Ayers says, "that a prime cause of hacks is poor software maintenance and late application of antivirus software. And what do we find? Phrases like, 'The supplier will install and maintain an intrusion detection system and keep it current.'"

A much better way of describing that critical obligation, he says, would be to pin down much more precisely what has to be done. So instead of the previous vague phraseology, Ayers prefers words like these: "The supplier will install an intrusion detection system approved jointly by the supplier and the client, and will apply all vendor product updates within 30 minutes of them becoming available."

It's just an example, but Ayers is resolute on the need to comb through SLAs looking forand excisingwooliness. He's also in favor of building into the contract a stipulation that the client will periodically attack their own systems in order to assess the capability of the managed security services provider to detect and respond to those attacks. "It's my experience that most companies fail to make such stipulations within their contracts," he observes. But how practical are such tough-sounding words? Even excepting the periodic targeting of a company's systems by its own personnel, some people have reservations about linking security issues to such tightly written metrics.

For public sector CSOs such as Jeff Ritter, director of IT for the division of employment and training for the commonwealth of Massachusetts, there's a legal hurdle to crossone that the private sector may not need to face. "Public sector contracts are worded very generally, and security is a very specific issue," says Ritter, who serves on the commonwealth's Enterprise Security Board. "A general contract at law can't possibly address the specifics of an engagement of this nature, in terms of particular releases and updates." Public sector contracts tend to be "blanket" contracts, he explainsgeneral in nature, lasting over time, and covering a basket of goods and services.

Massachusetts' own contract with managed security services provider Genuity, for example, calls only "for the vendor to make best efforts to provide the most up-to-date version," he says. Even so, Ritter can see the advantages of a tighter approach: "There's no real reason why such stipulations couldn't be in place, provided that the lawyers understood both the need and the technicalities to phrase a sensible contract," he says.

Managed security services providers aren't too sure, thoughand not just because they're objecting in principle to something that attempts to pin them down. "The speed with which patches and upgrades are updated is easy to talk about but much harder to do in practice," observes Patrick Cain, security advocate in the CTO's office at Genuity. "Patches can break what you've already got or just not work with it."

And in any case, he adds, there's nearly always more than one way to skin a cat. With many known threats, for example, it's perfectly possible to program the firewall to look for particular data packets and filter out the threat that waywithout running the risk of breaking anything until the stability of a patch or upgrade is well-understood.

In short, if such apparently simple issues can't be readily decided one way or another, it's difficult for any chief security officer to know if the deal he gets from his managed security services provider is a good one or not.

The mist is clearingbut slowly. Amit Yoran, vice president of global managed security services at Symantec (and another former DoD CERT alumnus) concedes that customer pressure is forcing change. "Users are getting more sophisticated in their RFIs and RFPs, and are getting to better understand the various value propositions on offer," he says.

For his part, Massachusetts' Ritter points to draft initiatives developed by the Massachusetts Information Technology Division's Cyber Law E-Government Advisory Roundtable with respect to website and software development. If there's a way forward, it might be there, he believes. With page after page of legalese leavened with healthy dollops of good business sense, they're not documents for the fainthearted. And nor, yet, do they deal with managed security services. But as a modelwell, yes, here's a bunch of lawyers with some sensible-sounding things to say about IT procurement.

Absent such progress, the business of managing your relationship with a managed security services provider will remain like nailing Jell-O to a wall. In which case, as the Romans used to say: caveat emptorlet the buyer beware.