Reinventing T-Mobile’s Security Function

Paris Hilton is the pink elephant in the room. For it was data from her wireless device that was hacked, and her wireless device was a T-Mobile Sidekick. A clever 21-year-old named Nicolas Jacobsen hacked the data. In fact, he had the run of T-Mobile’s servers on and off for more than a year. He took what he wanted from any of T-Mobile’s 16 million accounts, including Social Security numbers, account passwords and e-mails. (News of the Hilton hack garnered more attention than these breaches, dwarfing the fact that Jacobsen had also hacked the Sidekick of a Secret Service agent and published excerpts of sensitive Secret Service e-mails and documents.)

Jacobsen was caught in October 2004 and pled guilty four months later. (He was sentenced two months after that but the judge sealed the proceedings.) For T-Mobile, Jacobsen’s downfall was a mostly insignificant development. Because even before Jacobsen could be sentenced, a copycat hacker accessed Hilton’s account again and this time published some of her photos and data from her phone’s memo pad and address book.

T-Mobile had drawn national attention, but the worst kind, as it became the latest poster child of bad security. “T-Mobile is in the news again, with another celebrity cell phone hack,” jabbed the irreverent online IT news site The Register. The story, called “Big Company, Crap Security,” put T-Mobile’s misfortune in close proximity to another bete noire of the moment, ChoicePoint. “Combined with other high-profile leaks, T-Mobile’s internal security is not looking good,” the story said.

But all of that was more than a year ago. Now, in one room sit three of the top security executives recruited to effect change at T-Mobile by creating a new asset protection division. They are: Frank Porcaro, vice president and director of the new asset protection division; Ed Telders, director of information security, policy and compliance; and Rick Roberts, senior manager of security services. With them in the room, of course, is the pink elephant.

“If anything,” says Telders, not mentioning that celebrity’s name, “that thing helped accelerate the process, but the vision was prior to all of that stuff.”

The vision Telders speaks of is ambitious, because T-Mobile decided to put its security function through an extreme makeover. The overarching idea is focus. T-Mobile had security spread throughout its organization. Now the company wants to pull all of its security into one place, with one leader, to both reduce risks and increase efficiencies.

The asset protection group—Porcaro’s group—is the heart of the makeover. Asset protection will converge physical and information security and, at the same time, create two new groups, including an information security group and a full business continuity/disaster recovery group. In the past year alone, asset protection has grown from four employees to 18, with several of those new hires having CSO-level experience.

Meanwhile, as it’s under construction, asset protection is also being moved to another division, risk management and assurance, to be closer to related functions like audit and investigations. In the end, T-Mobile hopes to have one departmentrisk management and assurance (RM&A)through which all security functions flow.

Porcaro will know T-Mobile has succeeded when it has a fully realized asset protection group with coherent policies across the entire company, which can consistently show its bosses that security reduces risks and increases efficiencies. Porcaro puts the success of the massive effort “ideally” three years away. He says, “It’s a stretch goal, if nothing else.”

In other words, this is not a tack-a-CSO-onto-the-payroll kind of quick fix to T-Mobile’s security needs. The approach “is nice to see,” says Dave Kent, CSO of Genzyme who himself put his company’s security through a similar years-long overhaul. Kent says T-Mobile’s approach goes beyond the typical public relations-style reaction to a highly publicized breach. “What T-Mobile’s doing is a comprehensive, strategic approach. You always get acceleration of support [after] an incident, but they don’t seem to be just banking on that. That they’re going further and tying in all other ancillary functions into a truly converged operation is very impressive.”

Indeed, the plan’s ambitiousness and uncertainty are what make it worth observingso that other executive security professionals can see what real fixes look like, and how hard a full team of CSO-level executives must work to implement them. Here’s their story of the post-Paris T-Mobile asset protection division.

Before the Reinvention

Porcaro says that to understand T-Mobile’s security overhaul, one must understand T-Mobile’s itinerant history. In 1994, General Cellular and Pacific Northwest Cellular merged to form Western Wireless. Western Wireless launched VoiceStream Wireless in 1996, which gained about a million customers in five years. In 1999, VoiceStream spun off as its own company and entered what Porcaro calls the Pacman phase. It gobbled up four companiesOmnipoint, Aerial, Powertel and, later, MobileStarand also agreed to be acquired by Deutsche Telekom. DT made VoiceStream its mobile phone subsidiary and renamed it T-Mobile. By 2001, T-Mobile had 7 million customers. From there, growth continued through partnerships with companies like AOL, Borders bookstores, Kinko’s and Starbucks, and through new services for its phones like messaging, Wi-Fi, Web access and all of the other applications that have made mobile phones a growth business. Today, T-Mobile counts almost 22 million customers.

It’s the particle physics of such rapid growththe way all these companies collide and merge, fracture and fusethat explains how T-Mobile’s security arrived at a point where bad things could (and did) happen and where the need for an overhaul became starkly obvious. Companies simply can’t apply security policies or technology cohesively across so many companies coming together so quickly when all of those companies come with their own policies and infrastructure.

“The company got so large so quickly,” Porcaro says. “Internal and external audits suggested security needed improvement. And not just information security but physical security as well.” Internal politics compounded the problem, says security services manager Roberts. He says that before the overhaul (and before he arrived), the asset protection team had an “old-school mentality,” and “built barriers.” Roberts suggested that the security director took a “my way or no way” attitude to the organization and clashed with the head of the investigations group. It got so bad that the personality clash was codified into the organization, and the two groups were separated and made to report to different bosses.

Mike Morgan was an outside consultant working with T-Mobile at the time. He had designs on how to revamp security at T-Mobile. When the head of T-Mobile’s internal audit group left, Morgan stepped into the role, pulled asset protection under his purview and hired Porcaro, with his 30-plus years of experience, as director of asset protection.

Then, Porcaro says, Morgan “gave me the clay and has let me shape it ever since.”

Reinvention

In late 2004, after the notorious hacks of T-Mobile and just before Porcaro arrived, the security function was peppered throughout the company.

Asset protection was strictly a physical security function and it reported to the accounting department, below the CFO. Asset protection included a director and a four-person staff. Investigations, which used to report to the same place, instead reported to legal because of the political clashes between asset protection and investigations. Safety, which covers everything from cell tower safety to ergonomics in call centers, also reported to legal. As for information security, it wasn’t formally a function yet, just part of IT. It sounds egregious now, but at the time, during the company’s hypergrowth spurt, it wasn’t so unusual for information security to be just a few hires inside the IT department. You have to remember, Roberts says, “companies were growing so quickly then, people were just trying to get their IT to grow and work, never mind make it secure.”

In this arrangement, security was literally all over the map, with pieces under legal, accounting, the CIOand pieces missing. Such distributed security might work in mature organizations where security is an entrenched value, but it’s hard to make it work at a rapidly growing company where security hasn’t been fully developed, and where companies with different values are constantly being absorbed. Roberts says he had seen it before, when he worked at another telecom company where information security was in IT, business continuity reported to finance and “safety was out of the ballpark. The company lost cohesion and I wanted back into an environment with cohesion, because that’s how you’re effective, when you’re near each other working hand in hand,” he says.

More than anything though, when security is distributed, an organization lacks a real central focal point or leader.

Morgan’s idea was to make asset protection the security function’s much-needed focal point. It made sense to use asset protection because it was a more general security group compared with, say, investigations and audit, which have far more specific duties. Asset protection also already included the physical security function.

But focusing on asset protection meant elevating the function and bringing nonphysical security functions into the fold. Morgan’s plan would reduce risk by unifying policies and procedures, and also create efficiencies by reducing redundant efforts in different divisions. For example, why not combine access control to buildings with access control to network assets? A project like that (T-Mobile is still working on this) can work only if the physical and IT security teams are working together under the same boss.

Unifying the security front also served as a preemptive response to increasing regulatory pressures. “The [Federal Communications Commission], payment card industry, privacy [regulations], both at the federal and state level, all of this is coming at us and we need to be able to deal with it in a cohesive manner,” says Telders. Another way to say this is, if you’re going to get audited, best to be audited once in one place. Having security spread all over also increases the likelihood that audits will turn up less-than-best practices, since it’s harder to control security and apply policy when security is distributed.

With the focal point created, Morgan needed a leader. He recruited Porcaro. “The buckets were pretty well-defined when I interviewed,” he says. “Mike had a pretty clear sense of what he saw under the asset protection umbrella.” And what he saw is displayed in “Chart 2: Renaissance.”

A chart like that could make someone interviewing for the director of asset protection job flee in fear for the amount of heavy lifting that it implies is to come. Morgan may have known what he wanted, but as Chart 2 makes clear, he didn’t actually have half of itboth the business continuity management (BCM) and information security groups needed to be created from scratch. And the other half, safety and asset protection, would have to be redeployedasset protection coming from accounting and safety transferring from legaland then suffer through convergence with information security. And speaking of information security, “other than putting it in a box [on the org chart], we didn’t know how it would look or how it would take life at all,” Porcaro says. In other words, the information security department wasn’t even really an idea yet.

Morgan has compared his plan to changing all four tires on a car going 70 mph on a busy highway. But Porcaro didn’t flee; despite the quixotic overtones of the job he was applying for, he says he relished the opportunity.

There was one other absurdity: The entire asset protection function itself was moved, from finance and accounting to Morgan’s RM&A, where it would sit parallel to other security-related functions such as internal audit and fraud prevention. He was trying to create in RM&A the same gravity he wanted to create within asset protectionthink of asset protection as a planet with moons and RM&A as a solar system with other planets and moons.

Today, a year after Porcaro bought into Morgan’s four-bucket vision (Chart 2), T-Mobile’s asset protection function, in context, looks like “Chart 3: Enlightenment.”

This chart shows that the makeover is not nearly complete, but asset protection has made marked progress in a year. Bringing all these functions closer together on an organization chart also brings them closer together in the world, and Porcaro, Telders and Roberts report that the physical proximity is profoundly effective, especially in the design phase. “We’re building processes that have to have the experts from each area in the same room talking,” says Telders.

Convergence Visible

Notable, all three executives say, is how much they’ve converged physical and information security.

Roberts’ security services group, which used to be the physical security function called asset protection, now includes responsibilities for both physical and IT security operations. The business continuity management function, created out of whole cloth, also bridges physical and IT security. (BCM is cleverly divided, with a “fire inspector” continuity planning role and a “firefighter” crisis management role.) “The efficiencies you find are amazing,” Porcaro says, noting that even in areas he didn’t expect convergence to play a role, it has. For example, T-Mobile is building a 24/7 communications center for coordinating emergencies. Having IT and physical security together in the planning and designing phase has helped them see how the two will work together in the center. “Look,” Porcaro says, “in a crisisa network outage, a kidnappingit doesn’t matter, you have to pull on both physical and IT security strings.”

The efficiencies Porcaro and company can create extend beyond the obvious. “Even the RFP process is affected,” Roberts says. “The RFP for a single badge access solution is changed based on the fact we’ve converged and that single badge should now access doors and IT log-ons.”

Convergence also helps executives decide when things should not go together, says Jennie Clinton, senior manager of business continuity management. “For example, I once was at a place where they put safety and security operations under business continuity management. But those skill sets are totally different than BCM,” Clinton says. “Unless your organization is very mature, it’s not going to work, even though the bosses were saying that it was great synergy, that it looked great on paper. There are areas where the function needs to be not converged, and with all of us in the same group, you’ll hear firsthand when someone thinks convergence or overlap is a bad idea.”

“In security, you’re a bit of a one-off, a third wheel. At previous jobs, I reported to facilities, or legal. Youre there, but youre peripheral. Here at T-Mobile were clearly part of the bigger business model. I think were onto a good thing.”

Frank Porcaro, director of asset protection

And Telders’s information security function, focused on policy and compliance, also demonstrates convergence benefits. Porcaro notes that the group’s separation from the CIO and IT was important so that it could set information security policy as an IT outsider. “The goal here is to achieve an objective separation of ‘church and state,'” he says.

The progress hasn’t been lost on those closest to the asset protection function’s development. “Those who’ve been around get it. Within our team, everyone has bought into the convergence.

“But,” Porcaro says, “our challenge is enlightening the rest of the organization.”

Underscoring much of the team’s conversation, in fact, was a marked wariness. It was a successful first year, yes, but the three refuse to project that success into the future.

“We just had an offsite meeting and I threw something up on the board,” says Porcaro, “We can be where we want to be with the asset protection program they’ve wanted in three years. Trying to get into flying formation is a challenge but it’s a stretch goal if nothing else.”

“It’s doable, but I don’t want to blow smoke up anybody’s skirt,” Telders adds. “It isn’t easy.”

So far, all three men confirm that the board and top executives have shown good support. At the same time, Porcaro needs that support for at least another three years. That’s asking for a lot of patience (and a long investment) from the board and executives. “We have to demonstrate added value; it’s a big challenge for us.”

That challenge is compounded by the fact that the overarching plan is often interrupted by in-the-moment security issues. They don’t stop popping up. The speeding car getting its tires changed must negotiate potholes too. In a perfect world, Porcaro says he’d lock the team in a room for three years and come out when they are done with the project. Instead, the company continues to grow, and major unforeseen events develop. With business continuity and disaster recovery still in development, Katrina hit. Even as T-Mobile’s BlackBerry e-mail service grew, a patent infringement lawsuit threatened the very existence of Research In Motion’s BlackBerry service. (The suit was recently settled.) “So my only caveat is three years is ideal,” Porcaro says. “We’ll have to come back and revisit it.”

A Subjugation of Egos

A remarkable fact of T-Mobile’s new asset protection group is that Morgan and Porcaro were able to recruit so many CSO-level executives who were willing to report to Porcaro, a director, who reports to a vice president, who finally reports to the CFO. New hires Telders and Roberts70-plus years’ combined experienceare used to playing at the highest level of major companies. Why would they come into a place where the CFO was several steps up?

All three men say it was the entrepreneurial opportunity, the chance to build a security function from the beginning, that convinced them to join, regardless of titles or altitude on the org chart. “Sure I’d love to be high up there, but liking the job is far more important than liking the title,” says Roberts. (He also says, quoting a former Secret Service colleague, “I don’t care what you call me, just pay me right.”) “When I came out here to interview, I wasn’t impressed with the cost of real estate or living, and frankly I was thinking, it’s just an interview. But when Frank showed me what they were doing, it totally changed my mind. I thought, ‘We could do something great here.'”

“What attracted me personally,” adds Telders, “was that what Frank described was the CSO organizational model, even if we don’t use that specific title. We all share the belief that this is the right model for corporate America.”

“As I interviewed I was being recruited by another company,” Porcaro says. “Two things made a difference for me: One, the company seemed prepared to put their money where their mouth is. And two, I got very excited to be part of a bigger risk management organization.

“In security, you’re a bit of a one-off, a third wheel,” he says. “At previous jobs I reported to facilities, or legal. You’re there, but you’re peripheral. Here we’re clearly part of the bigger business model. I think we’re onto a good thing.”