Why BT’s red team strikes for real

While it was a Prussian general who said it first, Mike Tyson said it best: “Everyone has a plan until they get punched in the mouth.” Incident simulations, war games and tabletop exercises can go a long way to prepare the business and security teams for the worst, but there’s nothing like a trial by fire.

As part of its proactive approach to defense, UK-based BT allows its red teams to attack live systems without informing the rest of the business or the blue team defending it. These live exercises test the real-world abilities of both teams and inform the company’s defenses going forward.

BT’s Cytadel: Defense in-depth informed by risk

BT’s roots go back to the 1846 founding of the Electric Telegraph Company, and today operates multiple business units across the world. The company offers consumer services such as fixed-line, broadband, mobile services and television subscriptions and offers IT consulting and services for commercial customers.

Les Anderson, BT’s Group CISO, joined the company in 2014 from GCHQ and is keen that security makes an effort to understand and connect with the business to protect it without blocking the goals of business. “Enabling the business to succeed safely is the ethos,” says Anderson. “The business is here to provide service and features to consumers, which then generates profit. That’s why we can afford to have a security organization in the first place.”

BT’s defense-in-depth strategy internally is called Cytadel. Anderson explains that the strategy is based on the idea that traditional citadels are surrounded by walls, ditches, high areas, ground surveillance and moats. If a single element fails, levels of protection are in place. “What we’ve got here is defense-in-depth, as opposed to just a very high brick wall. If that one single brick wall fails, then everything falls, so this is about frustrating, and denying, and delaying bad things happening.”

Anderson is responsible for ensuring protections are in place for the physical estate, logical IT estate, and the people estate across the entire company. When he joined BT, the company had a traditional compliance cybersecurity regime. He was tasked with driving change and adopting a data justified, risk-based approach. “There’s many companies I think don’t do risk categorization. They have oodles of policy that they don’t link to those risks, and then nobody pays any attention to the policies,” he says. “They’re often unimplementable as well as so they’re ignored. Unless a policy is justified by a risk, burn the policy, because otherwise, what is it doing?”

“We consider holistically all that could happen to a service, a feature, a system. BT Sport, for example, it’s a great consumer service to offer,” says Anderson. “Imagine, then, if you want a badge of honor and try to take down a Premier League match. So, we’ve had to consider the risks of DDoS, for example, and we’ve invested heavily in anti-DDoS capability to ensure that that can’t happen.”

One example of risk informing security is turning the BT Smart Hub into an aggregating gateway to help prevent botnets of insecure home IoT devices from using BT pipes to launch denial-of-service attacks. “In an ideal world, things are secure by design, which is difficult to do in a very aggressive, disruptive technology marketplace,” Anderson says. “One option is to say you can’t attach to my network. Another is aggregating and making sure that the Smart Hub can adequately trust and authenticate to these devices or that it can put bubbles of protection around them in case the appliances themselves might be badly designed.”

Part of the idea of Cytadel is constant iteration–learning and adapting to new developments as quickly as possible. “I don’t like step function transformations. To me three- to five-year programs really mean that’s not going to happen. The shape of my team today isn’t going to be the shape of my team in 18 months’ time.”

“I’m trying to predict with a two-year forward window what the environments, actors and techniques are going to be,” he says, “and I will adapt the team, and our appliances, methods and defense-in-depth approach, to hold back that tsunami wave.”

As part of that prediction effort, BT runs a large botnet to attract the baddies to, as Anderson puts it, “show their leg” around what they are researching to weaponize and deploy against BT and other businesses.

Preparing for the worst and testing it

Whenever an incident happens – whether real or the company’s own testing — BT establishes what’s called a security threat analysis group. In those groups, the security team and others come together to perform root cause analysis and post-incident reviews to see what learnings they can take forward. Anderson and his team also spend a lot of time working on “black swan” events – the near-unthinkable but potentially cataclysmic events that could strike the business – with key stakeholders within the business, up to and including the CEO Philip Jansen.

These simulations include mock meetings, press conferences and press releases. Practicing for these near-unthinkable events, says Anderson, allows the business to be prepared for potential events and aids their understanding of the fallout from major incidents. It also informs how the red and blue teams approach offensive and defensive actions and formulate playbooks and responses going forward.

“You can’t stand there scratching your heads,” Anderson says. “You have to have done the playbooks. You have to practice. You have to have the governance in place to respond at the speed of an incident. We can only get better by learning from that which has gone before.”

A core part of BT’s approach to security is what Anderson describes as proactive discovery and remediation around penetration testing. BT has been performing red teaming “aggressively” against BT’s systems for over four years, testing the full spectrum of IT, physical and human testing. Even for a company of BT’s size and scale, his red team is large at around 100 people (another 400 are in the security department under Anderson, with 2,500 more across the wider BT Security division).

“I inherited what I would call classical penetration testers,” Anderson says. “It’s probably the largest in our sector and BT spent a lot of money training them and turning them into a force that’s capable of being a nation-state team.”

Anderson says they conduct around four to six red team operations per year, depending on the size and scope of his brief to them, as well as giving the red team opportunities to explore one area more in depth if they discover something potentially interesting.

Rather than narrow vulnerability testing and reporting, Anderson’s red team tests BT’s live systems for flaws and the blue team’s reactions, up to and including during live broadcasts of major sporting events. At the same time, there are ‘Chinese walls’ between the red and blue teams, meaning the blue team are fighting against the red team as they would any other threat actor.

“The red team are not practicing. They’re not simulating. They’re doing,” he says. “If the red team finds something alarming, I don’t stop them. They carry on and will create an incident because, in a sense, that’s what we’ve got. The blue team doesn’t know what the red team is doing and so as far as the blue team is concerned, they’ll just handle that as part of our reactive incident handling.”

Anderson says this blind red versus blue approach is an amplifying process as it helps drive the capabilities of both. Successful red team attacks help inform defenses, playbooks and the like. Successful defending by the blue team forces the red team to change up their tactics, techniques and procedures to find new ways in.

He points out that this is done safely and protocols are in place to ensure operational services aren’t inadvertently brought down. As to the risk of attacking your own live systems, he says it’s better for BT to find own flaws than for threat actors to find them. “If a hacker is able to get to a certain point and take out a live service, isn’t it better for us to have done it and then we can roll back far quicker than somebody else where we haven’t got a clue who they are, where they are, or how they got there?”

Anderson doesn’t tell the other parts of the business when these attacks are going on, and when asked if he faces resistance from parts of the business that might be uncomfortable about that, he says getting buy-in and sign-off from the CEO is absolutely necessary. “I can’t have any part of BT believing it can veto the security of the whole group. That’s why the CEO has been very key to support this concept and this method and give me the headroom I need safely.”