• United States




How OneLogin responded to its breach and regained customer trust

Apr 16, 20199 mins
Data BreachSecurity

After its second incident in less than 12 years, incoming OneLogIn CEO Brad Brooks saw the need to instill a security-first mindset in the company and relay that message to wary customers.

8 getting breached is bad for business
Credit: stevanovicigor / Getty Images

Data breaches have become ubiquitous in today’s businesses. In a world where companies of all shapes and sizes can become cyber attack victims, how you handle a data breach becomes critically important.

Maersk and Norsk Hydro were praised for their clear, concise, and transparent messaging and response to major ransomware attacks that crippled their operations. DoublePulsar’s Kevin Beaumont labeled Norsk Hydro’s “the best incident representation response plan I’ve ever seen,” and good incident response is good for business. Norsk’s share price at the time of this writing is actually higher than it was at the time of the incident, which resulted in the company reverting to largely manual operations in parts of its manufacturing business and estimated losses in excess of $40 million in lost productivity for the first week of the attack alone.

On the other hand, Equifax’s response to its 2017 breach was labelled a “haphazard and ill-conceived dumpster fire” by Brian Krebs. The company took over a month to reveal the breach and the terms of service of the website dedicated to responding to the incident included a waiver for joining any future class action lawsuits.

Identity and access management (IAM) provider OneLogin suffered two incidents in 12 months. Despite that setback, the company has recovered and recently secured a new round of funding, thanks in part to learning from its mistakes and showing customers how it has changed its security posture since then.

OneLogin suffers a breach

OneLogin is a cloud-based IAM provider founded in 2009 and headquartered in San Francisco. Privately owned, it offers user provisioning, lifecycle management, and multifactor authentication (MFA) services to companies worldwide including Airbus, British Red Cross, Dell, NASA and Mitsubishi Electric.

On May 31, 2017, OneLogin suffered a security incident involving its Amazon Web Services (AWS) infrastructure that led to the leaking of user information and data. All customers served by its U.S. data center were affected. The attacker used one of OneLogin’s AWS keys to gain access to the company’s AWS platform through an API from an intermediate service provider in the U.S.

By creating multiple instances of the company’s infrastructure to perform reconnaissance, the attacker gained access to database tables that contained information about users, apps, and various types of keys. OneLogin staff shut down affected instances and affected AWS keys within minutes of detection, but the attack had been active for around seven hours up to that point.

Though OneLogin told customers it encrypts certain sensitive data at rest, the company warned that it “cannot rule out the possibility that the threat actor also obtained the ability to decrypt data.” As a result, the company advised customers to change their passwords, generate new API keys for their services, create new OAuth tokens for logging into accounts, and create new security certificates.

This was the second incident, occurring less than a year after the company suffered a separate breach in which an attacker was able see information stored in its Secure Notes service in cleartext. 

How OneLogIn responded to the breach

Brad Brooks stepped into the CEO role at OneLogin shortly after the incident in August 2017. Though he wasn’t officially a member of the company at the time of the attack, Brooks was advising OneLogin on how to respond in the aftermath and says the company did a good job of getting the message out there. Briefly, here’s how the company responded:

  • OneLogin published an announcement of breach the same day it was discovered, highlighting that it been discovered and subsequently blocked and that law enforcement had been notified.
  • The next day the company published an update outlining the method of attack and customer impact
  • That same day, OneLogin issued a final update outlining steps the company was taking to improve its AWS-related security.
  • With the final update, the company sent an email to all its customers that had more details and pointed them to a support page.

“You’ve got to be extremely transparent with your customers and be honest about the fact that in most cases, you don’t know really what’s happened within the first 48 to 36 hours,” says Brooks. “Your first information is probably wrong, and being clear with the customers about what you do know and what you don’t know is important.”

Whatever the state of investigation might be, he says it’s important to have an ongoing conversation with customers to give them predictability around timelines for when you think you might know more and share information as it becomes available. “That transparency and that speed of which you get it out is critical. You can’t do like what Equifax did and sit on it for three or four months. It completely destroys trust and your brand and your company.”

Once an organization starts to learn exactly what has happened, it’s important to develop a clear action plan around what you are doing and will do, when it is going to be done by, and provide names of who within the executive team is leading these actions. “Giving very specific dates and very specific names within the executive teams gives the customers a sense of not just that you know what happened and that you know how to fix it, but also that sense of accountability and that it’s actually going to get done and there’s actually going to be a change.”

“Trying to obfuscate, hide or thinking because you expose your vulnerability that it’s going to be exploited again or encourage hackers to come after your system that’s not really the case. That fear of sharing needs to be overcome to actually accelerate some of the best practices.”

Long-term impact of OneLogin’s breach

Brooks says the most tangible outcome of the breach was that the company stopped growing for two quarters as some customers jumped ship and those in the pipeline cooled their interest to see how things played out.

“The sales process froze for about a quarter. Potential customers disengaged until they actually learned more and figured out whether we were actually going to be able to survive. That delayed the growth of the company by about two quarters.”

“Was I expecting the sales downdraft? Yes. Was I expected that it was going to be as severe as it turned out to be in those first few months? No.”

Brooks explains that accountability has become one of the key principles within the company, and so for the few customers that did want to change IAM providers as a result of the breach, OneLogin made that transition to new a new vendor as easy as possible. “We hope eventually that they’ll come back to us, but we also have the sense of accountability that they were our customer. We did something that offended them.”

How OneLogIn changed post-breach

In the wake of the 2017 breach, OneLogin has adopted a security-first principle within everything the company does. “The cultural mindset was something that I knew I was going to have to change coming on board,” Brooks says. “There is no other priority that comes before security. We will make any tradeoff. We’ll make dollar tradeoffs. We will make feature prioritization tradeoffs, and they will always optimize for security first.”

One of the first things the company changed was how it approached vulnerability management within its codebase. After an initial analysis, OneLogin found 150 defects in its codebase; within two months they were all removed. After that, the decision was made that no security bug should remain in the code for longer than 48 hours after discovery, regardless of prioritization or seriousness.

In April 2018, Justin Calmus was appointed CSO at OneLogin, having previously worked at bug bounty platform Hackerone, as CIO and CSO at Zenefits, and other security roles at Salesforce and Linkedin. “We’ve redone our entire security model since that breach and since [Calmus has] come on board,” says Brooks. “He thinks like a hacker, and that offensive mindset we feel has really helped us as a company, growing our maturity level.” OneLogin has taken these steps to improve security since the breach:

  • Implemented new controls such as multifactor authentication through YubiKeys when using AWS
  • Conducted external penetration testing (both on the company’s infrastructure and on site)
  • Joined bug bounty programs
  • Hired red and blue teams to help test and improve its security posture
  • Set up a security advisory board to help direct the company’s security posture going forward.

Brooks says he encourages customers with their own pentest teams to come in and inspect the company’s codebase to look for flaws themselves, which has resulted in vulnerabilities being found and fixed. “The idea here is that it’s never done. It’s a constantly improving environment, making sure that we’re making all the investments, and that it’s not myopic. It’s not just us looking at our stuff, but that we have the entire community working with us, hacking us, pushing us to get better.”

To measure customer perception of the company, OneLogin take a Net Promoter score on a quarterly basis with our customers. “Part of the questioning goes specifically into their perceptions of us around our security and our security processes, so we’re constantly measuring that.”

Brooks says perception of the company is at “industry-high levels, and improving” despite the breach, indicating that the transparency and messaging around the company’s new approach to security is resonating. The company also recently secured $100 million in new funding led by Greenspring Associates, and its products are still well-regarded in analyst assessments.

When asked what advice he would give to those going through a similar experience at other companies, he says that if companies are willing to change, “It is not the end of the world and there is light at the end of the tunnel.”

“Security is still thought of in many circles as nice to have, but it’s a must have, and anybody that doesn’t think that way is probably on the road to an issue of their own,” Brooks continues. “Regardless of whatever past practices led to a breach or to the issues that you’re having right now, you’ve got to realize that security decisions need to take priority over all other IT decisions, because you will lose trust with the customer, and I don’t care which business you’re in.”