How OneLogin responded to its breach and regained customer trust

By creating multiple instances of the company’s infrastructure to perform reconnaissance, the attacker gained access to database tables that contained information about users, apps, and various types of keys. OneLogin staff shut down affected instances and affected AWS keys within minutes of detection, but the attack had been active for around seven hours up to that point.

Though OneLogin told customers it encrypts certain sensitive data at rest, the company warned that it “cannot rule out the possibility that the threat actor also obtained the ability to decrypt data.” As a result, the company advised customers to change their passwords, generate new API keys for their services, create new OAuth tokens for logging into accounts, and create new security certificates.

This was the second incident, occurring less than a year after the company suffered a separate breach in which an attacker was able see information stored in its Secure Notes service in cleartext. 

How OneLogIn responded to the breach

Brad Brooks stepped into the CEO role at OneLogin shortly after the incident in August 2017. Though he wasn’t officially a member of the company at the time of the attack, Brooks was advising OneLogin on how to respond in the aftermath and says the company did a good job of getting the message out there. Briefly, here’s how the company responded:

• OneLogin published an announcement of breach the same day it was discovered, highlighting that it been discovered and subsequently blocked and that law enforcement had been notified.
• The next day the company published an update outlining the method of attack and customer impact
• That same day, OneLogin issued a final update outlining steps the company was taking to improve its AWS-related security.
• With the final update, the company sent an email to all its customers that had more details and pointed them to a support page.

“You’ve got to be extremely transparent with your customers and be honest about the fact that in most cases, you don’t know really what’s happened within the first 48 to 36 hours,” says Brooks. “Your first information is probably wrong, and being clear with the customers about what you do know and what you don’t know is important.”

Whatever the state of investigation might be, he says it’s important to have an ongoing conversation with customers to give them predictability around timelines for when you think you might know more and share information as it becomes available. “That transparency and that speed of which you get it out is critical. You can’t do like what Equifax did and sit on it for three or four months. It completely destroys trust and your brand and your company.”

Once an organization starts to learn exactly what has happened, it’s important to develop a clear action plan around what you are doing and will do, when it is going to be done by, and provide names of who within the executive team is leading these actions. “Giving very specific dates and very specific names within the executive teams gives the customers a sense of not just that you know what happened and that you know how to fix it, but also that sense of accountability and that it’s actually going to get done and there’s actually going to be a change.”

“Trying to obfuscate, hide or thinking because you expose your vulnerability that it’s going to be exploited again or encourage hackers to come after your system that’s not really the case. That fear of sharing needs to be overcome to actually accelerate some of the best practices.”

Long-term impact of OneLogin’s breach

Brooks says the most tangible outcome of the breach was that the company stopped growing for two quarters as some customers jumped ship and those in the pipeline cooled their interest to see how things played out.

“The sales process froze for about a quarter. Potential customers disengaged until they actually learned more and figured out whether we were actually going to be able to survive. That delayed the growth of the company by about two quarters.”

“Was I expecting the sales downdraft? Yes. Was I expected that it was going to be as severe as it turned out to be in those first few months? No.”

Brooks explains that accountability has become one of the key principles within the company, and so for the few customers that did want to change IAM providers as a result of the breach, OneLogin made that transition to new a new vendor as easy as possible. “We hope eventually that they’ll come back to us, but we also have the sense of accountability that they were our customer. We did something that offended them.”

How OneLogIn changed post-breach

In the wake of the 2017 breach, OneLogin has adopted a security-first principle within everything the company does. “The cultural mindset was something that I knew I was going to have to change coming on board,” Brooks says. “There is no other priority that comes before security. We will make any tradeoff. We’ll make dollar tradeoffs. We will make feature prioritization tradeoffs, and they will always optimize for security first.”

One of the first things the company changed was how it approached vulnerability management within its codebase. After an initial analysis, OneLogin found 150 defects in its codebase; within two months they were all removed. After that, the decision was made that no security bug should remain in the code for longer than 48 hours after discovery, regardless of prioritization or seriousness.

In April 2018, Justin Calmus was appointed CSO at OneLogin, having previously worked at bug bounty platform Hackerone, as CIO and CSO at Zenefits, and other security roles at Salesforce and Linkedin. “We’ve redone our entire security model since that breach and since [Calmus has] come on board,” says Brooks. “He thinks like a hacker, and that offensive mindset we feel has really helped us as a company, growing our maturity level.” OneLogin has taken these steps to improve security since the breach:

• Implemented new controls such as multifactor authentication through YubiKeys when using AWS
• Conducted external penetration testing (both on the company’s infrastructure and on site)
• Joined bug bounty programs
• Hired red and blue teams to help test and improve its security posture
• Set up a security advisory board to help direct the company’s security posture going forward.

Brooks says he encourages customers with their own pentest teams to come in and inspect the company’s codebase to look for flaws themselves, which has resulted in vulnerabilities being found and fixed. “The idea here is that it’s never done. It’s a constantly improving environment, making sure that we’re making all the investments, and that it’s not myopic. It’s not just us looking at our stuff, but that we have the entire community working with us, hacking us, pushing us to get better.”

To measure customer perception of the company, OneLogin take a Net Promoter score on a quarterly basis with our customers. “Part of the questioning goes specifically into their perceptions of us around our security and our security processes, so we’re constantly measuring that.”

Brooks says perception of the company is at “industry-high levels, and improving” despite the breach, indicating that the transparency and messaging around the company’s new approach to security is resonating. The company also recently secured $100 million in new funding led by Greenspring Associates, and its products are still well-regarded in analyst assessments.

When asked what advice he would give to those going through a similar experience at other companies, he says that if companies are willing to change, “It is not the end of the world and there is light at the end of the tunnel.”

“Security is still thought of in many circles as nice to have, but it’s a must have, and anybody that doesn’t think that way is probably on the road to an issue of their own,” Brooks continues. “Regardless of whatever past practices led to a breach or to the issues that you’re having right now, you’ve got to realize that security decisions need to take priority over all other IT decisions, because you will lose trust with the customer, and I don’t care which business you’re in.”