Incident response matters

When the social media management and sharing site Buffer was hacked over the weekend, it seemed like yet another embarrassing hack. “The incursion is no doubt a major black-eye for the upstart Buffer,” wrote David Berlind at Programmable Web. “[Buffer CEO Joel] Gascoigne has entered the dreaded damage-control zone that no start-up CEO wants to be a part of.”

[Understanding incident response: 5 tips to make IR work for you]

I think David’s post is an absolutely excellent overview of some of the realities and politics faced by developers when dealing with Twitter and Facebook (and by extension, other API providers), and I also think that his post accurately summed up the general risks faced by Buffer and Gascoigne.

I am going to disagree, though, that it was necessarily a black eye.

In this incident, Buffer showed that its concept of “radical transparency” – the concept and strategy that leads to the firm placing its revenues and other key metrics online, for all to see – has made the company look sensational.

Obviously, the reports and post-incident audit results aren’t in yet, and we could yet find out that Buffer did something really stupid. Its statement of, “We’ve increased security for how to store Twitter tokens and deployed a fix,” goes directly to my Raised Eyebrows Department. Last night I asked for a clarification from Buffer and got back (understandably, since they were hip-deep in the ca-ca) a, “We will get back to you,” Tweet.

But even if the firm did something monumentally stupid, it’s not necessarily a death-knell. People forgive even monumental stupidity if properly and genuinely apologized for.

In fact, even more serious breaches don’t always make the company lose momentum, customers or shareholders. Consider the case of TJX, which discovered in January 2007 that it had been breached for years, lost millions of customer credit card records, and was in violation of laws and industry rules. TJX had done so many things wrong in so many different areas of its IT that it is frankly amazing to me they manage to ever scrape together the brain cells required to get good deals on Roberta Gandolfi anything.

The company’s stock price took an initial hit of about a dollar a share but soon recovered, and over the next year — in the face of disclosure after embarrassing disclosure, the stock price rallied and steadily rose.

[Organizations ignore social media when it comes to business continuity planning]

TJX had managed to demonstrate two things: first, that it was reasonably working to understand and solve the problem, and second, that its customers loved it. Even though I believe that Buffer’s handling of this incident indicated that the company not only dodged a bullet, but may well emerge stronger for it. And they did it without being weasels.

I did cyber incident response full time for a few years and part time for several more, and what I saw this weekend was, for the most part, the kind of action that we always recommend and that never gets taken.For most incidents, the initial response should be some flavor of the following steps:

• Understand, as quickly as possible, that you have an incident, and communicate this to internal and external shareholders. Obviously the decision about exactly who are the stakeholders is highly variable, depending on an incredibly long list of considerations – I wouldn’t recommend everyone go public – in many cases that is exactly what not to do. But if the cat is out of the bag (that is, say, if a half-million of your customers are now advertising diet pills in their social media timelines), this decision may have been made for you.
• Understand, as quickly as possible, the initial scope of the incident (much of what you learn and assume in these early hours will be wrong, but you should work hard to get the most complete sense of what is happening and what systems are affected — you’ll be coming back to this step repeatedly).
• Once you have a scope, devise a plan to, in this order, stop the bleeding, secure what you have, and re-assess the scope and breadth of the incident.
• Develop an understanding of your available resources as mapped to the plan you’ve just made, determine the Deltas between what you have and what you need. This requires a brutally honest self-assessment, and almost certainly must be something you’ve considered in advance; you can develop this awareness after the fact, but you’re increasing exponentially the cost of the incident response — put another way, every dollar you spend doing this work in advance is worth $5 when the defecation hits the ventilation.
• Work with partners to fill the gaps between what you have and what you need. Rapidly.
• Repeat the last four steps until you feel you have positive control.
• Continue to communicate what you know, when you know it, to appropriate and approriately growing groups of stakeholders. Don’t make promises you can’t keep or statements not based on fact, but don’t shut up until you have facts if stakeholders are visibly or audibly nervous. “We have had a security incident that we understand has affected ____________, and with our staff and partners we are working quickly to determine the extent of the damage and we will report back regularly with progress,” is much better than not saying anything and allowing speculation to fester.

So how did Buffer do against these concepts? Pretty fantastically well.Soon after they knew they were hacked, Buffer management and staff took to Twitter and Facebook announcing the problem. Joel Gascoigne wrote a blog post entitled, “Buffer has been hacked – here is what’s going on”, in which he said in part,

“I wanted to post a quick update and apologize for the awful experience we’ve caused many of you on your weekend. Buffer was hacked around 2 hours ago, and many of you may have experienced spam posts sent from you via Buffer. I can only understand how angry and disappointed you must be right now.

Not everyone who has signed up for Buffer has been affected, but you may want to check on your accounts. We’re working hard to fix this problem right now and we’re expecting to have everything back to normal shortly.”

Gascoigne promised to update users on Facebook and Twitter. Updates were then made every two hours or so until the firm felt it had control of the situation.

He also added something personal that didn’t backpedal in the slightest:

“I am incredibly sorry this has happened and affected you and your company. We’re working around the clock right now to get this resolved and we’ll continue to post updates on Facebook and Twitter.”

[Business continuity and disaster recovery planning: The basics]

There are remaining questions. What happened? Why? How? How do we know it’s fixed? What steps will they take in future to assure it won’t happen again, and who is ensuring they’re doing it right? Who’s auditing the findings? How does Buffer know that the breach did not include user data? Credit cards? Personal information?

If Buffer continues to be as transparent about what happened as it has been about how much money it makes and its breach, I predict it will be much stronger than ever by this time next year.

Incident response matters.

Nick Selby is a government and law enforcement technology analyst, a Texas police officer, and CEO of StreetCred Software, Inc (a 2013 Code for America Accelerator company). He is also the co-author of Blackhatonomics: An Inside Look at the Economics of Cyber Crime, and technical editor of Investigating Internet Crimes: An Introduction to Solving Crimes in Cyber Space.