Monday
When the CISO asks to speak to you with that look on his face, you know the news isn't good. We were contacted by one of our third-party vendors, whom we had hired to do analysis on our website traffic.
It appears that we have been passing sensitive information to them over the Internet. This sensitive information included data, such as customer names, addresses and credit card information. Because we are a public company, there are many regulatory guidelines that we have to follow like Sarbanes-Oxley (SOX) and the Payment Card Industry's (PCI) data security standard.
Fortunately for us, our vendor has retained a copy of everything that we have sent to them.
Unfortunately for us, it was six months of information totaling over a terabyte.
Since our website is international, the legal department needed to obtain outside council to assist us in this matter. It will be a few days until I receive the data from the vendor.
Thursday
We have received the data from our vendor and my preliminary analysis is not good. It appears that we were sending the vendor every form field of every page on our website.
After speaking with the product team, it appears that the generator of the data is a piece of third-party code, which was supplied to us by the vendor to whom we were sending the data.
The first question that I asked was if this code was reviewed, which I was promptly told, "Yes!"
The code was reviewed before its initial installation almost a year ago. Even though the code had been in our staging and production environments for almost a year, we have only been sending the vendor sensitive information for the last six months.
I asked if the code had changed at all in that time, and I was told "most likely." The product team was going to talk to development to get me a list of all changes to the code.
The data is massive and there are over a billion records that need to be investigated. I am working on writing a small data-mining program to piece it all together.
Legal wants me to give them a list of every single person that is affected along with their location. In the meantime, they are investigating the privacy laws of every single state in the U.S. as well as several other countries that they suspect may be contained within the data.
After telling legal that it would take me six weeks to gather the information they required, I was told that I needed to move faster.
It seems that some of the privacy laws require notification within a certain period of time after the discovery of the incident. I told them I simply don't have the computing power to give them what they need any quicker. I was authorized to purchase several machines to aid in the data-mining effort.
Friday
My lab machines have arrived and I have been provided with a private workspace in which to work. I spent almost the entire day splitting up the data, and I am preparing to run my data-mining program over the weekend.
I have guessed that each machine will need about 16 hours of processing in order to complete. I will have to monitor the results over the weekend to make sure that everything completes on time. Other than getting the machines to work, I have been in many meetings with the legal department where the terms "data breach" and "customer notification" have been thrown around.
I immediately started to think about all of the recent news regarding companies and data breaches. I know I didn't want my company to be added to that list.
Monday
I met with the legal department this morning to give them a progress update. There were roughly 10 million entries in the data that contained customers and their credit card information, with six million being unique.
I have created a breakdown of all of the data based on state and country, and it seems that we may have to look at privacy laws in almost a dozen countries.
The product team got back to me and there were over ten changes to the third-party code since it was first put in place.
Unfortunately, they didn't get around to doing a code review on any version after the original.
The only real piece of good news that they gave me was that all connections to our vendor were done over SSL. At least this data did not go over the Internet in plain text. I will give this one piece of good news to the legal department at our meeting later today.
Three Weeks Later
Even though my work has been done for several weeks, the legal department continued to deliberate on whether or not to report this as a data breach to the customers that were affected.
As it turns out, the vendor who received the data had relatively good procedures in place and not many people had access to our data.
We were able to account for everyone who may have accessed the data -- and because the legal department feels that the data never left our control, they decided that this did not constitute a data breach. An outside forensics firm confirmed the data never left a controlled environment.
We may have dodged a huge bullet on this incident, but the required legal council, forensics and the time everyone spent working on it cost the company over $1 million.
Conclusion
Where did we go wrong here and how could we have prevented this?
Both of these were questions that were asked many times during the investigation. Of course it is easy to say that in the future we will never run third-party code on our website, but how realistic is that?
Large enterprises run third-party code every day in the form of open-source software, and we are no exception.
One major way we could have prevented this incident was to have a consistently followed SDLC process in place.
Code review is a major piece of any SDLC process, as is output validation. Someone should have been working with the vendor every time there was a software change to make sure that they were seeing the appropriate data and nothing more.
Our issue should have been easy to spot since we were sending much more than the required information.
It's amazing to me that such a small issue could cost a company so much money.
I shudder to think about how much this incident would have cost if we had to report it publicly.
The author is an information security manager for a company based in the Chicago area.