Home » Security Leadership » Exec. Communication

In Depth

Ouch! Security Pros' Worst Mistakes

We've all done regrettable things on the job, but does any valuable wisdom come of it? Four security pros candidly explain their biggest blunders and what they learned in the process

By Bill Brenner, Senior Editor

Page 4

"My first two thoughts were how would we ever clean all this up and where was our data. As we started the cleanup process, I had other thoughts: How do we secure our printed records while the clean-up crew is here? How will we document and destroy all these ruined records? And still, where is our data? We had water, sogginess and mildew to contend with so the cleanup process was much more involved than I could imagine. Security was a priority for us, and the whole team was on board to ensure everything was handled properly. We successfully sequestered sensitive paperwork in a locked facility, waited for it to dry, and then had it destroyed.

"We were lucky. Our primary server room remained unscathed, our servers, backups and main networking equipment was all intact (and dry). As you can imagine, after the clean-up the what-ifs started flying through our heads. What if the server room had been destroyed? In our case, the previous incarnation of our backup procedures would have saved us. Our critical data was indeed secured at an off-site location, but in its current state, it would have made maintaining business continuity a much slower process."

THE LESSON
"The most important lesson is this: Never displace your organization's business priorities with day-to-day 'emergencies.' You never know when something incredibly unexpected will occur."

3. THE TERRIBLE TYPO

Mistake maker: Andrew Cardwell
Position: Computing Security Officer, Director at Cardwell Security Ltd.
Location: United Kingdom
The incident: Mistyped serial number causing Internet domains to stagnate for a week

"I worked for an ISP that at the time was responsible for controlling the .org.uk domains. We where essentially the authority for any domains under that TLD [top-level domain]. This was around 1995, when domain names where all manually applied for, approved, updated and controlled.

"I had to update the main registry file and insert a new name and update the serial number which controlled the updates on the DNS server. The serial number was in the form of YYYYMMDDXX. XX represented the number of changes that day so in order to get it updated we had to do new XX = old XX +1. Sadly, I removed one of the digits so the serial number turned into YYYYMMDDX. As a result, the name server did not pull in the new file and update the .org.uk domains for a week until we discovered it on closer inspection -- and after several complaints."

THE LESSON
"This is an ideal example of lack of controls around the software, lack of a sanity check and human error. Over the years in places that now run the TLDs, controls have been introduced to ensure this kind of human error is sanity-checked through logical rules. I added something so the serial number should go up not down to help eliminate or reduce the number of human errors."