Battling bots on the front lines

    We’ve all heard about the threats and been briefed on the exponential growth of botnets, but last week we learned about this problem in an “up close and personal” way in Michigan.

 First, the good news: no sensitive information was lost. Our alert security team detected this bot/worm (initially called – W32.Spybot.worm and later W32.Rinbot.b) in about 15 minutes after initial infection. We blocked the addresses that infected systems tried to connect to very quickly, and contained the spread. We used all the tools we put in place over the past two years to their fullest, and without SurfControl, LanCope, forensic tools such as Encase, our IDS and IPS from ISS, and other tools, we would have been in much worse shape.  

Now the bad news, networks, servers, and customers were impacted. It took us about 48 hours to fully recover enterprise-wide and fix all affected servers and workstations. Although most customers were never impacted nor government offices closed, some networks did slow down with high traffic levels.

 Fortunately, we got a cleaning tool from Symantec – or we would have been rebuilding boxes.  The scary part on this one was that the signatures from Symantec only came out two hours before we were hit. The previous night’s signatures didn’t stop this bug. This was a true “zero day” experience, but at least we had signatures to stop the spread.    

Yes, we made the local papers. A bit more coverage in the Michigan technology news.   

We activated our Technology coordination center and our Public Information Officer (PIO) got the situation communicated quickly. All of that went fairly well, but it’s still hard to have a incident of this magnitude and not feel as if somehow we could have done better.

How can we make lemonade out of this lemon? Now the lessons learned:

1)      Finish the upgrade of old equipment. Finally, get rid of NT and Windows 2000 workstations and left-over servers. Yes, I’ve been trying to get this message out over the past several years, but now it happened – this bug hit the older systems hardest. The message to upgrade is now stronger than ever to invest in new OS – despite all the reasons that old applications can’t run on XP, yada, yada, yada. We now have a real cost in not upgrading for the business side. We made the case before, but now the “or else” has happened. Our CIO made this case with the legislature last Thursday morning in a Senate technology briefing. Good news: we’re accelerating our plan for replacements now.

2)      This infection came from a remote user connecting to us. This adds priority to our end point security project which we have already started. We need to get more protection out on the network edges faster. Again, nothing new here, but a great time to add emphasis and speed to our existing message and security projects.

3)      Partner with MS-ISAC, US-CERT, vendors, and others. We can’t go it alone. Partnerships are vital and work. We received invaluable information from our partners during this situation. Without their knowledge, we would have been in a much worse situation. We have also shared info with other states, who have had similar issues recently, and they have shared with us.

What messages did our internal customers (technical and business side) take away?  We’re not done with worms yet.  Security isn’t complete. We still need those security tools and skills and new equipment. The internet is as dangerous as ever. We also received more “thank you” messages and “well done” emails in the past few days than we have in the past year. (Kinda like the firemen were more appreciated after 9/11.)

I’ll wrap this up by saying that last week wasn’t fun, but it did reinforce our tactical and strategic security plans and priorities.  We’re doing the right things, but probably not fast enough. This was the worst hit we’ve taken since August 2005, and the entire enterprise took notice. Still, we also built better bridges with our internal partners – our exercises and planning really helped.

All in all – we did pretty well. Our security and infrastructure teams are stronger now than we were last week at this time. I guess Michigan is ready for “March Madness.” But while I hope U of M and MSU make the college basketball tournament, I hope we don’t experience another major round of bot problems anytime soon.