Govt’s $2.7 million KILL IT WITH FIRE approach to malware: Destroy all hardware

Finding malware on your own system or your company’s networked computers is never a pleasant experience. IT admins might grumble, or cuss under their breath about stupid users, but few would totally freak out about a virus. Yet the U.S. Economic Development Administration (EDA‘s) IT security department at the Department of Commerce decided upon the “kill it with fire” approach. This comedy of errors might be understandable if the malware was a maliciously crafted nation-state breed, but it wasn’t. But hey, why let common sense rule the day when you can instead choose to destroy over $170,000 in hardware like desktop PCs, cameras, printers, keyboards, and even mice? More evil hardware would have been destroyed, but the bureau ran out of money.

The EDA’s mission is “to lead the federal economic development agenda by promoting innovation and competitiveness, thus preparing American regions for growth and success in the worldwide economy.” Maybe it’s related to the agency’s “disaster relief” work, or maybe replacing mass amounts of hardware was its unique way to stimulate the economy. Regardless, the agency definitely took an innovatively incompetent approach, dumping over $2.7 million—almost half of its IT budget—to fight common malware.

It started when DHS’s US-CERT detected a “potential malware infection” before determining the “infected computers resided with IT systems operating on the Herbert C. Hoover Building (HCHB) network.” According to the Office of Inspector General audit, both EDA and the National Oceanic and Atmospheric Administration (NOAA) were told about a potential infection in their IT systems. NOAA’s security team quickly identified and remediated the malware infection.

“By contrast,” the report [pdf] states, EDA believed it had a “widespread malware infection” propagating within its systems and isolated its IT systems from HCHB network. “This action resulted in the termination of EDA’s operational capabilities for enterprise e-mail and Web site access, and regional office access to database applications and information residing on servers connected to the HCHB network.” After calling in an outside cybersecurity contractor, EDA was told that “preliminary analysis found indications of extremely persistent malware and suspicious activity.” Later, the contractor said that was erroneous, caused by false positives, “not actual malware infections.”

But EDA’s CIO and senior leadership were freaking out, believing it was a nation-state cyberattack, concerned that “the presence of extremely persistent malware that would prohibit typical containment measures, such as reimaging infected components for immediate use.” The report states, “External incident responders were unable to provide the assurance EDA’s CIO sought, because doing so involved proving that an infection could not exist rather than that one did not exist.”

EDA’s CIO concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity (which did not exist) was great enough to necessitate the physical destruction of all of EDA’s IT components. EDA’s management agreed with this risk assessment and EDA initially destroyed more than $170,000 worth of its IT components, including desktops, printers, TVs, cameras, computer mice, and keyboards. By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million. EDA intended to resume this activity once funds were available. However, the destruction of IT components was clearly unnecessary because only common malware was present on EDA’s IT systems.

EDA believed the malware attack infected more than half of their systems, 146 out of 250 components. The reality is that there were only two infected computers, according to the audit. But we the taxpayers coughed up $2,747,000, broken down as $823,000 to the outside cybersecurity contractor’s “investigation of malware infection and data cleaning;” $1,061,000 to procure “temporary infrastructure, pending long-term IT solution;” $688,000 spent on “contractor assistance for a long-term recovery solution.” Then there was an additional $4,300 paid out to destroy $170,500 in non-evilly infected IT equipment.

This all started in December 2011, but EDA managed to drag it out for nearly year. All the blame does not sit on EDA’s shoulders. In fact, the report states:

DHS’s draft report stated, “over 143 systems infected with common fake anti-virus” and “50 percent of EDA’s network is infected,” which portrayed a widespread malware infection. The NSA report stated that “the EDA network was extremely inundated with malware” and “the extent of the compromise and the state of the overly infected network will make it very difficult to deconflict the vast amount of indicators.” NSA did not independently verify incident information, but it presented similar information to that presented by DHS as fact. As a result, EDA believed these incident reports supported its conclusion regarding the extent of the malware infection.

The misunderstanding went undetected by EDA until December 18, 2012–and by the Department until December 19, 2012–when OIG completed its validation of events and informed both organizations of its initial conclusions.

“Despite NSA and DHS recovery recommendations,” eight months after isolation, EDA wanted to build a new-and-improved IT infrastructure. “EDA estimated it would need over $26 million disbursed in the next 3 years (an increase from $3.6 million to approximately $8.83 million, or about 2.5 times more, to the bureau’s average annual IT budget) to fund its recovery efforts.”

The audit identified three key issues with EDA’s cyber incident and recovery. 1) EDA based its critical cyber-incident response decisions on inaccurate information. 2) Deficiencies in the department’s incident response program impeded EDA’s incident response. 3) Misdirected efforts hindered EDA’s IT system recovery.

That seems to put what happened at the EDA more kindly than some of us would. You’ve most likely heard the saying, “Don’t throw out the baby with the bath water,” but the EDA did exactly that. In fact, it seems like the agency threw out the dirty water, the baby, washcloth, soap, shampoo, crib, and all the baby’s toys and clothes. Better/worse yet, we the people funded this government comedy/tragedy.

Like this? Here’s more posts:

• You might be a terrorist if…you complain about your tap water
• Breaking down latest leaked PRISM slides claiming U.S. ‘bugged EU offices’
• Surveillance court ‘secret’ rulings slaughter Fourth Amendment to help NSA spy
• NSA whistleblower Snowden: Even innocent Americans are ‘being watched and recorded’
• Happy Independence Day: Stop Watching Us, Restore the Fourth Amendment
• Hackers can wipe or steal data from security holes in 300,000 servers
• Hacking and attacking automated homes
• Former CIA, NSA director sounds off on PRISM, spying tools
• Rule of 7 applied to domestic surveillance
• Microsoft Research: MoodScope, a context-aware smartphone to sense and share your mood

Follow me on Twitter @PrivacyFanatic