The U.S. Department of Commerce's Economic Development Administration (EDA) destroyed about $170,000 worth of IT equipment including computers, printers, keyboards and computer mice last year on the mistaken belief that the systems were irreparably compromised by malware.
The bureau was poised to destroy an additional $3 million worth of IT equipment but was prevented from doing so by a lack of funding for the effort, a report released by the Commerce Department's Inspector General says.
The EDA's startling overreaction to an imagined threat to its networks appears to have stemmed from an almost comical series of miscommunications between computer security incident handlers at the Department of Commerce and at the EDA.
The problems started with an alert issued by the Department of Homeland Security (DHS) in Dec. 2011, warning the Commerce Department of a potential malware infection within its networks. Security administrators at the Commerce Department identified the potentially infected computers as belonging to the EDA and alerted the bureau of the compromise.
However, the Department's initial notification to the EDA incorrectly listed a total of 146 systems as being potentially infected, when in fact just two of them were actually infected.
A day later, the computer incident response team at the Commerce Department sent a second e-mailed incident notification to the EDA containing new analysis that identified only two systems as being infected with malware. However, the second notification was vague and did not clearly call out the fact that the first alert had been inaccurate, according to the Inspector General's report.
Instead, the second alert actually began by stating that the first notification had in fact been accurate and made no mention of any mistake in the previously provided information. Subsequently, incident handlers at the EDA assumed that the second notification was merely a confirmation of the analysis in the first alert and proceeded to assume that a major portion of their network had been compromised.
Over the next several weeks, incident response teams at the Commerce Department and EDA continued to work with a completely different understanding of the scope of the problem. The incident response team at Commerce assumed that their counterparts at the EDA had read and understood that the second notification superseded the initial incorrect alert while the EDA continued laboring under the belief that 146 of its systems had been compromised.
The EDA's impressions of a widespread compromise appeared to be confirmed when a forensic analysis of two systems showed them to be infected with malware. So, when the Commerce Department eventually asked the EDA to reimage its systems in order to get rid of the malware, the EDA responded by saying that there were too many systems involved for such reimaging to be feasible.
Rather than follow-up with the EDA to see what was going on, incident handlers at the Commerce Department wrongly assumed that the EDA had done an independent analysis of its systems and had identified many more systems that had been compromised.
"Unfortunately, both organizations continued to propagate the inaccurate information ... during the incident response activities," the IG's report noted.
In January 2012, EDA's CIO, Chuck Benjamin, decided to isolate the bureau's systems from the network on the mistaken belief that the infection was rampant and could spread to other networks. The CIO's decision to disconnect the system from the network also stemmed from, what turned out to be unfounded, fears that nation-state actors were behind the network infections.
A timeline of events provided in the IG's report does not indicate when exactly the EDA began destroying its IT systems in its effort to contain the imagined network infection. It does note however that Benjamin "concluded that the risk, or potential risk, of extremely persistent malware and nation-state activity [which did not exist] was great enough to necessitate the physical destruction of all of EDA's IT components," the report said.
"By August 1, 2012, EDA had exhausted funds for this effort and therefore halted the destruction of its remaining IT components, valued at over $3 million," the IG said.
The report slammed Benjamin and incident responders at both the Department of Commerce and the EDA for the snafu. It faulted the Commerce Department's incident response team for sending the initial incorrect notification, not properly documenting its communications, putting an inexperienced incident responder in charge of communicating with the EDA and then for not coordinating a proper response with the bureau.
The IG blamed Benjamin for not putting enough effort to properly validate the scope and seriousness of the reported infection before embarking on a needless and costly recovery effort. Even after an external security contractor hired by the EDA had identified only minor, easily remediated malware infections on the bureau's systems, Benjamin proceeded with his drastic recovery measures.
"In the end, nothing identified on EDA's components posed a significant risk to EDA's operations," the report noted. "Despite only finding common malware infections, EDA's management and CIO remained convinced that there could be extremely persistent malware somewhere in EDA's IT systems."
In total, the EDA spent $2.7 million -- or half its FY 2012 IT budget -- responding to the non-existent threat to its network. Despite fairly straightforward recovery recommendations from the National Security Agency and the DHS, the EDA focused on building out a new and improved IT infrastructure instead.
After disconnecting its systems last January, the EDA signed up for a shared service from the U.S. Census Bureau to maintain a Web presence and for email services. Last March, the bureau issued new laptops to all users and April set-up a standalone implementation of its core business applications.
In September 2012, the bureau submitted a request to the Commerce Department's IT Review Board for $26 million over the next three year to fund its recovery efforts, the IG's report noted. The request was denied.
In February this year, the Commerce Department's IT began a full-scale recovery effort and restored EDA's full operational capabilities in five weeks.
In response to the IG's findings, Matt Erskine, the deputy assistant secretary of Commerce for Economic Development noted that EDA had acted "out of an abundance of caution" throughout the incident. The response noted that the EDA had also continued to conduct and complete important work despite the disruptions.
In a separate repose, Simon Szykman, the CIO for the Department of Commerce noted that the department has launched a comprehensive incident response improvement project. As part of that effort, the Department has already completed a third-party assessment of its incident response capabilities, hired three experienced incident handlers and put a new security incident tracking system in place, Szykman noted.
Jaikumar Vijayan covers data security and privacy issues, financial services security and e-voting for Computerworld. Follow Jaikumar on Twitter at @jaivijayan or subscribe to Jaikumar's RSS feed. His e-mail address is jvijayan@computerworld.com.
See more by Jaikumar Vijayan on Computerworld.com.
Read more about government it in Computerworld's Government IT Topic Center.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.