A Contract Killing: The Drama of Government IT Work

Our anonymous CSO's tawdry tale of an IT services contract rollicks through software piracy, dope sales and worse. Who says government work is dull?

The government doesn't get its fair credit for drama. To wit, I was the security lead on an IT services contract with a government agency, on a job that went from routine to rollicking. A little background: This contract required training in a wide variety of areas—mischarging, sexual harassment, security and privacy, conflict of interest, and others that focused on confidentiality, integrity and ethics. Complete participation was required per the contract. Failure to do so could lead to termination. Emphasis on the "could."

After a mini St. Valentine's Day Massacre—a letter received on Valentine's Day alerting our company that our ratings were in the low 70s, meaning no contract bonus and the certainty that heads would roll—a new Program Manager was brought in. He was a fixer, brought in to correct the course of this contract. We soon found the fixer was really a "rule by intimidation and ridicule" type of leader. His job was to improve the periodic ratings in order to secure millions in bonus dollars. His compensation hinged upon this. Dollars were awarded based upon the ratings derived from specific measurements as per the contract.

Our job was peripheral to this bigger contract drama—until we actively scanned for vulnerabilities and found an anomaly we could not verify. Like an arsonist calling in the fire, a tech lead pointed us toward a couple of IP addresses that we could not scan. We traced the IP numbers to their physical location and found two servers located in an office. Per the requirements of the contract, we began to gather information off the two servers. What we found on those servers was quite exciting—and extremely disturbing:

- W2K3 running on both;

- Eval copies with cracked licenses now unlimited;

- Illegal copies of firewall software with rules specifically established to obfuscate any detection.

Firewall rules were created to allow by IP and name. Those named were part of the contract's two warring IT factions, IT operations and IT engineering, which were engaged in a struggle for control of the IT landscape. Dynamic IP allocation was required for all within the organization. Those with static IPs needed security approval. This had not occurred. Having a static IP allowed one such conspirator to access the servers in question off the internal network. His full name was on the rule.

The servers had never been patched or upgraded. The servers were running anti-virus software illegally acquired, loaded and never updated. They held 100GB of production data (including all server and desktop images for the organization).

One of the contract's employees was running a real estate business on one of the servers. All customer information (PII), financials and home listings, plus e-mailsnail mail distribution lists were stored there. Meanwhile, other documents indicated a love affair between four other employees on the contract. Their liaisons were dangerous, and not just because two of them were married—they were also taking place in this office (they fought about tryst schedules in the documents we had). There were rumors of sex videos stored on the servers. Despite our best efforts, we were unable to locate what would have been valuable evidence.

For those keeping score, there were several types of computer crime happening here, each a felony violation of Section 1030 of the United States Code (noninclusively):

- Fraud achieved by the manipulation of computer records;Intellectual property theft, including software piracy.

- Deliberate circumvention of computer security systems;

- Unauthorized access to or modification of programs (see software cracking and hacking);

-

We secured the servers and moved them to my office. We started to forensically image the servers, which had never been backed up, despite nearly 20 months on the job. I informed the Deputy Program Manager (DPM) that they were secured in my locked office and they were being backed up. The whole process would eventually take four days.

As each day came, the pressure mounted. The contract required us to report these incidents, but the Program Manager (PM), the Deputy PM and the IT Ops Manager wanted it hushed up. I was called into a meeting with the PM, DPM and the IT Ops Manager, who presided over the fornicating four.

The PM combined the physical features of the pointy-haired boss with a Mutley-like laugh. His management style was to glare menacingly at all near him, part of an effort to rule through continuous and multiple levels of attempted intimidation. He would, however, relax and beam with pride as he reminisced about selling dope on the library steps during college.

The DPM was a good-hearted sidekick who maintained a perpetual deer-in-the-headlights stare that was broken only by the incessant opening and closing of his mustachioed upper lip as his nicotine stick and caffeine drip passed into his needy, anxiously awaiting ecosystem.

We are required by contract to inform the government of any such incidents within a certain time frame, and it was getting late with respect to informing my government counterparts. Regardless, my peers would be informed (I knew something the PM didn't—that the CTO had decided to inform the agency CTO of the situation). This meeting was not to query what was being found. They already knew what was going to be found. You see, the IT Ops Manager had purchased these servers 20 months ago and had authorized their use as a backdoor way to meeting operational goals without federal scrutiny. Even so, the operational goals hadn't been met—unless you consider running a business from a federal government server an operational goal.

The meeting started with my chair positioned in a location under the direct gaze of the other three. They had prepared their line of questioning and felt confident they would achieve their desired results. Prior to the meeting, I had the facilities staff change the lock on my door. The new lock did not work with the master key for that office area (I acquired all keys to my door).

I came to the meeting armed with the initial draft report, distributing the evidence to the three interrogators. They peppered me with hostile questions about my intent in taking the servers and what I would include in my report. I informed them that any and all findings would be included in the report as per contract requirements and standard incident handling procedures. Why? It's standard protocol to review all incidents and subsequent findings/report prior to delivery. After they had exhausted themselves, we reviewed the draft report. There were no redlines, since the report was objective in nature—as required. The facts were stated and evidence provided. Of note, two of the four involved in this incident had left the company one week prior to the discovery of these servers. It is apparent to me that the warring factions had reached an impasse and new, more severe battles were taking place in the shadows. My role was that of a pawn for one and an enemy for the other.

That afternoon, I called my counterpart. He was initially stunned at the incident. He had a hard time believing that this activity could go undiscovered for more than a year and a half in an office in the same building as his. I sent the draft report to him before ending the conversation. In the evening, I secured my office and left with my laptop.

The next day brought a new round of questioning and inquisition. It was evident that someone had attempted to access my office after I had left. The DPM and PM were soon at my door to view the situation. I noticed the PM's interest in the doorknob. He worked the handle and examined the lock, with more than casual intent.

They were obviously anxious for an initial ruling from the customer. I soon received a phone call from my federal counterpart. He asked:

- When will the servers be back online?

- How will you prevent this from happening again?

I indicated that all software must be legal and that the PM had agreed to purchase all necessary software (minus the firewall). The servers would be placed within the data center and entered into the normal patch management and backup cycles. As for how to prevent this from occurring again, I promised I would perform a communist purge with those responsible sent to a gulag. Actually, we drafted a plan to perform more frequent vulnerability scans and network mappings as well as periodic announced and unannounced physical reviews of contractor accessible offices.

Even though the evidence was clear, concise and indisputable, no disciplinary action was ever taken (of course my options were but one). Should I have reported this to the Inspector General? It was in fact the responsibility of my federal counterpart. The felonies committed would be swept under the rug and the incident forgotten. Other copies may have been made of the data as methods of protection and self-preservation, but that is just speculation.

We returned the servers so they could upgrade the software per the plan, entering them back into production. Three months later, I left the contract. ##

Undercover is written anonymously by a real CSO.

Copyright © 2008 IDG Communications, Inc.

Microsoft's very bad year for security: A timeline