Undercover: The Company that Did Everything Wrong

The plane landed late afternoon at a small airport in California that looked like it could have been a scene from a 1960s movie. My team and I walked down the metal stairs (no Jetway at this airport) and across the tarmac to the one-and-only baggage claim carousel.

After gathering our luggage, we got into our rented cars and headed to the client site, where the CISO would be waiting for us.

We arrived shortly before 5 p.m., got our badges at the security desk and our contact came out to escort us inside. “Michael” is the CISO at “Client X” and the stress of the last few days has worn heavily on him. He looked like he hadn’t been home in a couple of days. His clothes were badly wrinkled, his face sported a two-day-old beard and his eyes were red. He escorted us into his office and updated us on the situation.

“Two days ago,” he said, “a number of people at the company, including corporate executives, their secretaries, HR personnel and others, were the victims of a well-orchestrated, well-researched spear phishing attack. The e-mail contained a message talking about a very specific program where Client X had just won a bid with the government. At the end of the message was a link that purported to be on our internal network, though it wasn’t. It linked to a site outside the company. Most of the recipients did not open the e-mail because they said it had a strange feel, but unfortunately a few of them did. Of those who opened it, most clicked the attached link. Unfortunately, the information security and information technology functions here are separate and report up two different chains of command. Days passed and no one was aware of what was happening. But then I got a phone call telling me that we were under a full-scale attack.”

I asked: “What kind of attack are you facing and how do you know?”

In fact, Michael said, his team at first had no clue as to what was afoot. They lacked the equipment to detect a breach and, even if they did, lacked the human resources to monitor such equipment. He told us his staff consists of one full-time employee and one half-time assistant who is shared with the help desk. The company was informed of the breach by a government agency that was able to watch the contents of one of the company’s hard drives fly across an Internet connection that the agency was monitoring.

“Trust me, that was a phone call I wish I had never received,” Michael said. “The officer couldnt say much because the investigation is ongoing and because I am only cleared to secret. He said enough to get us very, very worried.”

What did they say? I asked.

Michael: “They said we were hacked by a group sponsored by a rogue state. They targeted many different government contractors and saw the contents of one of our secretary’s computers fly past their monitoring station. It was only when I got this call that I started putting two and two together. We went to this secretary and asked if she opened any e-mail attachments recently. She said not recently, but the week before, she clicked a link in an e-mail, but the document that she was expecting didnt open. So she clicked a couple more times. Finally she gave up and called the help desk. This happened last week.”

So the intruders had been in the company’s network for at least a week. I asked what information the secretary had on her local hard drive and, more importantly, what information she had access to.

She had access to just about everything on the internal network, Michael said. And since the company had no network monitoring in place, nothing was preventing outsiders from monitoring network connections.

“What does your network topology look like?” I asked. “What kinds of equipment do you have in place and where?”

Michael: “It doesn’t look good. We are a large, geographically dispersed organization. We have about 10,000 users in 127 locations, with about 50 separate [access points] to the Internet. The organization is busy buying other companies like there’s no tomorrow and, with each new acquisition, our security posture gets worse and worse. For the most part, each one of the acquired companies is an independent subsidiary and security varies wildly between them.”

He said they had tried over the past year to start integrating them into distinct business groups, but the IT vision of that integration was a security nightmare. They set up network connections between all the units to facilitate communication and to try to consolidate some of the e-mail and time sheet programs. But these connections had been unfiltered and unmonitored. Data flowed unrestricted between all business units. There were no security enclaves, no defense-in-depth, nothing. The topology was a hacker’s dream.

“What about firewall logs? Router logs? Server event logs? How long are they stored?” I asked.

Michael: “What logs? Remember that each business unit is different, but here at corporate we don’t have logs. In fact, logging was turned off by the help desk because they got tired of responding to false alarms. Help desk reports to the IT director, not to security. You’ve heard the Jesper Johansson quote about computer network security: ‘Hard and crunchy on the outside and soft and chewy on the inside?’ Well, we’re not even hard and crunchy on the outside. Our perimeter has holes big enough that you could drive a truck through.”

My team had its work cut out for them. We asked Michael to set up a war room for us, and we handed him a list of things we were going to need.The makeup of our team changes depending on the needs of each specific engagement, but on this project, we had four team members on-site and an additional two available off-site: Bob, our senior forensics investigator; Victor, a seasoned systems admin; Sam, a junior resource; and myself, the PM and networking resource.The additional off-site personnel are Jerry, the best Python and shell scripter I know, and Dave, an independent resource who specializes in reverse-engineering malware. With these six assets, the work began.

Bob began with the computer that had originally been infected with the e-mail attachment. All of our data breach engagements include a forensics component, even if the client is convinced that they are not going to prosecute.

Aside from the fact that NIST (National Institute of Standards) states that making forensics copies is a best practice after a computer incident, having a pristine copy of the compromised computer has proven very helpful in the past. While Bob began the forensics copy of the hard drive, the rest of the team began their work: Victor asked to be escorted to the server room where he would begin poring through whatever log files the servers had; Sam began the process of taking meticulous and copious notes regarding all of the team’s activities; and I sat down with Michael in his office to gather more information.

We find that there are often nuances and subtleties within an organization that can sometimes help with the investigation, and this case was no exception. A few hours into the investigation, at about 9 p.m., we had already started making progress and I had some news to share with Michael.

The good news was that the forensics copy of the hard drive was finishing but, while this process was underway, Bob had been looking at the Exchange server with Victor. Bob found the e-mail that contained the link and had already resolved the IP address to a site in Spain. Using one of our hardened Linux computers and connecting out through a VPN tunnel to one of our remote computers in another country, Bob examined the questionable site.

The preliminary results showed that the site was a legitimate site that had been compromised. But because our test platform had not been compromised, we deduced that there must be another part of the site that the attacker used as a staging area.

The bad news came from Victor. As we had been informed, logging had indeed been turned off on all the servers. Even the hidden logging that Windows does had been disabled. The help desk personnel certainly knew what they were doing because all logging on the servers had been disabled. Worse was the fact that logging on routers and firewalls had also been disabled.

The client was being its own worst enemy, but at least there were no more of those annoying IDS alarms to investigate.

The lessons up to this point were clear:-The company had no security policy whatsoever; no guidelines on the right and wrong ways to handle e-mail, no access control procedures and no accountability.

-By shutting off all the logs, the help desk violated one of the basic rules of security: to keep and monitor detailed logs of activity inside and outside the network, larning along the way to tell the difference between normal network noise and activity outside the norm that could be malicious.

-The company allowed data to flow unrestricted between business units when it should have been segmenting the network to better protect proprietary information.

-The company dove into an acquisition spree without putting a review in place to find security holes and tighten them.

In this environment and with little manpower, Michael was way in over his head.

[Part 2 will appear in a future issue.]

The author of this undercover feature leads a Computer Incident Response team. He may be reached at the pseudonymous amonmadreen@gmail.com or eamonmadreen@hushmail.com