The Botnet Hunters

A self-proclaimed geek from the age of 14, Andre DiMino had always been interested in computers and networking. But it wasn’t until he entered his professional life many years later that he became interested in the security side of that world.

“I was a system administrator for a fairly large network that experienced a significant hacking incident one weekend,” said DiMino. “I became consumed with learning about the methods of attack, who might be involved, and where it came from. Right then, I became passionate about all aspects of security, as well as the various groups that carried out the attacks.”

And today, in his forties, it is DiMino’s interest in the dark side of security that consumes much of his free time. By day, DiMino is a professional digital forensic analyst. By night, he serves as director of an organization known as Shadowserver Foundation, a group of volunteers dedicated to sleuthing out cybercriminals and shutting them down.

DiMino, and another cofounder who is no longer part of the organization, launched Shadowserver in 2004 with the initial mission of tracking malicious activity online and finding some way to make it stop.

“We just kind of started chasing malware, chasing bots,” said DiMino. “Mainly we were interested in understanding what malware did, where it went, how it was developed.”

A good deal of their time was spent tracking malicious botnets, networks of compromised computers running software that is installed through virus or worms, without the owners’ knowledge; these systems are then controlled remotely by a “bot master.” They are used for various online crimes, including sending out spam, phishing, committing click fraud and launching denial-of-service (DDoS) attacks. Windows PCs are the typical target, although a Mac botnet was reported earlier this year.

Also see the interactive graphic What a Botnet Looks Like

Just five years ago, hunting botnets, said DiMino, was a much different game. The botnets were fairly straightforward, he said, and the primary method of communication was the IRC (Internet Relay Chat). DiMino and other volunteers were able to act like criminals by joining a botnet, watching its traffic to get an understanding of how it was architected and learn more its particular function. They found their efforts were worthwhile as they began contacting network hosts, alerting them that were supporting the botnets and seeing them shutdown.

“Things really started to snowball,” said DiMino. “We decided it should be a service to the community to improve the safety of the internet. And we started to build a cross-section of security experts to help out.”

Shadowserver now has ten of what DiMino called “carefully vetted” volunteers in several locations around the world. These cybercrime busters need to be of the utmost trustworthiness, he said, because the data which Shadowserver volunteers deal with is highly sensitive. And that is exactly what the bad guys want.

Tools of the trade

DiMino detailed the four-step process that Shadowerver employs to stop botnets. The group first detects malware by setting up honeypots (Internet-attached systems that are easy for hackers to find and infect), and they use many different types of technology to analyze incoming and outgoing traffic.

“In botnet/malware network analysis, we like do both dynamic and static analysis,” said DiMino. “Dynamically, we want to study full content network traffic to help determine exactly what is happening on the wire. So open-source tools such as Wireshark, Chaosreader, Argus, etc., are helpful. We also do testing as to how intrusion detection systems may detect malicious network activity, so we use Snort as well. Then there are the various open source honeypots that we use as part of our malware collection. Any organization interested in malware detection/collection should run some sort of server-side honeypot at different points on their network. It can give a very good indication of what they’re facing as far as potentially malicious traffic directed at them.”

These honeypot sensors capture spam and malware which is then analyzed. Volunteers want to know about its network touch points; where does the malware go? Who does it attempt to contact? These are the first steps to finding a botnet. (See How Gozi’s First Second Unfolds for a detailed look at one Trojan’s behavior.) Unfortunately, it is not very simple and requires a delicate balance that allows them to both obtain information without contributing to the problem.

“Bot masters now have ways to detect these drones and kick them off,” said DiMino. “Plus we don’t want to participate in an attack, so we don’t want of our monitor system to do a spam run or anything like that.”

All information is compiled into reports Shadowserver makes available to network operators, as well as law enforcement officials and other security-centric and defense organizations that might need the data for research or other purposes. Shadowserver will also contact a network operator to let them know if they are hosting a botnet. The only request for the free data, said DiMino, is that the host take action and take down the botnet.

It can be a thankless job: For as much as a well-intention network operator may disable a botnet, the machines in the network remain infected and criminals usually bring them “back to work,” so to speak, very quickly. For example, after Web-hosting company McColo Corp., which hosted several massive spam-sending botnets, was shut down late last year, spam levels declined by 65 percent, but then returned within weeks.

“It can most definitely get frustrating and discouraging at times to see the resiliency of some of the botnets. However we’re encouraged by the increasing interest and cooperation by law enforcement, various security organizations, and even international CERT groups that are able to track movement and continue to make an impact.”

The mind of a hunter

What kind of a person gets into this line of work, essentially a career in hunting malware and botnets? A person with extreme patience who not only has the passion, but the time, to do it, according to DiMino. Shadowserver volunteers often spend in excess of 12 hours of their own free time each day tracking malicious activity. DiMino, who has a degree in electrical engineering, would not have guessed his career would head in the direction it has.

“When I was in school, I never thought that I’d be doing this kind of effort to the extent that I am,” he said.

Steve Santorelli, on the other hand, had seen his future in IT security investigations coming at him from the beginning of his career. The U.K. native originally got into law enforcement with Scotland Yard with the intent of working in computer investigations. He eventually moved on to similar work at Microsoft before his current role as director of global outreach with the non-profit organization Team Cymru, a group founded a decade ago by four people who Santorelli said were motivated simply by the who and the why of online criminality.

“There was at that time an explosion, almost this perfect storm of organized crime that started moving into the cyber arena as the banks started to come online while at the same time the computer malicious hackers started to realize there was money to be made,” said Santorelli.

Team Cymru has 35 members around the globe investigating malicious online activity and working with law enforcement and others to stop it.

“I just find it is a fascinating way where you have to think strategically,” said Santorelli. “It is a modern-day game of chess.”

And in that game, as DiMino also described, one opponent is always striving to outdo the other, said Santorelli.

“In the Cold War, there was a nuclear-arms race,” he said. “Now just as the banks put in a counter measures, the criminals circumvent that. It is the same way with botnet technologies.”

Peer-to-peer botnets, like those known as Storm and Conficker, have brought the competition to a new level, said Santorelli.

“They are deeply disturbing. The only way you can really take down a peer-to-peer-based botnet is to kick down the door and arrest the guy who is behind it,” he explained. “Essentially the miscreants have examined the way the community conducts investigations and have evolved to circumvent countermeasures that we have put in place.”

“These are very sophisticated botnets,” he continued. “Even if you could hack into botnet infrastructure you are not allowed to issue the uninstall command. Most have a simple command that allows you to uninstall that. But you can’t do that because you are making unauthorized modifications to an effected machine in a jurisdiction no judge is going to give you permission to do. You can’t get a Texas judge, for instance, to allow you to make modifications to a machine in Tokyo. Technically even if you could, it would be against the law.”

From the shadows to the labs

The work of botnet hunting is done not only by volunteer and non-profit organizations like Shadowserver and Team Cymru, but in research units in many of the world’s largest security vendors, like Symantec. Vincent Weafer, vice president of the firm’s security response division, says his team is busy hunting malicious activity for actionable intelligence that can be baked into future security products. He also cites the constantly evolving threat landscape where the criminals adapt their strategies almost immediately after security manages to catch up.

“We deliver 10,000 new virus signatures a day,” noted Weafer.

It is one profession where time and experience don’t really make the job any easier, said Weafer. As botnets, and the criminals that master them, get stealthier and more prevalent, infections continue to climb. In fact, Symantec saw a 31 percent increase in the number of bot-infected machines from 2007 to 2008’s average of 75,158 infected computers per day.

Also see How a Botnet Gets Its Name

“A few years ago we used to talk about [people being infected by] going down the dark alleys of the Internet, at porn sites and various things like that,” said Weafer. “But these days most attackers cast their net quite differently. They find legitimate web sites with weak security and put in exploits on the sites with the notion that if enough people visit those sites they are going to get those exploits on machines.”

That means just about anyone is at risk for infection now. And having your computer patched and up-to-date no longer guarantees the user immunity. Weafer and others noted that these days, even if the actual malware is not sitting on the site you visit, all the criminals need you to do is run a script and you can be infected.

“A lot of these sites have little control over adverts now,” noted Santorelli. “We’ve seen a number of cases recently where people have gone to a legitimate web site and there is an advert up there hosting some kind of malicious code.”

Another popular tactic employed lately, even on legitimate sites, is offering rogue antivirus software, which relies on social engineering to trick a user into downloading what they think is security software that will scan for or remove malware. Instead, when the user grants permission for the download, they become infected.

“A machine that has been infected often doesn’t have one piece of malware, it has several pieces of malware,” explained Weafer. “And what it does next is phone home. It communicates back to the master to let them know “I’m online.”

At that point, the infected computer is part of the botnet; one of many computers there to do the bidding of the master controller, serving in some sense as a software-as-a-service for criminals who rent them out from the botmaster for various schemes, like fraudulent pharmaceutical scams, which accounted for 70 percent of global spam in September, according to analysis from security firm McAfee.

“We often call it botnet as a service,” said Weafer. “And there is quality of service and bandwidth to consider. These are all the things a botnet master is looking for because in turn he will advertise that out.” (This underground online world was examined in depth by CSO in Inside the Global Hacker Service Economy.)

Work that brings many worries

To hear about some of the things these security investigators have seen in their line of work is to hear tales of ominously growing infected networks with implications that have yet to be seen. And it is scary stuff. Both DiMino and Santorelli noted the rise of the now well-known worm Conficker as one of the most troubling moments in IT security history in recent years.

“It is one of the more disturbing peer-2-peer botnets because it is very big, and it became a media sensation,” said Santorelli. “But more disturbing than anything else about it is we haven’t actually seen what it is going to be used for yet. Conficker has infected, by some estimates, millions of machines around the internet, but it isn’t actually doing anything yet. A lot of people are very concerned about what it’s for.”

“Having been used to enumerating botnet drones in the thousands, tens or hundreds of thousands, seeing a multi-million node botnet rapidly propagate was quite alarming,” added DiMino. “We were initially worried at the infection rate and extensive propagation, but then considering how such a botnet could potentially be used, was especially worrisome.”

Measuring success

In a world where investigations can take months and years and the rewards are few, how does one measure success when it comes to hunting botnets? For guys like Weafer, the work has the obvious direct impact of enhancing products and helping customers. But for Santorelli and DiMino, the payoff is more personal.

“Personally, I love that feeling when you have when you’ve spotted a mistake a criminal has made,” said Santorelli. “So much about IT security investigation is about turning over ten thousand little rocks looking to see what you can find underneath. When you spot a mistake a criminal has made, then as a group you realize: ‘I’ve got ya.’

That feeling is still with him down the road, said Santorelli, through those long investigations. Even though after spotting that mistake there is still a lot of work to be done to identify the bad guy and get them caught, he said, realizing 6 months or a year down the line that this is the moment that will lead to an arrest and prosecution is like no other.

DiMino points to a volunteer collaboration effort formed earlier this year as the Conficker Working Group, an assembly of security industry professionals trying to contain the infamous Conficker worm, as one of the bigger rewards for him, and a good example of the progress that can be made when people work together.

Seeing several varied organizations with different strengths and goals quickly band together and plan a course of action was amazing, he noted. The CWG quickly grew and soon achieved worldwide involvement from some of the best people and organizations within information security.

“I think that was a sign of things to come in terms of how groups can work together when there is a controlled mission in mind,” he said. “That was pretty groundbreaking event because it got a lot of security researcher organizations together in room and said:”We have a real threat here, what are we going to do about it?”

FireEye Versus Mega-D: One for the good guys

In November, researchers with a small security-products firm based in California managed to deliver a severe blow to a notorious spam botnet known as Mega-D, or Ozdok.

According to Atif Mushtaq, a security researcher with the FireEye, after detailed analysis of the botnet’s inner workings, researchers decided that instead of playing a passive role, they would come forward and start working with third parties like ISPs and domain registrars to take it down. In a blog post on the FireEye web site, Mushtaq details how the research team worked in multiple directions simultaneously to work against all the fallback mechanisms so fast that bot herders wouldn’t get a chance to counter react. As we go to print, all the major Mega-D command and control servers have been taken down.

According to tracking data from Message Labs Intelligence, which tracks global spam activity, FireEye’s efforts were indeed worthwhile as activity from the botnet has significantly declined. Mega-D, according to Message Labs, has for over a year been among one of the top-ten active spambots. Now Mega-D’s ‘market share’ has dropped to a mere fraction of a percent. It now barely registers as existing, with only a few spam seen each day, rather than thousands, said Message Labs officials.

Security researchers know it is unlikely that a botnet will ever be completely wiped out. But efforts like those of Fireeye can cripple a botnet to a point where it will be a long time before it is able to regain its former standing, if it ever does. DiMino, who said Shadowserver has done some joint research and collaboration with FireEye, looked at the win as more proof of why botnet hunters need to work together.

“It was good work on their part and certainly an effort that provided tangible results. While the jury is still out on the overall effect of this takedown, it’s a great example of how a carefully coordinated and comprehensive plan can achieve success.”