Polymorphic Malware: A Threat That Changes on the Fly

When CISOs talk about polymorphic malware, they’ll remind you that polymorphism is nothing new. Known to researchers since the 1980s, this malicious code changes its attributes to make it undetectable by signature- and behavior-based antivirus and intrusion detection defenses.

Ten years ago, at the annual Defcon hacker conference, push-button-simple server-side polymorphic features were introduced with the Back Orifice 2.0 backdoor Trojan. Then came an outbreak of polymorphic worms in the early 2000s (Code Red, Nimda and SirCam). Then talk of them quieted.

Now polymorphic malware is being used to send multiple variants of Trojans, and bots are being sent out in short “bursts,” that last an hour or less and are gone before detection systems vendors even have a chance to write a signature, says Amir Lev, president of Commtouch, an Israeli-based OEM vendor of a widely used virus detection engine called Recurrent Pattern Detection technology.

One example is the Storm Worm, a spam e-mail attachment that broke out in January with subject lines such as “230 dead as storm batters Europe.”

Commtouch detected “tens of thousands of variants” of this spam message in January, Lev says. Another example is the Stration family of malware, responsible for worms and other forms of malware in late 2006. “Stration was changing so quickly—the encryption packaging, the compiler, everything. We saw up to 300 variants in a single day,” says Ron O’Brien, senior security analyst at anti-malware vendor Sophos.

The fight against polymorphic malware is an arms race: the bad guys against you and your security vendors. Vendors continue to add new scanning capability to their engines, which commonly include pre- and post-scanning of executable files in search of payloads and programming routines indicative of malware. The engines do this scanning in a controlled environment, or “sandbox.” This scan uses a heuristic routine or behavior analysis to detect potential problems. So, for example, if the executable checks to see what antivirus engine is running, or if it tries to contact the master controller, some scanning technologies would detect and block that, O’Brien explains.

Such a technique of scanning executable files “doesn’t work with today’s botware because when you try to run them in the sandbox, zombies won’t do anything,” Lev explains. “Later on, they’ll try to contact the master controller, but they do that when the computer is idle, usually late at night.”

Besides deploying intrusion prevention and anti-malware systems that use heuristics, your network defenses should include a layer of scanning that looks for potential malware variants. For example, polymorphic applications often use their own style of file compression formats because the encryption can be changed on the fly. A well-executed defense would flag such a file as suspect.

You also need to tune your network detection to work in real-time. “You must rely on layers of scanning and zero-hour protection and response,” Lev says.