Antivirus Taste Test: One Man's Quest for (Nearly) Objective Rankings

Security Consultant Chaz Sowers did a semi-scientific comparison of antivirus software. The results may surprise you.

Editor's Note: Chaz Sowers wants reliable, indepedently tested antivirus software with few false positives. But what really constitutes an "independent" test? Unsatisfied with lab ratings, he built his own malware testbed and put 35 AV products through the paces. Here is the story behind one man's AV rankings; your results may vary.

I started my research with an online company that just recently rated the "top" 14 AV products. They promised "independent comparatives of antivirus software" while at the same time stating that "since 2008 [they charge] a fee for various services we provide."

Call me a skeptic. But when a testing lab accepts money from a company to test its product, I have to wonder about the independence of the findings.

Even assuming the test results are truly independent, this business model excludes smaller companies that are unable or unwilling to pay the testing fees. A quick search online found over 40 AV products, many from companies I had never heard of before. I wondered how the lesser known ones might fair against the better known ones.

Since I already have a day job and didn't accept money from anyone for these test results, I decided to share my independent and unbiased comparison of AV products.

The results may surprise you.

Testing methodology, disclaimer and other stuff

My testing methodology was as unbiased as I could make it. After all, I had a vested interest in finding the best AV solution for my own computer. Of course my testing falls short of the double-blind scientific method, but I think it holds up well for publication in mainstream media. Remember this above all: I was searching for an AV product that would identify and delete the highest number of the test malware that I have. My emphasis in testing was on a high number of detections and my testing penalized software that reported a large number of "false positives."

Software

I used a fresh install of Windows XP, running in a Sun Virtual Box virtual machine, to run all tests. The installation of Windows was fully patched and updated (including SP3) as of Jan. 8, 2009. Each AV program was copied to the main machine from a shared folder and was the only program on the virtual machine not part of a regular Windows install. The test data resided on a logical D:\ drive and consisted of 36,438 pieces of malware. All of the malware has been, or currently is, in the wild. The virtual machine was restored to the previous, pristine state after each test.

Hardware

The system used for testing consisted of a AMD Sempron 2600-Plus processor, Asus A7N8X-E motherboard, 3 GB of 184 pin DDR RAM, a Seagate 190GB SATA hard drive, and an nVidia video card. The test bed was installed with Ubuntu Linux (version 8.10 Intrepid Ibex) which ran a copy of Sun's Virtual Box (OSE 2.1.0). Most AV programs ran without incident on the test bed but a few had problems which are detailed in the table that follows.

Source this website, which I augmented with additional AV found here. The Wiki sire had 35 unique AV product listings, including proprietary, freeware and open source. There were names that I have known for over 16 years as well as ones that I had never seen before. Of these listings, I eliminated those whose parent company no longer existed, those for operating systems other than Windows, those for whom I could not download an evaluation copy, and software at (or very near) the end of its life. What remained are the AV solutions evaluated here.

I found a list of AV vendors on

Download

All AV software was downloaded directly from the vendor's website (where possible) or from a trusted source (C-Net or SourceForge) where the vendor did not directly support downloading. In all instances the software I downloaded was fully functional but time-limited software and would be the same that I would install and keep. For the companies that offered free versions of their products, I still chose the trial version of the commercial product.

Disclaimer

Ultimately these findings are the true and factual results of my experiences with the software and hardware listed above. They should not be used as the sole basis for purchasing Antivirus software and none of these products is endorsed by me or any of the professional associations through whom I have certifications.

Here are my findings:

1 2 Page
Recommended
Join the discussion
Be the first to comment on this article. Our Commenting Policies