Chinese Anti-Virus vendor caught cheating on industry tests

Qihoo (CHEE-hoo) 360, a Chinese security company with hundreds of millions of users, has had their certifications from AV-Comparatives, AV-Test, and Virus Bulletin revoked after submitting products for testing that significantly differed from the products available to customers and users.

Each of the three labs confirmed that Qihoo had enabled a scanning engine provided by Bitdefender by default, but disabled their own QVM engine. Yet, when sold to the public in primary markets served by Qihoo, the setup was reversed.

John Hawes, Chief of Operations at Virus Bulletin, said in a statement that this sort of thing doesn’t help anyone.

“Independent tests serve both users and developers, showing which products are performing best and highlighting areas where developers need to work harder. If the products being tested aren’t those being used in the real world, nobody’s getting any useful information.”

With the Bitdefender engine being turned off, and the QVM engine acting as the primary resource used by customers, the situation is a classic bait-and-switch. Customers who opted to purchase Qihoo based on the lab results and reviews, would be getting a lower quality product based on stacked testing.

“According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives. Options are provided in the product to adjust these settings, but as the majority of users leave settings unchanged, most tests insist on using the default product settings to best represent real-world usage,” a joint statement from the three labs stated.

In response to the investigation, after being called out for their tactics, Qihoo said Baidu and Tencent (two other Chinese firms) also cheated the lab tests.

The investigation turned its attention to those companies, and “turned up some unexpected flags within their products, marked with the names of several test labs and implying some difference in product behavior depending on the environment they were run in,” the statement noted.

Indeed, similar flags were found in Qihoo products. But, there was nothing to suggest that those flags gave Baidu and Tencent any significant advantage, and in some cases the settings seemed to put the vendors at a disadvantage.

Eventually, Qihoo admitted to cheating.

The company said that some settings were adjusted for testing, including detections for keygens and cracked software, as well as directing cloud-based look-ups to servers located closer to the labs in order to make things look faster than they really were.

“Users rely on independent results to make an educated decision regarding their protection software. If vendors start to manipulate the testing process, they are hurting everyone involved,” said Maik Morgenstern, CEO of AV-Test.

At the same time, those who tune for marketing alone are also increasing the risk of further attacks.

“Misuse of such tests for marketing purposes will, in the long run, result in more successful malware attacks, making Internet users less secure,” added Andreas Clementi, CEO of AV-Comparatives.

Based on reported figures in 2014, Qihoo had 496 million Internet Security users, and 641 million mobile AV users, it wasn’t clear how many of them are using products that were recently stripped of certification.