Towards the end of 2016, Cylance, a feisty endpoint protection startup out of Irvine, California, published a series of blogs challenging the methods used by two certification labs to conduct endpoint protection tests. Cylance called the tests unethical, and accused the labs of fraud, bias, software piracy, and extortion.
The pay-to-play argument usually comes up shortly after a lab publishes a new testing and certification report. Outside of pay-to-play, the other argument common among endpoint protection testing centers on methodology - it's too static and doesn't actually mirror the real world.
Considering their comments and writing, Cylance feels that testing their products with improper configurations and without recent updates is stacking the deck. But their arguments, right or wrong, beg the question – is there any value in endpoint testing, or is it just a marketing tool?
IDC's Robert Westervelt says testing labs provide a valid service, adding that any organizations considering endpoint protection products should use testing data as one of the components in their evaluations.
"With that said, I think the best way to know the true effectiveness of any security solution is to test it out in your environment. Test results are often used by vendors in the sales process and I saw that a lot when I was covering the sales channel," Westervelt said.
"In most of the engagements I've seen, the CISO and other influencers in the buying process are more concerned about whether the product will solve their existing requirements or problems, whether it can be implemented while maintaining compliance, and increasingly, if the endpoint security product can integrate with the existing security investments they have made."
When it comes to conducting endpoint testing, most of the industry – including the testing labs themselves – tend to follow AMTSO (Anti-Malware Testing Standards Organization).
AMTSO has a number of documents related to testing a given protection product, from standard anti-Virus to newer endpoint defenses.
Membership includes all of the major testing labs and anti-Virus vendors, such as McAfee, Symantec, and Kaspersky, as well as second generation vendors Endgame, FireEye, Crowdstrike, and SentinelOne. Cylance joined AMTSO in December of 2016.
Cylance vs. Everyone
On September 13, 2016, Cylance published a blog accusing two testing labs of defrauding and manipulating of the public, because their published test results recorded 100-percent scores. The argument being, there is no such thing as 100-percent in security.
"The defrauding and manipulation of the public with these tests also stems from vendors who pay so that their test results will show 100% efficacy. These reports not only deceive the buyer, but they also set up impossible standards for the entire security industry," the post states.
In December of 2016, Cylance published two additional blog posts, naming the labs they have problems with directly. The December posts discussed tests performed by AV-Comparatives and MRG Effitas, where CylancePROTECT was matched against Symantec, SentinelOne, Sophos, Trend Micro, McAfee, and Microsoft. The test took place between September and October of 2016, and was paid for by Symantec.
"…it comes as no surprise that the major benefactor for the report also turned out to be the major beneficiary of its findings," the blog from December 1 states.
The post goes on to suggest legacy vendors like Symantec are allowed the choose the malware they're tested against, and for a fee they can have some editorial rights over the reports before they're released to the public.
Salted Hash reached out to Symantec for comment. In a statement, the company said they participate in both non-commissioned and commissioned tests from a number of labs, not just the two Cylance referenced. At no time did Symantec have any "influence over how any of these tests are conducted or what test samples are used."
"Benchmarking the efficacy of our products against both current digital threats, as well as competing solutions, is a vital and expected part of providing our customers with the highest performing protection possible. It’s particularly important for significantly updated releases that are being brought to market," Symantec explained.
"We are pleased with the outcomes of these independent tests, which clearly demonstrate that Symantec Endpoint Protection is an exceptionally powerful option for customers looking to block all types of threats to the endpoint."
Pirated and outdated software
When questioned about the Symantec test, Cylance argued it was flawed from the start, as AV-Comparatives and MRG Effitas were essentially using pirated software. As such, they were not able to access the cloud-based console or enable all the features in CylancePROTECT.
"We have no records or invoices showing that MRG Effitas, AV-Comparatives or any person associated with these companies purchasing CylancePROTECT," said Chad Skipper, VP of industry relations and product testing at Cylance.
Skipper also noted a lack of contact between the labs and the fact their product was tested with a default configuration. According to Skipper, testing in a default state will see most of the features in CylancePROTECT turned off.
There were two additional reports singled out by Cylance, where they took issue with the product versions being tested.
When they were tested against ESET in February of 2016, the labs used CylancePROTECT 1.2.1310.18 - released on October 14, 2015, months before the test started on January 29. Version 1.2.1340 was released on Jan 6, 2016.
The third test, a comparison with Sophos released in June of 2016, used CylancePROTECT 1.2.1370.99 - released on March 16, 2016, but the test didn't start until May 24. Version 1.2.1380 was released April 21, 2016.
Prepaid cards and VPNs
In a joint statement with MRG Effitas, AV-Comparatives said they purchased CylancePROTECT legally from a channel partner (Malware Managed) and the product was updated fully before testing. The statement also says the labs had access to the cloud-based console and used it. Their remarks go on to say all products were tested in a default state.
"Symantec commissioned this MRG Effitas and AV-Comparatives review, which we make very clear in the report. Symantec had no prior access to the samples used, no visibility of our test process, and no results have been falsified or adjusted in any way. In fact, Cylance is now trying to revoke all licenses we bought, and but not refunding the money. They try really hard to prevent that we can buy a license and they hide their product," the statement concluded.
When asked further about the licenses, a spokesperson for AV-Comparatives said Cylance blocked the credit card previously used to purchase a license. But that isn't going to stop the tests, the spokesperson said, because "there are prepaid credit cards, VPN and fake mail addresses, so we keep on testing them."
"Our tests are recognized as one of the most scientific and fair tests among the industries," AV-Comparatives said in a follow-up statement.
"We perform the most sophisticated Real-World Test. We are ISO certified for independent testing, we are EICAR certified, we are working together with the University of Innsbruck, to improve out tests."
Asked for confirmation on revoking licenses, Cylance gave one:
"[AV-Comparatives] used a non-company email address to gain access to our product. They obfuscated their corporate identification in order to acquire our product to test without our permission. Upon investigation and tracking CylancePROTECT clients to [AV-Comparatives] and [MRG Effitas], we revoked the license. Given their unethical testing practices as outlined in a previous email we believe that revoking this license is prudent to ensure our product is being fairly tested. Cylance has nothing to hide, we just want fair testing for every vendor," Skipper said via email.
AV-TEST develops new testing methodologies with Cylance
Last week, before the start of the RSA Conference, Cylance provided Salted Hash with the results of a new comparison test they commissioned from AV-TEST.
The test shared by Cylance is new though, and could shape the way endpoint testing is done in the future. AV-TEST said the new methodology will be introduced as new tests for the advanced anti-Virus solutions, which run twice a year.
The testing methodology was co-created by Cylance over a period of about six months, and consists of four different test cases. It places CylancePROTECT against offerings from Kaspersky, McAfee, Sophos, Symantec, and Trend Micro.
The first test case (the holiday test) will focus on products seven days out of date, with no access to the internet – denying them the ability to check cloud-based sources or apply updates. The second test involves executable files (malware) created by AV-TEST to simulate certain types of attack. During the third test, AV-TEST disables URL filtering and determines how well a product can detect drive-by malware and other web-based threats.
Finally, the fourth test looks at false positives. AV-TEST conducted this scenario by downloading 38 common applications and recording any blocks or warning messages.
The test results name Cylance as the clear winner, but while showing results for the others, the test doesn't name them directly. For example, the results of the first test shows Cylance with a score of 97-percent, and second place is displayed as Vendor 1.
On their blog, Cylance stated clearly: "…no one pays for a test that shows how poorly their product performs. They only pay for tests that claim that they perform better than their competitors."
Given that, Salted Hash asked additional follow-ups after reading the report.
"I realize we have bashed the pay-to-play model for testing, but our chief criticism is the lack of integrity and transparency into the methodologies, configurations used, lack of access to malware, testing against older versions and the inconsistent configurations set forth for each test," said Cylance's Skipper.
Cylance was primarily targeting the three paid tests from AV-Comparatives and MRG Effitas, Skipper explained, adding that all three have slightly different testing methods.
"Yes, we paid AV-TEST to do testing, but the methodology was structured to be fair to all parties, and furthermore we anonymized the data. The Holiday Test is not necessarily new as VB100 does this same type of testing. In order to truly test 'zero-days' AV-TEST created malware that no one has ever seen which is more fair than picking wild samples that some have already seen and simply turning off URL filtering is a real-world scenario."
The goal, Skipper said was to have different products tested against one another and publishing the results, while keeping level playing field for all "instead of a process where whoever writes the check gets the best results."