Vendors respond to Cylance's new testing methods with AV-TEST

Vendors were unaware of a private endpoint test co-created by Cylance

1 2 Page 2
Page 2 of 2

AV-TEST responded to questions, their full remarks are below:

Cylance has said that the Feb 2 test you did with them might be the standard going forward at AV-TEST, can you confirm this? Also, some of the vendors included in the Cylance test said the parameters used were a bit unfair, and they were surprised to see AV-TEST conduct "marketing influenced testing". Do you have any comments or statements in response?

We plan to introduce new test cases to our regular public testing. The current tests we perform are a good metric to measure how well AV products protect from malware attacks. What they currently don't cover are targeted attacks. This is something that we want to introduce as an add-on to the current test. The cases we looked at for this Cylance commissioned test, show what could be done in that direction. Test case 2 (simulated attacks) is what we plan to use for this purpose.

Commissioned tests are usually performed to highlight the strong sides of the product of the commissioning party. All other vendors in the test have been commissioning public tests in the past with AV-TEST or other testing labs with a similar purpose.

When this is done, it is important to state that the test was commissioned and by whom. It is also important to clearly outline the methodology and what the purpose is. This has been done by us. We are even explaining the caveats of some of the test cases and point out that they may not represent a usual/common case.

Question: The test states that you feel you were aligned with AMTSO testing standards (based on the public documents), have you discussed this test with anyone after Cylance helped create it, or were they the only AMTSO member (outside of yourself) to have any input?

Usually commissioned tests are not discussed with a third party before publication. Also AMTSO guidelines are really just guidelines and are not mandatory.

We agree that the "Fundamental Principles of Testing" are a good baseline to follow, which we did. For certain specific tests there are documents that give advise what to consider when testing those certain cases. However, they are not designed as step by step descriptions how a test shall be run. And they are not meant to prevent testing labs from running tests differently or come up with new tests.

In fact the test cases we used are not really new or unique:

  • Test Case 1 (Holiday Test) is similar to a test of another testing lab, that is regularly performed (the RAP test of Virus Bulletin). The Chief of Operations of this lab is chairman of the AMTSO board and we haven't heard complaints about this type of test from AMTSO before.

    In the past, there was a controversy which implied that next-gen products are just using multi-scanning services like VirusTotal to identify files as malicious. This test case, being an offline test, shows that it is not the case for Cylance.

  • Test Case 2 (Simulated Attacks) is covered by another AMTSO paper as described above.

  • Test Case 3 (URL Test) is something that is not directly covered by an AMTSO papers.

  • Test Case 4 (False Positive Test) is a standard false positive test that we are using in our regular testing as well.

We agree that Test Case 1 and Test Case 3 are not representative for the real-world performance of products. They highlight certain technical aspects, as we clearly outlined in the document: The test shows how products perform when disconnected from the Internet, or when multiple technologies are disabled.

The tests are primarily showing that Cylance is able to deliver a similar level of protection as the other products, even without up-to-date signatures or cloud connection (or by querying VirusTotal). We are also pointing out in the report that the other products are able to provide this level of protection, as shown in our regular tests, when they have updated signatures and can query the cloud.

Whether this is relevant to the user, should be decided by the user itself. We are not making this decision, but instead provide all the information to enable the user to make the decision.

Copyright © 2017 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
The 10 most powerful cybersecurity companies