The Value of Third Party Testing

We all wrestle with the challenges of security in today’s digital marketplace. The security landscape and potential attack surfaces continue to expand, and malware and exploits continue to become more sophisticated. However, one of the most significant security challenges that organizations face is simply deciding which solutions they want to incorporate into their security strategy. Vendors are multiplying at a dizzying pace, and anyone who has even been partway around the block knows that data sheets and marketing materials aren’t nearly as reliable as they could be. Moreover, given time and resource constraints, setting up a testbed and evaluating all potential solutions by hand is rarely a viable option.

Which is why third-party testing of security products and solutions plays such a critical role in thwarting cybercriminals. The reason is simple: organizations need effective security solutions that meet an evolving set of requirements. The fact that every organization’s network demands and business objectives are unique makes the selection process even more complicated. And to make things worse, far too many security vendors don’t do a very good job of providing data that enables a fair comparison between competing solutions. Besides often not providing enough information, data sheets can emphasize—and sometimes even inflate—good points, obscure product flaws, and rely on internal test results that don’t replicate real-world environments.

Comparison Shopping

It’s worse than comparison-shopping at the grocery store. Items next to each other the shelf may appear similar at a glance, but when you look closely at the labels, you find that one is priced per ounce, while the next is labeled with a price per unit. Another calls itself “healthy,” yet contains too many grams of fat. Fortunately, the FDA and equivalent agencies around the world are tasked with protecting the health of citizens, and so they have the authority to ensure that the labels on the food products you buy reflect what’s inside the package. And because they use the same measurements, standards, and processes you can make valid comparisons.

Unfortunately, there is no such authority for security solutions. Which is why third-party testing facilities are so essential. They provide a comparative assessment of solutions using standardized testing criteria and methodologies, allowing organizations to take an educated look at solutions through a common lens that would not otherwise be possible.

Everyone Benefits

It’s not just consumers who benefit from third-party testing. Vendors who regularly participate in these sorts of tests usually learn as much as their potential customers do from the results. Testing methodologies provide critical input to vendors about evolving enterprise requirements, while test results can help confirm they’re on the right track (or provide evidence for necessary course corrections)—regarding corporate expectations as well as through comparisons to other products on the market. Independent testing can even help manufacturers better understand market shifts in the options being made available by competitors to make informed choices about where to focus engineering efforts.

With the advent of digital transformation, for example, the networks that security tools were designed to protect are undergoing profound and often radical change. They are broader, more complex, and subject to a more sophisticated threat landscape than ever before. Effective testing methodologies often reflect these new requirements, meaning that yesterday’s winners who sit on their laurels can quickly become less relevant as test results reflect new requirements.

Staying Ahead of Evolving Security Requirements

To be effective, independent testing needs to be based on open methodologies (refined continuously based on enterprise requirements), impartially applied across available products, and then quantifiably reported.  Which means that organizations who rely on testing results to evaluate products need to do more than merely look at the results. They have to have confidence in the impartiality of the testing methodologies and ensure that the testing itself reflects the evolving challenges today’s network require.

Here are a few examples:

1. Organizations now expect NGFW solutions to be able to provide effective SSL inspection, and that functionality needs to be integrated into any firewall test results on which organizations rely. The same is true for technologies such as integrated sandboxing and SD-WAN, reflecting the changing nature of threats, connectivity, and traffic.
2. Datacenter security gateways not only need to continue delivering high performance, but also provide advanced security functions such as segmentation, deeper levels of inspection, and seamless integration with cloud-based data and workflow resources.
3. Endpoint security now needs to provide advanced exploit prevention and utilize machine learning to more effectively address today’s more sophisticated threats.
4. Breach prevention needs to combine the ability to not only detect known and unknown threats but also automatically respond to detected cyber events to stay ahead of fast-moving threats.
5. New tests also need to be continuously introduced. Web Application Firewalls are a relatively new area of testing, reflecting the growing need for dedicated protection of the web services portion of the network. Likewise, cloud services, IoT protection, securing OT environments, application integrity, and cross-functionality between traditionally isolated security solutions will all need their own, or to be added to existing testing processes to better reflect the rapid changes taking place in today’s networks.

Caveats

Of course, not all testing is the same. Which is why it is critical that companies looking at test results are also aware of some of the challenges. Here are two critical considerations:

1. Not all tests are created equal. When examining test results, it is important that you understand what was actually tested. Different tests, even conducted by the same lab, look at different things and may have different objectives. Some are very narrowly focused. Some value things like efficacy over performance. Most do not evaluate critical elements such as interoperability, visibility, or collaboration. Which is why it is important that the testing organization clearly publishes how their testing is conducted, and that you ensure that their methodologies match your own criteria.
2. Not all test results are created equal, either. It is also vitally important to know something about the organization that produced the test results you are reviewing. Many, such as AV-Comparatives, AV-Test, ICSA, NSS Labs, SE Labs, or Virus Bulletin operate with a high degree of integrity. A few other labs who are not members of AMTSO might not even follow the testing standards and provide a vendor with just about whatever test results they want.

Key Takeaways

For organizations addressing digital transformation, many of the current test methodologies being used by labs and testing centers provide critical insight into emerging requirements, enabling IT teams to evolve their security infrastructure appropriately. They help organizations narrow down potential solution candidates based on things such as superior effectiveness, performance, innovation, and value. They are an excellent place for organizations to start looking for validated solutions to ensure their security meets their evolving customer needs and internal digital business requirements. But also remember that most tests evaluate a product in isolation, and that the solution you choose not only needs to be at the top of its game, but also function as part of your larger security architecture. In addition to selecting a third party-validated solution, also be sure to look for things such as interoperability and the ability to share and respond to threat intelligence as part of a coordinated response that’s tied to an open security fabric. This will ensure you’re leveraging the right approach that unifies all security technologies to improve threat response time and better protect your network. 

Read more about the Fortinet Security Fabric and the Third Generation of Network Security.