• United States




What some cybersecurity vendors don’t want you to know

Feb 08, 20174 mins
IT SkillsSecurityTechnology Industry

When evaluating security products, you might be doing it wrong if you’re not incorporating assurance testing.

hidden data
Credit: Thinkstock

One of the most frustrating processes in evaluating security products is the proof of concept (POC). Call it a POC, bakeoff, evaluation, proof of value (POV), trial, whatever, it’s resource intensive and if you are evaluating a new product against an existing one or multiple net new solutions at once or over time, it’s slow and it’s problematic to have an apples-to-apples comparison.

Once you’ve done your homework which might include talking to industry analysts, reading third-party reviews, producing RFIs/RFPs, talking with references and reading through websites, it might be time to buy or it might be time for a POC. Regardless of endpoint security controls, firewalls, IPS, SIEMs, DLPs and related solutions, quality security vendors will want their solutions squarely compared to incumbents or new competitors in a thorough but fair evaluation.

Wouldn’t it be nice to know exactly how a new control will work when compared to other controls, in your environment, under the assault of real attacks, while integrated with your security management solutions – all in as little as an hour?

There are many great security vendors that welcome the evaluation of their solutions. If your vendor doesn’t afford this option, that could be a warning sign. A POC gives you a feel for how easy or hard it will be to deploy, configure, integrate and use the product in your environment. It also shows you the reality of the product as sometimes, not always, but sometimes, marketing and sales stretch the art of the possible. But how do you make your testing and evaluation fair across the various vendors and more importantly do it quickly, easily and thoroughly?

Leveraging security solutions that allow for security control assurance testing can prove to be very helpful during POCs. Solutions like these, often called security instrumentation solutions, are more commonly known to be used when evaluating your existing security controls to address questions like:

  • Are my incident prevention controls preventing attacks?
  • Are my incident detection controls detecting attacks?
  • Are my SIEMs and log management solutions collecting and correlating on these alerts?
  • Is my security team prepared to respond?
  • Are my processes designed to be efficient and effective?
  • More simply put – is my security stuff working the way I hope, pray and assume it should?

But this same level of scrutiny can be applied during POCs. For example, you can evaluate the capabilities of an endpoint control, firewall, etc. by safely executing attacks across the security controls to see if they block, alert, etc., and further, if the alerts show up in the solution’s management console, then further do those events show up in your SIEM.

You can run a variety of evaluations across endpoints and networks such as: malware execution, CLI attacks, PowerShell attacks, tunneling, data exfiltration, SQL Injection and C&C traffic. These attacks can be safely executed across everything from existing security controls within the production environment to security controls deployed in a lab environment.

The results of the testing will paint an apples-to-apples comparison of how these security controls preformed in the face of a number of identical attacks. Did they block the attack, did they detect the attack, were they able to log that information to a SIEM, if it was logged to the SIEM was the information valuable and usable, etc?

Now regardless of you evaluating the capabilities of your existing security controls against new controls, or multiple new controls against each other, you are armed with a valuable assurance testing solution that can yield results fast. These security instrumentation solutions allow you to quickly, easily and thoroughly evaluate security controls and with empiric evidence, know how those controls integrate with other security controls like SIEMs. Solid endpoint and network security vendors will welcome this level of analysis. Other vendors may not want you to know this option even exists. 

Now how great is it to know exactly how a new control will work when compared to other controls, in your environment, under the assault of real attacks, while integrated with your security management solutions – all in as little as an hour?


Over the last two decades Brian Contos helped build some of the most successful and disruptive cybersecurity companies in the world. He is a published author and proven business leader.

After getting his start in security with the Defense Information Systems Agency (DISA) and later Bell Labs, Brian began the process of building security startups and taking multiple companies through successful IPOs and acquisitions including: Riptech, ArcSight, Imperva, McAfee and Solera Networks. Brian has worked in over 50 countries across six continents and is a fellow with the Ponemon Institute and ICIT.

The opinions expressed in this blog are those of Brian Contos and do not necessarily represent those of IDG Communications Inc. or its parent, subsidiary or affiliated companies.