Most Security Products Fail First Certification Tests

ICSA Labs finds most products need several more rounds of testing before they can be deemed "secure."

In the software business, quality is often left behind in the rush to be latest and greatest. Security products are no exception, according to a study released Monday by ICSA Labs.

ICSA Labs, a unit of Verizon Business, provides third-party testing and certification of security products. The company examined 20 years of its testing data to create the "ICSA Labs Product Assurance Report". The report indicates that nearly 80 percent of security products fail to perform as intended when first tested, and generally require two or more cycles of testing before achieving certification. ICSA studied data from their seven certification programs; anti-virus, network firewall, Web application firewall, network IPS, IPSec VPN, SSL VPNs and custom testing, which are customized testing programs designed for specific clients.

Also see Broken Windows Revisited: Why Insecure Software and Security Products Hurt the Global Economy

ICSA found the most common reason why a product fails during initial testing is that it doesn't adequately perform as intended. Across the seven product categories, core product functionality accounted for 78 percent of initial test failures. Examples include an anti-virus product failing to prevent infection and firewalls not filtering malicious traffic, ICSA noted in a release on the findings.

The failure of a product to completely and accurately log data was the second most common shortfall. Incomplete or inaccurate logging of who did what and when accounted for 58 percent of initial failures. The report findings suggest that logging is often considered a nuisance and is undervalued. According to the report, logging is a particular challenge for firewalls. Almost every network firewall (97 percent) or Web application firewall (80 percent) tested has experienced at least one logging problem.

Rounding out the top three, said ICSA, is the finding that 44 percent of security products had inherent security problems. Security testing issues range from vulnerabilities that compromise the confidentiality or integrity of the system to random behavior that affects product availability.

Other issues identified in the study include poor product documentation and patching. ICSA officials said only 4 percent of the products tested in their labs pass their rigorous certification process in the first round.

Get the best of CSO ... delivered. Sign up for our FREE email newsletters!