On Tuesday, the OSVDB project outlined various problems with Secunia's annual vulnerability report, including instances where Secunia counted vulnerabilities multiple times, or under-reported them. The project also took issue with how Secunia classified third-party products, which the Copenhagen-based firm says are non-Microsoft programs, a definition that isn't shared by a majority of the security community.
"In the world of VDBs, we frequently refer to a third-party component a 'library' that is integrated into a bigger package," the post explains.
"The notion that “non-Microsoft” software is “third-party” is very weird for lack of better words, and shows the mindset and perspective of Secunia. This completely discounts users of Apple, Linux, VMs (e.g. Oracle, VMware, Citrix), and mobile devices among others. Such a Microsoft-centric report should clearly be labeled as such, not as a general vulnerability report."
The project acknowledged that their observations may be biased, as they are a direct competitor to Secunia due to the involvement of their commercial partner Risk Based Security (RBS) - but after looking at the source data, it's hard to ignore the numbers.
Secunia reported 13,073 vulnerabilities discovered, in 2,289 products from 539 vendors, but those figures are said to be tabulated by counting vulnerabilities multiple times.
In the matter of browser vulnerabilities, Secunia reported 727 of them in Internet Explorer, Chrome, Opera, Firefox, and Safari. However, the OSDVB project counted 756 of them, and noted that Opera isn't very good at clearly reporting vulnerabilities so the count for that browser is likely off, "something they [Secunia] should clearly have disclaimed."
The discrepancy in totals carries over to PDF readers as well. Secunia reported 70, OSVDB recorded 76. However, when it came to vulnerabilities in Microsoft programs, the difference in classification and reporting was much larger. Secunia reported 192 vulnerabilities in Microsoft products, noting that the total represented a 128.6 percent increase when compared to 2012. However, OSVDB recorded 363, altering the percentage increase to 175.3 percent.
"In conclusion, while we appreciate companies sharing vulnerability intelligence, the Secunia 2013 vulnerability report is ultimately fluff that provides no benefit to organizations. The flawed methodology and inability for them to parse their own data means that the conclusions cannot be relied upon for making business decisions."
The Hash has reached out to Secunia for comment. If they issue a statement, this post will be updated. Until then, the OSVDB blog post on the topic is an interesting read.Update:
Secunia said they're aware of the post, but that they wouldn't offer any comments.
"Yes, Secunia is aware of the blog post. It is company policy not to comment or engage publicly in criticism/accusations with competitors. Secunia firmly stands behind the data and methodology of its Annual Vulnerability Review."