Can crowd security testing be cost efficient for web apps?

Last week Bugcrowd published a comprehensive report explaining the current state of the growing bug bounty market. Almost at the same time, High-Tech Bridge released a web security trends report for the first half of 2016. Meanwhile, the Open Bug Bounty community helped website owners (including such giants as WordPress and Amazon) fix almost 25,000 XSS vulnerabilities.

(DISCLAIMER: Kolochenko is the founder of High-Tech Bridge.)

I have already written about the potential benefits and pitfalls of bug bounties, but some numbers from the above-mentioned reports appeared interesting to me — highlighting previously unobvious tendencies.

In this article, we will try to correlate the latest trends in bug bounties for web applications and web security in general to understand if crowd security testing can be a cost-efficient complement for existing web security testing technologies.

Overpriced XSS and CSRF dominate bug bounty submissions

Almost 80 percent of all websites are vulnerable to XSS, says WhiteHat Security. Meanwhile, according to the Bugcrowd’s report, 66.24 percent of all (categorized) vulnerabilities submitted via Bug Bounty programs are Cross-Site Scripting (XSS). CSRF vulnerabilities represent 19.71 percent respectively. All totaled – above 85 percent of submissions.

However, today even an automated vulnerability scanner, can more or less reliably detect various types of XSS vulnerabilities. Of course, scanners produce quite a lot of false-positives and require additional skills and time to convert their reports to something meaningful, however a proper Bug Bounty program implementation and management can consume much more internal resources (including your legal department).

[ ALSO ON CSO: Why bug bounty hunters love the thrill of the chase ]

Taking into consideration that, as per Bugcrowd, the average bug payout in the first quarter of 2016 was $505.79, buying a web application scanner annual license would probably cost you less than paying for a dozen of reported XSSs (of course if you pay cash and not by T-shirts). I haven’t even spoken about the Open Bug Bounty’s XSS vulnerability reporting program, where security researchers can be rewarded just with a recommendation or with a small “thank you” gift of your choice.

Gone are the days of remote PHP file includes (RFI), and in several years XSS will probably become as rare as SQL injections are today. Modern web security technologies are also evolving: a correctly configured Content Security Policy (CSP) HTTP header and SameSite cookie attribute provide quite reliable protection against the majority of classic XSS and CSRF exploitation vectors.

So, practically speaking, above 85 percent of all web application vulnerabilities reported via [paid] bounty programs cost companies more than purchasing a web vulnerability scanner.

Young talents from developing countries dominate the crowd

Bugcrowd report also says that above 50 percent of all the researchers come from India and Pakistan. Another interesting fact is that 75 percent of the researchers are between 18 and 29 years old. Being a youngster, I have to admit that I know very few professional penetration testers younger than the age of 27. And it’s not about the technical knowledge or skills, but about experience of delivering value to customers by reporting security vulnerabilities using right methodologies and appropriate format. Vulnerability discovery is just a very first step, afterwards you need to assess, wrap and present it in a meaningful and useful way, otherwise your finding is worth nothing.

So at the end of the day, you’d better contact one of the numerous cybersecurity companies in India to conduct more reliable and comprehensive penetration testing with some sort of insurance, deadlines, possibility to claim damage, and at a lower price.

Black Hats will get in, researchers will give up

Cyber mercenaries, or Black Hats, are motivated by big money (far exceeding even Google’s bounties) and desire to maintain their reputation of being able to break into any target. They are experienced professionals, often much more qualified than an average Bug Bounty researcher. They will work days and nights during weeks to get in, while according to Bugcrowd, 85 percent of the researchers participate in bounty programs as a hobby, 70 percent of which spend less than 10 hours a week hunting bugs. Obviously, Black Hats will find what they want and what the researchers won’t. Of course there are some exceptions, but exception proves the rule.

While the concept of crowd security testing can be leveraged with a lot of success for web systems or platforms designed for a very large audience, for small and midsize companies it can give a false and thus very dangerous sense of security. Some companies start thinking that if crowd is testing them, they have nothing to risk. They realize how dramatically wrong they are, only after being compromised.

Complicated vulnerabilities remain undetected by the crowd

High-Tech Bridge’s report highlights the growing complexity of modern web application vulnerabilities, as well as their exploitation techniques. Classic SQL injections or RCEs become very rare these days, while complicated application logic vulnerabilities, undetectable by automated scanners and often omitted by bug bounty researchers, still remain in many web applications. Many vulnerabilities are exploitable only in pair with other vulnerabilities, creating sophisticated exploitation techniques. The crowd, being paid by results, often won’t bother to detect them, quickly switching to more lucrative XSSs on the next newly admitted project.

Pure automation, as well as pure manual testing, are becoming inefficient and are currently declining. The new trend is a hybrid approach, when everything that can be automated is automated, while the rest is managed by qualified human. Well-established cybersecurity market leaders partner with cybersecurity startups to complement their automated network vulnerability assessment with managed machine learning technologies and manual penetration testing. Such hybrid approach is probably the right balance between technical efficiency and cost.

Conclusions

Being a multinational company like Google, Uber or Facebook, you definitely need to have a well-established and properly governed Bug Bounty program for your web applications. However, for small and midsized companies, or for web applications that are not designed to be used by millions of users on all continents, bug bounties may not only increase the overall cost of testing, reduce its quality and reliability, but also introduce additional risks.

Therefore, if you are thinking about complementing your existing web security testing portfolio with a Bug Bounty program – make sure that it’s appropriate for your web application size, complexity and expected usage in production. Think which types of vulnerabilities you are ready to reward, how much will you pay, and then double-check if the same vulnerabilities cannot be detected in a more cost-efficient and reliable way.