How do you measure success when it comes to stopping Phishing attacks?

Last week, Duo Security released a brief report on their Duo Insight tool for Phishing assessments. In all, 11,542 employees (400 companies) were tested, and 31-percent of them clicked the link that was included with the assessment email. Worse, 17-percent of those employees provided their usernames and passwords.

“Based on this data, in a real-world scenario, attackers can run a Phishing campaign that takes only five minutes to put together, and within 25 minutes, they’ve got access to corporate data that can lead to an organization-wide breach,” a spokesperson for Duo Security said.

So what’s considered a win when it comes to Phishing?

This question was posed to IT workers and non-executive types during the recent gathering of hackers and security professionals earlier this month in Las Vegas. Later, during a week hiatus, Salted Hash posed the same question to IT workers in Indianapolis.

The interesting thing during these conversations was that no two people had the same answer. Everyone had their own thoughts on the topic, and each organization measured success differently. It’s not surprising, but that missing uniformity is why anti-Phishing and awareness programs are so different.

Some measured success based on clicks. As such, if the employees avoid 80-percent of the Phishing emails delivered during an assessment, they see that as a win. From there, the assessment moves to focusing on the 20-percent that did click links.

No two Phishing attacks (simulated or real) are alike. If an employee avoids an obvious scam based on delivery notifications, but later falls for a scam related to financial documents, that’s a problem. Yet, some organizations stop testing those who are successful during a given round of assessment. This has the potential to create defensive gridlock.

The measurement of clicks can also be a problem. What counts as a click? Is a click simply following a malicious URL, or does that include attachments as well? Are hybrid attacks, those that use links and attachments counted as clicks, or are they measured differently? Depending on whom you ask, you’ll get different answers.

The general feeling among defenders was that an anti-Phishing “win” was a 10 to 20-percent click rate, meaning that 80 to 90-percent of the Phishing emails that went to the organization (testing or otherwise) were unsuccessful attempts. In this case, clicks were inclusive of both links and attachments.

Many also agreed that a layered defensive posture, as well as continuous assessment and training will help lower the impact of Phishing, but it wouldn’t prevent it entirely. Instead, better compromise detection, and improved response times should be part of any anti-Phishing program.

“When it comes to Phishing, if an attacker launches a focused campaign and had a 20-percent success rate, that should not be considered as a win for defenders. Whenever someone talks percentages when it comes to security, anything above one percent is not good,” said Daniel Ford, a forensic analyst and tactical security engineer, for Indianapolis-based Rook Security.

“Assessments count 80-percent as a solid success rate because most of the time if you are dealing with a large company, you would assume 20-percent of the people in the organization are either new, not familiar with security best practices, or will fall for a well-crafted email from the CEO or CFO.”

Even then though, Ford added, a 20-percent success rate should not be considered a win for defenders. He said the better metric would be to do follow-up assessment after security awareness training and seeing the results of that Phishing campaign.

At the same time, Ford’s colleague, Chris Blow, who is a senior security advisor at Rook Security, disagrees, because nothing in this world is one-hundred percent secure, and no one should expect it to be.

“The average failure rate (of the client) of a Phishing/spear-Phishing campaign is usually between 60 to 80-percent – a pretty astronomical number. However, if we carry those metrics through six months down the road after further security awareness training and tuning of technologies (spam filters, etc.); I’ve seen this number drop by as much as 30-percent,” Blow said.

“Blue teams can’t be expected to be the only folks defending against Phishing – humans are involved and we all know humans aren’t infallible. If I performed a Phishing campaign and the client had a 20-percent failure rate, I wouldn’t necessarily count it as a ‘win,’ but I would say that the company was quite successful in recognizing and defending against Phishing attacks.”

We asked a few experts a single question, their answers are below:

When it comes to Phishing, if an attacker launches a focused campaign and had a 20% success rate, could this still be counted as a win for defenders? If not, why do assessments count 80% as a solid success rate?

Josh Grunzweig, Threat Intel Analyst for Unit 42, Palo Alto Networks:

An 80% success rate can certainly be treated as good results depending on the situation. For example, if 80% success is purely users not navigating to malicious links or opening malicious attachments, and the organization has technical controls to limit or block the remaining 20%, that would certainly be considered a success.

However, if that 20% who fell for the phish resulting in a compromise of their machine, credentials, or personal information, then I personally wouldn’t treat that as good results. It really comes down to what other controls are in place within an organization to stem any potential compromises that may occur through a Phishing attack. While users should be expected to be aware of the attacks they may face via email, there should also be other defensive, technical controls in place to stop attacks in the event a user is tricked by an attacker.

Ryan O’Leary, Vice President of Threat Research Center, WhiteHat Security:

[A] 20% success rate on a Phishing campaign is a huge win for the attacker. Phishing usually targets an incredibly large number of victims, often tens to hundreds of thousands of people. A 20% success rate even off of 10,000 people is still 2,000 users that have been victimized.

The reason assessments count 80% as a solid success rate is that Phishing attacks are often quite difficult to prevent. The attackers change up the networks they come from, the emails they use and the format of their attack. Any reduction in the amount of Phishing attacks that are successful is fantastic, but there’s still a long way to go.

Troy Gill, manager of security research, AppRiver:

When considering the mitigation of Phishing campaigns, it is important to remember that the attacker only need a small percentage of return on the many attempts that they send out. In many cases even having gained just one person’s credentials at an organization can be the opening they need to breaching the entire organization. Therefore, allowing 20% of attacks to make it through is an unacceptable rate. Of course capturing 100% of these attacks all the time is probably an unrealistic expectation as well.

However in the area of user awareness training, these figures might be a bit more encouraging. If the end users at your organization are able to correctly identify 80% of Phishing attacks then that seems like a decent rate that can certainly be improved upon.

Tom Landesman, security threat researcher, Cloudmark:

The overall adoption of this baseline was probably informed, at least in part, by the antivirus industry and its standards. If you were to use a country as a target, having a 20% success rate infecting targets within the borders of a country would be a pretty poor return and labeled as a relatively “safe” country.

One downside of this is that it assumes all machines are equally valuable (probably true for most bots), whereas with Phishing this is almost never true. From my perspective, 20% is not a win for the defender based on the very immediate consequence of what can happen if just one person in an organization is phished successfully. [Salted Hash] had some great coverage earlier this year about what can happen to (a lot of) organizations when someone in finance gets phished for tax records — and it just kept happening for several months through the U.S. tax season.

Udi Yavo, co-founder and CTO at enSilo:

Any success, regardless of a rate is considered a win for the attacker. The attacker just needs one compromised victim gain success. The fact that 80% block rate is considered a success in terms of the company, demonstrates the failure of the industry. Since it’s not possible to block all infiltrators, the industry has set some sort of “acceptable standard.”

The threat actors mock us with this statistic, since as stated above, they just need that one victim. What we need is to re-consider such approaches and acceptable benchmarks and consider ways to level with the threat actor once within. So that regardless of compromising a system, the threat actor cannot guarantee their success of data theft and tampering.

Phishing is a problem that has existed for years, and because it’s a purely a human problem, there isn’t an easy fix.