In putting together awareness programs for dozens of clients, the potential to integrate phishing simulations always comes up. For the most part, it seems like a staple of awareness programs. But when the concept of phishing is raised, I always ask, “Why?”
Yes, the question potentially costs me money. Also while most people perceive the phishing simulations as a direct way to decrease phishing susceptibility, the decrease might not be relevant or significant. So when I looked at a recent CSO article that asked security experts what they thought “success” meant when it came to phishing simulations, I was a frustrated.
The comments from security experts mostly focused on a reduction in clicking on simulated phishing messages. I assume people believe that if fewer people click on a simulated phishing message, fewer people will click on a real message. That is not necessarily the case. This discussion is actually much more complicated than it appears, and it involves dispelling many myths and specious beliefs about phishing.
What is security success?
Before looking at success in phishing simulations, we must first consider what is success for overall security efforts. First off, there is no such thing as security. The dictionary defines security as freedom from risk. There will always be risk, so security is unattainable. An implementable definition of security is risk management.
Risk management is essentially the act of cost effectively mitigating loss. In short, security efforts are successful if you reduce your loss by more money than your security countermeasures cost. For example, if you invest $500,000 in anti-malware software, and you reduce the costs of loss due to malware by more than $500,000, your security program is successful. If you reduce loss by less than $500,000, your program, or at least anti-malware, failed.
There is a general problem with this measure, as most organizations do not adequately track security-related losses. Without the appropriate metrics, it is hard to prove that your security efforts were a success. However, the principle is straightforward. If you plan in advance, you should at least attempt to gather the appropriate metrics.
The problems with phishing simulations
There are several critical issues with implementing phishing simulations. The first one is the actual receipt of the messages. With all services, you have to white list the messages to ensure they get to the recipients. So, you are testing people with phishing messages that they would never receive, as the white listing is implemented to avoid the messages getting sent to spam files or from being deleted, before reaching the recipients.
Then there is the fact that just because a user does not click on one phishing message, it doesn’t mean they will not click on others. Some people might not click on cat videos, while they would click on a shipping message.
Then there is the sophistication of phishing messages to consider. I can purposefully manipulate the user response rate, if I choose. For example, if I want to show success in the program, I can create a very sophisticated message that uses inside information and is related to some timely event, and get a very high response rate. I would then follow it up with a more generic phishing message, such as a shipping message with poor grammar, and would get a very low rate.
The referenced article states that if phishing simulations get a 10 percent response rate, the effort is a success. As the previous paragraph highlights, a 10 percent response rate can mean little in actual effectiveness, depending upon the simulated phishing message used. However, even if you assume it is the most sophisticated simulated phishing message ever, that means that a significant number of people within an organization will still respond to the message.
More frequently, users begin to recognize the simulated phishing messages and do not respond, not because they are more aware of phishing concerns, but because they are aware of the simulations. Another common occurrence is that if one person detects a phishing message in an organization, they may then warn their coworkers about the message. The coworkers will then know to proactively delete the messages. In more than one simulation I was involved in, companies proactively warned employees that they will receive a simulated phishing message within a given time period for political reasons.
Phishing messages require technical failures to be successful
While security professionals seem to attribute responses to phishing messages as a demonstration of poor security awareness, it is actually a much more complicated issue. Again, there had to be a technical failure for messages to get to the user. More important, just because a user responds to a message, it does not mean that there should actually be a loss.
Click throughs to malicious websites can be blocked. Malware can be prevented from installing. So even if a user exercises poor awareness, there should not be a loss related to the user actions, as the attacker was prevented from achieving their intended goal.
What is phishing awareness success?
Given that reduced responses to simulated phishing messages is not a good measure of success, and even when there is success, you do not know if it equates to a loss, a different metric has to be utilized. To determine that, you have to understand the losses that occur due to phishing.
As real phishing messages typically intend to get people to either download malware or give up credentials, measures of related incidents should be used to determine the success of phishing reduction measures. You are looking for actual measures of phishing success.
The two primary attack vectors for malware is phishing messages sent to users, or unsafe web browsing. Both attack vectors target poor awareness. The losses related to malware are a better measure of phishing awareness than simulated phishing messages.
In the ideal world, you know the cost per malware incident. Assuming that the malware incidents on the network have decreased, you can attribute a legitimate value to the reduction of malware incidents. The reduced loss can be attributed to your awareness efforts, which might or might not be due to phishing simulations.
If the number of malware incidents increases or remains the same, or the severity of malware incidents increases, you then need to determine why that might be the case. It is conceivable that simulated phishing attacks will have no impact on reducing actual losses.
If your organization has other ways of tracking losses that can be specifically attributed to user related actions, or phishing, you can consider incorporating them.
Other benefits of phishing simulations
While I admit that I do not see a direct correlation for phishing simulations to decrease actual phishing susceptibility, there can still be benefits from the simulations. They do however have to be executed correctly to have an impact.
Phishing simulations can get people talking about phishing and security in general. They are made aware of the fact that they can be tricked, so they are more aware of the fact that they can fall victim to an attack. Most people think it will never happen to them. Simulations can create a teachable moment. How the awareness program uses the teachable moment then becomes critical.
If you have samples of actual phishing messages that were used to attack your organization or other organizations in your industry, it could be very beneficial to use those messages in the simulations. If users take an inappropriate action, they can receive the appropriate training, and hopefully decrease susceptibility to similar phishing messages in the future.
Phishing simulations also allow organizations to see how people react to potential phishing messages. Depending upon organizational policies, the desired reaction is to report the phishing message, so that the security team can triage the message, prevent other users from taking an undesirable action, and respond if it is determined that another employee responded to the message. In order for a security team to respond appropriately, they must know about potential attacks in progress, and a simulation might give the team an idea of the overall exposure and improvements that need to be made in reporting awareness.
It can be argued that success with phishing simulations has little to do with the actual number of people who do not respond to a message, but the number of people who properly report the messages. Phishing simulation success could be determined by the number of people who not only do not respond to the message, but also properly alert the appropriate authorities to the existence of the message.
How not to perform phishing exercises
Sometimes phishing simulations do more harm than good. Some organizations send out phishing messages too frequently, which disenfranchises the employee base. What is too frequently? That is hard to determine, but from personal experience, it appears that anything more than once a month is definitely excessive and can irritate your employee base.
If you highlight the “Gotcha” aspect of phishing, it makes people feel demeaned. Security already has negative connotations in most environments. You do not need your phishing program to further alienate people. Consider that phishing is a teachable moment, as is being pulled over by a police officer for speeding. It takes skill to avoid the leaving people with negative impressions, especially when people are told that if they continue to respond to phishing simulations, they may face disciplinary action.
Summary
The issues involved with phishing simulations are clearly much more complicated than people perceive them to be. While it appears that most consumers of phishing simulations are simply looking for a reduced response rate, there are many more issues to consider. Response rates are far from a definitive sign of phishing prevention given all of the potential variables.
There are more issues to consider that are related to psychology, compliance, regulatory, privacy, employee use of personal devices, among others. It can be specific to the culture of the organization, or the industry as a whole. What is important to consider is again that phishing simulations involve many more concerns beyond how many click on the messages.
In the ideal world, you would be able to equate phishing simulations to reduced financial loss. This requires a level of planning that most security teams do not implement. It is however critical to try to do so, if you intend to actually improve your entire security efforts. In the meantime, the cost of malware incidents experienced by the organization is a metric to consider. And as discussed, you must also consider not just a reduction in response rates, but an increase in reporting rates as a true sign of success.
Phishing simulations seem to be misapplied and improperly marketed, with an exaggerated value. It does not have to be that way. Minimally though, even if it doesn’t provide a positive return on investment, please make sure that you properly implement simulations in a way that don’t cause more harm than good.
Ira Winkler, CISSP can be contacted at www.securementem.com.