The war over SCADA: An insider’s perspective on hype and hyperbole (repost)

HP security evangelist Rafal Los sent me this post an anonymous friend wrote. This fellow has worked for utility companies and has a different take on all the recent hype around SCADA attacks.

It’s worth your time, so I’m reposting it here.

–Bill Brenner

The War Over SCADA – An Insider’s Perspective on the Hype and Hyperbole

Wh1t3Rabbit

[This post is written and sent to me by a close friend of over a decade who is a true industry veteran and insider, and by that I mean, they have direct first-hand knowledge into the security efforts being made on various SCADA power management systems. The person wishes to remain anonymous, for reasons we can probably all appreciate, so please address comments and thoughts through this blog post, and we’ll answer them as they show up, if you have any.]

Foreword:

Over the last few weeks there has been a tremendous amount of hype and hyperbole around SCADA systems, the ‘ease of hacking’, and whether foreign attackers are already in our critical infrastructure causing chaos and failures. While there is a great deal of momentum around critical infrastructure, SCADA systems, and some of the incidents that have happened – all dealing with security … it’s clear many of those speaking the loudest simply don’t understand the topic enough to be authoritative. As you will read below, this creates a panic and unnecessarily so. I will urge you to read this carefully, think it over, and then decide for yourself how you feel about all that is going on out there in the press, and on the wires. Thank you.

–Wh1t3 Rabbit

———————————————–

First, let’s be clear, the security of the electric grid is a serious topic worthy of discussion.

It is true that there are issues, serious issues, that need to be addressed. I am, however, constantly amazed by the number of reports related to the security of the electric grid made without any knowledge of how the electric grid actually operates. The North American electric grid is the largest and most complex machine ever built. To reduce the challenges it faces to a few buzz words and quotes is a gross oversimplification of an incredibly intricate problem.

This oversimplification leads to assumptions that are perpetuated by those who haven’t yet come to fully understand how the electric grid operates, and where the risks actually lie. When considering risk prioritization, the largest risks to the overall safety and reliability of the electric grid are three-fold:

–Natural – environmental, weather, vegetation, human

–Mechanical – equipment age and equipment failure

–Electrical – transmission capacity, load management

Those risks are, in general, not from cyber-based attacks. In the energy industry, everything is measured against impact to reliability, and there are at least five different ways the industry measures it. With names like SAIDI, CAIDI, and MAIFI, everything related to improving reliability revolves around improving those metrics. To date, cybersecurity issues have had no impact on those metrics in North America. This is not to deny that there have been cybersecurity events within the industry, because there have been quite a few, but none have ever impacted the reliability metrics. When doing a formal risk analysis, how much effort should be expended mitigating risk for an event which has never impacted reliability when there are events occurring on a daily basis that do?

This is not a “head-in-the-sand” viewpoint. This is a numerically reasoned viewpoint, based on years of operational history. It is true that things are changing, and that adequate protections must be built into new equipment deployment, lest the excellent track record of utilities so far be tarnished. However, media reports would lead the outside observer to believe that nothing is being done to improve the state of cybersecurity for our critical infrastructure, and this is completely false. A significant amount of effort is being expended in both improving the security of existing systems, and in the engineering of security for new systems. Efforts in industry organizations such as NERC, ISA, IEEE, and NIST are all working to address the concerns associated with cybersecurity for power systems, smart grid systems, and industrial control systems, each within their respective domains.

As for the hyperbole of security for utilities being in a “state of near chaos”, there is very little supporting data for this. References are made to “years of vendors selling point solutions”, “utilities investing in compliance minimums”, and “attackers having free rein.”

As for vendors selling point solutions, this is a true statement, but in and of itself, does not lead to chaos. Vendors sell point solutions in numerous industries, without those industries falling into chaos. A company can implement point solutions from any number of vendors — one for anti-virus, one for desktop firewall, one for network access control, one for identity management — with all of them feeding an event management console, and despite these point solutions, an extremely viable security framework can be built. It simply does not follow that point solutions lead to chaos. It may lead to management headaches, and additional staffing overhead, but these do not equal chaos.

With respect to investing in compliance minimums, this is an interesting statement to make, especially in the utility industry. In general, most utilities are required to comply with the NERC Critical Infrastructure Protection (CIP) standards. The CIP standards, along with many others that NERC manages, are created by the member utilities, approved through a standards voting process, and then “ratified” by FERC. Utilities are audited to these standards, and can be fined for non-compliance, with fines ranging up to a million dollars per day for critical violations. Utilities work very hard to meet these standards, with a strong financial incentive to do so. If there is fault, it lies not with the utilities for meeting the reliability standards set by their governing body, but rather that those requirements may be too low to satisfy some. The same might be said for any other standard, because none are perfect in all respects. Is there room for improvement? Absolutely, but this does not leave the cybersecurity of utilities in a state of chaos. In fact, all utilities with critical assets are likely to have a far more robust security program surrounding their critical assets than many corporations.

The exaggeration continues with the statement “attacks having free rein.” This makes it sounds like attackers are already wandering through the networks of our nation’s electric grid with impunity, and this is just not true. If it were, I think the chaos statement might be appropriate. In the state of the industry today, it’s far from chaos, and the very fact that your lights come on 99.995 percent of the time (the average electric utility customer experiences 200 minutes per year of outages) when you turn the switch is a pretty safe indicator of that fact.

While there are nuggets of truth in the statements, they simply do not support a conclusion of chaos. They do support a conclusion that the industry needs to look carefully to its future safety and security, and ensure that the things they are already doing today are sufficient to protect against the threats of the future. The creation of standards, which seems to have such a high level of visibility at the moment, while important, will not create security.

In the past few days, we have seen two reports of attacks against water facilities. In one instance, the assessment as to the source and nature of the attack is still a matter of discussion. In the other, it is pretty clear that simple security policies were not being followed in that 1) the system was connected to an external network and 2) that the password was trivial. We have seen far more sophisticated attacks against non-critical infrastructure than was in evidence in this attack. Again, these attacks were against the water infrastructure segment, which does not have a federal agency with the same power as NERC does over the energy industry governing its operations. I can say with confidence that in at least the second case, the NERC CIP requirements would have forbidden such a configuration, and a NERC auditor assessing the facility would have recommended fines levied by FERC for the infraction. The issue, as with any network, is not the standards, or lack thereof, but the lack of oversight in the design and implementation of the control network.

In June of 2010, the North American Electric Reliability Corporation (NERC) published a paper titled “High Impact, Low Frequency Event Risk to the North American Bulk Power System”. In this paper, NERC and the U.S. Department of Energy identify three event types that they classified as high risk, but low frequency. These three events are pandemic, geomagnetic disturbance and electromagnetic pulses, and coordinated attack. Coordinated attack in this case was defined as “a concerted, well-planned cyber, physical, or blended attack conducted by an active adversary against multiple points on the system.” The report goes on to say that no such attack has ever been experienced in North America. Run that probability through your risk calculator and see what comes out. This kind of event would be an act of war, and no private utility is able to, or could be expected to, defend against an attack funded by a nation-state. The cost of such defenses could easily double the cost of electricity.

The take away here is twofold. First, it is agreed that the energy industry, and the critical infrastructure segment as a whole, must pay careful attention to the security of their systems, but this is true of any industry! It’s true that the critical nature of the systems make security arguably more important, but whether we are discussing energy, water, telecommunications, transportation, health care, or finance, the security of all of these systems is essential to modern living. In none of the other industries do we see the same level of hand-wringing over standards and interoperability as we are seeing in the energy industry. Why is that? You don’t think it could be because the security industry smells fresh blood in the water with respect to smart grid, do you?

The second take away is that things aren’t nearly as bad as media loves to report. In the energy industry in particular there are already numerous controls in place, and an army of security people working to secure those networks. In addition, NIST, IEEE, and the IEC are all working on standards to help govern the security of communications in smart grid networks. The level of collaboration in securing smart grid systems is unprecedented in any industry, and instead of being lamented as the potential downfall of the electric grid, it should be heralded as a new benchmark for how security should be designed in from the very beginning.

Your lights came on before the smart grid, and your lights will come on after the smart grid, at least they will 99.995 percent of the time. And that remaining .005 percent of the time? Odds are good it won’t be caused by people attacking the electric grid.