The War Over SCADA - An Insider's Perspective

Saturday, November 26, 2011

Rafal Los


The War Over SCADA - An Insider's Perspective on the Hype and Hyperbole

[This post is written and sent to me by a close friend of over a decade who is a true industry veteran and insider, and by that I mean, they have direct first-hand knowledge into the security efforts being made on various SCADA power management systems.  The person wishes to remain anonymous, for reasons we can probably all appreciate, so please address comments and thoughts through this blog post, and we'll answer them as they show up, if you have any.]


Over the last few weeks there has been a tremendous amount of hype and hyperbole around SCADA systems, the 'ease of hacking', and whether foreign attackers are already in our critical infrastructure causing chaos and failures. 

While there is a great deal of momentum around critical infrastructure, SCADA systems, and some of the incidents that have happened - all dealing with security ... it's clear many of those speaking the loudest simply don't understand the topic enough to be authoritative. 

As you will read below, this creates a panic and unnecessarily so.  I will urge you to read this carefully, think it over, and then decide for yourself how you feel about all that is going on out there in the press, and on the wires.  Thank you.

--Wh1t3 Rabbit

First, let's be clear, the security of the electric grid is a serious topic worthy of discussion.

It is true that there are issues, serious issues, that need to be addressed.  I am, however, constantly amazed by the number of reports related to the security of the electric grid made without any knowledge of how the electric grid actually operates.  The North American electric grid is the largest and most complex machine ever built.  To reduce the challenges it faces to a few buzz words and quotes is a gross oversimplification of an incredibly intricate problem.

This oversimplification leads to assumptions that are perpetuated by those who haven’t yet come to fully understand how the electric grid operates, and where the risks actually lie.  When considering risk prioritization, the largest risks to the overall safety and reliability of the electric grid are three-fold:

  • natural - environmental, weather, vegetation, human
  • mechanical - equipment age and equipment failure
  • electrical - transmission capacity, load management

Those risks are, in general, not from cyber-based attacks.  In the energy industry, everything is measured against impact to reliability, and there are at least five different ways the industry measures it.

With names like SAIDI, CAIDI, and MAIFI, everything related to improving reliability revolves around improving those metrics.  To date, cybersecurity issues have had no impact on those metrics in North America. This is not to deny that there have been cybersecurity events within the industry, because there have been quite a few, but none have ever impacted the reliability metrics. 

When doing a formal risk analysis, how much effort should be expended mitigating risk for an event which has never impacted reliability when there are events occurring on a daily basis that do?

This is not a “head-in-the-sand” viewpoint.  This is a numerically reasoned viewpoint, based on years of operational history.  It is true that things are changing, and that adequate protections must be built into new equipment deployment, lest the excellent track record of utilities so far be tarnished. 

However, media reports would lead the outside observer to believe that nothing is being done to improve the state of cybersecurity for our critical infrastructure, and this is completely false.  A significant amount of effort is being expended in both improving the security of existing systems, and in the engineering of security for new systems. 

Efforts in industry organizations such as NERC, ISA, IEEE, and NIST are all working to address the concerns associated with cybersecurity for power systems, smart grid systems, and industrial control systems, each within their respective domains.

As for the hyperbole of security for utilities being in a “state of near chaos”, there is very little supporting data for this.  References are made to “years of vendors selling point solutions”, “utilities investing in compliance minimums”, and “attackers having free rein.”

As for vendors selling point solutions, this is a true statement, but in and of itself, does not lead to chaos.  Vendors sell point solutions in numerous industries, without those industries falling into chaos. 

A company can implement point solutions from any number of vendors -- one for anti-virus, one for desktop firewall, one for network access control, one for identity management -- with all of them feeding an event management console, and despite these point solutions, an extremely viable security framework can be built.  It simply does not follow that point solutions lead to chaos.  It may lead to management headaches, and additional staffing overhead, but these do not equal chaos.

With respect to investing in compliance minimums, this is an interesting statement to make, especially in the utility industry.  In general, most utilities are required to comply with the NERC Critical Infrastructure Protection (CIP) standards. 

The CIP standards, along with many others that NERC manages, are created by the member utilities, approved through a standards voting process, and then “ratified” by FERC.  Utilities are audited to these standards, and can be fined for non-compliance, with fines ranging up to a million dollars per day for critical violations. Utilities work very hard to meet these standards, with a strong financial incentive to do so. 

If there is fault, it lies not with the utilities for meeting the reliability standards set by their governing body, but rather that those requirements may be too low to satisfy some.  The same might be said for any other standard, because none are perfect in all respects. 

Is there room for improvement?  Absolutely, but this does not leave the cybersecurity of utilities in a state of chaos.  In fact, all utilities with critical assets are likely to have a far more robust security program surrounding their critical assets than many corporations.

The exaggeration continues with the statement “attacks having free rein.”  This makes it sounds like attackers are already wandering through the networks of our nation’s electric grid with impunity, and this is just not true.  If it were, I think the chaos statement might be appropriate. 

In the state of the industry today, it’s far from chaos, and the very fact that your lights come on 99.995% of the time (the average electric utility customer experiences 200 minutes per year of outages) when you turn the switch is a pretty safe indicator of that fact.

While there are nuggets of truth in the statements, they simply do not support a conclusion of chaos.  They do support a conclusion that the industry needs to look carefully to its future safety and security, and ensure that the things they are already doing today are sufficient to protect against the threats of the future.  The creation of standards, which seems to have such a high level of visibility at the moment, while important, will not create security.

In the past few days, we have seen two reports of attacks against water facilities.  In one instance, the assessment as to the source and nature of the attack is still a matter of discussion. 

  • In the other, it is pretty clear that simple security policies were not being followed in that 1) the system was connected to an external network and
  • 2) that the password was trivial.  We have seen far more sophisticated attacks against non-critical infrastructure than was in evidence in this attack. 

Again, these attacks were against the water infrastructure segment, which does not have a federal agency with the same power as NERC does over the energy industry governing its operations. 

I can say with confidence that in at least the second case, the NERC CIP requirements would have forbidden such a configuration, and a NERC auditor assessing the facility would have recommended fines levied by FERC for the infraction.  The issue, as with any network, is not the standards, or lack thereof, but the lack of oversight in the design and implementation of the control network.

In June of 2010, the North American Electric Reliability Corporation (NERC) published a paper titled “High Impact, Low Frequency Event Risk to the North American Bulk Power System”. 

In this paper, NERC and the U.S. Department of Energy identify three event types that they classified as high risk, but low frequency.  These three events are pandemic, geomagnetic disturbance and electromagnetic pulses, and coordinated attack.  Coordinated attack in this case was defined as “a concerted, well-planned cyber, physical, or blended attack conducted by an active adversary against multiple points on the system.” 

The report goes on to say that no such attack has ever been experienced in North America.  Run that probability through your risk calculator and see what comes out.  This kind of event would be an act of war, and no private utility is able to, or could be expected to, defend against an attack funded by a nation-state.  The cost of such defenses could easily double the cost of electricity.

The take away here is twofold.  First, it is agreed that the energy industry, and the critical infrastructure segment as a whole, must pay careful attention to the security of their systems, but this is true of any industry! 

It's true that the critical nature of the systems make security arguably more important, but whether we are discussing energy, water, telecommunications, transportation, health care, or finance, the security of all of these systems is essential to modern living. 

In none of the other industries do we see the same level of hand-wringing over standards and interoperability as we are seeing in the energy industry.  Why is that?  You don’t think it could be because the security industry smells fresh blood in the water with respect to smart grid, do you?

The second take away is that things aren’t nearly as bad as media loves to report.  In the energy industry in particular there are already numerous controls in place, and an army of security people working to secure those networks.  In addition, NIST, IEEE, and the IEC are all working on standards to help govern the security of communications in smart grid networks. 

The level of collaboration in securing smart grid systems is unprecedented in any industry, and instead of being lamented as the potential downfall of the electric grid, it should be heralded as a new benchmark for how security should be designed in from the very beginning.

Your lights came on before the smart grid, and your lights will come on after the smart grid, at least they will 99.995% of the time.  And that remaining .005% of the time?  Odds are good it won’t be caused by people attacking the electric grid.

Cross-posted from Following the White Rabbit

Possibly Related Articles:
Information Security
SCADA NERC Smart Grid Infrastructure ICS Industrial Control Systems Water Control Systems SAIDI CAIDI MAIFI
Post Rating I Like this!
Eric Gallant Great post! Two comments come to mind. First, I’m not sure that I agree that reliability is the best metric to use when discussing the vulnerability of the national electrical grids to cyber attack. The author is correct that we have an extremely reliable grid and much is being done by the industry to further improve security. However, the true peril of cyber war has more to do with leverage, political power, fear, uncertainty and doubt. The threat of action becomes a more effective weapon than actual action. The situation is analogous to America’s cold war with the Soviets. In that scenario, you wouldn’t assert that the Soviets and the cold war were not significant threats because we never lost a city to nuclear war. Likewise, you can’t downplay the threat of cyber attacks on infrastructure of national significance because downtime from cyberattack is low and grid reliability is high. Cyber war does not require an overt attack (or hot war) in order to achieve valuable objectives.
Secondly, I agree completely with the author that the electrical power generation/distribution industry is taking significant steps in a more secure direction. In cooperation with ICS-CERT, IEEE etc…a lot of really good work is being done. They may eventually succeed in taking the grid off the list of viable cyber attacks targets. However, ICS/SCADA use is not limited to the grid. While the grid may have its act together, weapons, bad actors and tactics that target ICS systems are quickly proliferating. It is becoming increasingly likely that another, not so proactive, critical industry will be victimized. (As the author points out, water treatment is clearly languishing without proper oversight.) Another example, what about the chemical industry? The threat of a cyber attack leading to a Bhopal gas tragedy type incident keeps me up much later than blackouts.
Many cyber security analysts are more concerned with cybercriminals targeting these industries for purposes of extortion rather than nation states acting against national infrastructure. I have focused my efforts on the data center industry and posted a blog with a threat analysis relative to that industry at:
Chris Blask Good balance commentary. We need to strike a tone between laissez-faire and panic, neither is warranted nor productive.

While I agree that the situation is not nearly as dire as the most breathless would have it, it is equally worth noting that it is not as rosy as the least interested might suppose. In some ways we are currently "statistically comfortable" in the context of the old analogy: one foot frozen in ice and the other stuck in a fire. Actually achieving a common temperature is a more useful goal than adding fuel or refrigeration to either foot.

There is indeed a great deal of activity by a large number of smart folks in regards to defenses. The Bad Guys may or may not be All That, but our Good Guys are nothing to trifle with, either. As I have noted previously, the current state of efforts at the aggregate level is much better than the average journalist tends to reflect and infinitely better than the most sensationalist (recent article: [sic]'Al Qaeda involved in Water attacks...' SRSLY??? Someone got paid to write that?)

The flip side is that historical statistics are not the best metric for forecasting future events, but instead in many cases provide unwarranted comfort. Given the emergent nature of the threat and that very little forensic capability exists in deployed ICS infrastructure, it is both difficult to empirically measure past experience or to predict the specific nature of future threats.

Every military epoch has ended with a great deal of confidence on behalf of the losing parties. This confidence has been robustly supported by previous experience which - in retrospect - did not apply to the new environment. While depth of experience regarding the application of edged weapons is not to be disregarded, applying that knowledge to the disruptive influence of gunpowder requires open-mindedness and concerted effort.

A non-state actor significantly bringing down the grid is a low-order of probability, and state actors are restrained by the ramifications of open warfare. However. Significant incidents with large impacts on limited populations of individuals and entities remain fully within the realm of capability of non-state actors. Not necessarily something to stay awake worrying about for the average person, but enough to make you really not want to be the exception.

Too much hype or too much complacency distract us from our goals. Diligence dictates that we continue to work through the process of cataloging risks and resources and evolving technical and operational solutions, nudging along the laggards while mitigating the emoting of the strident.
Secure ByForce I would have to partially agree with Eric and disagree with Chris. This isn't a balanced commentary IMO this sounds more like an industry insider trying to protect his own turf and he clearly has an view that is biased. While reliability may be six nines, if the possibility remains that someone or some group could take down a critical infrastructure for a few days your six nines stat becomes entirely worthless to me when I have to ration out water from my water heater or throw away all of my perishable food. The mentality of this author reminds me of web admins before code red and the references to political/governmental agencies for evidence to back up his ideas do not fill me with confidence, they actually do the opposite. Also another key point that Eric brings up is that SCADA is not limited to the smart grid. I also don't see how the author could compare the electrical grid to something like the financial industry, their security is tested 24x7x365 internally and externally not just now coming up with standards for security and I could really care less if B of A's ATM network goes down for half a day, I don't bank there and have multiple other streams of finance. I can say from personal experience and from other input that to take this kind of caviler attitude about security is a sure fire way to get schooled the hard way. Unfortunately for us their security incident could impact all of us, not just some companies defaced website.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.

Most Liked