ICS Cybersecurity: Water, Water Everywhere

Monday, November 21, 2011

Chris Blask



On November 17, 2011 Joe Weiss published a blog entry regarding a previously unpublished cyber incident at a water treatment facility. Later revealed as a site in Illinois, the attack persisted for several months before being identified and resulted in the destruction of pumping equipment.

In response to what was seen as a “nothing to see here” response from DHS an engineer by the name of pr0f penetrated the industrial control system of South Houston, Texas, and published evidence of his success.

Since then there have been numerous articles and events that have driven the public conversation about the security of the cyber systems at American water treatment facilities. The question at hand is whether this moment of attention will result in any improvements in cybersecurity of the nation’s water supply.

We have reasons to worry and reasons to hope. The reasons to worry are based in a failure to respond in the past. Hope is found in that facilities have more options today, due to the diligent labor of many quietly working behind the scenes.

Water Cybersecurity Timeline

Water was the first industrial sector to suffer a publicized instance of physical harm related to cyber attack. In 2000 a disgruntled contractor hacked his erstwhile client – an Australian sewage treatment facility – and pumped 800,000 liters of raw waste into a nearby river and the grounds of a resort hotel.

The sector has continued to be quite visible as an example of the escalating rate of cyber risk ever since. In 2006 a Pennsylvania water treatment plant was infected with malware. In 2010 a man posed as an army engineer to get information on water treatment operations in North Carolina. On March 29th of this year the Los Angeles Times ran a story on a California water system that had been completely compromised by penetration testers in the matter of a few hours.

In the last case, had the penetration testers been actual attackers there would have been nothing to prevent them from increasing the amount of chemicals added to the water supply and causing a direct, imminent and undetectable threat to human safety.

  • The Australian case was perceived as simply a new kind of disgruntled employee risk, and did not lead to much change in cybersecurity in water systems.
  • The second instance was an accidental infection, the malware author had not intended to compromise an industrial system. No damage to the system resulted and no significant alarm bells rang across the water sector.
  • The social engineering attack in North Carolina – someone physically entering a facility and engaging in face-to-face deceit in order to gain operational intelligence – has been published by DHS to the ICS community as an example of Insider Threat. Again, no demonstrable change in operational security at water facilities.
  • The March 2011 demonstration showed the scale of the potential damage. Anything a facility operator could possibly do with a water system could be done by an attacker. Still just theoretical in many minds, after a series of articles it has largely faded into the back pages.

And Now?

Now that we have an actual attacker intentionally compromising an actual water system and succeeding in causing physical damage, is there enough evidence of real and present danger to motivate significant changes?

Despite expected criticism of government, regulatory and industry processes, a lot has been accomplished since the sewage first flowed across the Hyatt Regency in Maroochy Shire, Australia. Like the rate of attack, growth in the conversation and structure in defense has been at first a low, long line and has spiked in recent years.

Joe Weiss’ own ACS conference and the Energysec Smart Grid Summit West conference this summer were epic and productive events. Both of these events have risen exponentially over the past few years from groups of lone voices in the wilderness to centers of world attention.

DHS and the Department of Energy have invested significantly in ICS cybersecurity in recent years and do not show signs of letting up. The DHS’ ICS Joint Working Group (ICSJWG), DoE NESCO and NESCOR, NIST SGIP and other government initiatives are providing a maturing framework for technical and operational solution development. These structures offer existing assets the water community can tap for processes and solutions.

Industry organizations such as the American Water Works Association (AWWA) will have a role to play in communicating to and coordinating with the water community. The quantity of sites and immediacy of the need will require the skills of industry organizations if we are to address the risk in appropriate scale over acceptable timeframes.

What Should Water Do?

Get Help

Every water facility today is within driving distance of someone who knows a lot about information security. At the very least, every water facility should engage with an information security professional to get some basic advice.

Best practices in information security are well known to be problematic to implement in industrial systems. Patching, password management and such tasks which are (relatively) well implemented in IT environments are often both operationally more challenging and bring risk of unintended consequences not found in office datacenters. As possible, however, water facilities and the organizations they answer to can and should improve the management of basic security practices.

Most information security consultants will not be intimately familiar with industrial control system devices and applications. Where water facility operators work with these subject matter experts on identifying the areas where their skills can be applied within water systems, however, both direct improvements in security posture will be realized and lessons learned for future planning.

Unplug It

Air gaps – however (rightly) criticized as security practices – have their place. While not a long-term or foolproof solution, unplugging any external connections directly into the ICS network - at least while considering how to adequately secure them - might not be a bad idea.

Most water systems will still need to remain connected to their corporate networks for operational purposes. Firewalled or not, these connections must still be considered an active channel for intrusion.

Monitor Your System

Monitoring of water treatment operational networks using common SIEM or log management tools does, however, offer the kind of capability that can address the immediate need for visibility into control system behavior. The ICS networks found in water facilities are deterministic systems with highly predictable behavior.

Current SIEM tools automatically baseline this behavior and alert operators to planned or unplanned deviations from normal operations. Unplanned deviations indicate either the failure of system components due to mundane reasons or attack and require immediate action regardless the cause.

The Open Source SIEM (OSSIM) is a free tool that water facility staff can download as an ISO and boot to an Intel chassis. When connected to a promiscuous span port on an ICS network switch, OSSIM provides a dashboard from which operators can view and validate traffic, establishing “Allow Only” rules that alert on any deviations.

Open Source and commercial SIEM tools will provide the visibility water facilities need, but setting them up and operating them may be outside the scope of their locally available resources.

Outsource Security Operations

Managed Security Service Providers (MSSPs) have evolved reasonably well in the IT sector. A small handful of them have developed services for ICS clients, providing water facilities the option of outsourcing an appropriate amount of moment-by-moment security operations.

A consortium of organizations has come together to create the LIGHTS program to establish a system of certified MSSP who will install and manage SIEM sensors at ICS facilities, providing active monitoring in 24x7 Security Operations Centers. LIGHTS provides facility operators a menu of options including assistance using Open Source security tools, managed security operations, and connectivity with industry awareness centers like Energysec Tactical Awareness Center (ETAC) .

The two sectors initially being addressed by LIGHTS are Electric and Water, with a focus on the large number of sites with limited resources.

Is it Darkest?

There is certainly a storm coming, as pr0f and the as-yet-unknown Springfield attackers have clearly demonstrated. The distant rumble of the drive-by attack of 2000 has been replaced by the crack of the pan-continental smoking motor of 2011.

Aberration has become Theory has become Fact.

But we have not spent all this time building these ships just to scuttle them at the first sign of rain. Most water facility operators should find it within their grasp to catch up with the threat that is now plinking their windscreens. And while no system should ever be considered completely secure, we have it within our grasp to reverse the state of cybersecurity in water management systems from worryingly loose to generally leak free.

Chris Blask is CEO of ICS Cybersecurity, Inc. Chris created the AlienVault ICS Group which produced the first SIEM for industrial control systems, authored the first book on SIEM, was Chief Evangelist at NSS Labs, built the Cisco firewall business, founded ICS specialist Lofty Perch and invented the BorderWare Firewall Server. Today he is Vice-Chair of the OpenSG Security Conformity Group, on the board of the Australian Wind Energy Institute, faculty at IANS, on the advisory boards of several ICS cybersecurity vendors and involved with industry groups such as DHS’s ICSJWG and DoE’s NESCO and NESCOR. 

Possibly Related Articles:
Information Security
SCADA Cyber Security SIEM Attacks Infrastructure DHS National Security hackers ICS EnergySec Industrial Control Systems Water Control Systems Joe Weiss OSSIM
Post Rating I Like this!
Freddie Neuman A correction to the Author:
City is of Houston, Texas, not Nevada. Old Galveston Road is Hwy 3 located in Harris County.
"In response to what was seen as a “nothing to see here” response from DHS an engineer by the name of pr0f penetrated the industrial control system of South Houston, Nevada and published evidence of his success."

Chris Blask Thanks, Freddie. That "South Houston, Nevada" had snuck into several articles. Always good to triple check the sources. ;~)
Freddie Neuman Chris,
You are absolutely welcome.
Man, wouldn't it have been cool to lure the pr0f Cat in to a honeypot where the reads looked hot and the I/O was not? Saweet! Perhaps, the low hanging fruit is the shaky puddin' of today?!
Aint no more scratchin and sniffin viewed lightly anymore.
Chris Blask Pr0f seems like a Good Guy in this context, though our law enforcement colleagues may beg to differ.

But I agree. Why should every single hacker/researcher have a 100% certainty that every single ICS device or network they encounter is real? If we can lower that to even a 90% certainty it would insert a degree of risk in the attack side of the equation.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.

Most Liked