Having worked in an enterprise security capacity in the industrial and manufacturing sectors, I'm one of the first to admit those two sectors haven't exactly been on the bleeding edge of security innovation over the past decade. The good news, if recent events hold, is the industrial and manufacturing sector appears to be going through somewhat of a renaissance. This is thoroughly exciting news for many of you who have been hearing stout opposition to your efforts.
After what appears to be decades of systematically ignoring security challenges, the recent climate of breaches seems to have shaken something loose. Purse strings have loosened. Boards have begun to ask security questions when they have never done so before. And most of all, I'm seeing several organizations formally hiring CISOs and giving them both accountability and control over the security future of the enterprise.
This makes me hopeful that change is in the air.
The problems with legacy drag
The latent risk many CISOs in sector companies are waking up to is potentially huge. Over the years, they've accumulated large volumes of perfectly siloed equipment, which was fully owned and managed by non-IT groups, and never connected to anything. As technology refresh cycles push forward, many of these previously standalone components (think about a set of computers attached to a machine that takes a raw piece of material and produces a machine-milled part based on a digital drawing, such as CAD/CAM) are getting network cards and are being connected to other shop-floor types of components. The design workstation is being attached to the manufacturing station, to the quality control booth, and all tied together to the raw-material-management system. All over IP.
Also, notice how I specifically pointed out that all these systems have not previously (and in some cases still are not) been available to the IT organization for management and maintenance. This obviously means that security likely didn't know they existed. Now they're being connected to the same flat, non-segmented, layer-2 network that the SAP and email systems are riding on. As these systems were previously managed by non-IT employees (in some cases it was an outside contractor) this translates to a lot of confusion and misunderstanding. Imagine taking one of these ICS systems, such as the assembly line control system, and handing it to someone in IT (and then enterprise security) to manage. The results have not been positive.
The other big challenge (as if we needed another) is that many of these systems easily qualify for the label of ultra-legacy. This means that they're greater than 15 years old and still functioning. In one example we've got a DOS-based application running off of a 1.44MB floppy disk on a 486/DX266 which manages the time cards of ~300 shop floor workers! This technology predates many of you reading this blog post, which means your immediate thought of “Why don't we just re-write this in Python?” is likely to break things in a way that will likely cause ripples through your supply chain and your bottom line.
Planning for the technology-driven future
As one of my favorite CIOs put it, “We need to get with it right now, while our competitors are still largely in the same position, because we are entering a time when industrial and manufacturing enterprises are no longer able to ignore their dependence on technology.” This is so true.
As enterprises start to connect Widget A with ancient shop-floor Thing B, we inevitably find that not only do those combinations create security issues, but the systems themselves are antiquated and unable to provide much in the way of options for a more secure implementation. This means that CIOs are conspiring with CISOs to modernize shop floors, and overhaul technology. Of course, Rome wasn't built in a day and clearly this desire doesn't translate into action as easily as that would appear. Lots of road blocks, integration challenges, and risks to be assessed.
The good news is this is a topic for discussion, and folks like myself and others are being brought in to support these transformations. Again, this gives me a sense that the manufacturing and industrial sector is experiencing an industry-wide renaissance of sorts. An awakening to the needs of innovation requires kid gloves from my fellow security practitioners — as you well already know we get maybe one shot at this.
Looking the future in the eye
Step one of this entire renaissance is understanding in what ring of legacy IT hell you're currently residing. This means spending a great deal of time reflecting inward and doing the equivalent of pulling at strings until yet another mystery unravels. I'm currently in the process with a few of these types of organizations of setting the guideposts for the next 12 months. There are a lot of hurdles to overcome and many engineers and line managers to win over with your charm. As I've already said, we will get one shot at this. The first time you crater a production-line system with a security patch because it needs to be applied for security reasons will likely be your last for a long while. Measure twice, then measure again and test before you make that cut.
The approach you'll be taking is one of assessment, transformation, optimization, management. Figure out where you are, make plans for making it better and execute to plan, slowly raise the bar over time and then make sure nothing falls through the long-term cracks. It's relatively simple on paper.
Your key trouble spots, from my observations so far, will be those legacy systems you've never gotten your paws on, your network, and your user base. In that order.
- Legacy systems. This should be self-explanatory as these are the siloed and previously un-managed or under-managed systems which you suddenly have responsibility for securing since they now reside on your global, flat network.
- Network. Speaking of your network, it may be high time to start thinking about segmenting and compartmentalizing ... this is, of course, much easier said than done—got netflow?
- User base. Your users are likely not accustomed to being ‘managed’ in any traditional sense, and while they've been running successfully with self-managed full admin capabilities, your meddling and trying to lock systems down and define user and admin profile will cause a stir.
Those of you in the manufacturing and industrial sector — remember all that complaining you did that your enterprises didn't find value in what you provided? You're about to get your chance to impress the business with your intimate knowledge of what it is your organization does, and how you should be supporting it going forward. You have a plan, right?
Cross Posted from Following the Wh1t3 Rabbit