Welcome to the Pipe Liners Podcast, episode 92, sponsored by InterSYS Corporation, providers of Poems, the Pipeline's Operations Excellence Management System, SCADA, Compliance and Operations Software for the Pipe Line Control Center. Find out more about Poems at InterSYSCorp.com. Simply means industrial control system. I offer that to hopefully make this a bit easier to listen to and understand.
With that, let's welcome Pascal. Pascal, welcome to the Pipe Liners Podcast. Thank you Russell, glad to be here. I brought you home because I think your experience is a little unique when it comes to cybersecurity.
Maybe a good place to start would be just tell the listeners a bit about your background, how you grew up in business and how you got into cybersecurity as a profession. I actually started my career as a control engineer, Beckett Holland in the late 1990s. And at some point, I was involved with a project over in the US and the company offered me a job. So I took to the Blanche.
I came over in 2003 with a food and beverage industry company, started my career in basically designing automation systems for plants. Cool. It's always interesting when you talk to people because when I got out of the military, it's been a brief period of time working in cryogenics as applied to food processing. So I don't know, maybe I don't think that's a conversation for the podcast, but maybe it's a conversation for another time just to compare notes.
How in your experience is business, classic cybersecurity different than ICS cybersecurity? So ICS cybersecurity differs from regular cybersecurity. And basically with ICS cybersecurity, you don't have the luxury of implementing the latest and greatest in protective controls. So for example, if you have a PLC, it was never designed to house controls that would allow you to implement, for example, password checks or authentication or authorization, apart from that because the installations in an ICS are critical and often need to be up to up time requirements of 99.999 or better percentage.
You don't have the time to implement updates. You don't have the time to do checks on it. Furthermore, it's hard to engage in these ICS installations with like testing or risk assessment because it's very touchy devices often. If you start probing with regular IT scanning tools, they tend to bug up and to fall over.
So it's really hard to get to a security cluster or even do a Ss with your security cluster as a moment. Yeah, I think that's exactly right. I think the other thing to, if you think about how quickly and how often Windows sends out patches, they send out patches pretty much every week. And if you're running a business computer, your IT department's probably applying all those patches to all of your machines every week.
And it's probably happening when you're not even aware of it. It's happening overnight, that sort of thing. Where with ICS, you might have a Nick card on a PLC, and that thing may have been out there 10 or more years and never gotten any kind of software batch. Or not even rebooted sometimes.
Yeah, exactly. I've seen I've seen up time, a time or so of five to 10 years on some of these devices. Oh, yeah, absolutely. I mean, they're designed to do that.
They're designed for that kind of reliability. So there's this kind of, I think, fundamental disconnect between those systems that we can take offline and update and take offline and update and those systems that we really can't take offline. And you made another really good point is that a lot of the things we do to monitor and test and maintain in the business network, if we implement those same tools and approaches in the ICS network, we can actually call this process upsets. And think about it, if you have a database that you accidentally take down on the IT network, oftentimes there will be a cluster of databases.
So when next one will take over, but if you're taking a PLC network, card down, then it takes down the whole process. And most of the time there is no backup for that. Yeah, it's never good to take a process offline and not do that in a kind of stepwise thoughtful or thought in a way. No, trust me, no.
Well, probably all of us that work in automation have some story about the one time we got the significant emotional event of not having it go the way we planned. Yes, I have a story with a quality scanner that I had to have, and I had to try it on the ICS network. And I didn't know that nicely. We had some downtime after I played with it.
Yeah, exactly. So we talked a little bit about how cybersecurity is different than ICS cybersecurity. So my next question is how is standard ICS cybersecurity different than pipeline ICS cybersecurity? So as we said with an ICS network, because it's off proprietary proprietary devices and because it's off limits a lot of times, it's harder to get to, but at least with an ICS with a typical ICS, it's all in closing one building.
So you can put a firewall on the edge, you can make sure that you do your due diligence, so that traffic between other zones within within your business, or like I'm talking about, for example, the office network is all restricted. So you can control that at least at that level within the ICS, it might be a little bit harder, but at least on the edge, you can still protect and monitor your device and your traffic really well. With a typical pipeline installation, you have substations all over the place. You might have substations in remote areas where people don't show up one month at a time.
So if there's anything to probe, or is there anything to infiltrate the attacker that's focusing on that area, we'll have all the time in the world. I've seen pipeline stations where they have an access point sitting there, and basically once you walk up there, you can probe and you can scan and you can try to break the encrypted for that access point until you get in. If it takes a day, if it takes two days, or if it takes two months, it doesn't matter because people often don't show up on there. And then once you do get on there, because because the way most of these pipeline installations are wired up, once you're on the network, you're on the internal network, that means that you can type back to the headquarters, you can type back to the core network for the ICS or for the pipeline the ICS installation.
Yeah, again, I think that's a really good point Pascal is that, and I try to always come up with kind of simple models in my mind of ways to think about this. If I think about a process facility, I can limit the point of interface between everything that's inside the process control system and the telephone network, the internet, to one physical wire going into one port on firewall. But I can't do that in a pipeline environment. In a pipeline environment, I probably have as many connections to the public network as I have sites in my system.
Unless you spend the money to do lease lines to all these spaces, which I haven't found a single pipeline company yet. No, and there's all kinds of reasons for that there. I've, yeah, there's all kinds of reasons. There's actually a whole other conversation around communications and legacy versus future.
I mean, everybody's like, well, IP, IP, because IP makes that way. If I'm inside the company, this is the way to think about it. If I'm inside the company network, and I can use tools to access VLCs and RTUs and flow computers, and I can monitor and change configurations and all that because it's an IP network, and I can do all that from my desk, that means that anybody who gets inside of your network and do the same thing. And like I said, if this insight reaches out to all your substations, and all we do is infiltrate a substation and we're on the internal network.
Exactly. And we can hop back to where all the other devices, yes, good points. Right. So I mean, and that makes, that makes pipelining very unique, because there are so many points where this is true.
So it's a, it's a challenge. I've been, I've been involved with a pipelining organization who crossed the border between Canada and the US, and it makes, that makes it a nightmare to do these kinds of things because you're involved with two different governments as well on top of everything else you have to do. Well, yeah, no doubt, no doubt. So we, before we get on the mic a little bit, we talked about your involvement with CPWE.
So maybe you could unpack that a little bit for the listeners, talk about what that, what that was and a little bit about the process and such, if you would. Yeah. So that's, that actually started when I, when I was involved with a company in Seattle, we started implementing IP networks to replace proprietary protocols like DevOpsNet and ControlNet. And eventually we started tying all of these systems together over IP because we could do it.
And now we could, we could go to corporate and we could program all the PLCs in our remote plants. And eventually I started thinking to myself, huh, if we can get to all of these devices, what's preventing a virus or an attacker of doing the same thing. So I started researching the ways of how to implement that securely. And at the time, and I'm talking 2005 and doesn't think there wasn't very much involved yet.
But eventually I came to an organization called RACOL Automation and they had done some of the work in this area and they had collaborated with Cisco and with bandwidth and come up with a converse plant by ethernet models. It's basically a set of documentations where they describe recommendations on how to properly subdivide your ICS from the, from the office networks and how to tie those together and make it so that you can safely, get information out and safely still control them. So I ended up, I ended up starting to work for them. I worked in the commercial engineering department that did the documentation, the cycles on that.
So I helped them out for a while. But I think CPWA is something that everybody should look upon when you're building a new network because it helps you visualize through the Purdue model and helps you implement security as an architecture. Yeah, that's actually one of the places I wanted to go in this conversation, Pascal. So you talked about, you said Purdue model.
So maybe you could explain to the listeners, what is the Purdue model? So when you start tying the ICS to the office network and then sometimes the office network ties over when to the, to the business network, those are, you're already talking about three levels of the Purdue model. So the Purdue model is basically a model that describes how the inter interrelationships between the ICS network and the, and the office network takes place. So at the bottom, at the bottom line, level zero is where your sensors and your actuators are sitting in the production floor.
Level one would be the HMISA control, those sections, level two would be your SCADA system. And you have a level three is basically your your PI database collection. And then a level three point five, which was an off the thought, is your industrial DMZ or the OT DMZ that makes for a secure communication between the ICS network and the office network. The office network is locally to a plan, for example, that's where your email systems are sitting.
And it's also where your VPN back to the level five headquarter network is. So that's the Purdue model in that case. Yeah, no, that's that's actually really, that's really good. So we'll link, we'll link this up in the show notes and provide some resources.
I think, I think one of the things that's interesting to me about this conversation around the Purdue model and conversion to ethernet and kind of the reality of, you know, many connections to the public network versus very few, I think that very few operators actually have an accurate understanding of their inventory of equipment and their inventory of things attached to their network. I believe that to be true as well. And I blame our relationship with IT on that, because a lot of times OT and IT and OT being operational technology and IT being information technology, a lot of times these are two different entities within an organization. And when an OT person wants a connection to one of his substations, he goes to his IT guys and says, hey, I want to go from A to B, they set it up.
And however they do it, that's the way it's going to be. So there's no conversation of, hey, this is what I'm going to do. What should be, what do we recommend to add on for security in that? I think, and I've seen it, the companies who do OT or ICS security did the best, are the companies that have a really, really well-defined relationship between OT and IT people.
And sometimes it's the same division. Yeah. Yeah. And I think that's absolutely right.
And you know, it depends on the nature of the company and the nature of its operation and its culture and a whole bunch of other things. But you know, it goes to, there needs to be some defined policy and procedure, there needs to be some defined architecture, and there needs to be practice and controls around how the architecture is protected. So, you know, set another way. I could design the perfect architecture, but if I, and I could deploy it, but if I allow people to just connect up how they see fit in the most expeditious way possible, then I might be able to get the work done, but I may be creating other risks.
And you're talking to an OT person here as well. So I was, I was in the same position and whatever it takes to get those two things to talk to each other, whatever it takes to get my process running, that's what we did. And then security wasn't off the thought. We're all guilty of that.
And I think, well, and there's lots of valid reasons why that's actually the case, right? Because when you're, you know, when you're trying to get a process up and running and, you know, producing revenue, you know, that that can often be a very, very important thing to the, you know, the health and sustainability of an organization. Absolutely. And we're not the only ones guilty of it, because if you look, if you look at most people, most organizations that develop IoT devices or software, security is still off the thought because you get a deadline and you need to get your product out to your customers, guess what's going to go first, not security.
Yeah, no, I got to get the, I got to get the new shiny sparkly thing out the market. And you know, security is not shiny or sparkly. No, unfortunately not. So maybe we could talk a little bit about documentation.
So obviously I mentioned the need for comprehensive inventory. We've talked about the need for an architecture that's understood. We've talked about policy and procedure. In your experience, how are organizations addressing those needs?
So I've seen people adapt ID procedures for this where they basically go out to say, how do we do this in the office network? And they take that and they make an ICES into it and they start, they start pumping out these rules and regulations for people to use. That's one way of to do it. But there's also organizations who don't have that and have to start from scratch.
And I think, I think if you, if you have to start from scratch and even if you have to adopt it, I think you should go out and look at what the NIST cybersecurity framework is putting out there. They have great documentation that helps you start this process. Yeah, I absolutely agree. And I know that the API is currently got a committee working to update API 1164.
And where that's a, that's kind of a recommended practice, if you will. I think one of the challenges here is there's lots of frameworks, right? There's NIST, there's NERC CIP, DHS has some stuff. It's really a bit challenging.
So I guess what I hear you saying is if I didn't have anything as I started from scratch, I'd probably start with NIST. And I'm saying that from experience, because the NIST is not just a good standard to follow because it because it covers a lot of stuff, but it has a really good structure of documentation that helps you get to where you need to be. They're not just saying you need to have two-factor authentication in place, for example. They also have documentation that says how you should get there and how you control that and how you check on that.
That's why that's why that's, in my opinion, NIST is probably for, especially for ICs and even for pipeline organizations. That's probably the best one to follow. Apart from that, there's the NIST also comes out with a profile that's dedicated to food and beverage, for example. I don't know if they have pipeline per se yet, but if they have other industries where they basically say, okay, we have the NIST framework, but now we put a little spin to it if you add this a set of documentation to it.
Yeah, absolutely. So if you were going to, if I were Acme pipeline company and I'm starting up a new pipeline operation, I've got a build a network from scratch, what advice would you give me? The first thing would be design, properly design your architecture with security in mind. And with that, I mean, right off the bed, make sure you have all of your systems that need to be talking to each other in a zone.
And a zone is like a subnet on a network, for example, keep stuff that needs to be communicating together and make multiple of those zones. And if there is a need between those guys to between those zones to talk or to talk up, make secure conduits, make secure paths to do that. So basically what I'm saying is design your architecture with security in mind. The benefit of doing zones like that is that you're creating choke points for your traffic.
And if in the long run, you want to put tools on there that help you do inventory, that help you look at packet traffic, you have these, you have these choking points where you put a packet kept sharing device. And it's easy to implement and it's easy to get to get everything that you need to get the next level of security. Yeah, so I'm going to try and put this in my own language, because I'm not, I'm not a network engineer of one of those things at this stage in my career, I'm trying to learn just enough. Maybe normally what I refer to then is if you want to get onto a tool wrote in order to get on the tool wrote and nowadays everybody's getting spas and or they take a picture of your license plate, you have to go to a choking point to get on the highway where everybody can be photographed or everybody can do their pass.
Right. That's how you should design your network as well. Yeah, it's kind of how they do security. This is even old school.
It's kind of how they do security in the surveillance and, you know, statecraft area, right? So they have they departmentalize it. That's what they call it is they so the people who need to talk together go into a room and they can all talk together and they all have the same clearance and they talk about the same things, but getting out of that room and into a different room, that's a whole different conversation. So it's kind of the same thing.
I want the things that need to talk together, put them into zone and let make it easy for them to talk together and limit the points where I go to another zone. Correct. And it has the added benefit of if everything in that zone can freely talk to each other and that's one part of your business. Let's say there is something that is going on in another zone, you cut the excess between those two zones and at least it's not going to cross populate.
Right. While you're continuing your business. Right. Exactly.
So what other advice? So you mentioned choke points. What's important about choke points? So the choke point is where we can collect all of the traffic that needs to go out of the zone or up to the office network, for example, by having a choke point, it's really easy to implement one of the new tools that you might have heard of like clarity or indigy or cyber X.
They have appliances that can sit there and sniff your traffic and they will tell you exactly what's on the network. So they can start inventorying, they can start telling you, Hey, your controller over there as an old firmware, you might want to update it or it says, Hey, somebody on the network is using nmap to scandal of your devices. That's not good. So having those choke points is a really easy way to add these bells and whistles, which I call security monitoring, which in my opinion is the second thing you should do when you design ICS security.
Yeah. So the other thing about a choke point, if I won't, if I have a problem, I have the ability to pull, shut that door and lock it. And now I may have consequences about the ability to share data, but I protect the network that's behind the door. And again, that's where proper architecture comes in place.
If you make sure that in that zone, there's only the devices or all of it, the device that you need to run that part of your production or your process or your organization, then you can safely shut that down. Yeah, exactly. Exactly. Exactly.
And then so the last part is monitoring. And you mentioned a whole bunch of tools. I think that's one of the areas where things are evolving very rapidly and cybersecurity is the tools that are available and the capability of those tools for doing monitoring. We have phenomenal tools right now.
Like I mentioned, clarity and energy and cyber X, these tools can give you an overview in like two weeks time, you can start mapping out or they start mapping out your entire network. And that's good. And that's bad because a lot of people will install these tools and say, okay, now I'm secure. Well, no, because you need to learn from these processes.
They need to learn your traffic, but you need to learn from them because once they start pumping alerts, most people are going to get overwhelmed pretty quickly because they're going to say, this is not right. And that's not right. And then you need to have the processes and the training and the involvement in place where you go and you look at these alarms, they say, okay, this is this is relevant. We're going to look at it.
And this is not relevant. So we're going to tune, we're going to tune the tool to be quiet on that part. So even though you know, if the tools install, that doesn't mean you stop, you stop learning, you stop evolving, it's continuous process that will go on forever. Yeah.
Yeah, I think I think when people say monitoring, it's not really monitoring what it really is is monitor, analyze, respond. Correct. Yes. Right.
Because if I'm just doing monitoring, there's no value in that. So I have data. So what you need to analyze the data, and then you need to be taking continual action to respond to the data. And refer me by somebody who knows your system, because if you're a rockwell shop or if you're a seamen shop, let's say you're a rockwell shop and you get an alarm from that system, it says your seamen speed is vulnerable.
And somebody who doesn't know what you're running will start yelling and screaming and shutting stuff down. But if you don't have a seamen speed, you'll see in your building, then you can safely discard it. That's an oversimplified example. But it's important.
Well, right, you've got to be able to understand what the what the what the data that you're collecting through monitoring, you got to understand what it means to you and to your unique network configuration and your unique process. To tie this quickly back to the pipelining business, I think monitoring right there with Arctic is probably the most important thing you can do for your security, because you're not going to be able to protect all of your sites and all of your devices 24 seven all the time. But you can have a monitoring and a learning system in place that tells you if something is wrong, and at least you can respond to it within a reasonable time. Yeah, I mean, it's the same.
It's a similar thing in physical security. You know, just because I have locks on all the doors and I have cameras on all the doors, I mean, I'm not going to have somebody come through the door. If I have locks on all the doors and I have cameras on all the doors and I don't have any way to know when somebody comes through the door, when it occurs, and if I don't have resources available to respond to that when they come through the door, well, they're just going to come through the door and I'm going to be I'm going to have information helps me find them afterwards. Yeah, and that may not be what I'm trying to accomplish, right?
A little spin on that though. If you do see somebody on your camera system come through the door and you know that he has a key, now you know, you need to replace your locks, right? Because somebody somehow got a hold of your key. Same security monitor and you notice something goes wrong and you detect that it's because if your control is not completely covering everything, now you have the ability to go in and adjust it.
Well, right. And just like putting locks on doors the way I lock up, you know, my car and what I have in my car and the way I lock up my house and what I have in my house and the way I lock up my office and what I have in my office, those things are all different. And if I'm a, you know, if I'm a jeweler, jeweler and I deal in precious metals and gold, what I want to put in place for security is going to be different than I'm a plumber. Correct.
You know, just because the value of, you know, the interest that an nefarious character would have in getting my stuff is different. So I think that's the other thing to keep in mind in all of this is that as you're doing this and you're trying to figure out how to do it, you probably want to approach those things that are most valuable first. Yes, almost vulnerable. Yes.
Right. Well, that's the calculus, right? How valuable, how vulnerable and then respond. And it's where stuff like risk assessments come into place.
You go out and you check your system and you see which ones are the most critical, you just check out which one are the most vulnerable and then you make an educated decision on where you're going to put your security budget. Right. So I want to try and wrap this up. And as I do many times, I'm going to try and put together several key takeaways.
And you can give me a score after we're done. See if I got it. So I think the first key takeaway is that, you know, ICS has got some different challenges and limitations. ICS being industrial control than just my normal business systems and pipelining has some different challenges than just ICS and specifically the fact that I have so many remote sites creating so many potential points of entry into my network.
The second is that I need to have an architecture. The architecture needs to be documented and understood and followed and controlled. I think the third thing is I want to end that architecture. I want to be very clear about where my choke points are basically controlling where traffic moves between zones.
And then lastly, I want to monitor to make sure that the traffic's going through the choke points and understand what traffic's going through those choke points and then take proactive action out of that. So I think I just summarized everything we talked about for 30 minutes. I listened to last part and it'll be done. That's why we put it at the end.
One little correction is that your architecture needs to be properly designed before you implement it. So sit down, make a proper drawing, go to the experts, talk to people and make it to your specific needs with security in mind. So yes. Well, that sounds perfect.
Hey, Pascal. So if somebody wanted to reach out to you and have more conversation about cybersecurity and ICS and pipelining, how would they get in touch with you? I'm a Pascal at Treadgen.com. So that's how they could hold me.
I'm on LinkedIn if you want to find me there. Okay. And we'll of course link all that up in the show notes and there'll be a if you want to go look at the guest page, you'll find Pascal on the guest page. Perfect.
Pascal, thank you so much for joining us. Thank you for having me. It was fun. Thanks.
I hope you enjoyed this week's episode of the pipeline or podcast in our conversation with Pascal Ackerman. Simply visit pipelineerspodcast.com slash went to in yourself in the drawing. If you would like to support this podcast, please leave a review on Apple podcast Google Play or your smart device podcast app. You can find instructions at pipelineerspodcast.com.
If you have ideas, questions or topics you'd be interested in, please let me know on the contact us page at pipelineerspodcast.com or reach out to me on LinkedIn. Thanks for listening. I'll talk to you next week.