Welcome to the Pipe Liners podcast, episode 76, sponsored by Gas Certification Institute, providing training and standard operating procedures for custody transfer measurement professionals. Find out more about GCI at gascertification.com. This week, our winner is Will Gage with Targa Resources. Congratulations, Will.
Your Yeti is on its way. To learn how you can win this signature prize back, stick around until the end of the episode. Well, this week on the Pipe Liners podcast, Adam Hill with Keptware Technologies Returns to talk about simplifying data collection. Adam, welcome back to the Pipe Liners podcast.
Great to be here, Russell. Thank you for coming back. So we talked last week about Fundamentals of OPC, and this week we're going to talk about simplifying data collection. I think this is a big subject area personally, because I think the way that we've been collecting data to get it up from the field and to the SCADA system and out to the measurement accounting systems, I don't think that's changed much in 10 or 15 years.
So maybe the way to dive into this is just talk about what's the nature of the market as it relates to data collection, what's going on, what's happening. Yeah, definitely. I'm giving another talk next week on this very topic. We work with customers all the time that have very unique architectures and challenge and requirements to collect both real-time and EFM data from devices, PLCs and book computers and whatnot.
The build blocks of the previous podcast will come into play with increasing complexity, where you're collecting data from a multitude of other fields, needing to serve up to data collection, even crossing over into IoT requirements and needing to get data into big data analytics packages. There's a lot of increased complexity that still resides out will, but wrapping around that and putting things in place like a data collector, for example, to streamline the communication bridge, reaching your devices and your client applications could certainly make a lot of sense. One of the things I touch on during this particular topic is what today's SCADA looks like and touching on the IoT architecture of what companies are doing to get data out of SCADA, for example, custom applications and whatnot, and then segueing into different avenues as we push things closer and closer to the edge, if you will. Yeah, so definition of the edge.
That's a good question. The definition of the edge, and I would ask you the same question, because I certainly understand your expertise. We are the software vendor. From our perspective, it's as close to the device.
Definition of device. As PLC, RTU, EFM in the field. Okay, so I actually have a little different definition of the edge. Okay.
This is actually interesting. Yeah, this is my definition of the edge is right by the instrument. I like it. Okay, however, right now, the edge is right behind the field device, the PLC, the RTU, the EFM, right?
Because you don't really have edge devices in our world and all the gas that are actually talking to the instruments, but that's coming. So the edge is basically where the data's at, where the data's coming from. You could ultimately think about the edges being the valve, the pressure, the the next word thing. That's my definition.
I like that. I like it. I'm learning something new every day. So that's good.
That's good. I like to learn. Learning is excellent. Yeah.
So one of the things I think about, you know, what makes data collection complicated. So I'm going to just do a little history, right? 20 plus years ago, SCADA in automation was a big monolithic thing, and you had devices out in the field, and you had telecommunications, and you had big iron systems, and all kinds of custom code that would get, you know, that would translate from instruments to numbers on a screen on a computer system. And that all existed as one big kind of monolithic block.
Like a lot of other technologies, what's been happening is that monolithic block has been kind of getting decomposed, right? So the instruments and the RTU are clearly segregated, and the RTU or the PLC has got intelligence in it. Now, communications has become a commodity. It doesn't matter if I'm moving email or data or graphics or what, you know, the communications is just communications, right?
So it's all becoming IP rather than, you know, I've got some cereal, and I've got some IP, and I've got some other proprietary stuff and satellite, you know, it's all normalizing. So I think one of the things that's going on is because communications, you know, become a commodity, and the tools and ability to move the data has become standardized, now there's a whole lot more people that want the data. That's one of the things that's acting would be what I would, what I would assert. I think the other thing that's acting is some of these people that want the data, want it for new purposes, and I'm thinking of particular data analytics.
Everybody's hearing about data analytics. Right. Basically, that's getting a, you know, a time series of data and applying math and statistics to it, to learn something meaningful to support decision making. So I think what's happening is we're actually seeing complexity and requirements evolve pretty rapidly.
And I guess, you know, you're a vendor, you guys working a lot more than just oiling gas. So what are you all seeing in that? Yeah. So oil and gas is definitely our most challenging, unique vertical with regard to connectivity and networks and whatnot.
Not everybody can, like we have customers in Food and Bev and Automotive and whatnot that have very, you know, brand new plants with perfect, you know, perfect ethernet communications throughout the plant and whatnot. So not every, you know, not every vertical is structured that way. So it's exciting to work with customers in oil and gas because of their, you know, unique requirements and the need to get data from sometimes, you know, from sometimes very old equipment that's been acquired through, you know, through, you know, divestitures or acquisitions or what have you. So, you know, certainly reducing the complexity we talked about that resides in the data collection piece.
And putting a data collector in between, you know, your client device layer, your SCADA to your PLC certainly makes a lot of sense. Moving into EFM data collection as well, you know, we're talking about two different types, simplifying two different types of data collection. The unique challenges of needing to, you know, extract EFM data from flow computers in the field that for obvious reasons due to custody transfer purposes is something we're seeing a lot more and more of customers needing to expose that level of data into different parts of the organization. So that's something that's unique.
Yeah. And, you know, I've been doing this a long time when I first started getting this business. It was not uncommon to go to a field site and you'd see one communications drop for SCADA, another communications drop for measurement, and sometimes even another communications drop, if you're at a compressor station or something like that, that was supporting machine diagnostics. So nowadays you go to station, there's one communications drop.
It's all the same data. There's no reason for, you know, three different sets of communications infrastructure to get it. Yeah. So that's certainly one thing that you've seen change.
I think the other thing that you're seeing changes that, you know, there's a lot of points of data concentration where before data concentrate again, 20 years ago when I got the business data concentration was done in PLCs or similar equipment. Nowadays data concentration is almost always done in software. So anyway, so what are the challenges of EFM collection versus real time data collection? That's a really good question.
So I think, you know, I would certainly like to hear your answer to this question to, you know, back to the learning piece as from software vendor to measurement, you know, expert and whatnot. But because that's the biggest thing for us as a software vendor is we really need to take a step back a lot of times and understand that we're not dealing with customers that are, we're not working with a particular customer that it has a line on a plant floor with perfect, you know, communications, wired type architecture. You know, we're dealing with very complicated networks here to pull real time EFM data from various devices. I think the protocols are challenging to support.
One of the things that we found is some of the older protocols for EFM data collection. We've struggled with being able to adapt and adopt support for those. So that's one of the, I would say that's one of the big ones with EFM data collection is make sure we support, you know, all types of flow computers, even the really old stuff. And you know, just keeping up with where the market's headed and some of the unique challenges that people are facing with respect to data collection.
One of the things, you know, we do well is modifying storage requirements for this data, being able to port data into different measurement applications, CSV files, database applications, wherever it needs to go, making sure we do that correctly, you know, and keeping up with standards. But again, from the software vendor perspective, it's very important to keep a pulse on the industry with regard to measurement data, not in addition to real time data, yes, but when it's the measurement piece, keeping up with requirements from these customers, because we're talking about a different type of data that's being collected from a different type of device. Yeah, and that's really what I was kind of going after. And you kind of alluded.
So, you know, for the listeners, you know, Adam and I know each other pretty well and Adam and those, my particular technical expertise, and I'm pretty deep in this whole measurement and custody transfer conversation. So this is a little bit of a kind of a softball for me, if you will. But that being said, I think the challenge, real time data is pretty simple. You know, it's small data sets.
It's a list of values, you know, and a timestamp, you know, it's pressure one, pressure two, temperature two, and a timestamp. So it's kind of simple. When you start talking about measurement data, I'm not really talking more about something you would more typically do in a database. You know, so all of these field devices, the EFMs and the PLCs and all that, they were, if you go back, you know, how they were originally built, they were all originally built as simple time series databases.
They would give you a list of registers and here's the value right now. And you start trying to history all that and bring that into a data set and I'm going to accumulate it for this hour. Here's my total volume and here's my total energy and here's my average temperature and my average pressure and my configuration was this and blah, blah, blah, blah. All of a sudden this is not a long or a simple data set.
It's complex data set. And I have to move the entire block of data. I can't, you know, partial block is not useful or meaningful. So that adds a lot more complexity.
So these EFM protocols, they need to be more robust. They need to be more reliable. And oftentimes they're, they need a more reliable communications network as well to communicate. So that'd be kind of my take on it.
So right now the, you know, the method of doing the EFM collection would be, I put in some kind of communications network and then I install a server someplace and that server is reaching out to all the devices and bringing the data back. So the server tends to be centralized. What do you think the future holds for this? Where we had it?
Great question. This is, this is really the nexus of where we're at today, you know, regarding what customers are doing and a lot of these are POCs where they're just trying new things, different, you know, different ideas, different architectures are being implemented. But really, you know, the current architecture for collecting real time data and EFM data for SCADA and measurement applications would be, you know, draw a dotted line between your PLCs, your EFMs, your RTUs, that's in the field and then above that is your enterprise. So that the data collector resides, you know, within the SCADA, the polling engine resides within the SCADA application or right beside it or what have you, you know, really where things are, you know, are headed obviously with today's SCADA and IoT architecture, there's a lot of custom development and interoperability issues to bridge the IT, OT gap to get data into big data analytics packages and out of SCADA.
Obviously, you know, measurement applications are, you know, involved there as well. There's various challenges to doing that. But one of the things we're seeing is this, you're very familiar with this concept, Russell and have talked with us about it many times, but this tightly integrated loosely coupled application idea, you know? Yeah, we'll gauge an idea to be in a podcast on that comment.
Yeah, love it. On that exact topic, it weighs back. Love it. And these behemoth, you know, think of these behemoth SCADA applications where the polling engine would reside within the SCADA as you know, a lot of that idea of decoupling that piece from the SCADA and then in certain customer architectures putting that south of SCADA to do the data collection piece and then move data where it needs to go from the central data collector.
The other thing we're seeing, which is rather unique, a company in Oklahoma City implemented a way to use our data collector north of SCADA. And so they put a data collector north of SCADA, which had a polling engine, but they, you know, they put the data collector north of that to poll data from eight servers, eight SCADA servers using OPCDA. So this SCADA application had a OPCDA server associated with it. And they were able to use a DA client driver to poll data from each instance, each SCADA aggregate the data and then using MQTT, actually publish data to a broker.
And then from there, other applications within the organization, other IoT applications. Yes. Let's talk about what is MQTT because I bet most of the people that are listening this have never heard that before. Yeah.
I'm certainly not an expert on MQTT. I will put that right out there. I'm more familiar with other protocols, certainly more familiar with OPC. But what MQTT is from just the bare bones and my understanding is it's an IT-centric protocol for data exchange.
And it allows you to move data in different ways. It's a different, you can think of it as a different pipe out of your data collector. Yeah. Okay.
So I'll give you my take. Yeah. And again, I wouldn't put myself out as an expert in MQTT. It is something I've looked at pretty hard.
But I think there's a couple things you have to talk about. The first thing is what is published subscribe versus poll response? Yeah. So most communications in SCADA and automation is poll response.
The host, the computer, ask for the data and the remote response. Okay. So I don't get any data unless I ask for it. That's poll response.
Pub sub is, and if you think about it, your email kind of works in a pub sub way. When I push send, I'm publishing my email. And then when somebody else gets on their client, they get it. So all the people that the data is addressed to get the data and they get it when they ask for it.
But not necessarily, they get it when they ask for it. They could be permanently subscribed and get it as a real-time fee. So it's a fundamentally different way of moving data. So MQTT is a pub sub.
Now pub sub has been around for a long time in the business world. There's whole companies that are built around that. But that pub sub is more of a TCP, IP, business network kind of communications protocol. MQTT is a low profile, low overhead, secure real-time pub sub protocol.
I know I just had a whole bunch of bull load buzzwords there. But bottom line is this, the data, if anybody who needs the data can go to the MQTT broker or server. So here's the data I need. And then whenever that data is available, they get it.
That is kind of creating a, but it's for real-time data, not like a data lake where all the data I ever got is in the data lake. It's just the real-time data feed. Correct. Yeah.
So it's kind of a different way of thinking about this stuff. It is. It is definitely. We're seeing two specific POCs.
POC being proof of concept. So be careful about those DLA. I know. It's exciting.
They're not full-blown adoptions yet. But they're trying new and unique ways. These companies are trying new and unique ways to move the way in which they're collecting the data around and how they're. And you touched on those concepts of poll response versus publish and subscribe.
So changing the methodology a little bit of how data is collected and served up to various applications. And again, you touched on it. We're talking about real-time data, not EFM data. I'm working with a few companies.
And I know very little about this yet, but new companies that are looking to expose even the measurement data to different parts of the enterprise. So there's a lot coming there as well. Yeah. Oh, I agree.
I think this is transformational for our business. It's, you know, right now there's a lot of people that they get data in the morning when they come in and they don't see data again until they come in the next morning. And the reality is very much that that's going to change. Yeah.
You know, we're all going to be able to see the data in virtually real time. And there's going to be lots of people consuming it. And, you know, one of the things that's interesting when you start looking at analytics is I need the data contextualized for the particular kind of analytic. I'm doing it.
If I'm doing a machine analytic, that's a different contextualization. Then I'm doing a capacity analytic. Then I'm doing a profitability analytic. They all use the same raw data, but contextualize a different way.
So that's where something like MQTT can really add a lot of value. Agreed. So anyway, so the purpose of this is to simplify. So one of the things I need the purpose of this conversation is to get some clarity about simplifying.
So what do I need to do to simplify all this stuff that we've been talking about making complicated? Yeah. I think you're, and correct me if I'm wrong, I think you're alluding to pushing things. And by things, I mean the data collection piece closer, if not to the edge, right?
Well, well, maybe certainly that might be one part of it. I think I would, I think I would talk about it at a little bit more conceptually in that the part that most people will agree is the most complex is when I'm dealing with the native protocol and I'm getting it right. That's the part that's the hardest. Right.
So if I have, if I'm not getting the data to measurement, if there's some kind of issue and I'm missing something or something strange is happening with the data, I don't understand why that almost always gets to what protocol am I using? How am I using it? And you know, it gets all the way down to that level of detail frequently. You got to actually understand how that stuff actually works.
And you know, when you start getting the data normalized, well, now I can just move the data around because it's normalized. Right. So I think I think part of it is data normalization. And then I think the other part of it is, you know, simplify the communications.
That would be my answer. And then when you start talking about, well, how do I simplify the communications? Then I think that's where you start saying, well, look, do all the data, you know, all the data collection and normalization at the edge. And then after that, use some standard to move it around like in QTT.
Right. So great. You don't have anything to add to all that? You want me to pontificate?
Well, you know, the edge discussion and a lot of, you know, some of the things you just mentioned, I'm, I'm even in a place, you know, at, at Kepware and a lot of us are where we're learning, especially with regard to this industry, we're learning about the way things are evolving within the industry and the requirements of certain customers. So this is, this is very, you know, the whole edge discussion, especially for me is, you know, something that I'm learning about each, you know, each day is very different for us too because we're supporting operating systems that we may not have necessarily supported before. So for even for our development teams and QA teams, there's, there's a lot happening, you know, that's, you know, allowing us to move in this direction. But, you know, things are definitely changing, you know, daily for sure.
Yeah, I, I'm with you. I don't think, you know, a lot of things, we start talking about the edge and MQTT and such as that. There are some people that are doing some stuff with it. Most of what's being done is experimental or proof of concept.
They're trying to understand it and determine what value they can drive out. There's not a lot of real projects. I do know of some, the ones that I know of tend to be more machinery centric, you know, compressor optimization, that sort of thing, they're not really measurement centric, although I think that's common. So it's, it's to me, it's a really interesting conversation.
I think the one thing I would tell listeners to take away when you get into this conversation, I actually got asked a question one time, how much data do I need to get in order to be regulatory compliant? Like how often do I need to get any data per second, every minute, every hour? And that's a really easy question with a really complex answer. I actually did a whole training thing and there's actually a YouTube video out on that.
But not unlike that, this conversation about simplifying data collection, if I put all the same stuff in the field, it gets easier. If I always use an ultrasonic meter and always use, you know, vendor, A's, EFM device, and I always use a cell radio, it gets easier. The problem is there's all kinds of reasons why that doesn't work. Acquisitions and the vestigures and economics and terrain and all that.
So, you know, to the extent you can standardize, it gets simpler. And well, I've had the conversation many, many times, you know, in this domain about business case, that being the specific combination of what are you doing, what equipment are you using to do it, what communications are, you know, everything from what's going on from the instruments all the way back to how the data is being used. Because every time you change a business case, it gets harder. All right.
So I'm going to do the same thing I did to you last time. I want to ask you, what are your three key takeaways from this conversation? Great question. So we talked about a lot of stuff today.
I think the big thing, you know, from an overview perspective is to understand that there's a big difference between real time and the EFM data. That's that's a key, definitely a key takeaway and then some of the elements we discussed today. Also, understanding today's SCADA architecture, the way in which companies are creating, you know, using custom developed tools to maybe get data out of SCADA and into other applications, but also understanding the options to introduce some of these tightly integrated loosely coupled applications for, you know, getting data, you know, using a data collector in a different way. That's definitely, that's definitely another key takeaway.
The final takeaway is that we're, you know, moving to the edge. Certainly, you know, there are applications out there and tools and an educated way devices that folks are using. So it's an exciting time and it's going to be exciting within the next, you know, five years or so or even less to see where things are headed. Well, I think the other thing, and I think we alluded to this a little bit in the conversation is that, you know, as things move to the edge, the distinction between OT and IT is going to blur.
Yeah, exactly. And the distinction between automation and IT is going to clarify. Yep. Well said.
So, yeah. Well, look, as always, Adam, it's a pleasure. Thanks for joining me. For those of you that are listening to this, Adam has already done his presentation edition.
So we're going to link up some of that content on the show notes, including other information about Adam and such. So we'll build a show notes page so you can find them and you can find the materials that kind of underlie this conversation because I'll tell you the pictures on the PowerPoint are very helpful. Definitely. Definitely.
All right. Thank you, Adam. It was a pleasure. Thanks Russell.
Take care. I hope you enjoyed this week's episode of the Pipelineers podcast and our conversation with Adam Hill. Just a reminder before you go, you should register to win our customized Pipelineers podcast Yeti Tumblr. Simply visit pipelineerspodcast.com slash win to enter yourself in the drawing.
If you would like to support this podcast, please leave a review on Apple podcast or whatever device you use to listen to podcasts. You can find instructions at pipelineerspodcast.com. If you have ideas, questions or topics you'd be interested in hearing, please let me know on the contact us page at pipelineerspodcast.com or reach out to me on LinkedIn. Thanks for listening.
I'll talk to you next week.