Skip to content
Episode 166: Alarm Rationalization with Doug Rothenberg episode artwork

Feb 9, 2021 32 min

Episode 166: Alarm Rationalization with Doug Rothenberg

from Pipeliners Podcast Hosted by Pipeliners Podcast

This week's Pipeliners Podcast episode features Doug Rothenberg returning to the podcast for Part 2 of a discussion about alarm management and pipeline safety, this time focusing on alarm rationalization. In this episode, you will learn about how to efficiently perform or support alarm rationalization to create value for your pipeline operation, the importance of gathering together controllers and supervisors to identify the best approach to rationalize the alarms, how automation can be used to optimize controller response to alarms, and more pipeline safety topics. - Access the episode show notes and full episode transcript at PipelinersPodcast.com.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Episode 166: Alarm Rationalization with Doug Rothenberg

0:00 32:36
of matches

Transcript Auto-generated, may contain errors

Welcome to the Pipeliners Podcast, Episode 166, sponsored by the American Petroleum Institute, driving safety, environmental protection, and sustainability across the natural gas and oil industry through world-class safety standards and programs. Since its formation as a standard setting organization in 1919, API has developed more than 700 standards to enhance industry operations. Find out more about API at API.org. And now your host, Russell Treet.

Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time and to show the appreciation we give away a customized Yeti Tumblr to one listener each episode. This week, our winner is Michelle Baker with Burns-McDonnell. Congratulations, Michelle.

Your Yeti is on its way. To learn how you can win this signature prize back, stick around for the end of the episode. This week, Doug Rothenberg returns for the second of a two-part series on alarm management, and we're going to conclude our conversation talking about alarm rationalization. Doug, once again, welcome back to the Pipeliners Podcast.

Thank you, and I appreciate it. Hey, I asked you to come back on and this time to talk about alarm rationalization, because I think alarm rationalization is often misunderstood, and it can be a challenging and yet very rewarding process. So maybe the best way to start is, what is rationalization and why is it important? So, rationalization, first of all, it's a funny word, and so let me change the word in our own mind.

Our alarm rationalization is a basically way to take a design for an alarm system and turn it into reality. What does reality mean? Reality means what are the parameters for the SCADA system or the DCS? To actually initiate the alarm, where the guidelines we provide for the controllers in the control room, to actually respond to that alarm, and all the other parts of the infrastructure that wrap around that, including how to acknowledge alarm, how to silence the alarm, and how to use the alarm response information.

So it's that whole collective thing. Yeah, it's basically everything about the alarm that a controller would need to understand it and respond to it. And the enterprise would need to implement it. And manage it.

Yes, exactly. It's all the data about the alarm. So, I would say, and I'm going to put this out here for you to tell me if you think I have this correct or not. But I would say that the purpose of rationalization is to equip the organization to have alarms that they can respond to in order to mitigate possible adverse outcomes.

I would agree with that. Oh, Ed Gimmett, you're arguing with me at all? That's no fun. Well, wait, we haven't finished the podcast.

Yeah, well, that's right. I sure will hit something. So, I think there's several pieces of this that are kind of easy to understand, and there's some others that are more complicated. I think the easy things are trying to capturing the adverse outcome and capturing what the response should be and what could cause it.

That stuff's really relatively straightforward. But I think alarm severity and getting that right is often misunderstood. So, what did people in your opinion, what do people miss about establishing alarm priority? The first thing, the first misconception is that a lot of people confuse alarm priority with words and decoration.

So, we won't use the words like urgent or critical or low or medium or high. Those are terms which have different meaning in all of our minds. So, what is priority? What is priority basically is the establishment of the level or the concern about any individual abnormal situation.

What does it mean to the enterprise if this abnormal situation cannot be managed by the controller in time, either because he runs out of time, he misses it, he does the wrong things or whatever. So, once we encapsulate that risk, that informs priority. What do we do? How do we do that?

We've foundationally start from the pipeline operator has as part of the infrastructure risk management, how he wants to do risk management. What we do in alarm management, we take his marching orders and definitions and reframe it consistent with what he's doing so that we can use it for alarm management. And so, alarm management priority incorporates what is the exposure to the enterprise to physical loss. What's the financial exposure?

What's the environmental exposure? What would be the loss of reputation if a certain incident would happen and the public finds out about it? So, we take all those risk shaping factors and interpret them for an alarm system. The way we use it is we bring up each individual abnormal situation that we design alone for.

So, let's take a low suction pressure for compressor, let's say. So, we say, okay, we want to protect the compressor for this. What is the risk if we don't do it right? We go and look at the alarm priority levels, we find the level that matches the situation that the compressor will see.

And that's how we establish priority. Yeah. And for people who have a background in risk management or have a background in safety, and particularly if they're putting together the overall safety program for a company, they're pretty comfortable with setting up these matrices of, well, this is how we're going to score risk, right? And this is a very similar thing.

I think in my understanding, or at least in my experience, I always say it, when I've tried to take the alarm management risk matrix and line it up with the overall company's safety risk matrix, that's both helpful and a bit challenging all at the same time. It's helpful in that if you can align it, it's easier to get the company to buy into it and agree to it and understand it and accept it. But oftentimes you've got at least in my experience, you've got to tweak it a little bit to make it appropriate for what's going on in the control room versus overall how the company's operating. Does that, is that what your experience or listening to your experience bucket and this done?

Russell, this is a really good introduction into what I'm going to say next. So in reality, when we start an alarm risk management assessment, we go to the safety guys and or HSE guys and say, hey guys, give me your risk management definitions and things like that. And they look at, well, why do you want to do that? And we tell them why and they give it to us.

And then we work it into an alarm useful form, and usually a matrix with different categories. And we show it back to them and they say, wow, I didn't realize that our information could be so useful. And a couple of other times, they basically, I didn't know that's what it meant. I mean, they were you never heard that exactly, but I've had a couple of experiences where the comment about we never knew that you could use this for anything useful, you know, what they're really saying is we never knew that this had direct application operations at that level.

Exactly. Exactly. It's kind of funny when you think about it, right? It's funny.

And it's also very satisfying because it provides a level of consistency between different parts of an organization that thought they might have been adversaries, but are now contributing partners. Right. Right. So the other thing I like about your work in particular is that if you set this up correctly, when you're going through the, you do a very similar thing, as you do in safety, where you come up with a matrix and the matrix has words in it, and then you put numbers underneath those words.

And then as you're rationalizing an alarm and you're asking what could the adverse outcome be, and you're saying, well, it could be this from a standpoint of safety and this is the most standpoint of environment, this from a standpoint of a reputation, you pick those words, but underneath it, it's calculating a number. And then that number is being used to determine whether this is critical or lower whatever in the standpoint of severity. And I find that extremely resourceful. I also find it really educational, because as you begin to look at how you landed your, how your score is actually landed, you'll find things like, I have too many critical alarms.

I have a bunch of critical alarms. I have no high alarms. Why is that? You know, that kind of thing.

And those are really resourceful questions to ask from an overall just automation and alarming program standpoint about, well, no, why is that? Is that real? Is that because of the nature of the process? Or is it how we constructed our facilities?

Or is it something we got wrong in our risk matrix? So this is a very interesting topic, because the we have to go back to how the controller uses alarm priority. He really doesn't care what names we give them. He doesn't really care whether an alarm comes out at the highest level, or the next highest level, or the next highest level, or the lowest level.

What he cares is, when he gets an alarm of a different, of a priority level, and he gets another one of a different priority level, this tells him which order to work the alarms. And this is fundamental. So if an enterprise ends up, because they've used these definitions, and as you mentioned, converted the definitions into an numerical score, they scored an alarm and they got the value, it turns out they have too many highest priority alarms. We know the way the alarms should break out in order to be useful in priority.

So what we will do is we'll adjust the numerical parts of the alarm priorities. So we get alarms in the categories we've decided at a proportion that makes sense for risk management. And I'll just give the proportions here, nobody has to take notes. But the very highest priority, we only want a dozen or two alarms in that category.

Because at that highest priority, if that situation is not managed, you're really betting the enterprise. So they really stay really staying down. So the next highest priority level, we think they should be about 15% of all the alarms should be in this category. I'm sorry, about 5% should be in this category.

The next highest priority alarm, which is the second from the lowest, should be 15% of the alarms. And 80% of the alarm should be in the lowest category. So from low to high, we have 80%, 50%, 5%, and just a very few. And when they're in this ratio, we know that when an alarm activates of a given priority, it accurately reflects the relative risk between priorities.

Yeah, so it really what you're trying to do is you're trying to set it up so that when the alarm comes in, the controller is going to work the highest severity alarm first. So you're really, what you're really doing is you're saying, if I have something going on and it's more than, you know, one thing and the controller needs to make a decision which to work first, I'm just taking that decision away. And I'm going to pre-sort the work and I'm going to work it by severity, which makes a lot of sense and can provide a lot of value to the controller if that's done well. But it also, there's a couple of other things that in your book you talk about, it thinks really interesting.

One is you talk about taking the overall alarm score and adjusting it a bit, either based on the time I have available to respond. In other words, I'm going to make something that I have less time to respond to, a little higher score than something I have more time to respond to. And I think that's kind of interesting and meaningful, right, in this context. So anyways, you know, it's interesting because I can really talk a great deal and really kind of minutia around this whole idea of priority.

But the dotted line is you're just trying to help the controller do their work more effectively. Exactly. And that's the sole purpose. So let's say, what does it mean when we pick the alarm of the highest party first?

When we go back, in the previous podcast, we talked about some foundational principles for alarm. And one of them is you have to provide enough time for the controller to be able to do the job. Well, if you have two or three alarms activate around the same time and they're different priorities, first of all, it's two or three activated around the same time, you pretty much know that the controller can't work all of them to completion. So which ones would you rather have been work first, knowing that the other ones won't get a chance to be worked with the ones with the highest impact.

So right away, we satisfy our own gut feeling of intuition. Well, if we're going to take a hit, let's take a hit for the lowest priority ones, because that's less of a hit than the highest priority ones. Yeah, and you were talking about a compressor station, right? It would not be uncommon that if I had an alarm at a compressor station, I might have more than one.

Right. Right. So rolling those alarms together by a site has some value because that kind of reduces my alarm count that I've got to work because I'm really only working one upset even though I have multiple alarms. And then the other part about that is I want to focus on the thing that has the most severity associated with it.

So if it's just a matter of I'm going to lose a little production, that's different than I'm going to lose a compressor set is different than I'm going to have an environmental release. Correct. And I want to deal with that thing that has the most impact first. So that's the whole point of the whole risk management.

And you know, what I've seen a number of people do is get very bound up and trying to figure out how to do that. And your approach to taking a risk management and creating a frame and some scores really simplifies and streamlines that conversation. And not only does that, but it makes everybody feel comfortable because everybody's dealing out of the same card deck and they're using the same risk management protocols that both society feels they should be doing and the enterprise does. So everything changes the whole accountability from guesswork to engineering and careful work.

Right. So one of the things I want to talk about is this because the thing about doing this rationalization is there is a very long piece of rope that is the entirety of all the things I could put into the rationalization. How do I know what to rationalize and how do I know how far to go? Okay.

So what Russell's really asking is that if I have a pipeline operation and I've got a controller sitting there and I've got an engineer looking over a shoulder and the engineer says to the controller, you know what? I can alarm almost anything. And I can do it. I can let me and what he really means is I can have maybe 15,000 configured alarms for you.

You have a flow? I could do a high, high, low, low, low rate of change. I can do all these things for you. And so alarm management basically has to say, okay, slow down.

Let's take this capability of alarming everything and turning it into a tool. And the way we do that is we don't look at alarms that we could configure. We basically start with one of the ways this pipeline system can get into trouble. And Russell, I think you wanted to talk about abnormal operating conditions.

You want to pipe in with that and then I'll pick out afterwards and finish my discussion. Yeah. So my this is always a challenge because I like to start an alarm rationalization process by getting an inventory of all the AOCs that could occur. And then asking the question, how would I alarm for this specific AOC?

And in pipelining in particular, almost everything is is shown with some kind of pressure change that the challenge is knowing what's the root cause of the pressure change. And so, you know, what do I want? Do I want a high pressure alarm? Or do I want to know that I have do I want an alarm that says, Oh, I have this blockage in the pipeline, right?

What I really want is I want to know that I've got a blockage in the pipeline. I don't really necessarily want to know that I have a high pressure. It's just that that's the way I indicate it. And that conversation can get real circular real fast if you're not careful.

So but I do think it's more valuable because what happens if you start looking at the AOCs much like when you're doing a process as an analysis, what you start finding is, Oh, what I really need is an alarm that does this thing that I don't currently have available to me as an alarm. Okay, so let me try to converge on Russell's points because they're all valid. So the way the way to manage the situation is basically one have the people in the room who understand the problem, the people in the room who have to manage the problem after you understand it. And then they discuss and reach a consensus on what's the best way to tell the controller he has a problem.

And once we tell him he has a problem, what's the best way for him to get himself out of that problem. And that's a negotiation process, that's an expertise process. And that's exactly a cornerstone of what we do when we basically do an alarm response activity information. Right.

So how do you know where to set the activation point? What do you understand? How do you determine the set point, the value at which I want the pressure to enunciate this is high pressure? Okay, so the principle we went on is wherever we set it, we have to give the controller enough time to be able to do the things that we think we know he needs to do and we're going to document that after letting him get enough time to do that.

And it's easy to say because, you know, time, but alarm systems don't work in time, they work in engineering units values. So if you have a high pressure situation you want to protect, you have to find a pressure value that translates into time and you have to make sure that's enough time for the controller to manage it. Well, how do you convert? Well, we as engineers and designers understand a principle of process, variables, conditions, they have a normal way they respond.

So how fast and we're not talking about catastrophic failures here because catastrophic failures, you know, it's over and done with the operator, the controller can't really manage anything, he's we're now talking about disaster management. So in control room, how fast can this pressure change when things are going wrong, but they're going wrong in a reasonable way. And you say, okay, it may change in about maybe 10 pounds every half minute or it may change in terms of flow, a half a barrel, if it's liquid or flow. So you know how fast these things are changing, then we have to say, well, okay, how how hard a job is it for the controller to do the things we need him to do?

We've already identified what he needs to do. He needs to check this, look at that, make this change aside. How long has that taken time? And we okay, maybe take some minute or two for him to be able to do that or three minutes we'll do that.

And we know how fast the process is changing, we know how much time you need, you're multiplying together and you get an answer of, well, it's going to be 20 pounds. So you basically set the alarm point 20 pounds away from the trouble point, which is a point you're trying to protect. And you have a reasonable confidence that in those, in that difference of 20 pounds, there's enough time for the controller to do all he needs to do to be able to manage it, if managing is in the cards. Yeah, where this gets complicated is if I'm looking at pipeline facilities where I have to send somebody to manage the response, and that sending somebody's going to take 30 minutes or 45 minutes for somebody to get there, then it starts getting really challenging trying to determine, well, this is going to move this fast and I've got 30 minutes and now you start getting into, well, is it even operable at that level?

But it does point out that this calculation of time is critical from a warm management and rationalization sample. Correct. What it also does is it informs that situation which you just said, if you need to send somebody out in a season of half an hour, an hour away, or even longer, and in that period of time, you can't recently expect the process to stay manageable, then you already know that you can't solve this problem with alarm. You have to solve it with some other way, maybe some MOVs in the field, some shutdown, some ways of automatically reducing pressure flow a little bit, until you get a chance to get out there and do something more extensive.

Right. Well, the other thing is managing an alarm is taking a process to a safe state, right? It could be, it could be basically just restoring some abnormality that an adjustment will solve, and you can keep running. Well, that's right too.

That's right too. Yeah. So I want to, I want to transition and talk a little bit about what you think that the world needs in the next generation of a warm management system. So what do you think we need in alarm management?

Okay. So Russell just asked is that if we have an enterprise that's done a really good job of what we now understand is proper alarm management. We've gone through, we've identified all the abnormal conditions, we've identified the best way to do alarm them. We've found the alarms, we've decided on the criticality of the alarm and the alarm at a patient point.

We've decided all the things that control and needs to do. We've got that all in place and working very well. So what's the next step to allow the controller to do even better job of what we've asked him to do? Okay.

So there are a couple of things and these are, these are blue sky things and they, I'm not looking at them from a point of a SCADA implementer like Russell is or DCS, but I'm looking at it from the point of view of a controller. So the first one on my list would be let me make sure that the alarm response information gets displayed to the controller right there now. So if we've already figured out what he needs to do, let's show it to him in an automated fashion. And then number...

I got to insert that we've been doing that in our solution for 10 years now. Right. And that's one of the... I wouldn't say easy, but it's one of the responsible things to do right out of the box.

But I'm going to extend a little bit further. In the alarm response sheet, there's a lot of things like check this and look at that and make sure this to the extent which the SCADA system or DCS can actually do that checking, like check for the level and something, we should do that automatically. We shouldn't let the controller have to go and find that manually. If we've got sensors, we've got measurements, or if it's important to check it, let's have the alarm response set up, do part of that checking.

Okay. So the next one I think remembering the time is the commodity for controller. If you had enough time to do anything, everything is possible. So safe operations is really time management.

So what I would do is to take the alarm information on a SCADA system and every alarm is displayed, I'd have a countdown clock which says here's how long it's going to take until something bad happens. Remember, we already have to know that information when we set the alarm activation point. So we already have it. So let me just let's just let it to the operator.

Another thing is that when the alarms come up and the alarms change, let's say the operator's controller is working on alarm A, alarm B comes in, we might want the alarm summary information in the SCADA to automatically reorder the alarms for him. So he sees which alarm poses the most risk in terms of for the enterprise and not running out of time. One of the big problems in pipeline operations is keeping track of all the equipment that's not working up to snuff or maybe out of service or reduced service. And usually around those things, there are problems that develop.

So let's keep track of all those kinds of things in a more organized automated way. So either we tell the controller, hey, this is not working so well, so watch it more carefully, or we may want to inform the alarm system. Hey, this is not working so well. Do you want to change alarm limits to make sure it's not getting out of management?

Yeah, that starts getting that starts getting a little more difficult, you know, changing alarm limits. Generally, that's under some kind of MOC and controlled process. So doing that on the fly is that that infers some complexity that is beyond just what you can do in software. Yes, but remember, if you're going to do it, it has to be pre-engineered.

Yeah, right. Pre-engineered also means pre-approved. Yeah, that's good. Yeah, yeah, but it's yeah.

Cool. So Doug, again, I appreciate you coming home and talking about all this. What do you think that my planners ought to take away when it comes to rationalization? I think that even though it looks hard to do, it looks like it might take a lot of time, but there are very efficient ways of doing it with the right amount of guidance, it can be done very expeditiously.

And everything you put into it, you get out in spades. So I would absolutely agree with that. And I think that the thing that is often the most effective approach is get to something that's workable and put in place the organizational support and culture so that you can be continually improving it. And the people who have the best ideas for improvement are the ones that are on the consoles working the alarms.

Exactly. And the team that closely supports them. Exactly. Right.

Well, listen, this has been great as always. And thanks for being on the podcast. Thank you for asking me and those of you are listening. Thank you for listening.

I hope you enjoyed this week's episode of the Pipelineers Podcast on our conversation with Doug Rothenburg. Just a reminder before you go, you should register to win our customized Pipelineers Podcast Yeti Tumblr. Just visit pipelineerspodcast.com slash win to any yourself in the drawing. If you would like to support this podcast, please leave us a review on Apple Podcast, Google Play, or whatever smart device app you happen to use.

You can find instructions at pipelineerspodcast.com. If you have ideas, questions, or topics you'd be interested in, please let me know either on the contact us page at pipelineerspodcast.com or reach out to me directly on LinkedIn. Thanks for listening. I'll talk to you next week.

Frequently asked questions

How long is this episode of Pipeliners Podcast?

This episode is 32 minutes long.

When was this Pipeliners Podcast episode published?

This episode was published on February 9, 2021.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Pipeliners Podcast episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.