I'm Marianne Kolbasak-McGee, executive editor at Information Security Media Group. Today, I'm speaking with threat modeling expert Adam Shostack, who is founder of security consulting firm Shostack & Associates. We're going to be discussing the importance of medical device cyber threat modeling. So, Adam, for starters, when it comes to medical devices, who should be conducting threat modeling, for instance, the manufacturers, health delivery organizations, both, and what type of cyber threat modeling should they be doing?
Both should be threat modeling and both should be using the four-question framework of asking, what are we working on? What can go wrong? What are we going to do about it? And did we do a good job?
And the way they answer those questions is different because the things they're responsible for are different. But the threat modeling technique or the threat modeling framework of those four questions leaves us to everyone who's working on a medical device should be asking, what can go wrong with the parts that I'm working on? And so their answers might include things like, the device doesn't require the end user to change the password, or we have no mechanism for update and we really need one, or all of the software on this medical device is running with full administrative privileges rather than using a more limited account, so an attacker who breaks in doesn't instantaneously get full control over it. The HDOs should be asking, what is this thing I'm buying?
Did they do a good job at threat modeling? Does it have a credible security plan? Can we operate it in our environment? How do we do so securely?
How does it integrate securely? And so there's a partnership that's needed where we need device makers to make things that are secured by design, and then we need them to deliver them in a secure, in a way where it's secure by default, and they help the HDOs roll them out in a secure way. If an organization hasn't started on medical device threat modeling, how should they get started? And I'm assuming that with the FDA's enhanced authority over medical devices for new devices that are being, you know, submitted for approval by the FDA, I would assume that, you know, device makers are now having to do this, perhaps, as part of the FDA process.
And if they haven't started, how can they start? And same with the HDOs. How can they get started in terms of the medical devices that come into their environment? Let's start with the device makers because the FDA has been announcing loudly for years that this is coming.
The new authorities that they got in recent, recent changes to the law that allow them to refuse to accept devices whose security is not up to snuff so that the device isn't safe and effective, are in effect now. They can reject your submission if you are not threat modeling. And the place that I would recommend that everyone start is the MITRE MDIC threat modeling playbook. MDIC is the medical device innovation consortium.
And they, MITRE and Shostak and Associates worked together to create a playbook that helps device makers understand how to threat model, how to get started, and what success is going to look like. Then, as you're developing those technical skills to do the important work, you need to be engaged with your regulatory professionals to make sure that you are doing the work to a degree of rigor, that you're keeping the appropriate records in the device file, and that those files are ready for submission when your device is ready for submission. So in light of the evolving threat landscape facing the healthcare sector, what kinds of threats should be part of this modeling? You had mentioned before, you know, does the device have a password?
Does it not? What type of like external threats should these, you know, this modeling kind of take into mind when the organization kind of assesses this? This is an interesting question because a lot of people are looking at the impact of insufficient security, which is an increase in ransomware. And when I talk about threat modeling and when we talk about threat modeling, we don't necessarily mean the threat actors and their behavior.
We mean the potential future problems, which these threat actors are taking advantage of. They're things like insufficient authentication, insufficient integrity control. And so the threats, we often use a mnemonic STRIDE as we threat model. And STRIDE is spoofing, tampering, repudiation, information disclosure, denial of service, and expansion of authority.
And these threats are things that we as engineers should be looking for. And so we should make sure our devices resist these threats. And that protects us against future problems like ransomware. When it comes to some of the sort of developing medical devices that we see, we're seeing, you know, AI and machine learning enabled medical devices.
We are seeing a lot of robotics sort of medical devices, you know, especially in the surgical area. How does that impact threat modeling? So the fundamental questions, the things that can go wrong are pretty consistent over time. And this is one of those things that's a little tricky for people to understand because we hear about all this new stuff that's happening.
And the metaphor I like to use for this is a musical one. You can recognize jazz or zydeco as a genre of music. And it doesn't change very quickly or very much. Right.
There's a new zydeco song. There's a new country song, what have you. But if you understand the genres, you understand what's happening. And so I think of threats like spoofing and tampering as these genres.
And with AI and ML, and particularly with large language models, we're seeing some new genres of attack that are available against these things. There's prompt injection, as an example. There's memorization where the LLM can be convinced to spit out its training data. And, you know, in my mind, memorization is a little bit of a genre crossover between this ML version of a threat and the classical information disclosure genre of threat.
There's other threats like the one we call hallucination. And we used to just call it garbage in, garbage out. But these one of the big challenges which we face with these new tools is they're so exciting and they give us such intriguing answers that are so believable, so interesting, that we want to set these things aside. And I'm optimistic that the FDA is being very thoughtful about what makes a medical device safe and effective.
And I'm going to be honest with you. I am very, very skeptical about the introduction of machine learning into medical devices. I'm not saying that we can't do it. I am saying that we have to be very thoughtful about do we replace algorithms that are designed by humans, tested by humans in, say, a surgical device with a statistical model that we may not fully understand?
I think that we need to be very careful to say, what should it do? How do we make sure that it does that and only that as we're rolling them out, as we're developing them? And we can use the four questions of threat modeling to say, what are we working on? Incorporating an LLM into a chatbot to talk to patients.
What can go wrong? Oh, my gosh, a lot can go wrong, from biases in what it's doing to making up answers to sounding more convincing than it should. There's lots of answers. But the fundamental reason that we use the broad-based four questions for threat modeling is they lead us to answers in these new situations that we're finding ourselves in.
So it sounds like the AI and the ML or the large language models themselves might be threats, perhaps? You know, I'm going to quote Blade Runner. A machine is either a benefit or a hazard. I don't think of the LLM as a threat in and of itself.
I think of it as an unpredictable element. And that's threatening. But, you know, by itself, the LLM just sort of sits there and says, I'm ready to give you random output when you give me some input. So, Adam, what are some of the common mistakes that entities make with medical device threat modeling and how can they avoid those mistakes?
Oh, great question. The biggest mistake they make is not threat modeling at all. The second biggest mistake is to threat model either late when you're done with the product and there's fewer changes you can make or to have threat modeling be a paperwork exercise. And I like to think of threat modeling as the measure twice, cut once of cybersecurity.
So if you're building a building and you decide we're going to get going early, we're going to pour the concrete today without having double checked where the flags are and where the molds are and all that stuff. And you pour the concrete wrong. It's very expensive to redo. And if you incorporate an LLM into your MRI machine to, I don't know, read read brain scans, you're going to spend a lot of money training that MRI, training that machine learning model, incorporating it in, testing it, and then you're going to discover how badly it does.
Doing your threat modeling late rather than doing your threat modeling at the beginning when everything is on a whiteboard, on a cocktail napkin, so that we can avoid the mistakes that we're going to make. That's the big mistake people make is not getting started with threat modeling by the people doing the development work early in the process so they have the most choices as they gain an understanding of the threats. So, Adam, looking ahead, what emerging sorts of threats or threat issues are most concerning to you when it comes to medical devices and why? You touched upon some of the issues with AI and large language machine learning, etc.
What other things are you concerned about? We're all concerned with legacy. I'm also concerned with how we balance innovation, speed, and security. One of the things that I've learned as I've engaged with the medical device community, and this is obvious, but it's worth stressing, is the people who use medical devices are doing so to cure or treat some medical condition.
And so it's easy for me as a person with a background in security to aim for perfection. And if we aim for perfection, The thing I'm going to be talking about is, you know, I just talked about the need for speed, or the desire for speed. And so I've been exploring how can we use large language models to help us threat model so that we can use their, their valuable capabilities to help us gain an understanding of the systems we're working on, help us learn what can go wrong faster. And I'll tell you, one of, one of the nifty things about this, this upcoming meeting that Archimedes is hosting is it's going to be between the two weekends of the New Orleans Jazz Festival.
And so I used an AI to write some code to create Spotify playlists to get familiar with some of the music that attendees might have a chance to see. And then I used an AI to help me understand what that code did. And I got to say, I was disappointed in the results. For me, it was interesting to explore how LLMs can help us code and deliver code faster.
But then we end up in this scenario where we might not understand the code, and that's okay for creating playlists, but it's not okay for creating medical devices. And so I'm going to be sharing that and some other lessons about how to use AI, what I see as the state of the art in using AI to help us threat model is. And I'm hoping to lead a bit of a conversation about, because I know that FDA is aware of these things and is spending time and energy, but I hope to contribute to that conversation about the current state of the art in using AI in software development for medical devices and more broadly. So I'm looking forward to the event at the end of April, beginning of May, and meeting folks there.
Well, thank you so much, Adam. I've been speaking to Adam Shosak. I'm Marianne Kolbesak-McGee of Information Security Media Group. Thanks for joining us.