Skip to content
Siloed Security? Forget AI Adoption episode artwork

Nov 16, 2025 55 min

Siloed Security? Forget AI Adoption

from Invisible Machines podcast by UX Magazine Hosted by Invisible Machines

Omar Santos is a Distinguished Engineer directing AI Security at Cisco. He’s here for a frank conversation about the realities of security in the agentic era. As more software is created on-the-fly by AI agents at the request of humans, security has to become an ever-present layer. Security will be built into complete agent runtime environments and will require constant human oversight and intervention, augmented by the ability to simulate outcomes to avoid risk. Omar is also the Co-Chair of the Coalition for Secure AI, and these are the things he’s thinking about on a daily basis. He sits down with Robb and Josh at the end of a travel blitz that included work surrounding OpenAI’s Stargate Project, a four-year $500b plan for new AI infrastructure in the United States. The trio discuss how the ongoing training of models and the rising demand for inference continue to push the demand for security across burgeoning technology ecosystems. ---------- Support our show by supporting our sponsors! This episode is supported by OneReach.ai — creators of Generative Studio X (GSX), the first complete AI Agent Runtime Environment (V1 circa 2019). Forged over a decade of R&D and proven in 10,000+ deployments, GSX lets enterprises design, build, and orchestrate secure, scalable AI agents and systems. Trusted across healthcare, finance, government, and telecom. Use any AI models Build and deploy intelligent agents fast Create guardrails for organizational alignment Enterprise-grade security and governance Avoid vendor lock-in. Backed by UC Berkeley and recognized by Gartner. Before you build or buy another AI solution, think about getting an AI system. Book a Demo: https://onereach.ai/book-a-demo/?utm_source=soundcloud&utm_medium=social&utm_campaign=podcast_s6e9&utm_content=1 ---------- The revised and significantly updated second edition of our bestselling book about succeeding with AI agents, Age of Invisible Machines, is available everywhere: Amazon — https://bit.ly/4hwX0a5 Chapters - 00:00 - Intro and episode setup 00:33 - Meet Omar Santos and his role in AI security 01:00 - Security as the new programming 02:20 - Coalition for Secure AI and security as a new language 04:45 - Identity, access, and AI agents 06:09 - Scaling models and mega data centers 09:04 - Training vs inference and the compute explosion 12:54 - Budgets, compute, and hybrid human–AI security teams 15:16 - Checklists, guardrails, and spec-driven development 20:00 - From IDEs to agent swarms and background agents 25:19 - CodeGuard, rules for coding agents, and secure SDLC 32:00 - Why doing nothing is the biggest AI security risk 39:30 - Validating AI, AI safety levels, and open source dilemmas 46:00 - Private networks, insider AI agents, and embedded security 51:00 - Simulation, digital twins, and business-wide risk modeling #InvisibleMachines #Podcast #TechPodcast #AIPodcast #AI #AgenticAI #AIAgents #DigitalTransformation #Cybersecurity #AIInfrastructure #AIOrchestration #AIManagement #TechLeadership #Innovation #ResponsibleAI #AIStandards #Cisco #OpenAI #StargateProject #AISecurity #Technology

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Siloed Security? Forget AI Adoption

0:00 55:54
of matches

Transcript Auto-generated, may contain errors

Bye. Hey, welcome back to Invisible Machines. I'm Josh Tyson, a contributing editor here at UX Magazine, and I'm joined again by our resident visionary, Rob Wilson, CEO and co-founder of OneReach.ai. Rob and I are co-authors of the bestselling book, Age of Invisible Machines, and this podcast is where we continue our explorations of agentic AI. Today, we're excited to welcome Omar Santos, Cisco's distinguished engineer of AI security. Omar is also co-chair of the Coalition for Secure AI, which brings together leaders from Anthropic, Google, Microsoft, OpenAI, NVIDIA, and others. Today, we're going to talk about the ways that security is becoming a dynamic

ever-present layer as more and more software is being created on the fly, driven by natural language. The very idea of a security department feels both antiquated and woefully inadequate, and as Omar is here to explain, it's truly an organization-wide endeavor. So we're going to explore how to manage teams composed of humans and AI systems, what the advanced orchestration of AI agents actually looks like, and how increasingly sophisticated simulations will be used to mitigate risk. Enjoy this percolating conversation with Omar Santos. Yeah, we really, Rob actually brought something up kind of interesting as we were prepping for this call moments ago, this idea that security is now the new programming in a way. The act of actually programming software is becoming more and more sort of conversational and ephemeral and apps can just be kind of put together at a moment's notice to accomplish a goal. But over top of all that is the myriad security concerns that arise with technology this powerful and sort of this probabilistic and sort of unpredictable.

And I wondered what your sense of how that interplay works now, like between programming and security and how it all fits together. Yeah, as a matter of fact, actually, that's the perfect timing, even though I have been traveling a bit. But we did a few announcements. I want to say we, of course, I work at Cisco, but I lead the Coalition for Security AI with Google and Drop A, company AI, et cetera. We're trying to see what is the most relevant and timely advice that we can give to the rest of the industry whenever it comes to AI and security. And it's exactly what you mentioned. Security becomes the next language, if you will.

It has to be embedded. It cannot be. So one of the big things that we have been working on, the Coalition for Secure AI, is redefining, one, how we handle identity and access for AI agents, right? Especially making sure that these systems get enough access exactly whenever they need it, just in time permissions model, right? That scenario. I mean, there's no biometrics. Absolutely not. We also started with the typical analogy that if you go back in time and you ask...

you know, one of those blacksmiths and they say, hey, if I can automate, I think that we talked about it in the previous call, but, you know, if we can automate your production line of, you know, horseshoes, and of course they're going to think that you're crazy, they're going to figure out about their job. The underlying technology that actually changed, it was not so much of the horseshoe, it was actually the whole transportation. movement. It's like we're not going to give our AIs a fake eyeball so that they can log in or fake face because that's a security reuse. That'd be like the horse and buggy thing. Again, it's not a replacement to the horse. It's not a replacement to the buggy. It's a whole new vehicle. Yeah, and the challenge is that...

I think that we still have in mind of the problems of yesterday and nobody knows what's going to be in the next 18 months. The scalability of these models now, right? So they're becoming better. We were talking about coding, right? So they're becoming a lot better. That's the kind of returning investment that most organizations are getting. You move faster, you code faster. However, these models are running in what I call, you know, the crawling mode. Why? Because we're about to put them in a jet, which are the Stargates of the world. You know, this huge...

But yeah, what is happening out there? Like what's coming as you're out looking at these massive Stargate data centers? So what we're seeing, what we're talking about is actually the scaling loss on where these models now are behaving pretty well whenever it comes to actually coding. And of course, depending on the metrics that you use, you may get different results with them, but definitely they're increasing. They're going back and we're definitely hitting, we have hit a few walls, but we have actually circumvented them with mixture of experts and et cetera. But when it comes to infrastructure and these new data centers,

you know, the magnitude of, you know, this multi-gigawatt data center nowadays makes you think that these models that we are very impressed and they're changing every week nowadays, how are they going to be? perceive or how they're going to scale in 18 months whenever we start putting them into this infrastructure. So one of the things that we are actually working, and we just announced that, so I'm putting my Cisco hat on, is not about just running models in one data center, but what if you don't have a lot of power and a lot of architectural elements, can we put two or three of these data centers working together, right? And yes, as a matter of fact, we created a new chip called the P200, where now we're putting this you know, these designs of these data centers. So if you have one in Texas, the other one in Arizona, you know, up to a thousand miles, we can actually connect to 51.5 minutes per second. So my first reaction to that is like, oh, we're building Skynet, right? Yeah, totally. I think it's important to point out that a lot of people is, you know, this concept of model is a little vague, right? Like, you know, I have more specific granular version of model. And so when we talk about more infrastructure, we're not just like simply talking about the ability to train these models. We're talking about the fact that the solution to...

an LLM building an application correctly may not just be that the model itself, the underlying algorithm improves, but just the sheer fact that we do it 20 times concurrently and pick the one that worked, right? And so as we think about like scaling data centers and scaling the models, like, is that a new model or is it just the same model, but then threading it, just like you would generate an image and get four versions, you know, why not get 10 versions of code and then deliver the one of the 10 that worked, which essentially, yeah, you could say that's a model because someone might slap. a model product name on the fact that it splits the workload and runs it in 10 and then you know map reduces it back to like one result but in essence that's what we're talking about right that compute exponentially now not writing one program or writing 20 at the same time picking the one you like the best or it picking the one you like the best and then that making it more reliable which is which is crazy compute wise because we're really talking about compute solving the problem not data not algorithms but actual just computing hardware yeah you're spot on and

Make it for the audience. The two major categories that you have in these AI workloads, one is training and model that you mentioned, right? And that was very, very computationally intensive and still is. We haven't solved that. Second, it is pretty expensive, right? But it's getting cheaper, which is interesting. And just like you mentioned about training models, you go back one year. I'm not talking about 20 years ago, just one year.

you see the significant difference on the way that you do evaluations and you automate a lot of things. And data, at the end of the day, is the core of AI because, you know, there's no AI without data. However, now you can also generate data. So for training, exactly what you mentioned is that now you have federated learning with different models. You kind of dispose them and you train two or three of the variants of, quote unquote, the model. And then it's not just one model itself. It's a mixture of experts, right? So you have one that is specific, one that is doing distillation.

diffusion techniques for images and so on, one for audio. So that's what you see more in the frontier AI companies. But then the other side is the inference, right? What you mentioned on desktop compute and so on, because you train a model and of course, you know, it's a continuous process because you're not going to just sail in there and just have, you know, GPT-5 for the next 20 years. But at the same time, what the main demand is actually inference, right? And that's the users. Especially now that some of these companies are actually building this agent capability built in within their frameworks. Like you saw last week, actually I was at OpenAI Dev Days last week.

whenever they introduce their agent kit and they introduce, you know, the agent platform. And you're going to see, you know, that Google, you know, follow it after that. And all that is compute, right? Some way it has to run. So it's exactly what you mentioned, the techniques that you're going to see coming up.

in the next few months is around consensus scores. Whenever it comes to test time compute, like you have two or three outputs, so we call it inference. And then at the end, you come up with, you know, some type of consensus score to give you the better answer, whether it's to create a call, to do a meme, you know, or to, you know, do something in the scientific element. So pretty exciting. Yeah. I was thinking like, you know, at some point getting a project done and I think real soon, you just sort of give the system a budget, tell it what you want.

and it'll it'll just use up all that compute and give you the best it can within the confines of like that budget so you sort of say whatever i want to build a game you know of chess and i want to do it for two dollars that's my budget and it's just gonna it's gonna work on it and then at the end of that budget you'll get you'll get the best that it could come up with it might have read it out to 10 different um like you said different models it might use the same model just with different context uh prompting uh both right and um and then ultimately like deliver you something for two dollars uh and if you want a better game like you up your budget and you say you know well what can i get for ten dollars um and that's a it's interesting because that's like how we do it with people that's exactly right you're so spot on as a matter of fact one of the points that we made in that uh announcement in the coalition for square for the next 12 to 18 months is how to create guidelines for hybrid human AI security teams, right? So in our case, it's our security, but it applies to everything. Exactly what you just mentioned right now. Think about it, how humans will do this. You know, I will hire a team. I will give them a budget.

I will have some type of onboarding process to the engineers and project managers and so on. I do some type of evaluation type of criteria. If you're going over budget, I'm putting more precious. If you're going under, I'm going to give you some rewards and so on. All that definitely can apply to an agent. So you have to figure out how to manage teams where some of your team members are actually AI systems. Some of them are not AI systems. And the other one is the scaling norms.

Human in the loop, in my opinion, this is just my opinion, the Cisco Decoalition is kind of dead, right? So it has to be human on the loop, right? So there's a lot of new conversations about self-critics and critic systems and so on, because you're not going to be able to escape. Yeah. This episode is brought to you by OneReach.ai, the runtime for AI agents. Their GSX platform provides turnkey private architecture and tools for building enterprise-grade agentic solutions. Recently, I worked with a team at OneReach to create an agentic companion for the second edition of Age of Invisible Machines. So now when you scan any of the QR codes found in either edition of the book, an agent is there to provide additional insight using a custom knowledge base. There's also a newsfeed feature with updates on agentic AI as well as this podcast and a book resources skill.

that provides clickable galleries of footnotes from each chapter, all of it built using the revolutionary Generative Studio X platform. Learn more at onereach.ai. There's a book. I'm going to pump a book here. It's called The Checklist Manifesto. It's a great read. I think like a lot of the answers are held in there. It's focused on health care.

It really gets into the idea that far fewer mistakes would be made by humans if we operated more in a checklist basis. And I think these are going to be critical for AI agents. It's pre-flight checks, just like we do in airplanes. We worked on the Boeing 787, the Dreamliner, and checks and checklists are key, right? The whole thing, the reason you can fly in a safe environment is because of checklists. It's one of the number one tools. I don't know him, so I'm not pumping his book for any other reason than I think.

as we work with agents within our company and with customers, checklists we find to be absolutely not just instruction, not just guardrails, but you can think of checklists far beyond guardrails, but also inclusive of guardrails. And I think that kind of gets into your point we made in our last conversation where security is not a thing anymore. Everything's security. There's no, if you have a security department, you're going to be completely an outdated company. It's like the 8-track department. You're so spot on, and as a matter of fact, I really hope that...

folks around, you know, that know me listen to this because this is exactly the message that we're trying to actually drive in the last few months. One, if you think about it in any organization where they have a security function, maybe large, I'm spoiled. So mine is large. Some of us are not. But they say, well, we have a security team. Security is taken care of. Let's innovate somewhere else. Now we have to actually flip that 180 degrees saying security has to be embedded.

And I really love, and you can tell that this is actually not even planned, because I want to write a book now to actually see. Apply that to before, during, and after. So before, there's a lot of, I call it hype, but it's actually reality. It makes complete sense. Whenever you ask an agent to do something, most people are thinking about just a normal prompt. Like I'm going to give it a task and then the agent will figure it out. If you do that, you're going to fail. Especially even in coding. So there's this hype now or this term called spec-driven development, which I love. Because you actually have a really good set of criteria.

on what is going to be the product, the end product, what are going to be the techniques of the planning phase. And then if you think about it, if you think about security, you can start embedding some threat modeling in there. At the end of the day, you know, those checks are threat models. The other one is during. So during, of course, guardrails, a lot of these AI coding agents, I'm honing in to coding now, but...

like whether it's cursor codecs cloud code etc now they have rules well you can implement this type of checklist and and so on not only for security but for other things and then after for code review what is the code review a series of also checklists yeah i it's great you're bringing this up because i am being a coder myself i am completely surprised that it's kind of gone like still more sick for me even um first first you add ai to my ide right But now when we get into the world where it's going to make 10 apps concurrently or 10 functions concurrently, the IDE has no place. So now, like, why do I have this IDE? So first there's like this moment in time. Like, it's just a bridge to get you to the other side where I don't need. So what do I need? I need a new IDE that is just all about security checklists. It's like check, check, check, check, check, check, check all of these things that are within our company's policies, within our canonical knowledge, within.

you know within the sort of security check the same thing that a Q quality control person might do in a factory when something rolls out you're like testing and checking and double checking that everything that happened in the factory was correct and you're putting that little stamp on it to say this was checked by person Y so that they know who to hold responsible if that thing goes out wrong so you're spot on whenever it comes to AI coding agents I think that a lot of people were focusing on the ID. And then what you had is a chatbot with an ID. And then after the chatbot, it evolves into workflow files, rules, and so on. But it's still human-driven, so human in the middle. And what you saw with companies like OpenAI and Anthropic, they started actually going outside of the ID. Whenever Codex was released, I think OpenAI was actually a little bit ahead of themselves a bit because it was agents, right? You give access to a GitHub repository, that agent will read all your GitHub issues. So as long as you actually have a good GitHub issue, it's like, go ahead and code, right? I'm going to come back. The challenge was,

the longevity of the, you know, the planning phase, you know, the multi-step operation that an agent will do. Short-term memory. Now, we have agents as of today, probably that will change tomorrow, that can reason for 30, you know, that can operate for 30 hours, right? And as we're going to see now with, you know, the evolution of all this compute, that it's going to be like, hey, at the end of the week, there's going to be 5, 10 agents, you know, 100 consensus scores with tasks. And they're probably going to figure it out.

The questions I never even thought about of asking were given tasks, right? So background agents was definitely a thing. And then, however, since most of the software developers still operating in IDE, that's where you started seeing something that was pretty particular. And this is, again, my opinion, another opinion of it, but you started seeing the windsurf, cursor, you know, the co-pilots of the world in the IDE space and in between the CLI. Why? Because, of course, the way that you interact with your computer is changing. And now I can have an agent that is not only about creating code, but organizing this, creating FAQs. You can have sub-agents now that is not around just creating the code. It's like every time that an agent creates a code, another sub-agent documents it. Another one is actually publishing out. Another one is doing it. Three of them, code review. Another one is a security review. I love this because, like, the Git, to me, it gets, like, the next IDE that at some point...

We don't need Git repositories for the purpose of deploying code because a lot of code is just going to execute on the fly. But what we know is that code is a container for our business logic, for our company rules. AI agents can just read the code to understand the canonical knowledge and processes of a company, not necessarily to read the code to write code or to read the code to update code, but just to read the code to understand the company better because nobody wrote that into a document somewhere. And so now it's kind of like Amazon Q, how it's migrating mainframe to AWS services, right? The same idea that you can have an agent read the code to find out what the onboarding process might be for X, Y, or Z, just to execute it without doing anything to the code.

When can the agent be reading the code and working through checklists too, right? Creating a checklist from the code. Yeah. And right now they actually do that. One of the most powerful things, still human driven, but most people, whenever they inherit code from somebody else, they're really, really careful of not breaking that, right? Right. Especially a new engineer, right? New engineer, you know, you don't want to touch this thing. You're driven to the task. Like you have to be really prescriptive. Now with all these agents, it not only can explain it to you.

but in many different ways that you can actually understand it. But at the same time, you can start documenting all those nodes to itself. So then it can actually iterate in the code later. So if you look at all the planning phases of these agents right now, the planning mode is what most of the companies actually include, is that they create their plan and you can edit it, but they create their plan and then they create subtasks and then they go from there. But you mentioned something extremely relevant.

And it's around the checklist, what I call before doing and after, in security. We are releasing an open source project called CodeGuard. And what it is, is basically exactly that. It's checklist or rules that we translate it and we have automation to translate to all the coding agents. So Windsor, Courser, Copilot, you know, the clouds of the world, et cetera. And what it is, even though all those models have all the security best practices already in their training data, they're non-deterministic. Now you can decrease temperature and be more deterministic, but in some cases actually render some elements, you know, more challenging or I will not call that useless, but more challenging.

But what we do is that if we create a service of rules that think about that is the SDLC of agents, like in the past 25 years, we train a human. And then that human at two o'clock in the morning has to remember the training of not to introduce a SQL injection, but it's already overworked. So it's probably going to skip that anyways, even though if it hasn't, but think about an AI model probably has it here. But then one deterministic element is going to introduce vulnerabilities, and that's what we're seeing nowadays. So if we use this type of rules, quote-unquote checklist, I mean, this is perfect definition that books all those checklists one before, so all the threat models during, so it will...

prevent the AI agent to introduce hardware credentials or signal indexes or the vulnerabilities that we don't even know that it's just, you know, now this technology is going to change. And then thereafter is the code review. And also, if you think about it from a true end-to-end perspective, if it is finding vulnerabilities that was also present in code, you can't even generate the test cases, what we call exploits, right? Which is kind of scary, but it can generate that, but it can test in runtime. Nice.

if that's actually exploitable or not. And that's one of the biggest nightmares for pretty much everybody in security. Yeah, we call it simulation, but you're right, it's just testing in real time. Yeah, that's where NVIDIA is headed too, right? I mean, they're looking at it, I think, explicitly in terms of like robotics and like how you'd be able to train a robot to do the dishes without breaking plates, but it applies to this as well. Like the more robust you can bake in these simulations where it can run through checklists and then explore possible outcomes, the more it can narrow the ability to prevent security threats, right? Yeah, and also what we're doing. So,

whenever we build any agentic solution now we always have an agent uh eval which so the old world was like you run through code you get an error you give the user an error call support or try again later well now that's an agent but let's say it's like an integration challenge an api failed or it failed to understand or read some object it can edit that function and code and re-attempt it for the user in real time. So essentially, you're not like reporting the bug, then waiting for a human to pull it out of a ticket, then go reproduce it, then go fix it. You're talking about someone runs into an issue. It literally fixes it on the fly, repairs, keeps the user moving, and then puts it in a queue for someone to review and permanently add, which is crazy, really. It changes everything, doesn't it? And to your point, now it's not about writing code. It's about...

how to make sure it doesn't do something wrong in that process, something insecure, like the checklist. A hundred percent. As a matter of fact, I think the future, what you're going to see, the technology will take us there hopefully, but is that if I am interacting with a third party or getting a third party technology, I may actually have my evaluators. So it will be my checklist to evaluate your program, your product, et cetera. And at the end of the day, everything in security.

It's, of course, around authorization identity, but it's trust, right? And the trust, whether it's regulated trust or self-inflicted, you're going to have to have that type of checklist. So imagine now if you're acquiring a new product and you have your agents that instead of sending, hey, do a pen test on these elements, you have your agents to automatically evaluate what are the simulations and potential opportunities that an attacker can actually do this on the fly and then do all these simulations in a way that goes way beyond a traditional red teaming or pen testing. Right. Because it's not about... Oh, yeah, go ahead. Yeah, and to close on that, it's not about the vulnerabilities you have at that time, because it's all point in time, all pen testing and retina. It's simulating the different ways that you can actually configure a product, deploy a product, deploy a software, whether it's software as a service and you do it a couple of tweaks, or an infrastructure device that is controlling half of the internet, right? I think that this technology has some possibility of doing that type of simulation in different configuration modes. Something I wanted to hit on, this...

I feel pretty confident when I say this. I know it's a bit of a prediction here, but I think you're living in the future, so you can tell us what's there. I think there's this concept people have that if I don't do anything, my security is fine. There's this false sense of not understanding that the whole world has become a much bigger threat than it ever has, and that all of your traditional software has gone from safe to unsafe. And that doing nothing is like not locking your doors and windows because you lived in a nice neighborhood and you didn't have to. It's not if you do nothing, you'll stay secure. Just to add on to that, too, I think last time we talked, we were talking about this idea of outbound AI in the hands of consumers and how consumer adoption seems to be outpacing organizational adoption significantly enough that that's kind of a hidden threat that's hiding in plain sight, right? Like people could very easily mount all sorts of novel attacks on organizations that they're barely even thinking about.

Yeah, you're spot on. And that strategic update that I mentioned to you, we were talking about the crumb kitties, right? Basically, right now a teenager with 100 bucks and access to Claude can potentially write way better code than probably somebody that was a nation-state attacker to compromise the system, for sure, right? And to Robert's point, if you don't do nothing... you're going to come to the realization that you have to do something because if you're a business, you have to do business. You're in a business and you just stop. And then everybody else is actually doing something with AI. Everybody else will actually expect that you will talk to some type of AI system, some automated system.

to do transactions, it's like going back in time and say, hey, I'm going to actually walk to your office in New York from Raleigh, North Carolina, and then I'm going to hand you a whole bunch of coins to do business with you. Of course, people are going to look at you and say, what? And then even if you go back in time and just 20 years ago and you said, hey, in order for me to do business with you, you have to be API driven. So API first, same thing. Yeah, but also security. I think that's just the part that how vulnerable we all just became.

And I think I fear, when I fear AI in those moments I do, which is sometimes, I don't fear AI itself. I fear the panic buying of AI. I fear companies waiting, thinking they have time, and then finding out that the rest of the world is sort of way ahead and that they're behind. And then they just buy. And they're not educated. Like this new security world isn't about buying a new firewall, right? It's educating people. It's culture. It's so much. It is buying a new firewall, but it's all of that. As a matter of fact, it just brought something that, you know, we're discussing the coalition is if AI is actually going to be creating code way faster and hopefully a little bit better in the future. Typically, whenever you buy, whether it's a security product or some third party, you know, software, it has a hundred features. You use one.

I have an agent that is actually just going to create your new security product. because you can do it and have all those checklists and evaluations in place and everything. And since you're going to just use one of the features, probably the other one is library, you know, and some people probably wants to have that risk to somebody else. So, so time will tell, but definitely, you know, now that more organizations are going to be adopting these coding agents and hopefully they will become better. The notion of actually just buying a security product, you know, in the future, you will probably be comparing that product to your own, you know, quote unquote, you know, internal to your organization, right? And you may even move faster with less money. Yeah.

I've struggled with this a lot. If I do it in the human form, it's like every employee now is going to come with a security guard that stands over their shoulder and reviews everything they're going to do before they do it. And I just don't believe in that anymore. This security has an afterthought or don't worry about security. We'll hand that to someone else to worry about. So examples is these models that appear, what they call them, like LLM firewalls or whatever. I don't use them, so I don't know the names of them. But these are...

These are like supposed to protect you from prompt injections. But what I worry about is that it sort of becomes this checkbox security thing again, where then the customers don't understand the nature of the vulnerabilities that they're protecting against. And they just think plugging in one of these things makes them safe. And that allows the teams that are building these things not to put in the checkboxes, not to put it like that security has to be woven throughout the whole thing from beginning to end. So maybe you have an eval of some kind.

That's fine as long as it's part of a greater design. But this world where you just go buy products, plug it in and go, oh, yeah, I don't have to worry about that because I've got a generative AI firewall, whatever the heck that is. Yeah, I don't know. I wanted to know your thoughts on that. Yeah, I think that if it's only putting guardrails everywhere, you're probably not going to address the host. Definitely not going to address the QT at all. The other one is I look at the defense of AI.

in four different type of personas. You still have to have some type of course monitoring for probably AI agents to look at the system. And if it's something that catastrophically deviates from the norm, you know, if it's definitely a PII in an output or a prompt, block it, right? I mean, that's a given. But the other persona is around validation. And that's another challenge too. Because how do you validate your AI application or your agent, et cetera, you can red team a model to infinity. These are non-deterministic elements.

You can talk to it and generate prompts, malicious prompts, and it will reply to you, right? And then you're going to evaluate that and you're going to continue until the rest of the infinity. We have to come up in the industry with somewhat of a passing score or a feel-good score that you say, okay, I'm going to actually go ahead and put this into production. And unfortunately, there's no standard for that, right? Yeah. You're done with it and you move on. Yeah, I've been talking a lot about this. I think, you know, I like to borrow these things from mature industries. I think it's always surprising me how little transfer we actually do, especially in tech.

you know, looking at other industries that have been doing a long time. And I think about drug trials and drug creation because we've sort of moved into that mode where drugs have like an efficacy rate. Like they're not, they're designed with the idea that there will be some failure. There will be some, you know, some negative side effects, some, you know, benign side effects. It will work on some people. And they have a set of, unfortunately, complex algorithms that we're going to have to lean into complexity on this one that say, well, you're really kind of putting these scales on that say, well, what is the benefit? If 10% live, but 90% have benign side effects, it's okay. If 10% live and 10% die, it's not okay. And so I think we can borrow from an industry that develops drugs, that tests them.

that has very significant analytics around that testing. Specific algorithms that you can employ, there are certain ones where you can minimize the amount of negative side effects. So if it does come into one, it stops and sort of optimize. And then there's also the like, you know, trying to pre-qualify before you give them the drug. And then there's also the simulation of let's simulate this person getting it before we actually give it to them. And so I think...

I know it's an imperfect system, but it sure is a lot closer than our software systems are today. I don't know what your thoughts are. No, I think you're spot on. The elements, even if you look at what Anthropic is doing with ASL and ASL, the AI safety levels, it was not around just... you know, software engineering or AI and so on. It was around the bio, you know, in their case, you know, biochemical weapon, you know, safety levels. There's a lot of safety levels that we can actually adopt that, right? Now, it's a little bit subjective, you know, depending on who you ask and what are the levels of an AI model right now. I think that most people are agreeing that we're at level three. And exactly what you mentioned, and I wanted to pick your brain on something. Imagine if we go to level four, which again, we're going to put these models into a bigger data centers. We're most definitely going to hit level four. And I'm not talking about AGI or super intelligence, you know, I leave that to somebody else.

to talk about, but if we go to ASL 4, should that technology be open source, for example? So in that case, you know, if we go to level 4, should that be open source? Well, you may argue, it's like, well, you know, the more open that we are, the more bigger the ecosystem actually can look at explainability and, you know, actually advance the technology. So again, I'm contributing to open source all my life, so I can vouch for that. But whenever it comes to these levels, it makes you think, right?

To me, it's like, would you ever do open source cancer drugs? Exactly. Exactly. And that was the same analogy. The argument was, you know, if you want to cure cancer, should you be invited in an AI lab and not be, these models do not even be exposed to the internet. Right. So then you start talking about AI sovereignty and you start talking about, you know, who's actually invited or not. And it gets super complicated. Yeah. It's, um...

We get into the world of like, is there a right or a wrong? And the answer is, it really just depends. Someone's going to open source it. So I think we can talk about what we think is right or wrong. And I think to me, it's clear. You don't open source cancer. I mean, you can take whatever you want yourself, but you can't give it to anyone else. But what I would say is it doesn't matter what I think because it's going to happen. So it's more like, how can people protect themselves?

from it right and and um from this sort of thing happening and i think again i don't think there's like specific things kind of as you've been talking about it's more of a mindset it's just it's being there it's staying on top of this stuff carrying not thinking security is boring not looking at this podcast and being like oh security that's not for me It's like, oh, it's for everyone now. And this is all there is to talk about, really. If you want to deploy this stuff practically, I'd say the center of the conversation is security because there is no deploying it without that. But yeah, it's an interesting thought I am always constantly battling with.

you know, should, should have open AI, open source this or not. And, and then I realized like, it's not going to matter. It's going to get out there. Someone's going to do it. Um, so then it kind of comes down to more like their responsibility to create a safe place for it. You know, it's almost like an expanded version of what we're talking about without bound AI in the hands of consumers, right? Like you, you have to start planning for that contingency. Not because I want it, but because I just think it's going to happen. I think the whole private network thing is where this is all going. Like I just, I, people are going to need a safe place.

That doesn't mean that the internet won't exist, but I think it's more than ever, they're going to need a safe network, a trusted private network that they can go to that has specific rules to be able to get onto it. Cisco's world, really. Yeah, and it's a pretty complex set of questions that we have ahead of us whenever it comes to, as you mentioned, somebody is going to distill your model anyways if it's actually out there. It's going to create somewhat of a replica. However, the other thing, as you were talking, is around...

Going back to agents and specifically coding agents, we were talking about that whenever you onboard a person, you go through the onboarding process, you introduce it to a team, you go into the guiding principles of the company, how you can take VTO or PTO, whatever we call it nowadays, and then all that. The onboarding process of an agent has to be with security in mind from the beginning. It's already automatically became your insider threat. So exactly what you mentioned around the security. In my case, I was opening some remarks the other day that I said security is dead, but then I kind of fixed it afterwards because we have to make security easier. It has to be embedded. If we don't make security easier, then we're going to lose the security problem. And if you think about a lot of the security...

capabilities that we embed into coding is very reactive, right? Yeah. A lot of companies, a lot of actually even open source projects, if they do have a nest bomb, you know, they do it in the, at the end of, we have to make security simpler and we have to make it that it's embedded, that it's difficult. Yeah. Not easy task, I can turn on the thing. But now you're thinking about these agents that are now entities that became your insider threat, but they never sleep. That they operate in some cases for 30 hours or for same cases for 30 milliseconds.

They're getting access to the same data that you have. And of course, there's no true one-size-fits-all identity and authorization solutions out there for them. We're getting better. So all those are going to be definitely the questions that we have to address in the next 12 to 18 months. I have no choice. Yeah. Well, Omar, what do you think about like the composition of like a security team and security leads and companies? Because it's easy to imagine, like Rob brought up Amazon Q and we talked to Tim Wood, who was a design lead on that. And he described that a lot of the work for companies using it is like almost like swipe right, swipe left. Like as things are migrating over, it's like, do you need this? Can this look like this? You know, and you're making those kinds of decisions. So you can imagine like someone on a security team spending a lot of their time, like being presented with choices on things that are coming through the system and making those decisions. But then there also might be this element of, that kind of relates to simulation, right? Where your job is to try and

feed the agent, give the agent ideas for things to simulate. It's almost like trying to figure out ways to break the system, but asking like, well, what would happen if someone did this? What happens if party A does this and then party B comes in and does this? Like, how are we compromised? How can we work around that? And it is almost like that efficacy thing where then security is always present and it's always a little bit in flux and it's always being fine-tuned. And so you're kind of always moving towards being more effective in that way. As a matter of fact, the concept of simulation, of course, exists for quite some time. It's just that now we're better at everywhere.

We used to have these digital twins that now people call out about agents of themselves, which is, we're calling these twins of this, digital twins for infrastructure devices. We had that for a long time. We have automation on where can this device be configured in a way? Can this device be a different way? And then you simulate the traffic that was going into it. You simulate our attack and the attack path. And what if you didn't have visibility like NetFlow back in the day, or still today? And so that type of simulation is...

is definitely what is going to be the next attack surface management. What you're doing for attack surface management, you depend on a combination of scans and shodan and OSINT, et cetera. But now you can have an agent that don't sleep that is actually trying to actually see what an attacker would have done or can do from inside, from outside, from the site, from everywhere. And then at the end, you may have these entities that will compare nodes and they will decide what is actually the most effective mitigation technique for you. And then you should be able to spend some time because it's absolutely impossible for a human to evaluate all these type of scenarios. Yeah, you almost want like a paranoid system, right? Like a system that believes it's always under attack. So it's just always on edge. But one thing I'd also like to point out to your point of silos or the breaking down of silos, if you can simulate an attack on your system.

or you could simulate a vulnerability of a physical product that you send out. You can simulate business decisions. You can simulate what happens if you raise the price of something or lower the price of something. These are not in a silo anymore. It becomes a gray area of what is the simulation objective? Is it to secure the product? Is it to sell more of it? It really starts to merge in, and you can see that you can't just relegate this to one silo of the company because it...

You know, the data you need to properly simulate has to come from everywhere. And then the implications of what you learn need to go back out to everywhere. Yeah. It's like a business function. You brought something super relevant. Imagine, of course, we're talking about security here, but we have a simulation of an attack or many, many permutations of an attack against a system organization company.

And then you can simulate what happens afterwards. You know, customers calling your technical assistance center, all that, your costs, you know, how long does it take to remediate based on previous incidents? You can start simulating all that and you have the end-to-end business impact, not just that I launch an attack and compromise. That's super interesting. You're right. Yeah. What is the, what is the cost? What is the risk? What is the cost to mitigate the risk? It's like your own actuarial data on a per vulnerability basis.

It's like crazy. You're like becoming an insurance company. I think that cyber insurance has to move that way, right? Then you create a profile of your company and then they start doing simulations and probably your premium and insurance is actually going like this. They were weak. They were not. Crazy.

Well, this was great. I love this as much as the first conversations. Anyway, this was great. Thanks for coming on. Thank you for having me. Hopefully this was useful. All right. Well, thanks for hanging out with us again here on Invisible Machines. Don't forget to like and subscribe wherever you get your podcasts. Always remember that you can watch this podcast too on our Invisible Machines YouTube channel. Head over to uxmag.com. We have a podcast page there where you can explore our back catalog and you can look at episodes centered around popular topics. Many, many thanks to our incredible production team here in the US and in Ukraine. Elias Parker, Michael Lipinoff, Alison Harshberger.

Alla Slessarenko, Vera Pakulkov, and Darina Moskovchuk. Till next week, Owagi.

Frequently asked questions

How long is this episode of Invisible Machines podcast by UX Magazine?

This episode is 55 minutes long.

When was this Invisible Machines podcast by UX Magazine episode published?

This episode was published on November 16, 2025.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Invisible Machines podcast by UX Magazine episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.