AI agents have transformed how software gets written, but the operational side of running software in production has not yet experienced a similar revolution. The same team is responsible for keeping systems healthy, investigating incidents, and managing reliability are still doing much of that work manually. Mesmo is a production AI company that makes autonomous operations fast, efficient, and safe. Their open-source project, Aura, is a declarative agent framework specifically designed for SRE and platform engineering workflows.
It takes a Kubernetes-inspired approach where teams define what they want agents to do rather than scripting every step of how to do it. Andre Elizondo is the head of product at Mesmo, and he has a background in systems engineering, SRE, and observability. In this episode, Andre joins Kevin Ball to discuss what makes SRE agent workflows fundamentally different from coding agents. How Aura handles context engineering, or as declarative configuration model, the spectrum of agent autonomy, and where the role of the SRE is headed as agents take on more of the operational work.
Kevin Ball, or Kate Ball, is the vice president of engineering at MENTO and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI and action discussion group through LatenSpace. Check out the show notes to follow K-Ball on Twitter or LinkedIn or visit his website, kball.llc. Andre, welcome to the show.
Thanks for having me. Yeah, I'm excited to dig in. So let's start with you. Can you give us a little bit of your background and how you came to be at Mesmo?
Yeah, absolutely. My background at my core is, you know, I started my career running systems in a lot of different forms, both enterprises, small companies, both as a practitioner, as well as in the ecosystem at various different companies across cloud, infrastructure, observability, security. And so my view on the space is primarily from an operator or an SRE back in the day we called them Sizadmins, right? And over the last few years, I've been heavily involved in the AI ecosystem, especially around agents.
And so, you know, here at Mesmo, when I'm focused on, I head up our product team and we're at my team that's very focused on building, you know, really the ecosystem around how we enable SREs, platform engineers, really everybody in operations to gain the same benefits that we've seen with coding agents over the past few years, rather, for how we better use them to reinvent, how we build, maintain, troubleshoot everything around production. Nice. So can you give us actually just that, like, quick, high level, what is Mesmo? And we're talking a lot about agents that will talk a lot about agents, but like, Mesmo is a company.
What do you do? Yeah. So when you think about what the software factory is that a lot of folks are talking about right now, usually that's around, hey, how do I use something like a coding agent to build software faster, test it faster, and get it to production faster? What Mesmo is focused on as a company, and we have a, you know, a long past around observability, making observability really useful, really kind of like toned in on making it kind of like an next iteration of how we use observability, where we've really focused our time within the last six to eight months or so, is really around how we take it at a next level further, right?
So when you think about how things like observability as a good anchoring point, change in the land of agents, well, not only do you need agents that know how to handle that observability, but the observability systems look a lot different, right? Instead of an agent looking at a dashboard, it's going to prefer the raw data, it's going to prefer insights, it's going to prefer, you know, like specific tools around how it uses that data. And so, you know, when you think about the mission of Mesmo, we're really looking at the latter half of the software factory, right? Like creating that software factory and ultimately the dark factory for operations, not just how we get software into production, but how do we really change the entire lifecycle around, what does it mean as an operator to make sure that that software is healthy, to make sure that that software, I can optimize performance, I can optimize cost, and it's a big topic that I bring up a lot, which is like, you know, what if you had an SRE with an empty jerky?
What would they do? And there's a lot of things that fill that particular topic. And so, you know, at Mesmo, we're really focused on not only building the agents, but building the operating environment, building the ecosystem for how we enable teams to do that today. Totally.
I feel like, yeah, as we automate more and more of software delivery, that makes things like observability that much more important because you need to be able to understand, hey, my agent said this was fine, but is it actually behaving? How is it going? Totally. Yeah.
So, you're on agent legible observability. You went more than that. So, how did Aura come into, and maybe just a very high level, what is Aura, and how did it come out of what you're working on? Yeah, great question.
You know, when we first started to go down this path of the next iteration of observability, right, when you think about like, how do you distill patterns and logs and traces and metrics, we focused on initially, like most folks like exposing our MCP server in a way that people could consume it through agents. And you know, that meant that instead of your agent consuming an entire, say like hour of traces or week of traces, which would immediately look the context window, you know, we wanted to make sure that that was tailor made for agents being the primary consumers more than humans in the long term. Now, where we saw some problems internally as well as how folks were consuming that MCP server, is that, you know, S3 workloads look a little bit different, right? So when you think about like the types of workloads, the types of context engineering, the types of human in the loop, and like what that looks like for things like coding agents and things that are, you know, just building and testing software locally or in a controlled environment, that looks a little bit different than some of those problems.
Like I mentioned that, you know, you have an S3 handling a large swath of data in front of them or hurt to figure out, hey, what's going on? And there's a lot of, you know, kind of like looking at graphs, looking at dashboards, looking at all these different pieces of information. And what we saw, again, from our own experience as well as various different partners in the space is that, you know, what folks typically were doing is they were taking a very general purpose, very unappinionated agent framework, right? And like trying to put together that framework in a way that, you know, hey, I can do some pattern recognition on logs.
I could do some, you know, pattern recognition on RCA on traces or metrics or something like that. And really what's interesting about why we went down that path initially is, you know, we really just wanted the outcome of getting better RCA's, faster RCA's, be able to troubleshoot our own systems better, use that internally better. And what we found was like our experience of going through that journey and some of the folks that we initially partnered with were very similar to a lot of the things that we've heard from various different SRE teams and platform teams in the space. It's like, you know, it's like the building of agents, especially for managing production, it's very much like the iceberg analogy that we've seen for a lot of different things like Kubernetes, like cloud, et cetera, where, you know, everybody's thinking about the tip of the iceberg and like, you know, hey, how do I just take a model, wrap it in a simple agent framework and give it a prompt?
And in reality, there's, you know, a lot of things that are needed to actually make that reliable, right? Like how do you make sure that the model can consistently call tools and those tools don't just like randomly fail? Or how do you orchestrate something, you know, fairly complex, like a multi-agent workload without making somebody kind of get dragged or glass to get to that point where they're able to actually realize that outcome of solving that use case. And again, when we saw that with operations workloads specifically, we just saw more and more of that kind of like upfront effort in order to make that reliable workload and it's really like hampering people's ability to be successful with those frameworks.
And so, you know, we really just wanted to kind of like shorten the circuit, right? And so that's really, you know, going into where Aura came into play is like, it was our own learnings. We looked at, you know, a lot of different frameworks and harnesses at the time in the space. And you know, a lot of them were focused on either being insanely, you know, general purpose, kind of like one harness for everything, one framework for everything.
And they typically didn't make those hard things. Like I just mentioned very easy. There were like a lot of LEGO blocks that you had to figure out what the desk are looking at the end of it. And what we wanted to do with Aura was, you know, kind of like what the experience a lot of, you know, SRTNs or platform teams are familiar with, which is like, hey, if I'm deploying something in Kubernetes and I do a Kubernetes deployment, well, you know, I tell the Kubernetes like what the outcome should look like, I don't necessarily need to tell it imperatively every single step that it needs to do to manage those workloads, scale out those workloads, ensure that it's reliable at the end of it.
And we wanted to create that same type of experience for how ops teams built and manage and maintains agents. So again, like shortening that process to getting to the outcome and ultimately making it so that, you know, the tools that we expose, like our own MCP server and others just become much more valuable because you can get, you know, you can get your outcome closer to day one than if you were to kind of go through that process of building all the individual pieces that are frail and kind of like have the tendency to fail on their own. You know, we wanted to make sure that that was a, not only a short and workload, but a very reliable workload to get to that outcome. Yeah.
So that's, there's a bunch in there and I'd love to dig in a little bit, but let me, let me pinpoint one of those things because you mentioned, SREs are used to working in something like Kubernetes, which is essentially like a declarative environment, right? I declare my relationship, my, my objects are declared their relationships and the system is responsible for bringing them up, making sure that they get into that state and kind of getting there. And talking about that with regards to agent definition is fascinating to me because like, I think most of us are familiar with like coding agents of different sorts. It's probably the most widely spread agent harnesses out there are things like cloud code and that sort of thing.
And there, I think a lot of us are still being pretty, or imperative. And we say, I want this, I want that. You know, so what does it look like to have a declarative agent definition? Yeah, I think the proof is really, you know, the benefit of us open sourcing Aura is that you can look at the repo, you can look at exactly what the examples are and like, we operated very much on like a lot of the Linux fundamentals or philosophies of like having things like saying defaults, you know, where if I don't need to tune every single part of the process, you know, I can operate on the default built into the system.
But at the same time, I have the ability to, you know, tune the reasoning budget, tune the quality score where as the agents is maybe executing its tools or going to an end state and trying to solve a problem. Like, how does it self evaluate itself and self correct itself as part of that loop? Also, you know, enforcing the loop itself, right? Like ensuring that we were kind of opinionated on those things underneath, but also gave the folks that were using Aura the ability to kind of customize them if they stuff it.
And again, it's like, it's the same thing with a Kubernetes manifest, right? Like, I may just want to deploy in Gen X or Redis and I may just be all I need to do or I may need to scale that out. I mean to do like a bunch of different pieces along the way in order for that to be very tailored to what I'm working on internally to my workflow. And so we wanted to do the same thing for agents.
And so again, when we looked at the ecosystem, you know, there were different ways of like defining the agent in a framework. There were different runtimes. There were different kind of pieces along the way that were like very unappinionated and very much like, hey, you can do this, you know, you do a little bit anything. And again, with the types of problems that we were saying with the industry, we wanted to really just operate on the path of like, hey, you can start using Aura, you can pick up our container tomorrow, you can run it in your cluster or even on your laptop and give it a simple config file.
And so it's also the beauty of Aura is when you look at, you know, how you build these agents declaratively again, going on the manifest ideology here, we use a simple tomofile and in that tomofile, right, I define the agents, whether I'm saying, hey, here's a simple agent, maybe with one tool and maybe that tool is my observability tool, maybe that tool is my cloud provider. And a simple system prompt that says, hey, how should, you know, what should you do or what should you focus on? But I don't need to define, you know, again, the reasoning loop, I don't need to define the self correction loop, I don't need to define, you know, anything along the way. But as I'm starting to get into some of these more kind of complex things, right, where hey, maybe I'm dealing with a tool behind that agent that I know is going to return a lot of data, right, like maybe there's like different MCP servers, like for example, the Prometheus MCP server, I'm sure a lot of people have tried to use that MCP server, and you just end up getting this context blown of like, you know, every possible thing that's returned by that MCP server.
And so, you know, again, where we wanted to make things easy was we introduced different concepts along the way, like, for example, scratch pad, scratch pad is something that's built into Aura natively, and it's something that you could very easily turn on with a single line in that config file and say, hey, start using trash pad. And basically what it does is it acts as kind of a layer in between the tool and the model to give the model the ability to interact with the outputs of that tool via like a simple file on disk, and then ensures that, you know, it'll kind of like look at whether or not that tool needs to be recalled, whether or not you can kind of respect the previous tool call, like, you know, how you can kind of share that tool call across different agents. And so, again, it was really operating on that idea of same defaults on day one, make it really, really easy to get it started, but also get it running and ultimately achieve the outcome, but also not alienate folks that really, you know, cared about tuning every part of the process along the way. That's actually something probably worth calling out is like, you know, when we also looked at folks in the space, especially a lot of like commercial solutions or things that are like built into, you know, the various different obstacles that people are using today, like there's an embedded agent inside of anything.
A lot of those were black boxes, right? And so one of the really important things for us in like our day one, you know, opinions on how we've built and released and continued to develop on Aura is really that we wanted to make sure that all those things that we're doing are fully transparent to the user. You can not only see what they're doing, you can, you know, interact with it, you can tune it. And so that's, you know, transparency on day one was also an incredibly important thing for us along the way.
Yeah, that's super cool. And what I mean, even just the tiny example of scratch pad is really nice because I feel like I operate a lot in coding agent land, it's like you learn these patterns of, oh, let's dump this to a file and let's interact with it, or oh, let's dump this into a SQLite database and then query it rather than just raw, putting everything in context. And when you find those patterns, you make it easy, declaratively, just say, Hey, agent, here you go. You know how to use it?
Run with it. It's really powerful. You know, the target with Aura is this is an agent framework for SREs for kind of production use, which is once again, like different than most of us first encounter agents nowadays and coding agents or things like that. And maybe we're building our own somewhere, but let's like flesh out a little bit more of those differences.
So we talked some about the declarative versus imperative. What are the other big differences in building an agent framework for a production environment for SRE as compared to I'm running this as a coding agent on my laptop? Yeah, there's really like three big things that I look at across the, across the landscape of like, what does it take to have a really successful SRE agent for any team? You know, a big pillar across this is how we handle context engineering.
And so again, like when, when teams are just kind of picking up a framework and just putting a model and putting a simple prompt, like oftentimes they're not thinking about context engineering on day one and folks that may be aren't super motivated with context engineering, right? We always have the limitation of any model, you know, big or small is the context window, right? And so we're always operating within that context when I'm sure many of you listeners know about this, but just to be verbose on that topic. And you know, in reality, operating within the context window, it can be very simple for very simple use cases, right?
And it looks very different for things like coding agents where, hey, the context window is just the code or the understanding of the code base that's like right in front of me. But for SRE is right that context engineering looks very different, right? Like, you know, when you think about the amount of runtime data, everything, you know, understanding the code, understanding of the infrastructure, understanding of a lot of the observability data, right? So like, traces, metrics, logs, et cetera, you know, you can't just say, hey, go give me like a last week of traces and logs and metrics.
And I'm just going to shove it into the context window and hope that the model not only has the ability to have all that within its context window, but also doesn't like hallucinate along the way, right? And so the thing about context engineering for these types of workflows is they automatically become like at scale, right? Like typically because of the nature of, you know, us operating this in production, we're dealing with scale and those don't present themselves on day one. And so context engineering is a really big pillar of that.
The other piece is really like how we handle things like, you know, multi-agent orchestration. And so when you think about like some of the advanced patterns that are starting to pop up that are becoming, you know, really useful in this whole agent ecosystem is like, hey, how do I have, you know, as part of any different type of agent, maybe that's like an agent that's doing, you know, root cause analysis or whatever, like I may want different agents focused on different tasks that, you know, can collaborate with each other. And that's going to look very different in different types of use cases. Maybe I want a security reviewer, maybe I want somebody who's, you know, just focused on investing in the cluster, just focused on investing in the code base.
And that helps me to, you know, do this in parallel. So it's actually a lot faster to do that. But it also makes sure that like each one of those agents, you know, isn't dealing with again, like a single context window as its limitation across the way, right? And so like orchestrating multi-agent workflows is typically a non-trivial thing and pretty much any harness or any framework.
And so again, that's like usually a day three of, you know, if anything, maybe a day four type of problem that not everybody's thinking about on day one. And so it's really limiting them the ability to gain those, those benefits until they've actually hit the problems with like, you know, what they're doing in the simple agent or maybe the simple agent plus one or whatever. But we just wanted to make that again, like very easy to solve on day one because we believe that the nature of these types of production, SRE workflows benefit a lot from this orchestrated multi-agent workflow. And then the last one is, you know, how do we handle things like human in the loop memory, things that are opinionated to the task?
And again, that's where, you know, we think about like what memory looks like in coding agents, what memory looks like for pretty much anything else. It's very general. It's very like you can kind of have the agent provide memory and then the memory system is like either just a simple graph or maybe it's just a simple, you know, kind of collection of like summarizations across the context window, whereas, you know, what you would want in, you know, an incident investigation is really a memory system that's looking at the context of, hey, what are the previous incidents that I looked at? Like what were the previous resolutions to those incidents?
What happened along the way, right? And so like, this is also another problem that like is unique to these SRE workflows is really how we handle those longer horizon tasks, those longer horizon things that the agent is dealing with. And then the last piece I'll just kind of highlight as part of that is things like human in the loop, you know, how we have agents that we can, you know, give a little bit of autonomy or a lot of autonomy to, right? Like we believe that should be not a either or but really like a graduated process, right?
Where I maybe want to start with something that looks more like a co-pilot and then be able to very easily grant some more autonomy where, you know, it looks a little bit more like an assistant. And then once I truly trust that assistant and I actually want to, you know, change the way that I'm running production to really be agent, I can really be autonomous, well, then I want to graduate to more of that autonomy that looks like, you know, really that like dark factory when we feel like we're comfortable with turning off the lights. And so we wanted to make sure that Aura kind of took in all those different concepts that I mentioned and just make them kind of like available on day one and something that, you know, we have our opinions baked into the harness, but at the end of the day, you know, you can plug that harness into, you know, our tools, you could plug it into your own tools, your own, you know, observability systems, cloud providers, et cetera, behind the scenes and you know, really kind of like provide that for the community to build on as a common set of practices that we believe will enable this like industry to move forward. You have highlighted like some of the key things.
So it's okay. I'd love to die another level of detail deeper on probably each of these the one I want to start with is orchestration because as you know, like I feel like this is a problem everybody's trying to figure out right now in different forms or another and you have people building their own orchestration frameworks, you haven't been built into different things. So first from a developer experience standpoint, what does orchestration or multi-agent look like in Aura and then maybe we can peel back a little bit and be like, and just sort of look at what are the primitives? How is it built?
How would it, if I wanted to customize it, what would I plug into? Yeah. So the way that we handle orchestration is meant to be like as simple as possible, right? So, you know, there's a lot of different things that folks can customize again if I just picked up a like a general purpose framework, like Linkchain and said, Hey, I want to do, you know, multi-agent orchestration while I'm thinking about the type of reasoning loop I'm trying to think of like the type of retry as a resiliency across the way.
And the way that it looks like with an Aura is again, that simple tomofile that's driving the whole process. I define an orchestrator or a top level agent. That agent is very simple. It's anywhere from like five to 20 lines of like, you know, how detailed I want to be on my system, basically that's like the majority of the context there.
And then I basically just kind of like note that as my orchestrator agent. And then what I can do is I can define various different workers. And so again, like just another subsection of the tomofile where I'm saying, Hey, this is a worker that is named this and I'm going to give it this description and this preamble. And I can also limit its tools so I can give it access to everything or I can say, Hey, you know, these are the tools that I just want this worker to have.
That's again, like limiting some of the potential of context float or, you know, kind of limiting the ability for that agent to be really good at one task. And like again, it can look basically as simple as that, right? Define the orchestrator. I define the L on that I want to use.
I define the workers and you know, everything behind the scenes is actually driven by Aura out of the box, right? So I'm not having to think about how the orchestrator is going to handle its planning, right? So like the first thing that the orchestrator will do is kind of create a plan, create a go forward like what am I going to do, right? Like how am I going to fan this out?
What workers are available to me? It's going to go ahead and provide that to the workers where it's basically going to go into kind of an execution mode. And then it's going to go ahead and kind of like self evaluate, right? So like as I get the results from those workers, hey, do I have enough of a confidence that will actually sell for itself along the way and say, hey, you know, this is a confidence I feel comfortable with moving on to the next step.
Or you know, you can kind of make it either very high of a quality threshold or very low, depending on what you want to do with it. Typically we see like, you know, 70% confidence is what we see a lot of folks kind of define initially there. But you know, I'm not having to think about like how the tools are provided to those sub workers. I'm not having to think about how that planning and self correction loop is defined for the orchestrator.
I'm not having to think about like how the orchestrator provides the relevant context to just the workers that it's currently. That's kind of what I was going to ask about is like, what is the communication path back forward and back? Like is it just dumping files in place for them? Is it, you know, standard in and standard out?
Like, can it hand off more complex things? Like if I have a sub agent that pulls together a database for me, can I hand that to a different sub agent? Like what are the tools available in the tool chest? Yeah.
So we kind of go for like a single path and single path out for right now. That'll change as the harness develops. But you know, it's meant to be, hey, the orchestrator does the planning and then hands it off to the workers and the workers hand it back. And the orchestrator can hand it to different workers if it determines that, hey, I've built some context but I need to clarify it.
So I'm going to get a little bit more context, execute another tool or maybe execute the same tool again because it didn't give me the result I was looking for. And so the orchestrator becomes kind of the important part of that process. But the thing that we're starting to see more and more of is like, you know, these multi-agent workflows, you know, they have a lot of tendency to become really complex really quickly. And so we want to make sure that was simple on day one, but like, as we see more and more of these patterns emerge, we definitely want to make sure that Aura kind of enables you to like gain the benefits of either like performance or reliability out of what that like orchestration flow looks like.
What we can also call out with orchestration, which is pretty interesting is that, you know, when you think about like how you are gaining better economics out of the tokens that you're using in some of these workflows, again, you're dealing with scale. And so like you want maybe the bigger model, maybe you want an opus on how you're doing the planning and the reasoning in the top level orchestrator. But then on those sub workers, you can actually declare a different L, a different model behind the scenes that you know, operates as more of a delegate a task worker, right? Like, hey, maybe I want to use haiku, maybe I want to use an open source model.
And so that actually is also something that Aura makes insanely easy to do using our declarative format so that you can say, hey, I want to not only use a multi-agent workflow, but I also want to use smaller agents for the workers and ultimately get better kind of like token economics out of the whole process. And that's kind of like one of the things that we saw done day one that we've seen some pretty surprising, like pretty exciting results out of just enabling that to happen and kind of embracing the idea of a multi-agent system with a lot of different models and mix. Totally. Yeah, there's a lot of like classic like data categorization extraction, etc.
That is trivial to do with a small model. And you really don't need to throw up is that it? But you want that as a sub piece of your broader thing. So yeah, making that easy is super helpful.
So to play back, making sure I understand you're sort of right now operating on like sort of single orchestrator set of available workers, the orchestrators deciding what and just sort of spawning things out and then it can dynamically decide, hey, I got this from here, but actually I want to send it over there doing things, but it's kind of single level of delegation. Exactly. So we're to your earlier point to where we're making it as simple as possible. A lot of our interactions are just utilizing flat files on the system.
And so that's something that pretty much every LMM agent is interacting with is a file on disk. And so we're not introducing a lot of custom processing. We're kind of operating on the simple, most reliable thing that when breaks, you know, why it breaks? But ideally it doesn't break because it's a simple enough mechanism.
I want to come back to talking about memory and human elukuses are interesting, but that points in another direction that I think is really interesting to explore, which is like the observability of the agents themselves. Like we've been talking about how you're using these to tackle observability data and things like that. And so we're just as or have for making it self-observable so that as you said, you know, when it breaks, you can tell why it broke and how to fix it. Yeah.
Being an observability company, this is like one of the things that we made a very hard requirement for us to solve basically, you know, even when we didn't GA or when we were kind of in a pre-release, like an early access type of thing, we instrumented everything in Aura natively with open telemetry. So we're heavily invested in like open telemetry for the agents. We define everything again in the ream, you can see exactly how we're instrumenting that, what pattern we're using for open telemetry, which we're built on top of open inference for open telemetry. And so you can see exactly in the spans of that trace of the agent interaction, you know, how is it doing the reasoning for the next step?
Was the tool call successful, et cetera? So we organize that automatically and we build on top of open telemetry to make that happen. And the other thing that we have natively built into Aura is a really rich set of server side events that can be returned back to the client that can really show you like what's happening live as part of the investigation process or like what is the agent doing behind the scenes. So, you know, definitely want to get to the point where it's like, hey, I give this really complex task to maybe a multi-agent workflow and I just kind of like wait on the spinner or anything like that.
Like as that's executing, we have a CLI built into Aura so you can very easily just like start interacting with the CLI. And that is, you know, it will very easily show you all those different events that I just mentioned, right? So like you can do slash stream, you can do slash expand and it will show you like, hey, what are all the activities that I can see from the agent happening, you know, behind the scenes as I'm interacting with it? And then, you know, because it's built on top of open telemetry, I can send that to my observability system and that makes it really easy for me to understand, you know, the auditability of like the interactions from the agent, what did it actually do and why?
Right? So like the reasoning steps across each process are also captured in that trace. And so we just wanted to make sure that, you know, one, it's built on open standards, you can send it to anything that supports open telemetry and it becomes a great way for you to start building trust in your agents, right? And again, like this is a big barrier in the industry I'm saying is that, you know, for SRE is like anybody who's dealing with like a production use case, you really have to understand exactly what's happening behind that agent in order to trust it.
Otherwise, you know, you're not going to put into production, you're not going to, you know, offload more tasks to it, you're not going to really integrate it into your workflow. And so the observability or the transparency of what's happening behind the scenes, like that was an insanely important piece that we want to solve one day with. One of the big patterns I'm seeing right now, once again, doing the metaphor of coding agents, because that's where I live a lot of my time, is like looking at traces, essentially of how are things going and using that to iteratively improve. And if you're talking about observability, agent or an SRE agent, it is the same thing, but you can't just be looking at files on your laptop anymore.
You've got to have those deep observability traces. Totally. Yeah, exactly. Because we're built as an observability tool, you know, our platform enables you to send those traces directly to us.
So obviously we have the ability to pull those in, visualize those, show you the different, you know, service mappings and interactions. Most people have an observability tool in place, and so it's easy for you to take over and just point it, you know, whichever open telemetry provider you decide to use, you know, the visibility that you're going to get into those workflows is not going to be limited to, you know, using our commercial product. Let's come back and talk a little bit about memory, because I think this is also a really interesting domain. And I'd love to get, once again, one or two levels deeper on how you approach it.
As you commented, a lot of memory implementations out there are pretty naive. I'm going to use an even simpler example. You look at chat GPT memory or something. And it remembers everything.
It doesn't give me any visibility into what it remembers or why. And you know, when I'm working on some thing for my son's project, it says, Hey, because you're a director or you're a VP of engineering, you should be thinking about it. That's right. It's just this one big undifferentiated mess.
Obviously that's not what you want in a production SRE context. It's you mentioned examples of exposing incident logs or incident histories and things like that. So like, how does your memory architecture work? Once again, first from a user perspective and then like diving in under the hood.
That's a good question. And you know, a lot of this you can actually see in our PR is like how we're developing the next iteration and what we're already starting to improve with the NORA. But the way that we look at it is like there's different levels of memory. And so, you know, there's memory like more turn based or session based basically like that you want the LM to be able to recall, you know, what was the last tool call that happened on the last couple seconds, right?
Like, or what was the previous run of this agent able to come to as an conclusion that maybe is is right in front of the agent at that moment, right? And that's more of the turn based or session based memory, which is kind of like the lower levels of where we want that really fast. Kind of like when you think about memory architectures with like, you know, having DDR as like, you know, my RAM versus a hard drive versus, you know, something that's like a offsite storage. We kind of took the same approach with ORA where there's very simple memory that's built for turn based and session based that again, you can tune along the way.
And then what we're looking at like the really interesting problem is when we start to get into some of those like longer horizon memories. And this is actually where like, you know, we are working on this within the platform itself, within the Mesma platform to offer this as something that, you know, is a little heavy to do in line, but it's actually great to do kind of as an asynchronous background process that just gets exposed to the agent that doesn't have to be tokens on. And so this is where we're looking at like again, more of the kind of needle in the haystack incident over incident investigation, but like tailored to the types of workloads that S3s would deal with where those memories are not just about the last tool call or, you know, the last iteration of this agent running. But it's like, Hey, what happened in the last six months?
What happened in the last 12 months? What happened like with this service that I started to see maybe a little bit of service degradation and also a change that somebody pushed from GitHub and also, you know, started to see some funky stuff in the logs. Like that all becomes a collective memory that we're kind of creating as a simple briefing to provide to the agent that just makes it easier for that agent to either solve the next thing, you know, the next time better or potentially just short circuit the process entirely and say, Okay, you know, normally I would kind of come to this conclusion of doing this next thing, but I can see the last time I did this thing and I can kind of compare that to ground truth in the background so I can kind of validate whether or not that had the intended impact that I needed. And maybe that'll change what I do the next time that this happens because now I've actually evaluated that in the background.
And so that's kind of like where the longer horizon memory is like something that we're very interested in because again, if you look at like, you know, just taking either the graph, you know, memory MCP server that most people are familiar with that you can just grab off of a kind of the standard MCP servers or something like M0 or MMP or any of the kind of very general MCP memory servers, we wanted to make sure that that that was again, shortening the circuit to the outcome or shortening the path to the outcome where we know the types of tasks that SREs need to solve for. And we know how memory becomes really important to some of those longer horizon tasks. And so we're just going to provide that on day one. What does that end up looking like technically that is that I kind of rag like implementation, is it exposed via another and MCP or it's a tool that separate tool that you can query?
Like how does that end up actually playing out in the agent? Yeah, so the way that we're exposing it today is just taking like the simplest and lowest common denominator across the board, which typically ends up being another cool. So the way that we look at it, we call them investigations. And so as I'm handling either a new investigation or looking at previous investigations, that's just another tool available to the agent that we exposed to our MCP server.
How that practically looks is we're providing kind of a briefing as part of that tool call. So it's kind of like the summary and then where the agent can go for the next steps. Something that again, because we're focused on building on top of these lowest common denominators in the space, which a lot of this looks like just an MCP tool, you could plug what we're doing in the background for memory or what we're doing in the background for context engineering longer horizon things into Aura. And that's what we see as a great path to consume that.
But you can also plug it into, you've already invested in that agent framework that's, you know, you've already gone through the last to get to that outcome that we were just talking about before, or you want to plug it into cloud, you want to plug it into wherever, you know, you can easily do that. We did experiment with a little bit and again, you can see this in our PRs doing things like providing a virtual file system to the agent where we can kind of like seed in those memories without the agent actually having to do the tool call. And some of that's in kind of an experimental phase where we're looking at like, how can we better expose these memories that don't actually require the agent to do a tool call and is rather just like transparent to the agent where it's not wasting tokens. But some of that stuff is like kind of in a research preview and again, you can see that in our PRs and kind of follow along with it.
Yeah, no, it's super interesting. And maybe this connects a little bit to the context engineering question, right? Of like, what sets of things do you want to embed in your initial context? So it just kind of knows it versus make it on demand, go and call it versus, yeah, have it in an explorable database or find, you know, some other way of finding it.
And there's also one last thing that I want to make sure I also layer on here that's related to memory. We don't always call it memory, but it's like on the same exact path, which is things like institutional knowledge, right? So like more of the institutional memory or institutional knowledge, right? Like where, for example, every SRE team, right, has a set of runbooks that they have internally, maybe that lives in confluence, maybe that lives in GitHub, maybe that lives in, you know, insert your favorite place to store runbooks, right?
And so that becomes incredibly useful grounding for the agent. And so, you know, we also expose a really easy way for you to either connect this to where those runbooks are currently stored. If they expose something like a vector database, it's really easy to plug that in and use that as grounding for the agent. So the agent will always kind of ground and respect itself based on the workflow based on what you've defined as the source of truth being your runbooks.
We've also actually published a blog post on like, hey, if the agent has the ability to not only consume those runbooks, but as it starts to see things during its investigation process, how can it actually contribute back? And we've published a blog post on this, how we're actually doing this internally where we have our runbooks store in GitHub. We actually use Aura for a lot of our own SRE workloads internally. And as we identify things that like, hey, this workflow has changed based on what's in the runbook, instead of that becoming a human process where somebody's maintaining the runbooks, and it's only a one directional, you know, like consumption by the agent, we've actually seen a ton of really positive success with providing the agent a feedback mechanism to basically iterate on those runbooks, open a PR and say, hey, I did this investigation and I need to update the institutional knowledge.
And that's just like a really good example of kind of the opinionated memory where like everybody has institutional knowledge and somewhere to form. Yeah, well, I love that because it closes the feedback loop and it gets us to the last sort of pillar that you talked about that maybe we want to dive into, which is like the amount of human in the loop you have. And sort of you described here a version of human in the loop, which is like, I did a whole bunch of stuff, but here let me suggest to a human improvement. But yeah, how do you think about, you mentioned, which I love this idea of having the sort of gradient from almost entirely human with CoPilot going on to completely automated, but like, what does that end up looking like?
How do you end up configuring it in Aura and where do you see different types of SRE tasks typically falling out in terms of that level of autonomy versus human run? It's interesting because the way that people are looking at human in the loop today is like very dependent on the types of systems behind what the agent is interacting with. Right. So that's like one parameter, one dynamic.
There's also like how confident are you in the workflow itself? And then there's also like how confident are you in the data that has been returned through the workflow, right? So there's like an aspect of how you provide the ability for somebody to say like, Hey, these types of tools, which maybe are read only or sorry, right? You know, they have the ability to write or they have the ability to do something that, you know, it's got a one way door, right?
Like I want to make sure that I provide a very hard kind of barrier like for the agent to be able to know that that is something that's insanely sensitive and it needs to ensure that it asks for help before continuing on. Like that's kind of one piece that, you know, we see everybody with some level of like, Hey, there's remediation, but like I need to make sure that there's always a human that hits the go button before it does remediation because I know that that's a, you know, one way door that's really hard to come back from that's kind of like one aspect of like limiting or providing human to the loop on the tools themselves. Then there's like, you know, how far do you want the agent to go in the process before, you know, ask for a person to kind of let it continue on with go, right? So like if you're not limiting access to the tools themselves, but you're limiting access to like how far in the process the agent can get to, right?
Like you can basically act as an assistant, but at the end of the day, the person who is implementing the remediation to what you've investigated is always going to be a person. I'm not going to provide you those types of tools and it's always going to be, you know, you're read only, you're only getting me up to the point where you're creating a nice package for an SRE to go execute a script remediate a problem, but you're coming to exactly what needs to happen before, you know, that point becomes like a simple place for the agent to determine that it needs to continue on or just completely hand off the process. And then the last one is, you know, where we start to really get into, I think, like the true benefits that people are looking for, which is when I'm thinking about like a truly autonomous workflow, right, where, hey, instead of SREs handling this end-ten, it's actually an agent handling this end-ten. That's really where we start to get into, you know, like an aspect of, hey, how do you use some of the data that we were talking about before with the observability data, right?
And like grade that over time where even if you think that that's a pretty good workflow and you have the right data and you have the right workload to find, you want to make sure that it is tested a certain amount of times before you give that the full like thumbs up, continue on, operate autonomously, right? That's kind of like the graduation process. So like there's, there's like three different areas there. Now how that looks, you know, specifically with an aura, it's all driven through the config of all where it looks a little bit unique is like, as soon as we start to get into those like longer horizon things again, where it's like, hey, I'm not just limiting access to the tool itself, but I'm telling the agent where it needs to go ask for help.
That's where we're exposing a tool within our MCP server to basically provide to the agent. So like the agent doesn't need to go figure out, hey, you've told me that I need to go ask for help here, but like, I got to go figure out who's on call. I got to go figure out how to get access to Slack. I got to go figure out how to do, you know, X, YZ.
Instead, what we're doing is like, that's a process that we're seeing across, you know, a lot of SRE workloads that needs to happen. So like, we're just exposing a simple tool for the agent to be able to ask for help. And then what we're doing behind the scenes is like some of that correlation that I mentioned, hey, who's on call, who do I need to go ask, trigger notification via Slack, somebody give me the thumbs up and then provide that to the agent versus the agent again, having to do that itself. So that's kind of like how we're not only defining the different types of human in the process, like governed autonomy is kind of like what I think about it, but also make it so that our platform kind of provides the simple mechanisms for the agent to kind of like ask for help from a single place without having to blow a bunch of context to go figure out all the possible places that it could go as for help.
And so like I said, the way that we operate with doing that, right, like more is open source. We always expect people to be able to use Aura with or without Mesmo. And so you know, you can definitely point Aura at a different tool that's, you know, a different human in the loop process. And we publish kind of the spec there of like, if you wanted a different human in the loop process, here's how you can do it.
So you can define it. And then we kind of like operate on, hey, we open that spec, we've defined that spec, but then for the actual implementation, okay, great, we're going to make the simplest path for you to realize that implementation, which ends up being just, hey, our MCP service going to naturally provide this to you on day one. So you can consume it right off the bat. Yeah.
Well, and I think that's a really nice way to have a pairing between a commercial product and open source of you. Look, all the doors are open. You can do all the different things. And here's some things that we know you're probably going to want to solve a lot of times that we solve well for you.
So here you go. That kind of gets into an interesting question around sort of the future of Aura and in particular around like governance, right? So is this right now? This is a Mesmo run project and you all are doing most of the work.
Are you looking to get it into third party foundation or how are you thinking about governance of Aura going forward? Yeah. So right now Aura is a small but mighty project. And I say that because, you know, the primary contributors are from Mesmo.
We're starting to see more and more people interested in the space and the fact that we've gone with the Apache license makes it a lot easier for folks to be able to contribute without having to jump through a bunch of hoops. We're a member of the AAF. So the agent foundation that's underneath the Linux foundation and we're active contributors there. And so, you know, there's definitely a future here where I think as we start to get more and more interest in Aura and you know, we have no intent of this just being like, you know, the Mesmo way to run agents, we want this to be as like general purpose of a best practice standard in the industry.
And so I think like, you know, not to put the cart before the horse because we still want to get as much proof and partner with as people as closely as possible where we can very quickly, you know, make it a race and where it makes sense. But I think as it starts to mature as a project, definitely makes sense for, you know, potentially contribution to, I think CNCF is probably pretty crowded. But like the great thing about AAF is it's fairly new, fairly green filled and you know, there's a lot of folks that are interested in kind of moving the space forward through that. So definitely looking at that as a good option potentially in the future here.
Yeah, totally. Well, and I think it makes a ton of sense to keep it tightly controlled to start with, right? Like I've definitely seen people try to open things up a lot too early, right? And it leads to chaos oftentimes, right?
Like the benevolent dictator for life, even if the dictator is a company, like that's a good way to start a project and keep it moving. If folks have not used Aura, they are curious they want to get started, maybe they want to try contributing or tweaking it. What is your sort of on-ramp? What do you recommend?
Yeah. So, you know, everybody from Asma is there, you can kind of get help. You have an idea. We're very interested in kind of like fielding, you know, what people want Aura to look like in the future and what ideas they have in the space.
And you can definitely open an issue on the GitHub repo like you can for many things. PRs are definitely welcome as well if you're interested in donating your tokens, right? Like, I mean, we try to make that as coding agent-friendly as possible where we're, you know, heavily leaning on like the Cloud MD and agents MD and things like that. And so we're trying to make it as easy for folks to contribute to.
But then again, that Slack community is a great place where like, if you don't know where to get started or you're having some problems or you just need a little bit of help or have some questions, you know, pop it in that community and very interested in helping, you know, helping where we can. And we're also, you know, very interested in partnering with people. Like, if you're looking at how you want to change the future of SRE and you're, you're aligned to this conversation that we just had and, you know, you want to partner with a company like Mesmo that's kind of like looking at building this, you know, first hand, and we have engineers actively working on this, you know, definitely reach out and we're like happy to provide any kind of support or like implementation and things like that to make sure you're successful. So a lot of different paths based on, you know, just community contribution versus like true commercial partnership and design partnership there.
Yeah. So, you know, we're going to do a chance at coding, but having that first and figuring out how do you design your workflows and your processes and your automations and everything to make that friendly. I mean, that's to me, that feels like what makes sense in this world we're now in. Totally.
Now we're in this point where the code is cheap, but the ideas are expensive, right? And so it's like the easiest thing for us to do is make that process, you know, streamline is possible, right? Like, we know that coding agents are the future and they're here to stay and, you know, it actually lowers the barrier, right? There was a while where like in order to contribute to an open source project, you had to really be in depth in that project and like actually understand what's happening, maybe be an expert and go, rest or whatever, you know, because or is written in rest, not everybody knows rest and we totally understand that, but every agent knows rest, right?
Like anybody who has access to opus or any of the coding agents, right? You know, you have an expert in front of you and you have your tokens. And so, you know, now the donation to open source becomes, hey, if you want to donate your tokens, these projects like ourselves and other folks in the space, we just need to make it as easy as possible for you to do that. Well, I realize we're getting close to the end of our time here.
Are there any things we haven't talked about yet that you think would be good to talk about before we wrap? You know, there's a concept that we're really keen on advancing kind of the industry and kind of how we operate as Mesmo, which is, you know, there are two kind of like axes on this in the space, right? There's what production AI looks like as far as like AI in production and like all these things we're talking about, right? Like multi-agent workflows, there's, you know, how you reliably call tools, there's memory, there's things like that.
And that's all like how does an agent operate well in production? And then there's another access which is like AI for production, right? So like that's where you think about like what are the types of tasks, what are the types of skills needed, what are the types of tools needed that really enable these agents to handle these types of tasks that we see high value and something that's, you know, inherently hard to solve. So, you know, production AI and like how we're trying to shape the industry around like, this is a problem, it's really the intersection of those two things where we want to enable SREs to not only own their own agents and like kind of go through this process safely and reduce the risk to get to the outcome, but like what we see as a healthy side effect, which I personally am really excited about just from being a practitioner earlier on in my career, is really like, this is our mechanism to also elevate the whole role of what SRE means, right?
Where we're more, we're architecting reliability more so than responding to things that we've implemented, like we're taking more of a design approach for how we can make systems reliable more so than being firefighters all the time. So, I think that's a really interesting opportunity that we have to move the space forward. And like I said, I'm just, we're really excited about like where this is already starting to show success and where we see this kind of advancing in the next months and years and things like that, that really bring all the SREs today that are used automation, they're used to, you know, handling these really high risk scenarios and just giving them the tools and the ecosystem to make that ready for the agent there. I love that.
It fits really well into something I've seen in a bunch of AI spaces where like a lot of people are trying to delegate their thinking to AI, but where it really shines is when you can use AI to elevate your level of thinking and really do something that's more impactful, deeper, more just leverage all of those different things. So can you maybe pull one example out of how using this at Mesmo has helped you elevate your SRE approach? Yeah. So one is, you know, again, like we've taken the approach of making it less like a firefighter and more like a reliability architect.
So all of our SREs, right, like we have all of our engineers on call as well, right? Like we take the approach of like the SREs are the ones building the guardrails and building the systems building the workloads, but that makes it easier for other folks to also take that pager, right? Like be on call for the systems where, you know, all that tribal knowledge that typically was trapped in SREs had, like agents are a really good mechanism to make that kind of an extra duration of like as code, right? Where like we get that out of people's heads and make it more automated, make it more standard practices.
And so that's really helped us to really iterate internally on what's the next iteration of SRE and really use our own ability to, you know, capitalize on our own harness and like our ability to kind of like incorporate our own learnings there. And we really just like elevated our SREs internally. It's so much to the point where like it's made the role really exciting and really given people internally the ability to kind of see what the future looks like, which then makes them also excited to talk to other SRE teams and other companies, which has been kind of a cool site to see as like we're getting to the point where we can actually share knowledge, build the community, build, you know, kind of like these standard practices that extend outside of them as well as well.