Advanced software systems have long been more complex than any single engineer can fully understand. Observability is the established solution to this problem, but with AI agents now generating code, deploying changes, and operating autonomously, the challenge of understanding large software systems is entering a new dimension. Grafana is an open-source observability platform, and one of the most widely used in the world. The company builds tools that help teams collect, visualize, and act on telemetry data across logs, metrics, and traces. They are now extending that capability into the Agentic Era with AI-powered investigation and monitoring tools. Anthony Woods is a co-founder of
Grafana Labs. In this episode, he joins Matt Merrill to discuss how AI-generated code is straining software operations, why telemetry data volume has become as much a problem as a solution, how Grafana is adapting to a world where agents are the primary consumers of observability data, and what keeps him up at night about where the industry is headed. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in back-end development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at Dept Agency.
You can learn more about his work at code.theothermatm.com. The background in tech is a network engineer, then building automation tooling into one day. I found I was writing code full-time and becoming a software developer. That was a lot of fun being able to bring together all different parts of the system we talked about. About 12 years ago, Grafana started a company called Grafana Labs based on the open source project that took the talk we created. We took that love of open source and of helping people understand the data
and the telemetry that getting out of the systems and turn into a business. That is awesome. I am super excited to talk to you because I am a backend nerd. I've done pretty much back-end stuff my whole life. I've always been at the intersection of backend and DevOps.
I want to talk about software engineering operations in the age of AI. I want to go into how you are doing this at Grafana itself because that is a company at scale in this age. Also, what you are seeing happening with customers. I am sure you are seeing a lot of it. I think where else start is, when you look at how software teams operate at five years ago versus today, what do you think are the most significant shifts you are seeing in software operations?
Obviously, things are shifting and changing very quickly. That is one of the big changes is just the speed of change, both from capability and tooling of what is available, but also just on the demands on engineering teams, the pressure for them to continually deliver more and faster. We have seen the big shift from building monoliths into microservices and the benefits that had four teams to have smaller teams with smaller scope so they could move faster. That has really worked, but the consequence of that is now we have got these very complex distributed microservice architectures where no one knows how the whole thing works together. They might know their own little piece.
We have seen that both internally for how we build our software. We find that model works really well for velocity of development, but also as that complexity. The way we combat the complexity internally and the way we see our customers doing it is we have better observability tools. It just means collecting more telemetry data to understand what is happening. We look at all these small little microservices and they are really just little black boxes that are going to push into your production and observability is that tool that turns that black box into a glass box. When it inevitably breaks,
they always do. You have got visibility to see inside, see what happens when we are wrong. That is becoming increasingly important now with the shift to AI where a lot of the code is not being written by a human, but it is still going to break. You need to have that visibility to know what is going on inside. This is another thing that we have seen 10 years ago. Maybe the conference is
maybe monitor on the other side. We have to measure all the things just because people wanted to capture more telemetry. That has come back to bite people. We have the costs. We have so much data now. We are
ingesting expensive to ingest all that data and process it. We have to have too much data and when things go wrong, they don't know where to start looking. They are just overwhelmed by this ocean of data that they have got. Trying to help people trim that down and focus on what is the important data that you need to be able to solve the problem, not being overwhelmed. That is really important. That is the problem to have. The other big thing we are seeing across the world is open
source and advantages, especially on the operational side. We are seeing technology changes fast and historically it was open source is a risk for a business. You can trust it. You can look after it. Whereas businesses are looking at open sources as a must have right. That is where innovation is
happy. What we are doing with the ecosystem is a great example. Organizations are realizing that just a future proof of what they are doing on the operational side, they need to adopt open source technology, open source standards. They get access to the innovation that is happening. But also move away from the kind of vendor lock-ins that people want to avoid today.
So I am admittedly not familiar with open telemetry. Could you say a little bit about what that is? Yeah, definitely. Anytime you are building an application and you want to instrument it, there are a lot of choices for how you do that. When I write my code, I want to commit logs, I want to collect traces or metrics. There have been a lot of vendors that have provided proprietary,
kind of SDKs and things that you can use. It is open to the open standards. It has been around for a while. But now we are seeing where the velocity is actually using it and getting a lot of value out of it. It is a growing ecosystem. It is a standard practice for how you instrument your
applications and you collect that telemetry. It has very good kind of semantic conventions. Things look the same. That way when you have different teams working on different projects, building things like that, it should all look familiar. If I have not worked on that team before, I can go and have a look at their logs or metrics and other traces. It should look familiar. Things
should be named consistently where things are understandable. Just having that consistent approach really helps. Obviously, the huge advantage here is you do not want to have to reinvent your applications every time you change your observability vendor. It is a thing that you just want to do once and never have to think about again, but still have that ability to go and change your observability vendor over time, or add additional tools or different capabilities. So having that open ecosystem, open standards and vendor neutral approach is very attractive for a lot of organizations.
The HTTP of logging. That is the way that they are. That is awesome. I cannot believe I am not familiar with that. That is very interesting. I am assuming Grafana supports this. You are supporting
the project itself too. We have more opinionated solutions of how to do the application. So what you said about open source and open standards to me also intersects with AI quite a bit, where I want to spend a lot of our time. One of the many benefits that I see of open source is that it is well documented. It is on the internet. These models can scrape all this information and
support it very easily without some vendor lock in or pay wall or something like that. Are you seeing that happening already in this space? Or perhaps not? Is it not cut up yet? No, definitely. We are really excited. We are excited about the capabilities we have
been building into the products. We had a team where we saw AI coming 18 months ago, how they are going to impact operations. I am not saying anything useful. We did see that big shift with some of the new models that came out beginning last year, just like the end of 2024. We thought we were behind. We had a team of two people during one of our
quadle. We had one together. It was amazing at how well it moved. It was able to understand what people wanted to do. We knew how to use Grafana. We were like, how did you guys do this?
How does it work? They are driving the island. We thought more about it. The thing we realized was we spent more than a decade building this great open source ecosystem. We have 25 million plus users around the world who love and use Grafana. They write tutorials, guides on how to use Grafana
to solve certain things. Everyone has got public GitHub repos, they have got the dashboards in it. They have got configuration for how they have employed it. They have got a whole bunch of public information. It is information that the foundation model is from Google or open a train. Out of the box, the model is no technology. They know what users are trying to do.
They know how to solve those problems. We have been able to leap ahead of our competition by not having to go and train expensive models. We can use the foundation models and be able to tie integration tools. That is a huge advantage. We found in great minds, we had 12 years ago, we decided to focus on open source. It is a great plan. It has helped. We see this a lot. We are
asking code to instrument open telemetry. It knows how to do that. You can add that to your code and make sure you need it. That is amazing. It almost seems like good karma. You put up this good thing
into the universe and now you are getting something back. That is a big value for our organization. We have this manner that comes from the right thing. We do the right thing for employees. We do the right thing for our customers. When you do that, it comes around. People do the right thing for you.
I am a believer in doing the right thing. It is working out really well for us. You used to have a saying that a company I work for is do good work and the rest will follow. We need more of that. You don't have to answer this if you don't want. Are you afraid that somehow
this might turn around and bite you in the button, somehow replace the tools? I don't personally see other might happen, but I am curious if that is on your mind. I am very optimistic that is it. I think there is certainly risk there. We can't just sit around and hope for the best. We are definitely seeing a change with observability. I think the main thing,
especially for us, which is scary for our lab, we have built our reputation on being the company behind the dashboard and the visualization tool. It is something that users interact with. We are finding more about our internal use cases as well as our customers. It is more and more. It is not humans interacting with the data anymore. It is agents going in doing queries, looking at what is
happening in your environment, trying to understand and derive insights from it. We are seeing that shift where there is less about the dashboard itself. We are definitely making changes within our product to support that. We have a project which is a unified command line tool. It is designed for agents to interact with the data and access it that way. I still believe that one of the things
that is important when it comes to AI giving you information is the trust that verifies. It is pretty smart. It is pretty clever at what it can do. Sometimes it gets wrong. You need to have that information that you can go and look at to say how did you come to this conclusion, how did you make this decision, etc. And often the best way to implement that is still graph. We have that in
the environment. When you do an investigation, you will come back and explain to you, I saw this trend or this spike. That is why I have gone and done this rabbit hole. So being able to explain that graph is really easy. That is something that the user is still going to want to be out of a tune. That makes sense. It makes a lot of sense that at the end of the day, mostly what you are
looking for is a question that is causing a problem or a pattern or something like that. Are you doing things proactively? Is the tool looking at these patterns proactively at this point? Yeah. We have a few different things that we do. The age of AI, I think we are in it. But there is
a lot to it. Everyone now just associates with LMs and genoger. There is a lot of other AI practices that we use internally to get better quality data. The other ones are great when you can give them a lot of context and quality data. But it is the classic junk in junk out. You want
to make sure they are getting clean data coming in so they can make the right decisions. We use a lot of AI tools internally to understand your telemetry data. The one thing we do that for is acknowledge graph, look at all the telemetry and build a graph of all the different entities and how they relate to each other. You can say I have this application, that is right, but I know it is running on a pod. That pod is running on a node. That node is in a cluster in a certain region. You can know the
relationships and that is right when we hand that off to the LMs. Suddenly when they do see a problem, I am saying a latency spike for this application. They then can traverse the telemetry and understand what I can find, what node is running on. I can see that that is a resource contention problem. Your latency spike is because you have a noisy neighbor who is consuming all the resources. Simple things like that. It is not simple to us. Being able to understand all that
telemetry becomes really important. The speed of which things are changing. In the old days you would have a same-to-be database. You would be like, I can describe all my assets here and it is a static thing. The reality is every time you have one of those, they get updated, it is something dynamic and generated by the data being ingested in real time. It is always up to
date. It is always a representation of what your environment looks like. From here and you write, this is different layers and applications of different machine learning like AI models that feed into reasoning models that can help you identify those patterns and things like that. That is awesome. I think that is important. I think that is what we are seeing. The LLMs are very smart, but you have
to give them the right data for them to be able to correct conclusions. That is what we put a lot of effort and focus on is how we make sure we are curating the data set properly. We can use a lot of machine learning as well as leveraging things like the open standards. Leveraging things like open standards means that we know the naming conventions and the semantic conventions. We know what the data is representing. That is public information. The LLMs can understand it as well.
That is awesome. Just a pivot slightly. I think what I think on my mind is, observability has been about humans understanding what is happening in the system. As we use LLMs more in the applications themselves, we start to lose a lot of control and understanding of what is happening there. How is operations changing with that paradigm shift right now that you are seeing with your customers?
Yeah, from our perspective, we see the shift of this kind of like a genteic model of software design. It is just a new design frame. We had mainframes, model lists, and then we moved to microservices. Now we see this as a genteic as a new way of building software. It has got some different problems that it introduces of how we understand what is happening. The tools we have for the most part
get you a long way there. As we think about how we understand what these agents are doing, it is typical applications of ability or APM. You want to be able to collect that to let me know what is happening. Now we have some new types of data and things coming out that we want to collect as well. A lot of the conversations to understand. We can go and write our e-vowels
and do interesting things. You are giving correct responses. When things go wrong, try to understand how did it get there? What was the input that caused it to go off the rails? We are seeing different demands across the market where people want tools that can just do this, that can just build into their agent framework. That is something we have to do internally as well as we are
building our agents and capabilities. We need to understand this. As we do it, we are just just scratching our initial problems and share that with the world as products. We did the same thing. We announced last week that we have in Grafana Cloud our AI observability, which does this right. It builds on top of open telemetry to collect additional things.
Then builds an experience where you can go and have a look at the things that are important to you when you are building agents. It is reliability and quality as well as cost. How many tokens are going for all these conversations? Which kind of workflows within my stack are the ones that are using the most tokens? That could be a good thing. Often we care about token usage because we
want to see adoption. We are in the growth phase. It will come a time in the near future where suddenly people will be like, let's think about how we can spend less on our tokens. So we want to give that visibility to our users so they can understand what these systems are doing. We have to dive in and when things do not go as planned, we have the data they need to understand what went wrong so we can make changes so we can improve things over time.
One of the questions I had was what you are seeing your customers, what does it mean to monitor an AI feature in production? I think you explained a bunch of that. What I am wondering too is, what are those hooks into something like open telemetry? If I am thinking about how I might prompt a model to do something to me, are people saying as part of what you are doing, log out your reasoning steps to open telemetry or something like that where you have another agent that does that or is it more deterministic coding before and after calling? What are you seeing in terms of those? Are they integrated with the APIs themselves in some way too?
Yeah, so for a lot of the integration, you think about a lot of the tool calls. They are not really any different from any other kind of RPC call that you are making. What you are going to measure is actually getting a response. Am I getting valid responses coming back? You are going to want to look at latency. How long are these things taking to respond? That is going to impact the user
experience. Talk about what does monitoring mean or what does it mean? Why do we do it? The main reason for it is to make sure that we are meeting our customers' needs or our users' needs, meeting their expectations. One of the ways we do that with the ability and the simplest approach we have is around the concept of SLOs, service level objectives. Rather than saying, hey, I am going to go and
monitor my CPU usage. Do I care if I am using 100% CPU? Not really, because I am paying for a CPU, shouldn't I be able to use 100% of it? What I actually care about is, what is the quality of service I am giving my user? Am I seeing a latency spike? That is something that I care about.
We are going to pick what are those core user experiences that I want to make sure my application is delivering on. But even like latency, depending on what your business is, you will have your own objectives. Those are the things you will want to measure over time. As they change, that is when you know something is wrong. If your user is getting 500 hours, you know something is going wrong.
Or if you see traffic drop off on your site, maybe that is an indication that something is wrong when you want to have someone go and look at it and give you an idea of what it might be. But there is ways to collect all this data. One of the challenges with the challenges we have with Tracer is the volume of data. We have to use supplemental data stores to store that information. This is where we see one of the nice things we like to do in Grafana. We built this ecosystem
around integration and interoperability. We are able to bring data together from lots of different types of data sources. We have to draw those correlations. We can build a new database that is storing all this information. We can just stitch that together and be able to correlate between my traces. We have traditional data. We also tie that to the chat history or context information
that you want to go and store. Back to what you are doing inside of Grafana itself. You mentioned that hackathon assistant thing. In terms of the day-to-day operations, where are you seeing AI have the most operational wins for you? Maybe it is outside of observability itself.
Yeah. We are definitely very bullish on AI capabilities across the organization. We have a mandate of everyone trying to use AI wherever you can so we can take advantage of it where possible. Obviously, we are seeing huge adoption in software development. It is great for building software, for getting code out there and just being able to innovate quickly. We are actually quite fortunate
because we see the role of a software engineer is changing. It is less about the code and it is more about engineering becoming more product managers. We are really about understanding the user problem and then being able to explain that to a general goal of the code for you. That is something that we are, again, fortunate with Grafana. We have always had these bottoms up kind of culture where our engineering teams have been responsible for the roadmap. We have a great product team,
a great product manager, but they are there to facilitate for the bring the information in from our customers, bring the information for what is happening in the industry and we can see in the coming, facilitate the conversations. At the end of the day, we like our engineers to be responsible for the product roadmap because we want them to own it. Part of that is because we are building products for engineers. They do have good intuition and insights that are needed. We have always had engineering teams and had to focus on giving them autonomy to make decisions about product and to think about what other needs of our customers and how we can go and so that is now helping a lot with the tools that we have got where they are already experienced. They have been
product managers themselves and so they are able to do the tools quite effectively to help you build things and get new products shipped and delivered. The other area we see is more than the original site is investigations, understanding what has gone wrong. Again, I talk about the current system started as MVP. One of the things we like to do with our product, we are scratching our niche and before we release products to our customers, we want to make sure this is a product market fit. The way we do that is we make it available to our internal teams and if they start using it without anyone telling them to, then we know it is a good product. We saw this with your system.
Soon it was in there, people started using it. They get paid for something. They just send this and say, can you go and tell me what you think it is? It is very remarkably good at doing those investigations and finding the problems. That has been really powerful. We leverage that a lot.
It is not always right. Even if it is not right, it is going to reduce the surface area that you need to go and look at. It will come with suggestions or it might exclude some problems. It does accelerate our ability to find that, restore those problems. The other area we are seeing more and more is, which is a little bit stereo. As we see, I shift from the development side into more of the deployment
side. When do we start giving agents keys to go and just, you have written the code, why don't you just go and deploy it? This is where I get uncomfortable. It is a scary prospect. I don't think we are far away from it. Obviously, we see a lot of things in the news cycles with agents going to
people's production databases. People are a little bit scared about these kind of things. But me personally, I don't think we are that far away from it. I think one of the things we see is that we don't inherently trust the AI to do the right thing. That is okay. We shouldn't. We should
be cautious. If there is an opportunity for it to do something wrong, it will find that opportunity eventually. For me, I don't think that is any different to how we treat people. The number of times I have said to someone, we don't need to worry about the normal process. It is just a small change.
It is not going to affect anything or just deploy it in production. I didn't expect that was going to happen. We need to protect ourselves from ourselves already today. We have a whole bunch of tooling in place in our society. It prevents you from being able to do the wrong thing. You can't go and do a
change with a blast radius. It has to be small blast radius, limited impact. If something does go wrong, it is not going to impact all of our customers. We have things around cost management. When we do a change, we have a check that runs and says this is going to increase spend by 10x. You can't do that.
We have all of these physical gates in place that prevent people from making mistakes. That also works for agents. Once you build these gates in place with hard blocks that stop the agent from accidentally shooting itself in the foot, you get closer to this reality where you can start to let it do some things in the environment. I don't think we are far away from that. Having those controls in place is an organization with a jury thing. It has taken us a long time to build that. We
add new checks in place every time something goes wrong that we didn't expect would happen. That is easy enough to do with people. We move much slower. The scary thing for an agent is they can move very fast. They can make a lot of mistakes very quickly. We have to make sure we have
the right controls in place. That is going to be the thing that will allow us to give the agents a little bit more control to go and deploy things. They can write code. They can observe it and see if it is working correctly. If it is not, it makes them more changes. Just shorten that
feedback loop so they can base their changes based on direct feedback from users and what is happening inside the production environment. It is a scary thing. I think it is an exciting challenge. We want to get to that place where we can free it up. As our engineers, our job is to balance the how do I deliver faster and balance that against how do I keep the system reliable and meeting my customers expectations. We can write a lot of code right now. We can develop a lot of new features
but we can get that in production and validate. Once we get past where we can give agents a little bit more control, then that whole cycle will accelerate faster. We can get better insights and keep iterating and going. It is a scary prospect and there is going to be a lot of pain along the way. But I do think it is coming. We just have to find ways to do it in a control as safe as possible.
There is definitely a risk analysis that needs to happen there and how you step slowly into that. If you have the right checks and balances, I cannot wait for some of the stories that will come out from this. It is going to be unbelievable. I will give you a gap of what we are doing in the final lab. One of the things we are automating is when we see we do a deployment and be small scribes into a specific set of users and we can go and look at that and make sure it works.
We have automated checks that we can leverage for final assistance. We can say, hey, are there any problems that you are showing up? If it does find problems, we have automatically open a PR to rollback. Those actions we are seeing have a user and look at that and approve that PR before it happens. It is a simple thing. We can improve that and rollback. We are seeing these things where we are
slowly giving the agents a little bit of operational control because they can do things much faster than we can. That is only people in the way. It depends on where that change or what change it is making. I don't know how I feel about an agent deploying a data migration on millions of rows of data. But can I have an unfix? Something that could be easily rolled back? Why not?
Exactly. I think that is where we are going to start. It is safer operations. We are going to be more comfortable giving the AI tools. We will definitely say there will be a bunch of young startups where they are very risk tolerant. I just came back and I will do whatever at once.
That is okay. Hopefully not important ones. Let's talk about the people using these tools. As AI handles more of this operational work, how do you see the role of SRE DevOps engineer, platforms engineer changing? Do you think they are getting bigger? Smaller?
I think right now they are bigger. Simply because the volume of new software is growing so quickly. That is it. They are getting left with the hard problems. A lot of the simple work that SRE is used to do can be done by AI now and be done by the agents now. Which then
leaves the SREs to focus on more high value. How do I go and think more strategically around how we build a reliable way of doing things? How are the AI not supporting the business or helping you drive things forward? Right now I think this is just growing for it. We are shipping a lot of code.
It is of questionable quality. Sometimes we do need more of that kind of practices to make sure we are thinking about reliability and make sure we understand how we measure it and make sure that building cool features and you take is a different value we want to have in meeting our customers' expectations. When things go wrong, do we have the data to understand what happens? What went wrong? We can go and fix it. I think that is definitely the case. As that changes over time,
I think we are still going to need those practices. The thing that scares me the most both from an SRE perspective is also software development is just that because all of the easy things are being taken care of. That is the work that you give your junior people. You just have to go and do it and as you are doing it, you learn how these systems work, you learn what to do. But as we take that opportunity away, it is how we are going to give people the opportunity to develop the skills they need to become these experienced sinuaries that we are still going to need. That is the thing that scares me a little bit is what is going to happen in
10 years time when we realize we don't have anyone who has the skills that we need. I don't know how we fix that. I think it is the same with software development. We still need real experience software engineers. The new grads and juniors aren't getting the opportunity to develop the skills where they can have that experience. They have learned things the hard way
and have gained that experience. They have earned it by making a lot of mistakes along the way. We are taking that opportunity away from people. I think that is the scariest thing for me when we think about the future. The high level thinking that is just keeps getting more and more and more.
It goes from reactive and detail-oriented things to proactive high level thinking. I am in the software engineering game. I wish I knew who the author was. One potential way out of this is to go back to a apprentice journeyman master model, which I am a huge fan of software as craft. I thought that was really interesting. The only way you are going to get this experience is by
sitting shotgun to an experienced person and feeling the pain with them and seeing their line of thinking. That is especially interesting with SREs because you are reacting to these potentially nasty problems and trying to unravel that. I thought that was a real question. I think it is an approach. I think the scary challenge is an expensive approach where you are not getting immediate value from. I think a lot of businesses and organizations are going to be like,
sure, why don't we let some other company be the ones that train all the people and we will just keep using the agents. I think it is going to be very difficult to find that. I think one area where we do see this opportunity is still open source. We love to engage with our community. We do get a lot of young people who come and contribute and that is our opportunity to coach them.
We are happy to invest that time to nurture those people so we can help them develop their skills and so on. I will say there will be more of those community driven. I do think this idea of training people up to be productive so they can leave and go get a job at another company. It is not going to be an attractive option for level reasons. I do think more of that as software engineers, as a community, I think it is our opportunity to find ways to nurture those people coming through and give them opportunities. I think I can also support that. Pro-based is opportunity to write
code. There are ways we can communicate and share ideas and help train people a little bit to give them the experience they are going to need. Also, not all apprentices in the trades are paid either. We have been spoiled in the tech industry but they have been high paid internships that lead you to a high paying job. The playing field may be leveled but yeah, good points. I think even a lot
of the people I talk to, even still people we hire today, I love to go to our onboarding, you know, in person and get people out to go and talk to the new starters, what's happening. A lot of them have similar stories back around to me, right? They are very self-taught, right? I think that is true within this industry where a lot of people love the technology, they are excited about it, a lot of the fact that things are changing all the time and there are always new things to do. You have to have this mindset of wanting to go and teach yourself and learn new skills.
And so I think that is still going to happen, right? I think open source is a great avenue for that where it is an opportunity for you to go and do something off your own mat to develop those skills that you need in the future because the speed at which things are changing in this industry, right? You have to be committed to a life of learning because you blink in as a new JavaScript framework. I hadn't played on talking about this but I feel like it could be fertile ground. So are you seeing open source projects struggle with the amount of code that agents are producing or what are you seeing in your area? A little bit, both from the volume as well as the quality.
Yeah, so it's very easy to have an agent to go and build a feature, right? But just because it works doesn't mean it's code that you want to maintain over time. So that's I think where we're seeing the challenges. I think this is still where there is that challenge with the artistic kind of code tools where we put a lot of thought into the architectural design and guiding principles for how we build code, right? And it's every organization is different but you end up building your view of the world and how you want code to look, right? So it's got consistency and so it's much easier for
your teams to move around between your teams, right? Because the code bases are all similar, kind of layout, they're using similar kind of practices for how you go and build code. And so things look a little bit more consistent, whereas when you start getting the AI in, it doesn't care about those kind of things, right? It's just going to find a solution that works and go and throw that code in. And so there's a lot more work that has to go on to think about and care about maintainability, right? So that certainly affects open source, right? Where you don't just
want to accept any pull requests, right? That adds any new feature, right? You want to think about is that feature, right? For this project, right? Is that something that we can maintain over time?
How does that impact the rest of the ecosystem? So that's where we're seeing the challenge is that you'll get people that will just submit a pull request, you know, there'll be 20,000 lines of code that you know, the AI is generated for them. And you're like, well, no, right? We're not going to just to review this. It's too big. It's not how we do it. It's not how we do it. It's a small chunks.
And so it can be reviewed. So it can be maintained over time and understood. And so I think that's the challenge that we're seeing is code is a collaborative business, right? And especially for, you know, open source projects. And so the agents have to fit into that collaboration mindset, right?
Of not trying to overwhelm people with too much change all at once, because it's just not going to accept it. It's too hard to review, too hard to maintain over time. And to your earlier point, these things might be learning moments for people that are less senior, too. Yeah, definitely. Yeah.
So let's move towards the future a little bit. So what's the operational problem that you think AI and these tools are going to crack in the next two or three years that people are still doing manually right now, even right now with the existing tools that we have? Yeah, I mean, I think right now, we're at the stage where AI can be that first responder, right? When something goes wrong in your environment, right? So I work at fire because you've got an SLM, and it's like, hey, you know, it relates to spikes. Something's wrong. You're going to page,
you're on call person, but at the same time, you can go and spin off something in your platform. You can go and spin off investigation. So hey, go and try and work out what's wrong, right? By the time your uncle and you get in front of their PC, right, there's probably already a response to say, hey, this is maybe what the problem is, right? They've worked it out or it may excluded a whole bunch of things. So having that, I think is where we're going to see over the next
few years, and that's just going to get more reliable and more accurate over time. We're going to build more playbooks. We're going to build more skills for it so it can do more interesting things. But I think that's what we're seeing is that first responder, as soon as you know, something wrong in the environment, it will go in and try and troubleshoot that and find that request for you. So that way, instead of taking an hour, right, to find that request and kind of solve the problem, we're now talking in 10 minutes, right? We can go and get things patched up and move on. We're still
going to have the follow-up process, right? Which, you know, is so important for our, sorry, so easy to actually don't just end up leading, let's go fix the problem. So that's still going to be important. But again, you can leverage my tools to kind of process more data and kind of do those kind of things. But I think the near term, so the next couple of years, that will just be standard, right? Like you'll just be able to make a quick decision that 60, 70% of the problems
are kind of resolved before your on-call person even gets in front of the computer. Yeah, because right now for resiliency in systems, we rely on redundancy and things like that. But I think what you're seeing is what we're starting to trend towards is actually in a limited sense, but a self-healing system, a dynamic self-healing system, which is pretty amazing. Yeah, and I think that's going to be really important just because of the speed of which software they shipped, right? If you want to move fast, right, you're going to inevitably break sooner, right? And more frequently, right? Because you're trading speed for accuracy, right? And so the need
for better visibility just becomes more apparent, right? You want to make sure you've got the data that you need so it can go on fix. But also, we're just seeing that AI is having able to use that visibility data to understand what went wrong. And that feedback can be really fast. And that's just going to help us be out of shipping faster, but also maintain the level of reliability that customers expect from us. And I think the other thing that we're seeing AI very good at is obviously
processing huge volumes of data, right? Going and looking for those kind of trends, those hidden things that a user hasn't noticed. We do a lot of that internally where we can just say, hey, just go and query a huge volume of logs and find what's interesting. So we're seeing that use case change. We recently made changes to like, so a log aggregation system to just be able to do that smarter and better, right? To fit with that use case of what AI agents want to go
and do. So we've made a whole bunch of architecture changes and performance improvements so that it can query 10 times faster. Yeah, more importantly, when you give it a query, it can reduce the volume of data that it needs to go and scan and look at to go and do that. So that's- I'm curious at a high level what those architecture changes look like. That sounds like a very interesting problem.
Yeah. I mean, so one of the things- Again, yeah, yeah. We announced it last week at GraphineCon. So it's an open source, right? So yeah, I can definitely talk about it. I mean, I'm allowed to talk about it whether I understand it as
well as the people who have written it. That's a different question. So what we really think about is, you know, lucky it was designed for cost efficiency, right? So how do we store huge volumes of logs as cheaply as possible? And the way we do that is just don't index them, right? The
use case we designed for was a developer use case where you just want to kind of correct, right? Basically through your logs where I grabbed actually, you know, look at the log message and you just keep excluding things or you might have a- I'm looking for a certain string- personal edition, right? And so that's what we kind of optimize it for. But more and more, we see people wanting to do more kind of analytics, all queries, or needle in a haystack queries more often where like, I've got a transaction ID and I knew it happened some time over the last four days, go and find it everywhere in the logs. And so, lucky it was designed, you know, to be cheap to ingest data, right? And probably wear the cost on the queries side, right? Where we just
root for it. So we're just going to look at all the data and look what it's trying to do. And I can do that quite efficiently, but it still takes time, right? When you're talking about terabytes and terabytes of data that you want to go on process every second. So one of the big things we've made is we kind of introduced kind of similar to like-blowing filters, but it's a slightly different kind of approach where we can know which portion of the data to go and look at which one we don't need to worry about.
So if you're doing that needle in a haystack query, we can very quickly say, hey, actually, you don't need to go and look at the petabyte of data we've got. You actually need to look at these 10 terabytes that we've got here and it can do that very, very fast. So that's one of the ways. The other way is we've just put a lot of effort into the query engine itself to optimize it and just make it faster, being able to parallelize it as much as possible. Because that's easy to do, right?
We've got a computer system today, right? Where we've got a computer, we can just go and spread out the workload and get things done as fast as possible. That works insanely well for us as a cloud provider. We've got the benefits of the system, we can use it all using it, we can reach a very large infrastructure state and not use it all at once. So we can say, hey, this is one user's query. The second is they can go and use a thousand calls. And the next second is different
users. So we can get really fast response times because we've got a bigger environment that we're right with time. Fish and season scale. Exactly. I would love to just talk about that for an hour.
All right. So a couple of wrap up questions. This one I'm really curious about. So what's an operational scenario that keeps you up at night, especially related to AI, something that you think the industry isn't taking seriously enough right now? Well, that's a good one. I think one of the challenges for me is just a lot of the AI
agents that are being built today are just black boxes, right? And we don't know what they're doing. Sometimes we don't want to know. So I think that's the scary thing is the speed at which these systems can operate and make changes and do things, right? Because we've moved from the chat right to the subject and what we're like, we're like, hey, I do things, right? And not just
respond to questions. So that scares me when you've got these black boxes that are just no one knows what they're actually doing, right? We have a mandate internally where engineering teams are leveraging AI wherever they came to be built software, right? But at the end of the day, the person who merges that they're responsible for that, right? Like they're accountable for that. And so I get scared about when that starts to erode, right? And it's like, well,
who's actually accountable for the things that are getting deployed into our production? I was having a conversation. Someone actually just yesterday, right, talking around like this governance problem, right? We're seeing more and more agents, agent communication, right? What happens when you've got, you know, an agent in your organization talking to an agent in another person's organization, and they're just doing things, right? And something goes wrong. Who's accountable for that,
right? Who's responsible for kind of remediating that and taking that? So that's the thing that scares me the most is around, well, I guess two things, one is the black box aspect of it, like what are these things actually doing? The other one is the accountability, who's responsible when things go wrong, if we are giving all this autonomy to AI agents. Some really interesting legal and contractual things that'll come out of that as well. Yeah,
that's a really good point. It always comes back to humans anyway, right? You know, like, what's the throat to choke if something goes wrong? Yeah, that'll be the most lucrative career in the future, right? Whereas just AI, you know, scapegoat, right? Yeah, and yeah, okay, what's going on?
Yeah, okay. AI insurance. Okay. Hey, you want to make one of me want to make a new startup? No, I'm just going to do it. So all right, let's close with, I was like to leave the audience with something. So if you could
give one piece of advice to our audience about how to evolve their operations for an AI first, what would it be? Yeah, definitely. My story being about this is don't go and build an AI solution, make AI part of the solution, right? So we've got so many great products out there. You don't get to go and write a new product, right? That's AI first or something. Really where we see the value
in AI is when it's behind the scenes, right? When it's not in your face, there's just things just seem to magically work, right? And you don't think about it and you don't know why, you just know that it works, right? And you've got the response that you want. And if you think about AI, that's the approach that we want to have is leverage it. It's great capabilities, but it doesn't
need to be the only thing, right? That you're doing, right? I think, you know, think about when the internet came along, right? It was a new and shiny thing. And everyone's like, great, it's now internet enabled, right? Now we don't care, right? It's just an expectation that things should just
work and it happens to work because it's leveraging internet and technology that we have. I think AI is where in that sense, right? Now we're in that pipeline, where everyone wants to label everything with, oh, it's AI, AI, AI, AI, AI. We're really losing our care less than that at a time. What they actually just care about is does it work, right? Is it doing the thing that I want to do? And it
doesn't make happy. So like, I think that's the focus is just making sure that AI kind of works with existing kind of workflows. It's coming to meet people where they are and help kind of accelerate what they're doing and not trying to force them to have to go and do things they don't want to do. Well said, well said. Well, thank you very much. Really appreciate your time. No worries. Thank you.
It's very interesting. Thanks. Thanks for being here. Thanks a lot. It's been a pleasure.