The world that the AI lives in is all digital, and it's the actual world, and those things are not necessarily the same. When we started doing AI for coding, it was autocomplete. Nobody really knew to what extent you can actually build complex software using AI. And then it basically broke through the expectations because we were able to model more and more complexity in the systems. I just don't know if operations is constrained in the same way.
Where's even the data to learn how to build a system that understands all of the operations? My assumption is that we have all the data. The question I think is maybe about complexity. And to what extent can you pick the things that you've described that are operational and convert them into something that an AI can reason about? The dirty secret for the longest time of running like a distributed cluster was just human beings managing it, do you think we're at a spot where you could use AI to run one of these services?
I think that there's enough to be able to. AI can now write remarkable amounts of production code, but writing software and running it are two very different problems. In this episode, A16C general partner, Martin Casado, sits down with rapper CEO, Jad Naus, a systems engineer Martin first met when they were both at Stanford to ask what happens as AI moves from coding into operations. Jad explains how his 10-person team uses coding agents today while he thinks the strongest engineers are getting even more leverage from AI and how work they once took months can now take days. But the harder question is what happens after that code shifts, production systems face unpredictable workloads, failures, and an enormous number of possible states.
Martin and Jad debate whether AI can learn to troubleshoot those systems by detecting what change, forming hypotheses and running experiments, or whether operating in the real world is fundamentally harder than generating code. And that leads to the bigger question behind the conversation. How far can AI go when the answer isn't already contained in the world that was trained on? Jad, thanks for joining me on the podcast. For those of you who don't know, Jad is the CEO of Grepyr, but also we went to grad school in the same group.
And when I first met you, we had a new student, I walked into the office and there was a motorcycle helmet and a pack of marboros. I was like, this guy must be unbelievably cool. So my question is what happened to all your cool. Yeah. Question number one.
Maybe just for a context, maybe give a quick overview of your background. So just for those that are listening, Jad is one of the best systems thinkers I know. I love talking to him. We're going to go kind of do a deep dive of engineering from a systems perspective in how AI is impacting us, especially when it comes to things like operations. But just to ground it, maybe a bit about your background?
Sure. So you met me at one of the peaks in my coolness. Yeah, you're very cool. A motorcycle in marble cigarettes. I remember thinking, I was like, wow, our cigarettes even illegal.
It was so just corn. I remember mixing me one day smoking outside of Gates and looking at me and going, like, Nick, Nick, the Q and the Zare Advisor, Gates was the building. I was one of my favorite. I don't remember if I was a private car at the professor. And of course, my favorite thing about him is that he smoked.
I was like, the only professor. I just thought it was like great. Just to be clear, I don't smoke anymore. No, too old to middle age for that now. But let's say I went to Stanford with you.
You were always the guy I had to look up to. And you're like, hey, this is the guy I want to be. You want to grow up. I started four years before you did. So how was it?
And so you were four years ahead of me. You graduated a while. I had me went and started in Icyra. I stuck around, worked more on OpenFlow, helped build a bunch of stuff on the hardware side. From a security perspective, actually, one of the most fun projects that I worked on there was I built the first set of SHA-3 implementations that people were considering using.
And I built them in hardware to kind of show whether they work or not and whatever for one of my projects for my thesis. That was fun. And then I decided after a while that I actually don't like networking because it's just about moving bits from one place and putting them in another. Not hard to do. I think it's very deep.
I mean, there's something interesting about it. But ultimately to me, that's not it. It doesn't do it for you. Yeah. And then I actually went to the postdoc at MIT with Nikolai Zaldafitch, who was also amazingly deep and super smart.
He's phenomenal. I did some application security work at the time, started a company based off some of the research I had worked on with Ramesh Chandra over there, who was a PhD student at the time. And ultimately technology was too early. So what we did was, well, ultimately a lot of the work I was actually reading Ramesh's. And his PhD was on, well, hey, if you get attacked and you get exploited, can you recover from that by rolling back the state of the whole world that's related to that application and then roll it over on top of a patched version of the application during the next point?
Yeah. And so we started doing that in the real world. And we're like, that sounds really hard. It's actually kind of relevant to what we're talking about. Yeah.
To what extent can you actually bound the state for operational systems, which I would argue is very hard. But yeah, we almost got it working. Almost. I feel like for all long failed disciplines, this is why people get into flight track for smart people. It's like, you see a bit of progress, but you're ass and toting.
But anyways, I digress. So then I joined App Dynamics. I was there for about three and a half years. They went from about 200 people to 2,000 people, got acquired by Cisco. I was a Cisco employee for one day.
And I said, and then you're like, Chad, I don't know how you can work with me. Come do this VC thing with me. I'm like, what does VC? Well, I'm an architect, I do engineering stuff. What do you guys like?
Great. The less you know, the better. Literally what you said. It's true. And then the first day I started here, you said, slow down.
It's actually people that move from, like actually not bringing all the VC. It requires a lot of reworking the brain. Yeah. Just like going from a PhD to operating, by the way. I got to undo the brain damage.
That's what real world is like. And then I stayed here for a couple of years. A bunch of work with you. It was a really fun time. Worked with a lot of great people, like Jennifer, Mark, and Ben.
Satish. Satish. Matt was here on a first, almost towards the end when I was leaving. Yeah. And then joined the pliers, their chief product officer, helped them build their engineering product teams, kind of scaled from about an engineer, like 15 people, to over 90 across product engineering, and for second design.
And after about three years, I kind of had, for each one of these pieces that I owned, and I kind of was sitting around just coordinating and doing meetings all day, and wanted to go do something else. So left, and then started talking to a lot of engineering leaders about the problems that they're facing, and what they're trying to do. Everybody at the time, this wasn't 23. Everybody was saying, we're trying to really spend. Now, some of what you're spending money on, everybody would say, first was infrastructure, and then observability.
And so I started looking at that problem, trying to understand why do people think it's too expensive. It turns out there's a few things. First, people were spending somewhere between 5 and 20% of what they spend on infrastructure on observability. But ultimately, what they were really trying to spend it on was reliability. They wanted to buy reliability.
And of spend, like, predictability of spend. Reliability of their infrastructure and applications. SESC. And so they were putting money in observability as a proxy for reliability, and for the longest time, the more money you spend, the more reliability you got out. But as architectures changed, and turned into microservices, that are much smaller, that are much more vertically integrated, the volume of data that's being collected at every layer expanded, there was much deeper market penetration for observability, and kind of the volume of data that you're collecting and the cost of observability was actually starting to grow faster than the units of function that you're outputting.
And the marginal rate of, or the marginal increase in reliability for every additional dollar your spending on observability was just flattening out. And so that's why people thought it was expensive. But people were still willing to pay 5% to 20% on reliability. And so the question was, can we actually solve that problem and bring back that increase in reliability, that marginal increase in reliability, not necessarily by spending more money on observability, but some other way. And so the thinking was, OK, great.
If we can figure out what is the signal, what is the noise, can we actually turn the observability data that you're collecting that's mostly fluff and mostly insurance at this point, and do that kind of extraction of information upstream, do whatever you need upstream, and be able to ultimately massively reduce the cost of observability downstream. And if you can extract that signal in real time upstream, then you can, one, reduce the cost. But two, you can now feed it into other systems and use it for driving reliability operations afterwards. And the distance at the time, that was something that was called AI coming into the picture. And we're like, well, great.
If we are in the data stream and we're extracting the signal, then we are in a strategic position to feed that to whatever is going to come in the future, because that's going to be necessary. And so that's what we built. And so, Grappler helps companies. We started with helping them reduce their spend on telemetry massively. And so we put us in the data stream, plug and play, pass the data through, reduce your cost by 90%.
And now we're building out that second layer where we're feeding those signals into the AI layer. That's now going and troubleshooting, figuring out when things are going wrong and fixing them. I love to just talk through your experience and how AI is changing dev generally, which is a lot of people who could ask this question, a lot of people answer. But I just feel like, you're like the engineers engineer. You've been an architect.
You've run product. You've run engineering teams. Like, actually, how big is your company now? How many people? Wait a time.
Yeah. So you've got to build a lot of software with a relatively efficient team. So maybe just a high level sketch on, like, if you're a senior engineer like you that has architectural experience, what sort of gains do you think people are seeing? Maybe talk through how you use AI, what gains you're actually seeing, what is the stack, what is your process, how you think about it? So maybe I can give you a historical view of what's been happening.
I'm running a team and so on. And so when we first started AR, we're still kind of auto-complete. I remember at the time we were using Co-pilot and IntelliJ and I'm like, oh my god, auto-completed these tests. I'm like, yeah, but it's just auto-complete. And very rapidly, things started evolving.
And what I've noticed is that really good engineers are able to somehow put it into their workflows in a way where the AI starts training them. Oh, interesting. In terms of how to talk to it, how to get the best out of it. And what I noticed is that I do think that's model specific. Like, do you learn how to use AI writ large or do you get efficient at a given model?
I think there's some nuances that different models bring. I'm sure you're looking at your tweets. You've got to. There's nuances, but there is a way to work with AI. And I think that there's also a way to work with different harnesses.
And I think we tried, for example, using open code or different harnesses or using codecs versus cloud code. Different ones have different ways of managing contacts or thinking through problems or when to spin up the right sub-agents and what to tell it, like the kind of prompts. For example, with cloud code, I find that if I don't tell it, don't spin up 20 fable agents to do basic research. It'll go and spin up 20 fable agents to do research and burn all your tokens. So a lot of these things, you start learning about the harness and the models and how they interact with each other and then how they can interact with the rest of the world.
But I guess I can just kind of. In my mind, this is what I started noticing is that some people were adapting and learning and kind of doing it that way. And others weren't as adaptive. And then as we started hiring more people into the team, that disparity between OK engineers and really good engineers just multiplied 10x. Can you predict who would be good?
Was it good engineers got better and better engineers didn't? Or was it really OK? Yeah. So if you were good engineers, it's a better good engineer. There's some bad engineers.
All of a sudden, you can get engineers. No. No. It always ends up being you have to really understand ultimately the setup. Because the more complexity that your brain can handle, the more you can break down the tasks and kind of assign them.
I think this might actually be a non-consensus opinion. That I mean, I do hear it sometimes. But the idea that the better engineers are much better and those are hard, hard, hard, hard, less. You found this to be pretty universally true. I mean, look, my team is still small.
We haven't done it across 100 people. In my experience, for the things that we do, that is the case, because we build distributed systems and distributed algorithms and your brain has to handle the complexity. Because AI doesn't do it. And so for the problems that we solve, none of the models can actually think through the things that we do. Like, there's a lot of, for example, we have a query translation layer.
We were just talking about query optimization. And so you have to really understand the semantics of the query and how to get to the semantics of the query, and then explain it to the model, and then have the model go and build the translation layer, let's say. Yeah. So for the people that are listening that, what do you mean by semantics? So semantics are really the meanings of things.
Like, the semantic of a dog, of the word dog, is the animal that actually walks in four legs and barks. And the syntax is the fact that it's just three, a three-letter word, and it has three different letters. The shape of the word is different, and that shape can be different between different words. But different words can actually also mean the same thing. And so they might have the same name, semantics, but they might have a different syntax.
So a dog and a puppy might be slightly different semantics. They refer to the same. But they still code dog. So the way that, and just for like computer science, the way that I describe it very simply is syntax, is like the characters you type, and semantics is what the machine actually does. And so it's like the actual runtime properties of the system.
So you're the computer science, I did electrical engineering. You became a computer scientist, and I became a VC. So that's the problem, I see. So for people that are listening, I just want to understand, like there's the actual syntax which is writing the code. And then when you talk about syntax, you're just talking about the code itself.
And when you talk about semantics, you're literally talking about you have a machine that interprets that code, whether it's compiled or not, and that machine does something, and semantics is what the machine actually does. And these things are clearly very different, right? So for example, some semantics you can't even derive by looking at this syntax. You don't know what the machine is going to do, because whatever, you kind of have to test it. You have to test it, you know?
And they can go all sorts of different ways when you test it. And a very classic version of this is SQL, for example. Like you can write some SQL query, but how it actually performs, how long it performs, depends entirely on what the schema looks like, or what the data looks like in the database, et cetera. But that's, I would not call that semantics though. I would call that performance characteristics.
Oh sure, yeah, I think I'd be like, I would say like the semantics part of that would be like, depending on like what the shape of data is, or like what the configuration might end up with different semantics, because like you said, you want null to be treated as false. And you won't know that unless you also know the configuration of the system and so on. Okay, so for the purpose of this conversation, I think we should just say that there's like, the actual code you write and then just how this behaves, and we'll call the second one semantics just so we can have the conversation. There's a, well, you want to call, we need to dive into the semantics of this, and make sure they're clear. Because there's a lot of, okay, okay, well, we'll obviously be at the level.
Okay, so let's go back. So, you know, actually before we started the podcast, you and I were just shooting the shit, and you were talking about like how you can do products, projects now that take a week, where in the past would take a team in months. Can you maybe walk this through a little bit about like, how you personally kind of do projects, and then kind of the gains that you see. So like a little bit about like your stack and your process. So I use clock code.
I wanna say something that's actually pretty funny, but I use Windows. What? Was that created by this company Microsoft? And I use WSL. So I use Linux for everything like development and so on.
There's no like X or UI for it. I'll just use the command prompt. But how do you use Windows? Because I just never really wanted to learn the shortcuts. Are you a gamer?
It's like you tab over to Counter Strike or whatever. Like, why else? I'm not hitting on Windows. I mean, you know, like I used in the 90s. Really, like the only reason is I never wanted to learn the shortcuts.
I hate the one button mouse. I hate the one button stuff. I hate the number of modifiers you have to type for shortcuts and stuff like I just never, I'm like, why, why, why? Why waste time? Okay.
So I'm like, yeah, I've used Windows Windows here. And so WSL I use clock code. Where's WSL? That's Windows sub system for Linux. So it's a, they actually run the Linux kernel within Windows in a hypervisor and provide you access to it.
It's great. They've done a great job. Windows WSL and the clock code. And like, my latest, the latest thing that I do is I use Opus 5.5. I tell it.
It's a really good model. It's an amazing model. I've shipped in the last two weeks, these massive 20 PR things that I just dumped on my team to go and review. And they're like, oh, I'm like, don't worry about it. It's horrible.
It's like, it's like the new reporting structure. Like the people below review the PR for the CEO. Is that where we meet? I was like, no, everybody's like writing the same thing. Everybody's like writing.
Yeah, I, I, I review my PRs. And like everybody's like just, just PRs flying around. Yeah, yeah, but like the thing that has changed a lot in the review cycle is that like we started much more focused on, to focus much more on like the, the tests and like doing things like golden expectations and like, are they shifting in different ways and so on. But Opus 5 and like the, the trick that I recently learned about is that you, instead of using subagents, you basically tell Claude to spin up command line subagents because those maintain their persistent session over time. And so you can have it spin up these agents separately.
You can actually run them from other sessions and so on. And that allows you to be much more efficient from a context perspective. How many is this your run? How many? How many agents agents?
Typically I work on like three or four session windows at the same time because I found out more than that. I just start losing context between the different things because like they start finishing faster and then I, then they wait. And do they interact with the other PR networks? The subagents like together? No, they interact directly through the command line.
Oh, well, okay. Yeah. So like I asked like the main, like the main thing that does the implementation to have a tech lead reviewer and like they basically ask it questions, does this make sense? Does this not make sense? Review might be our et cetera.
And so they iterate on these PRs together. And then they have a PR doc that basically they work on with the design doc and everything in it. Like the first thing that I do is I ask that agent to do it, I ask the tech leads to review the PR doc. I go through the PRD, I go through the design doc, make sure it makes sense, go through each piece in like detail, make sure that all the algorithms work and like all the pieces work, the division of labor works, the vision between PRs and the relationships work. And then I thought it'd just start doing it.
Figure out if there's like a parallelization that you can do and then spin up other agents to actually go do them. But ultimately you have to kind of put it all back in one stack and be able to go through it as a human one by one because otherwise it just gets messy and bushy and you can't really tackle it. So how much time, so can you give a sense of like a project you do now versus I don't know, six years ago, like time differences? So the latest project that I shipped was something like. This is like, to be clear, this is production code.
This is like that one of the products. The statement you're about to make is the gains that you get for like shooting products. So I'll give you an example of the last, there's the last PR that I worked on or like the last set of PRs was what we call auto promotion in a data lake. So in a data lake you store all the data and typically the data is column oriented, parquet files and iceberg and so on. And when you get data from logs, they're semi-structured, they're arbitrary columns and so on.
In order for it to have the best layout of the data and to be able to query it efficiently and fast, the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows. And so we call that auto promotion or shredding. And doing something like that is incredibly complicated. You have to go and figure out how does the metadata system work, how do the commits of the metadata system work, how do you coordinate across different writers and so on. And that took maybe about 10 days.
And previous days, that would have taken maybe about six months for a single person. Well, and was this you? That was just me. It is insane. Yeah, so we're like factor of 10, factor of 20 improvement.
Yeah, production code for your products. Yes. You know, one question I always have is like I can very much understand, you know, these models are trained on all the code ever written, let's call it, right? So I can understand why they're able to generate code, which is the syntax piece. But then like for, you know, you're doing core infrastructure, let's say you're doing distributed algorithms, like, like the properties, like how it handles itself when it's running is very, very different.
Like often you just can't even look at those, the syntax to know what's going to happen, right? Like a lot, let's say, let's say you're writing some distributed system. And then like the way that it will perform the availability, the correctness even is going to be dependent on what the workload looks like, you know, like what are the access patterns, what is the traffic. Are you talking about like the performance aspects of it though? Or are you talking about like the availability?
Like availability, you may have to lock things. So it's not available for a period of time. You can talk about correctness. Like maybe you've decided to do a ventral consistency. And so like if you do a read, it's not going to be consistent because there isn't to lock.
And the performance, like, you know, maybe you cache, maybe you don't cache. So these are all kind of trade-offs in the design space, right? Yeah. And like unless you know like the access patterns, you won't know how it's going to perform in these things, right? And so I just think to myself, okay, I've got an AI, I can understand how it will write, you know, like code for a generic distributed data store, for example.
But I don't understand as how it will know to like make the trade-offs needed because it doesn't even know what the access patterns are. And it's not like these things can like, like where's even the data learn how to like build a system that understands like all of the operations, right? Like today we rely on human beings to like manage clusters and to like, you know, optimize databases, et cetera. And so can you maybe talk about how you think of like, I get the writing the code, but like you have these deep operational systems, like how do you play into that problem? And like, hey, have we made any progress or all or is this an outstanding problem?
The question I think is maybe about complexity. And like to what extent can you take the things that you've described that are operational and like that happen in production and convert them into something that an AI can reason about? And exactly. Like how can an AI reason about this system? Well, a first off, like to what extent do you see that writing the code that it can actually predict what's going to happen?
Like how good a job is it to do that? And the second one is like, okay, now how do we make it reason about this? It was asked the first one, like if I use your method, the JAD method of writing code, I build something distributed system, like how often does it actually get a performance correct? How often is it like actually operationally like pretty sensible? And by the way, I work on this distributed Gaussian splatter and I would say it's shit, like it's horrible.
Like because like, you know, you've got millions of Gaussian splats that you've got to page in and out and stream and like, it just totally depends on like the running system. So like none of the models do a good job. So in our world, what I've found is that we've defined basically really well what performance benchmarks look like and how the system should run and what the constraints are actually building within like our stream processor look like to get the right performance. But every once in a while, it'll do like my latest project has been all about query optimization and query performance. And so I was like, hey, you're asking examples.
I was like, we are basically generating code that corresponds to SQL and we want to run it in a pipeline. And we're like, okay, well, you're generating shit code, fix it. And it started basically going through and what it didn't realize was that it wasn't actually doing initially any optimization on the code. It's like, oh, it's fine. It looks really cool.
It's a problem. And I said, well, look, there's these conjuncts over here. You're doing and then like this particular term is repeated and it should have been taken away. And it's like, ah, okay, let me figure this out and so on. And then like it basically went on a thread to try to figure out what are all like it built a bunch of hypotheses around like what are all the things that it can go and optimize to make this better.
But ultimately it was able, because it was able to ground that in performance benchmarks. And I was like, look, here's a data set of the queries and like what the users are asking and so on, go work on it. And so it can basically do a relatively good job that way. But when it comes to saying something around like, how much memory should I give the system? It has no idea.
And that has to come from actual experiments that run in production. And so what I think actually these systems seem to be doing well at this point in terms of operational stuff is that it knows that, hey, I probably won't be able to answer this question. Let me run an experiment somehow and come up with that information. But of course, when it comes to operations, like production systems don't behave the way you expect. And ultimately that's why incidents happen.
So have you found, are you doing any of the AI SRE stuff where you actually have a running system that has logs and you feed it to an AI? And that helps you tweak it? Yes. And how does that work? So we're just, as anecdotally, my experience with other people doing this has not been that great.
But of course, I've been getting anecdotally. So a couple of things. The first one is we're doing that for both operational pieces and for security. But the way that we're tackling it is not the way that other people are doing it from an AI SRE perspective. A lot of those systems basically start from a lagging indicator, like, hey, this metric spiked or this alert happened.
Go back through a very complex system and try to determine why that happened. Like, that's why it doesn't work. Because backtracking into a root cause is actually very problematic. And so instead, what we're doing is doing it the other way around, which is can we actually start by figuring out if there's something different in your environment that has changed? Like, can we do a large scale clustering, a large scale, kind of figure out a needle in a haystack, something that is different?
And now zero assuming that it was working fine before. And now it's not so something happened. But it could be just like a change in normal traffic patterns, right? That's fine too. That's fine too.
And so the thing that we're looking at is like, when things go wrong, usually the system will tell you something went wrong somehow, right? If it's working fine and scaling as normal, nothing will happen. But sometimes you'll see some message which says, error connections ran out or memory exceeded or resource buffers emptied or like some kind of indication, like some kind of really early warning, like something happened. And that's the one message that we want to capture. And then go from there and see, is that an actual problem that we need to investigate and deal with?
So how does it do? So far it's doing pretty well. I mean, we're running it internally in our systems. We ran it in a couple of our customers environments. And in the cases where there was a couple of cases where we ignored it and those turned into production incidents.
Really? Oh, wow. Yeah, it's kind of just from an intuitive perspective, like I can understand how these models understand syntax because again, like I said, they're trained on all of the code ever written and it just turns out that it covers a lot of useful cases. It's hard for me to actually even think of a bound on the actual operations of software, right? I mean, this is a ridiculous statement, but it's actually kind of true.
It's like every time you add a bit to a program, the state space of like what it can do operationally increases by a factor of two. So you very quickly get to like more states than atoms in the universe. And arguably, this is why product approaches to operations have never really worked, right? Like the KS engineering stuff never really worked. And actually, the dirty secret for the longest time of running a distributed cluster was just human beings managing it.
So I'm just wondering. And then in order to even train an AI to do it, you have to have enough data and enough configurations. So I'm just kind of curious, do you think we're at a spot where you could use AI to run a distributed cluster or run one of these services? Or do you think that's still on the come? I think that there's enough to be able to have it start that loop, right?
Like, I mean, ultimately, like, I think that, I agree with you, there's infinite space. There's a lot of different things. But ultimately, humans also operate on runbooks. And like, sure, there's a lot of things that happen in the introduction that you ultimately need some human to go and like, hey, this is too complex. There's like some state that's hidden that I know of because I talked to this other guy, like at the water fountain the other day.
And he told me about this thing he deployed. Like, there's a lot of stuff that potentially isn't visible in the state of the system. But I'll argue that the goal of what we need to be able to do is to get to have enough telemetry and enough understanding to make everything really visible from, hey, here's what the system looks like, here's how it's working, here's like whatever. And then I think ultimately, most of the issues that end up really showing up in operations, assuming you have the system previously running OK, is due to some kind of change. And if you're able to just focus on the goal of figuring out what the change was and kind of going through this as a stupid question on this, which is, I never think of these generative models that could predict stuff.
And it sounds like you're talking to modeling like you predict a change or you baseline and you detect an anomaly. Like, am I thinking about this wrong? No, you're not. Because the goal is not actually having the agent try to predict, but have it get a signal that something changes. Oh, that's a deterministic.
That's something I see. I see. Yeah. Wait, wait, wait. So there is a separate model that does a predictive stuff.
And then that will alert the AI. Yes. OK. Yeah. So what we do is we basically give us your petabyte of logs per day.
And we scan all of them. And every time we see a new kind of pattern emerge in your logs and use things that we haven't seen before, we lift it up to the AI. So that's a predictive model that actually does what predictive models are. Good at which is determining out of norm things or whatever, whatever. Exactly.
And it goes to the AI to the AI kind of uses that to figure out what to do. Exactly. So the AI basically can then reason and drive that down. So I can say, oh, just message. Let me actually look at the code and figure out where it comes from and understand what the code does.
Let me see if it's actually happened before. Let me see. So it basically runs all these hypotheses and heuristics to try to understand does it make sense as a meaningful and some kind of way. And then it can determine what to do with it. So how far away do you think we are?
Let's just go back to query optimization and like log to something like, so how far away do you think we are from having an AI basically do the job that a human used to do when it comes to query optimization? I think very close. Really? Yes. Wow.
I think running a distributed cluster. I just think like, OK, you've got James Cowling, PhD from MIT who runs a distributed cluster. And the cluster's down, the company doesn't work. And then if it's up, it's running. I don't know.
Like I read at the point where you would say that an AI can do that. No, not yet. OK. I think it's, I think, I mean, I think like very, very simply, when we started doing like AI for coding, it was auto-complete. And nobody really knew to what extent you can actually build complex software using AI.
And like when we were looking at it, we're like, AI is never going to be able to build X or Y or Z or whatever. And then it basically broke through the expectations because we were able to model more and more complexity in the system. I just feel like I'm a very simple thinker on this stuff. And to me, these models are distribution and distribution out. And it turns out like you can actually train on all the code ever created.
And it's actually like that manifold is pretty manageable. I just don't know if operations is constrained in the same way. If that feels to me more like kind of like the universe, like heavy-tailed and exception-based and power laws and all the tough stuff for these models to learn. But I don't know. I mean, you're the one that's in the field.
So maybe that's not the case. Maybe they're manageable. But I will tell you, the history of systems, as you know, is you always want to build a smart algorithm that does the efficient thing. The smart virtual memory management, the smart congestion control, the smart routing. And what do we do instead?
Every single time. We build a lot of provision. We know we over-provision every time. Every single time. Like remember, like swap memory when you like swap to disk.
If you did that, you were fucked. Like that's that your computer was dead. So like we literally over-provision RAM. And like routing, you know how many adaptive routing algorithms are like, we will respond to traffic. But you have oscillations like fuck it.
We over-provision. We just use DC&P. I just feel like the history of systems is that you don't predict stuff you over-provision because you can't. And I feel like in the operation side, so either we build the systems up front, they're over-provision, so you don't have to do kind of complex operational stuff. Or maybe I can do it the way that a human being.
But I think it's very different than code, because you're dealing with exceptions. There's not a lot of data, et cetera. I just feel like a different domain. But I'm a VC. I'm just like, you know, I kind of like the reason that I'm more optimistic here is because I live in the world where like what we do is actually deal with the observable world.
Like obviously you're the literal expert on it. So it's silly for me to even like ask you these questions. No, not at all. Because in my world, my assumption is that we have all the data, is that we know exactly when like a new thing happens, or like an exception emerges, or like a workload changes, or like whatever. In my world, all of that has already been collected and streamed through.
Yeah, but the model that is not trained on a grounded source for that stuff. No, no, that would be really hard. Right, but the story here isn't like, hey, Mr. Claude, here's a bunch of logs, is something going on, right? And it has to be much more able to run experiments and kind of dive in and see like what are the different shapes of possible causal chains that actually fit in the environment that exist.
So yeah, this is what you said before. Now I see what you're saying. You're saying that like the universe is heavy-tailed, the number of bad things that can happen is probably an exception thing, but like the set of steps used to debug it is a management thing. Exactly. That's what you can teach one of these models that's not happening.
And therefore, if you give it the right signals, it can go through a number of steps to constrain this problem. Exactly. Interesting. Exactly. And it really is taking the role of the human.
Because remember, remember the whole chaos engineering wave? Yes. And the reason it didn't go anywhere, is like you're very quickly like trying to like, you know, reduce like a state space that was atoms in the universe, like super quickly, like it actually makes that manageable. And it was like the human being that would go through a step that would do that on the actual debugging. And this is what you're saying the AA would do.
Yes. Right. So the goal isn't to like have the model just be like, this is what happens. It's, hey, run through this workflow to determine what happened. And like we're trying that also on the security side, for example, where like, hey, in security, you have something different changed in your like, statistics, like some behavioral difference in terms of what entities exist, the cardinalities of various things that exist.
And so that becomes a perturbation in the whatever the universe. And so we take that signal and we feed it into an AI and we're like, hey, AI, come up with a bunch of possible causal chains that exist and then go through and verify if any of those fit the pattern of the data that we actually see. And so that becomes a much more like, hey, here's a, here's like a, I don't know, like fit this square into find the peg or like find the whole that fits this peg. That seems even harder to me because the operations one of the argument is, the system can get in states, into broken states that are kind of like very rare. And there's a lot of those rare states.
And so it's just not something that you could, you know, train a model on just because there's so many rare states that can get into that are that are bad. And so maybe you can reduce that by doing the steps. The security one seems even harder because this isn't just like the natural order of our state. You actually have an adversary that is trying to like intentionally find rare states and intentionally find stuff that you can't do. And so it may be the case that like code becomes solved.
It takes a little while, you know, a lot of operations that take a lot longer. Yeah, but like also like think about, I mean, I don't know, to me like the whole world is sort of quantized to a certain extent to plank length. Is that what the physicists say? I don't know. I don't know if the last time.
I think there's like, I actually, I literally don't know what I'm talking about right now, but I actually think that like like space time is just good ties into like, but. But yes, I mean, I agree. Like to a certain extent, what you just said, but for operations and systems in general, like there is a discretization of like the possible states and the past through them that are possible. Yeah, right. I just think when you're dealing with, there might be a ton of them.
There might be a ton of them. I'm not sure you've ever seen our, was in our grad school group. Yeah, soon there, yeah. Oh yeah, yeah, yeah. So I remember he came to me once and he was looking very sad.
He's like, well, I just had this really sad realization that, you know, if you take any MP3, I remember he's like, yeah, he said, MP3s have a finite, let's say, you know, four minutes. It's like four megabytes, you know? And that means there's a finite number of songs in the universe that comes, no, because, you know, because it's all made of bits, right? And it's four megabytes. And that means you can just do right.
Every one of those concerts, I'm so sad in this notion of like the, the fine-ditus of music, right? This is like, we're walking on the dish when we had this conversation. I'm like, motherfucker, dude, you know how quickly you've like overcome like the atoms in the universe. You haven't even gotten to a cane, right? It's like two to the two to the two to the two.
Like what are you talking about here? Be happy to do the two four megabytes. Like what are you talking about? Like there's so much to say, like be happy. That's effectively infinite, right?
It's like what are we talking about here? So I do think at some level, like we do underestimate like the number of states that like, you know, systems can get into it. Like maybe, I will tell you from the perspective of VC that looks at these companies, the ones that are trying to tackle operations or having a tough go at it. It is a, it seems much, much tougher than those that are tackling the code. Cause I think it's a much tougher problem.
I 100% agree. It is a much tougher problem. And it's not just because like the systems themselves are, you know, there's like external loads that are random that kind of happen to you or that, you know, somebody just shipped a new thing and rolled it out into production didn't tell anybody. Or like there's some state that actually happens when like this thing happens and this thing happens and then you end up in a deadlock. It's also because it involves humans a lot more and not just like one human who's the coder.
It actually also involves pools of knowledge or distributed to exactly. I'm like the interactions between like what it takes to like connect the pieces and put them together. Like, but ultimately, I mean, I think about like where AI is and like what are the possible futures and like the depths and the forward looking capacity of all the things. Like I think code is just, to me, it's a microcosm of like, maybe I'm just too fatalistic. Maybe I'm just like too pessimistic about how the world works.
And I think that everything is like following some kind of probabilistic path. And so to me, like if we can make it work in code, we can continue expanding that level of complexity bit by bit and kind of adding layers and layers and layers until we can handle those problems of operations. It's not fatalistic is the opposite. It's very optimistic is what people would say. For me, I don't think the universe gives up secrets that easily, but like we'll see.
I mean, like this is kind of the big question. And I would say, as yet it's not an answer question. No, no, but I don't know. A lot of things that as humans we think of as being random or impossible to tackle. Yes.
That's what we saw with coding. Like, you know, initially people thought like the possible world of all these things that you can implement in code are kind of limited and you can't really get a system to potentially understand the full complexity of a human brain that's required. I think over time it all, it's so funny that I'm actually telling you this and you're like the VC, I was looking into the future and being like, no, well, I mean, I'll tell you, my view is like the study of these models are you train on data and either that, I mean, the, what's the right to say? So you train on data and that represents a distribution that distribution is just for or not.
And so like when you train on language, the fact that it's useful, that's to me says that humans are so remarkable because they've looked at the universe, which is a very high dimensional system. And then they've reduced it to a manifold that's actually useful for a model, right? Yes. Right. But that's different than the universe.
Like that's, there's a reduction from the side dimension to the language is less dimensional. And I think that code is a reduction of all the operational states of a machine until like the structure that make it run. And so to me, there's still a disconnect between like the instructions or the description and the actual, like the actual running of the universe and actually running the system. I think we can make gains toward it. But what I will say is like, you're kind of converging on simulation, like there are things that are computationally irreducible, like you're going to have to run it to get the answer.
You can't just like, you know, predict it. And so like maybe we just are at an event horizon where like, models will solve anything that you don't have to actually simulate to get the answer from. But why? Why do you have to like just think about it in terms of, now as opposed to being able to simulate and sorry, being able to plan its own execution to gain the information that it needs in order to be able to. Yeah, maybe that's where we're going.
Yeah, that's kind of what I'm fair enough. And I think like there's a lower dimensional, there's a lower dimensional space, which is like the steps that you would take that we can reduce a whole bunch with AI. And then whatever steps that you need that that can't do, it can create the code to do or it can simulate. And the point that you're making though is that like, hey, look, the model can't actually go and figure out the additional dimensions of the world that we need to go and find to like model the universe into its world. And then you can see that the world is bigger.
That's probably true. But can it actually simulate a scientist sufficiently to be able to get to that point? Like maybe, maybe. Cool. Can I ask you a question?
How much of the VC work do you think is something that an AI can drive? So my model for VC is its very relationship driven. I think, so it's OK. So I just have to believe that markets are actually pretty efficient. And so it's not about finding the company that nobody else finds.
I think people all find it like the right company. And then I also have the model VC that tech is positive. And I'm kind of optimistic about all tech. And I don't worry about TAM. And so I think the key is raising money.
It's access. It's ability to win deals and it's ability to add value after that happens. And a lot of that is stuff that startups can't naturally do, like, recruit big execs or help with that. And so the way that I think about venture, you kind of have to be a human being to do. It's like, because that's the actual topic.
So for that, I don't think like an AI. Like, if you have an AI simulated universe where it could like, whatever, reward hack, building trust relationships, fine. But I think that that would have to be to be. That said, I will say I have seen an analysis done by AI. That is better than any genialisome ever worked with.
So for the analysis piece, they're great. But I think this is more an indictment on VCs, ability to analyze that it is, like how powerful the models are. I think of all a Kabuki show in bullshit. I just think the right VCs are like, tech is net positive. Techways always add value.
Like, let's find the best company and investigate. The markets are very, very efficient. So I just feel like the answer to this question is almost kind of a technicality. Like, my answer is almost a technicality. So how, like, you know, one of the things that I came here, I wanted to ask you this real question, which is like, you know, we started talking specifically about ops.
But to me, like, there's this world of AI writing code, which is kind of like this abstract simulation world that it lives in. And then there's the rest of the world. Exactly. And the entire discussion is this. It's like, there's like this, there's like the AI, the world that AI lives in, which is all digital, and like, whatever it is, the training data, the indestribution world, and there's the actual world.
And like, those things are not necessarily the same. They're quite often not. And so the question is like, to what extent do you need to train an AI to be able to survive in a world? And like, what are the things that it actually, I don't want to answer that question, like, what do you need to train? But like, I'm curious about the open spaces.
So we talked a little bit about operations. You said, I don't know if it's going to be solved or not. I'm like, yeah, Martina's going to be solved. And in my mind, like, there's so many other things. Like, for example, self-driving cars.
Like, for a long time, we thought that's not going to happen. But it's clear that it's actually going to be done. And it's going to happen. It's just taking time because of a lot of different things. And so what are the other next stages, the next areas that you think are kind of like lined up right after code?
I mean, my sense is, AI is going to continue to take off in a wildfire for a constrained domain, where the solution space is in a constrained domain, right? Where the distribution of the solution space is the same as the distribution of the trainings. The distribution of the syntax for code is the same as the distribution of the trainings. The distribution of the generated images is the same as the distribution of the trainings set. In those areas, I think it's going to continue to take off as a wildfire.
The domains where the distribution of the solution is the natural world, I think it's going to have, it's going to be much harder to take longer. And I think the economic equivalency is going to be much more difficult. Self-driving is a great example. I remember when Sebastian Thrun, one of the DARPA grand challenge when I was at Stanford, remember that? And we were like, hooray!
You know, some type of cars is around the corner. Seriously. And it took 20 years and $170 billion investment to get to worse the economics than Uber, right? So I think that solving real problems where you have to deal with the distribution of the real universe is just much, much harder. And so I view as AI as a general tool.
It's a very effective tool. It's one where you have a distribution of data and then hopefully you can apply that to some problems. And we'll apply to a bunch of stuff. And I think economic equilibrium is going to determine what works and what doesn't work. Because I think it's a very, very general tool.
It's like a universal function approximator. And wherever that nets out, I mean, big dent. One of the things that I said earlier was how AI was training humans to use it. And I very often think about some of the problems that seem intractable is because the world is kind of that way. And then we're trying to fit AI or the tools that we're building to manage the world as it is.
But in some situations, we actually tend to change the world to fit to what we can actually do for 100% of course. I think this is where the budget was. It's not exactly the 100% of the data warehouse. The automated data warehouse. And so the question like that always comes to mind when we were looking at these data warehouses and whatever.
And the only one that worked was the one where it was like, you know what? Just automate the whole thing. Yeah. And I wanted to like what extent as we kind of improve or change how we kind of expand the domain of what we want to solve with AI tools and so on, that we actually started modeling the world or changing the world and the things that can actually happen to fit what AI needs it to look like. Yeah.
I mean, I think that right now a lot of AI just operates in the digital world. Yeah. And for that is literally the solution set is the same distribution as the training set. Yeah. And so that's fine.
And I think that again, like in the case of equilibrium, like I think like when you're dealing with the real world, like you can solve any problem you want. It's just a matter of like compute and money. And the equilibrium point is going to be an economic one. And I think one way you can impact that function that you're doing equilibrium over is to constrain the environment, constrain the system. And so I think that we're going to like we're in now in this meta economic equilibrium flywheel where we're kind of optimizing this is super general, the very inefficient process with a solution set that people value.
And I think this is about 30 years of this and a lot of value is going to be created. You know, you want to hear a funny story? Yeah. So we built this AI thing and we're like talking to people, we're like reaching out to customers. We're like, hey, we built this AI thing.
We're helping customers become more reliable. Here's like what we're doing. We're like, great, we want to talk to you. And we go on the call and we're like, hey, here's what we do. We built this system, help you optimize your data, it helps you reduce your costs, and then it helps you become more reliable.
Because now we can find these problems and solve them before they hit your incident. They're like, wait, wait, wait, wait, wait, wait. Roll back. You said you can save 90% of our data dog bills. Yeah.
That's why I want to buy. But the AI thing, they're like, no, no, no, no. That's what we want to buy right now. And it's interesting, like a lot of these, like for code, I always think about it, it's like, oh, for code people just get it. They're like, oh, shit, it just works.
And on the upside or anything that's beyond that, it's very still to understand the actual value and put a dollar value to it and be a pay up to it. And 100%. And I also think from the cutting standpoint, literally is engineering productivity, which is one of the highest line items on these things. So it's just very true. Exactly.
But you can also think about it a little bit. Sorry, productivity. Or it's like your business going down. Maybe you can save an incident at the cost of, I don't know. It's weird.
I think like how people think about it outside of the. Yeah. I also just think like, this code was so magical because it was so good. Nobody thought that could happen. And I think the operation stuff now, it's still like, pretty early.
It's still pretty early. So yeah. All right. Well, listen, chat. Thanks, Deputy.
Come in. Thanks.