The fundamental idea here is to build AI that improves itself and I think we're getting there. For the first time, we could swap the optimizer as they optimize the and this leads to a reinvestment of cognitive improvement into itself. You can think of RSI at many different levels of the stack. Prompts, hones, weights, and you can go all the way to basically chips. Our goal here is to optimize all the levels of the stack.
Our approach to RSI is that we have a multi-agent system working together that's been recursively improved. I'm so impressed by your belief that this idea of a model improving itself. But it's not a belief. It is a dramatic statement which is, I think it's a first-time effort you say it. Because we have been so thorough, we have not released things until we're super sure it's correct.
Do envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement. We can't share too much, but one of the very important things that we've tried to do. A16Z general partner, Martine Casado, sits down with makermaker.ai co-founders, Daval Ajada, and Owen He, to unpack recursive self-improvement, or RSI, and separate the increasingly popular term from what it would actually mean for an AI system to improve itself. Daval and Owen distinguish true RSI from simply using AI to automate pieces of AI development. Their goal is a closed loop, where the system being improved is also doing the improving across components including models, harnesses, and eventually more of the AI stack.
They explain why closing that loop still doesn't guarantee runaway takeoff, and why measuring real progress requires accounting for both capability gains and the resources required to achieve them. They also share what they're seen in their own research, including early signs that self-improving systems can increase performance while improving efficiency. And they dig into the practical challenges of proving those gains are real, from variance and reward hacking, to the enormous difficulty of distinguishing genuine improvement from noise. Welcome both of you. We're going to be talking about everybody's favorite topic, which is RSI, recursive self-improvement.
Maybe to start would be great to just introduce yourselves and your background. So we'll talk a little bit about what you're working on now, and then we'll dig into the content. I think of myself as a physicist of some kind. I studied that, then I did the PhD in reinforcement planning at MIT Media Lab. I really like building things, and I'm very excited to be building closed startup, maker maker of Owen and Grace.
My name is Owen. I was first studying math, and then one day I realized that all mathematicians do is out-proofing theorems, and I was taking a class called Formal Language and Logic, and they were showing how you can build a theorem prover. And I was like, oh, that would be cool. I can just build a machine that proved theorems for me. That's how I got into AI.
Amazing. And then from my PhD, I'll continue learning and math learning, then work for a few years at DeepMind at the research scientist. Now I'm working on RSI. I see a lot of pitches come in, and a lot of them use the term RSI. And I'll be like, oh, so what do you do?
And they say, we use AI to create AI. That sounds very interesting. So what do you do? We use AI to create GPU kernels, which that doesn't, to me, kind of map the definition without considering the RSI. So maybe can we start by you defining what you think, or what you would include in RSI, definitionally?
Yeah, I think RSI is definitely an overloaded term today. Many people want to say RSI, they mean different things. So I think the important thing when you're differentiating them is to ask the question, what is the optimizer, and what is the optimizer? When you're self-improving, who is improving? So there are people who say, OK, we're doing RSI.
We have human using AI to develop better AI. But that to us is just autocatalysis. In a sense that the improver is still human with tools, even though these tools are getting better and better. But the ideas and the improvement come from human brains. I mean, to interrupt, but to be clear, most technical epochs have had autocatalytic effects.
There's no way you can create a computer chip without a computer. You can argue that the internet development was sped up by the fact that people that weren't geographically located could coordinate. I mean, the steam engine, you could build better steam engines because you could get cold faster, et cetera. So there's no issue in using the thing to build a thing faster. It's not new.
And so you're saying something different? Yes. I think for us, in history, or a civilization, we have this autocatalysis using computers to build better computers. But the idea for this change, for this improvement, come from human brain. But that human brain was never the improved E, never the optimized E.
Now, for the first time, we could swap the optimizer as the optimized E. So the brain of the agents, which is LM, the model, can be the thing that you're optimizing. And this leads to a component, this reinvestment of cognitive improvement into itself. But there has to be a reason that you've decided to take on this hard problem as opposed to what anybody else would do, which is like to have pieces of the problem. Yeah.
I mean, before I answer your more important question here, the reason I did this is I think it's a hard problem. It's fun working on the hardest problem. And I think. But do you have an intuition that this is the actual right answer as opposed to something that's more pragmatic? Yeah, I think a pragmatic thing would be just from a control theory perspective.
If you have a gap in the system, you can have a closed loop. And autocatality has no closed loop. So whatever you improve, there's still a gap. Humans have to come in. You have to point other resources to make it work.
But if you have a closed loop, you still have to point resources. But the marginal gains ideally get either constant or larger over time. And that's the fundamental difference. I think what's really different here can be explained pretty easily. So if you think of capability, what we're trying to do here is we're trying to improve the rate of improvement of capability.
So the first derivative of capability versus time is the thing that everybody is trying to improve. And now you can put that to be equal, driven by two things, the denominator and the denominator. The denominator would be performance, which is what everybody is trying to improve. And then the denominator is some kind of cost, that's the capital, or difficulty. And this one is pretty set, because the higher the capability, the higher the difficulty of the next step.
Now, the numerator is an interesting part. For things like steam engines and whatnot, that was fixed. So then you end up with a very simple differential equation, where you basically plateau. Now, if you have a system where the things being improved is also the thing that's doing the improvement, you now have a new way to think about it. And the simplest case would be that the numerator is also correlated with capability, which makes sense.
Because the better you are, the better you are also improving. And if you do that, the math can start out, and you basically get constant improvement of the time. That could be one level of RSI. And then there's one more case, which is now that the numerator could be correlated with a higher power of capability. And then you get exponential takeoff.
Can we dig in a little bit more between RSI's having closed the loop and actual convergence properties? For some reason, when people see a graph where there's a circle with an arrow to another circle, and it goes back, and they look at that close control system, and they say, take off. Right? Which, I mean, if this is a nonlinear control system, it's very difficult to know. What the hell happens?
Anyway, most people don't seem to understand this distinction, because we very quickly go take off and fool me on that stuff. And we talk about these things. Well, as a network guy, you know that it depends on the exponent of your function. That's self-referencing. And what we found is that it's almost not useful to speculate on what we think could happen, because this is really an empirical problem.
And the real way to know what's happening is to test things. And testing things is really, really hot. Like, reward hacking is one of many problems that we have to deal with. But the question you're asking is very empirical. And for some of those interesting stuff, but so I'm a networker.
That's how I would have answered. I would have thought math people would have been like, we've got this differential equation. And I took out my pencil and I wrote down the answer. I mean, it is possible to write down the differential equation, but you still have to impertically measure the variable they exponent. So there's no analytical solution.
I mean, honestly, we don't even have a learning theory of deep learning. We don't know fundamentally why it works. And then now you're like five, six layers on top of deep learning. And so we don't have a good theory. I think the closest things we have are scaling those.
And these are empirical laws. I think one misunderstanding people have about our approach to our side is that they think the entire system has to be fully automatic and digital. But actually, if you look at our mathematical foundation, we have a multi-agent system, not just one digital agent. And in this multi-agent system, human researchers and engineers are actually special case of these agents. And it's this hybrid system of multiple agents working together that's been recursively improved.
And we don't want to leave out any component to not optimizing them. Because if it's possible to optimize in model, why wouldn't we do that? Because you will have more gains if you do include those. Anything that you don't optimize will later become your bottleneck. So one observation that we have made here is there are claims of speed as improving, which is actually the case.
So we look at the model release time that it is reducing. And so on one hand, you can be like, OK, so clearly, we're in some sort of a rapidly improving phase. But if you divide by the actual amount of capital raised by these companies, that's no longer the case. And so you're like, well, maybe they just raise a whole bunch of money. And then AI allows you to apply that money.
So how would you, this is not for yourself, but how would you encourage people that are either evaluating claims about this or even working on them to think about not fooling yourself by throwing more money at the problem? Like you're actually showing that you're making improvement here that's meaningful with the denominator. I think for us, it's pretty straightforward. It's because if you get improvement and they just come at more and more resources being put in, it's not RSI. You have to be able to back your claim of how much improvement you're getting.
But to include all the man hours and experimentation that went into that, because I feel like at some level, like that, I guess they'll talk about the actual flops that go into money that go into reality. This is the 23rd experiment. And it's actually a human-friendly, a lot of mental capital to get to that final configuration. Like it actually includes that in the denominator. We do.
And I think it's good company discipline to do that. And we've tried to very intentionally from the beginning, have a culture of tracking where things go, cost, time, all that kind of stuff. I should have funny things. Internally, we have this thing. People talk about leaderboards, which is like school.
You know what I'm about to say. We have something called a bleederboard, where we keep track of the cost of experiments, of everybody who's running experiments. That's awesome. And everybody can look at who's high up on the bleederboard. I can also give an example from AI.
I don't know if you know this nano-GPT benchmark you're running. It's basically asking you to retrain GP2, which when it just came out, it's a very expensive model to train. And weeks, I think, on multiple GPUs. And right now, people can train it on single GPU within hours. And that number is shrinking constantly.
But that's also probably separate, right? That's just like whatever more is law. Like, no, no, it's the same GPU. It's the same hardware. Using the same hardware.
And then you might prove the techniques of retraining. Oh, interesting. What has improved is algorithmic data techniques, kernels, all of the kind of stuff. How much it costs to train GPT2? I think it's a few hundred dollars.
I just said, oh, wow. So the same GPU. Same GPU, same data. And this is just algorithm. And do those come from humans?
Or do those come from agents? A lot of them are all the research results. But that's definitely the denominator. That's not the numerator. What you want to see is you can do that with GPT5, which we're not seeing.
The cost of these things is going up. Well, I think one of the most closely guarded secrets in the industry is exactly this thing, which is how much compute you can get out of this. Because in some sense, even if all you were improving was your denominator, you could get faster training, because now you can get more rollouts. And that means now you can sample that what you're describing, like the manifold of the space of language more extensively. And then you can get better trajectories, which means you can get smarter models when you use that to train.
For sure, it probably does not matter. It's good. For sure, it probably does not matter. It's good. I've yet to see, which is great that you have glimmers, where you can improve both the numerators and the denominator.
I mean, that's a pretty meaningful statement. It's awesome. What we found is that a lot of the papers that claim to do RSI when we tested in our internal infra, we find that they don't work. And a lot of it is because they're cool ideas that humans try to introspect and say, oh, you know, this is what I think should happen. They're complicated.
They sound cool. They get you a good paper. But then when we test it at scale, you know, on benchmarks and real economic problems, they don't scale. And what we found is that the simpler thing that closes the loop is the thing that scales. I would say one thing that we do that's probably unique is that we always start with a very grounded mathematical framework that basically lays out the map of the whole space.
And the reason is because this space is so large that if you don't do that and you don't have this very principal way of doing it, you get classed very quickly in this complexity. And the other reason is that often when you have this really complex system, there's two extremes of how you can build it. You can take an approach where you basically build like 200 different systems. And you hope that like this, this, you know, what is it, the root goal of a machine will work out? Yes.
Or you take the other way. And you kind of basically spend some time thinking about these 200 things. And maybe you realize, you know, they're actually just 12 abstractions that generalize these 200 things. And so what we have tried to create a culture of this really thing from the first principles, let's look at the math. And then we realize what assumptions we make to be practical because you have to do both.
And then we see what works. And when you do that, you then can see that, oh, you know, these multiple papers or ideas are actually subclasses of the same general framework. And if you do that well, you can scale. And I think that's one thing that I see that's missing. And that's hard to do.
And it takes a very unusual group of people to really make that work. I think Richard Feynman said once that there are two kinds of sciences, there's a STEM collections. And then there are physics. So without a unified math framework, all these research are sort of just STEM collections. You could, like a lot of these paper we're reading, they're just ideas inspired by some human behavior.
And then they try to mimic that and say, oh, look, we have RSI. But then if you don't put into a unified system, you don't know if the overall system is actually improving. Or you're just locally applying an idea that's getting better. So what we have is that we're putting all these components that you could possibly recursively improve into a unified framework and look at it from a holistic view. So that improving one part does not actively impact the other part.
I see. I see. So for example, there were people who were doing recursive improvements just on harness. And then there were people who were only doing on weights. But the ways and harnesses are not isolated component.
We have impact on each other. You can think of RSI at many different levels of the stack. You can think of prompts, harness weights. And you can go all the way to basically chips. And our goal here is to optimize all the levels of the stack.
It's efficient because we care so much about the denominator. And then we started doing weights, weight updates. And what we found is that you get more gains. But you still have to do harness stuff. And you can keep going down the stack.
And my guess is that you probably will need to do a lot of the kind of co-optimization of many different parts of the system to make it work. Can you talk a little bit about the problem domain that you're going after just to prove that this works? We are trying very hard to pick areas of the AI space that are as close as possible to AI itself, again, which are to be recursive. And if you think about it, there's not that many problems that are super close to AI and gives you improvement on both the numerator and denominator. And one of those is the inference problem, right?
Because the more you can spend on, sorry, the more you can get improvements and efficiency out of inference, the more everything unlocks. And it's not just for technical reasons, but also the most promise of both the numerator and denominator being improved. It kind of breaks the mind, right? So just for everybody that's listening to this, there's two improvements going on here. One of them is the ability of the model to make inference faster.
And the second one is now that the inference is faster, it can do that faster. And again, that does not necessarily mean takeoff. That can still ask them to at some point. But it's just wild to think of this kind of compounding impact. Yeah, yeah.
We're starting out with another research lab. And one of the reasons we chose it is because everybody in the world right now wants to serve their own model. And everybody wants to use the latest, for example, OpenWade model, KMEK3 or whatever, just coming out. And they want to be able to serve it as efficiently as possible, right? So it's an economically extremely important problem right now.
Just for the sake of discussion, if I wanted to make the inference for KMEK3 faster, would I use KMEK3 in a recursive way? Or would I use a separate model and system to do that for KMEK3? To improve the serving stack of KMEK3, you mean? Well, our system is not just using one LLM. You have many different LLMs that are doing many different things.
Oh, so you're doing a different question? Are you doing a different type for all of them? I mean, again, it's the most effective, because it's being used the most time. But the interesting thing is the choice of which LLM to use for which thing is also not always up to us. We'll let the model choose.
So be honest, today is not really us. We're deciding what to optimize. It's our agent. It's our agent. It's a recursive way of improving.
And what we're trying to do is not to limit itself to only one component. So it chooses whether to start with it. We try to be as hands-off as possible, let it do its thing. Do you have a single master prompt? Is it like, improve yourself?
But even though master prompt has been improving. Really? It improves the master prompt. And it's not one RSI system, because you can think of it as the thing that does the improvement. But then you also have the thing that knows what to improve.
And that's both of dynamic problems. And they are very cool linked. How general do you think this ends up being? Right now, you're focused on speeding up inference on a specific domain. Do you think that whatever approaches you're coming up with here generalize?
Or do you think there'll be a different one per target solution space? Mathematically, we have defined it so that it's one system that generalizes to everything. I think that the one part of the system. And by everything mean any problem domain that you can point to AI to? Correct.
But then there are physical limits, for example. And one big limit of this is the human element. I think the way that a lot of AI companies are thinking is let us use AI to replace humans. But what we found is that there's so much that if you have good researchers and good engineers that they can bring to the table, that's going to be very hard to replace by agents. And so that's why from the very beginning, we've tried to think of our physical anti-company as a multi-agent system, where humans are a kind of agent that's gohabitating with other AI agents.
Can we at least reduce bounds? So for example, do you envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement? We can't share too much. But one of the very important things that we've tried to do in our company is not just work on the numerator, which is what I was describing earlier, which is your performance. But to work on the denominator.
Which is how much you can get out of the same amount of physics, money, whatever you want to call it. But do you think it's possible to have a system where the numerator continues to increase at the same way with the denominator being fixed or decreasing? It's not just possible, I think it's happening. Really? Yeah.
So it takes less energy to get better results. Yeah, over time. You think there's a universe in which that continues without asymptoting. Because if that was the case, then you actually have to take off. I mean, that just seems to disobey information theory.
And it's a lot of physics. I mean, I think at some point you're going to reach plant constant, plant energy, plant time. So it will have some total at some point. But I think... Oh, I see.
And so you think that asymptote is basically the computation of the universe. Yeah, I mean, how many particles are there 10 to the 80? We know speed of light. We know what it is. You multiply those two and you basically get an upper bound of computation.
And so given that's fixed amount of computation, you can do every second. We know we have a limit. I'm so impressed by your belief that this idea of our model improving itself. But it's not a belief. No, you have to be able to do that.
That's amazing. And it's a benchmark accuracy. It's upper body by 100%. You're never going to see exponential improvement on that because eventually it's about 100%. So for a lot of time, we have this question.
How do we even know if something has ticked off, right? If we don't have the right metric. It's not a belief that this is happening. I think we're already seeing the numbers of data happening. But it is a dramatic statement, which is, I think it's a first time I've heard you say it.
But is this been kind of a continuous result or is it relatively recent that you're saying to see climmers of this? No, we've been seeing these results kind of growing, growing, growing over the past few months. How do you have confidence in the results given, you know, like it's not working yet and you don't know? No, it's not a belief. It's empirical measurement.
But also as a company, there's a very funny thing that is happening, which is because we have been so thorough, we have not released things until we're super sure it's correct, right? And every few months, there's an announcement of something RSI related. We should have really established results or something. And we look at it, we try to replicate it, and we realize, oh, it's now it's chirping. And so it's very, very hard to know in this sea of AI, I don't know what you want to call it.
Everybody's trying to do our AI RSI stuff to know what to do or not. The technical answer is you have to be really, really good at measuring things. So for example, we have one of our engineers that we named him the biceroy of variance. That's a real thing. His job is to go and destroy the results we have and find what's wrong.
And we even have little slack emojis that we have. There's one with red laser stuff coming out of his eyes where he finds that there's bad variance and stuff. One of the things that we found is that when you're dealing with a new benchmark or a new problem of economic value that you want to apply these two, so far we have not found a replacement for what we call quality time. Literally running hundreds of experiments, looking at it, developing an intuition, trying to figure out what is wrong, what is not wrong, what is just variance, what is not, what is this weird thing that actually shows that there's some kind of reward hacking, like two levels down the agent, that built the agent, that built the agent, did hack something and now you have these weird results.
It's not easy. So when I have dealt with nonlinear control systems, there's a bunch of, in networking, there's a bunch of degenerate behavior. It's not just convergence and diverging. It's like oscillations. And this is a very multi-dimensional problem.
So you have to deal with all of that stuff, like potential oscillation. On and then on top of that, you have noise. The lens are fundamentally super stochastic. So what we find is that specification on a specific benchmark, there's levels of noise that sometimes are unpredictable. And some of them are extremely weird.
Like you even have some weird noise that comes from this GPU and this other GPU, they're both B200. But they have slightly different behaviors on the physics. And you can see some of that being replicated at the high level. So that's why it has been a critical thing. So from that, you could already see that intelligence is not a single scalar.
It's more like a multi-dimensional vector. And there will be domains and field where this RSI will give you a take off and there will be dimensions where you might plateau or you will have relatively slow improvement. And I think the defining factor is what is the slowest component in it. So you're an optimization person. You know that there's this Amdos law where the speed up you can get by optimizing components.
You just take it to zero, yeah. I think I'll stop it. Exactly. So the digital part we could optimize and accelerate, you could get like 1,000 times faster or 1,000 times faster. But if it's only 20% of your orbital time, you're only going to get 20% speed up.
So for domains where you still need to move atoms that interact with physical world, that's really slow, you're still going to be bottlenecked by that. So let's say every evaluation you need to do, let's say drop discovery, you still need to do wet lab experiments and then clinical trials and that take months, then every new idea have to go through a month or multiple months evaluation. So those field probably won't be accelerated as fast as coding or mathematics where you have very fast and cheap evaluations. But what you have to see is anybody put a bounce on that. To say, in order for me to improve this by X, to solve this problem, I need this much net new energy to do it, right?
And if you have it, you can actually predict, like if I'm like, listen, how big of a warehouse do I need to solve like a drug discovery problem? Or how many different robots I would need to solve? So do you think that we're getting any closer to actually having bounce? Because I think this is such a critical question for all of us to answer. One of the internal jokes we have in the company is that the secret to RSI is somewhere in a garden who is company in Wisconsin.
And the joke here is that if Einstein hadn't discovered relativity, he would not have these laws that tell you how speed and contraction of time works. And if some unfortunate engineer had to design the GPS system, they would have had to find a correction of time. And the pro would have said, no, no, it's clearly not the fabric of space time that's contracting. But I don't know why, but this is the formula of devices. It's like a curriculum.
Yeah, exactly. Exactly. Exactly. And then people later would have been like, holy shit, this is like space time contracting. And we have this theory that RSI right now is being thought of, let's solve all these super hard, kind of fashionable problem.
But what we think is that the complexity of the world is such that the real challenges probably exist in spaces that are not being applied to. And so that's the joke that there is some workflow somewhere in the garden who is company in Wisconsin. That if we solve it, we get fum. I love it. I just want to quickly summarize.
So you're going after RSI. In your case, this is not auto catalytic effects. This is actual RSI, where the system improves the full system and freo is an actual system, which includes the harness and the weights. If you think about the problem is you've got a numerator and denominator. The numerator is like how good the system gets and the denominator is how much energy goes into it.
You believe your sink limits of the side were both improved at the same time using this type of approach. Is that correct? Yeah. When can we expect to hear more from you? Very soon.
Amazing. Yeah. Thanks so much both of you for talking here. Thank you. Thanks for listening to this episode of the A6-D podcast.
If you liked this episode, be sure to like, comment, subscribe, leave us a rating, or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcast, and Spotify. Follow us on x, A6-D and subscribe to our sub stack at a6-d.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only.
It should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A6-D fund. Please note that A6-D and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a6-D.com forward slash disclosures.