Noam Brown – Agent swarms, alignment, & recursive self-improvement

Today, I'm chatting with Noam Brown, who is a researcher at OpenAI. He was one of the foundational contributors to what became a one in the reasoning models, and now he's working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the million price problems with a system of 10,000 different AI agents that spent 130 billion tokens over eight hours. One of the reasons I'm talking to is I think you were in the first people maybe two or three years ago who was thinking about how the reasoning models would allow us to see into the future, because if you scale up inference compute, you can see what the base capabilities of models will be in the future. And if you're in a similar position now, to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now.

So the way I think about it, when you plot the performance of these reasoning models with test-not-compute on the X-axis and performance on basically any reasoning benchmark on the Y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is like a very natural thing. It's the thing with people. If you're taking the SATs, you have five minutes to go through the entire exam, you're not going to do very well. If you have five hours, you're probably going to do a lot better.

The AI models are pretty similar. And they'll spend that time doing this model out of themselves, figuring out, going through different cases, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don't want to sit around for three years waiting for a response. And so what you can do is what a lot of people do is they paralyze. They just get a team of people.

If you're going to find a company, you want to get a group of people together and go faster. It's the same thing with these AI models that it helps to just have multiple agents working on something because they just go faster. And so multi-agents is a way of scaling test-not-compute in parallel instead of purely serial. And it is less efficient because it doesn't have, it's not like a single agent has all the context to itself, but it is a very effective way of scaling test-not-compute if it's not. I'm going to mention that you're questions because these systems, so this is an unreleased model. So we haven't publicly seen how these systems work. And so I just have a bunch of

ways in which I'm confused about what the qualitative properties of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what 130 billion tokens are, if it was a single human thinking as a full-time job, stretch back to back, 130 billion tokens would be a human thinking for like 4,000 years, eight hours a day or something, working on a work week. So starting from the ancient Sumeria of today, a single sequential human thinking that long concentrated in 88 hours. I feel like qualitatively that is a super important consideration. And I'm surprised that there isn't a bigger parallelization penalty.

Then you can just have 10,000 agents collaborate and because maybe the agents are better collaborating than humans might be, they're going much faster, that they can actually productily collaborate at such a big scale. Or maybe there is a big parallelization penalty. Yeah, let's talk about the parallelization penalty and then we talk about the qualitative stuff. Because the truth is that we don't have very good science on multi-agent scaling up to this kind of scale. So when we released 5.6, I think that was the first time that we had a proper multi-agent system in our models. And we actually did in the blog post show some plots of the scaling performance of

multi-agent systems. Because we have it as an option, it's ultra mode. And the default is four agents, we can set that to hire. And in the plot, we show, okay, here's what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together. And what you see, and it depends on the benchmark, but for some of the benchmarks, basically, if you have four agents working on the problem, it is done twice as fast. So you're

basically paying, because there's four agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern, it's like a little less efficient, but you continue to see that performance. Is it a linear or a real-time speedup or a sub-linear speedup as you increase the number of parallel agents? I would say it's slightly sub-linear, though it does depend a lot on the problem. So math, for example, is quite parallelizable. It's not the most parallelizable thing, but it is very parallelizable.

I think web search, things like doing a deep research report, we have to look through a bunch of sources that's extremely parallelizable. I suspect that something like writing a novel would be very unparallizable. So you would probably not see a big benefit from having 10,000 agents working on a novel together. In the same way, you probably not have a big benefit from having 10,000 people working on a novel together. So the performance does depend on the domain. We

do measure it up to 16 or so agents in our published blog posts. The problem is it's very hard to push that science to 10,000 agents, because it's just so expensive. You guys just dated over a week? That's one data point. We don't know how long it would take a single agent to solve an opportunity.

We haven't done that experiment yet. And maybe we will, but that's also only one data point. We want to do a thoroughlation. It's actually the experiments are just too expensive to go to that scale. So we have to do some kind of methodical science about what happens when you go to 64, 128, 256 or something and get a sense of that behavior. But it's going to be very hard to push that

all the way to 10,000 and know for sure what was the benefit that we actually got for using 10,000 agents versus 1,000. One thing I want to make clear is that the effort to get, it's not like, they have to get a blending price problem. This was not due to multi-agent. I wouldn't even attribute like 10% of the credits of the multi-agent. The reality is we're trying to open AI has trained a very powerful model.

And we can get that model to operate over a long horizon. We can get it to think in parallel. But at its core, the reason why we're able to do this is because we just have a general purpose, very strong model. And I think things like multi-agents are flashy and are new and that probably gets disproportionate credit for that reason. But the core reason is this is just a very powerful model. So the generalization is actually quite shocking to me. The US systems,

I mean, I don't know how these systems were trained. But presumably they were trained how our training happens. You have a bunch of checkable synthetic problems. You do a bunch of RL against them. And nowhere in the training process, I'm guessing was like the model solving anything as ambitious as a millennium price problem. But the generalization was strong enough that you could

have this much easier, verifiable problems generalized to this much parallel effort on such a hard problem. I think that is true. First of all, we do train the model on very hard problems. So there is definitely a gap. We see if we train on some kinds of tasks, like it's able to do tasks that are more ambitious than that. There is an interesting challenge that

as the model becomes smarter and smarter, the kinds of questions we can ask them, just a lot of them are too easy. And it's hard to challenge the model. And I do think that that's going to be an interesting thing. If I had to make an argument for why you might not see AI's like LMs go the same path as AlphaGo and AlphaZero and all these kinds of game playing as, it might be that this kind of problem that in things like AlphaZero where you have self-play, you have an infinite curriculum, you're always playing against an AI that's equally strong. Whereas for things like training in a land with reinforcement learning, at least the ways that are out there right now, you give the model a problem and you ask it to solve it. And if the problem is so

easy, I can dissolve it in a second. It's not really learning anything. So if we run out of problems to ask it, that challenge it, then that is a plausible scenario where actually like, okay, it becomes much harder to make progress. Now, I do think there are ways around that. And so we haven't really hit that as a wall yet. I mean, I think that if it ever became a serious

problem, there would be ways around it. But it is like a plausible scenario. Yeah. And just for the audience, when you were referring to AlphaGo, you're saying like getting superhuman relatively fast after achieving human performance. Yeah. I mean, if you look at the trajectory of game playing as like go, they, within a span of like

a year, went from being a European chess champion, being like on a number 15 in the world, to beating the world champions, being unimaginably as a matter if you were stronger than any human life. And it's possible that domains like math, we see a similar trajectory, but I think there is a very plausible scenario where actually that doesn't happen. Yeah. Yeah. So I want to understand that if in six months, people will have active multi-agent systems, how to one model what it is like to collaborate with or hire a multi-agent system? Yeah. I should start by talking about how these

multi-agent systems actually work, which is I think a very different way than a lot of multi-agent systems in other AIs. So a lot of people that approach multi-agents for things like LMs tend to take this very scaffolded approach where, for example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task and the children work on it and they return their answer. And this seems like a very sensible setup, very sensible scaffold. It definitely helps, but there are a bunch of limitations with these kinds of stuff. So for example, if in this setup, you have a coordinator that's sending tasks to children that children work on it and then return their answers, well, what happens if two children are given similar tasks?

Can they talk to each other? And usually the answer is no. And that's very inefficient, right? If you're given a task and it's actually really helpful to talk to somebody that might know an answer to a question that you're working on or a part of something that you're working on, it'd be really helpful if you'd just be able to ping them and say, hey, can you help me out with this thing? But a lot of systems don't have that setup and adding it just increases the complexity. So you have to get

into the scaffold that you have. Another thing is like, what if the child doesn't really understand or has a clarification question? So it has to choose then between, okay, do I just return and ask the question instead of solving the problem? Or do I solve the problem just like making assumption about what the parent wanted me to do and just solve it that way? And so any scaffold that people come up with, there's always limitations involved. And the approach we wanted to take was to

just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use and figure out for themselves how to use it effectively. So we give the agents the ability to message another agent and when a message is another agent, it is inserted into the context. And then it can do it like a few other similar things. But that's basically the core of it that it can just send a message whenever it wants just a tool call and it can send it to other agents and they figure out for themselves the best way to coordinate around that. And it turns out that if this is done well, you get very sophisticated behavior. And to me, it looks a lot like how human

collaborators work over something like Slack, for example. When we were working on this project, it was really exciting when we finally got working to see these agents working on problems together. I remember one example, so we get the agents the problem and then one agent says, I think I've got the answer. And then another agent says, actually, I've got a different answer. And then they have this whole discussion about, well, how did you refer that answer? Can you explain it

to me? And going back and forth and trying to clarify what could have been wrong in each other's reasoning. And then they finally converge and like, oh, yeah, okay, that seems right. And then it just broadcast to the other agents like actually I changed my answer. I think he's right. And it just

felt like a very natural conversation. It's kind of felt like when you see chain of thought for the first time that's trained through the reinforcement learning, and you're like, oh, this is just kind of like what a person would think if they were running down their thoughts as they're thinking them. It kind of felt like that. So it is really cool to see this kind of behavior. And so I think collaborating with these things honestly, it feels a lot like collaborating with a person. So it's a

very natural flow. Except one qualitative difference that might become salient in the future is that the systems will be thinking, I don't know, more than 10x as fast, right? You just look at how many tokens per second the output versus how fast a human talks. And they're working all the time. They're not sleeping. And they're collaborating with each other at a much more intense pace than humans have

the capacity to collaborate with other humans. So I'm trying to go to qualitatively expect in a year. And is it like a shadow organization that is moving a hundred x faster in my company than the human level is? That's the iteration cycles much faster. What would take a human organization a year to do is happening within a week within this shadow organization.

So will it feel foreign? I actually found that it's pretty surprisingly natural to work with these things right now. I think that could change. So for example, we have these ultra-fast modes that enable sampling to be like 10 or 15x faster, whatever. And then it's going to be pretty hard to keep up these things. I think the idea is these agents when they're communicating with each other,

yeah, they can go super fast. But then also they understand when they're talking to an agent versus when they're talking to a person. And they're behaving in different situations. So the main example that we have publicly of sophisticated multi-agent systems is unfortunately the hugging phase one. And the thing I found interesting there, I mean, a lot of things are concerning, obviously, but the thing I found interesting is just like the spontaneous emergence of hierarchy of middle management. And it sounds like you're saying this level of

organization emerges spontaneously from training. I think the details are spontaneous, but I mean, just because we're giving a lot of flexibility to the agents to decide how to give each other in the optimal way, it doesn't mean like we are still giving them a starting point. We're giving them a prior about, oh, this is what reasonable communication might look like. They're also, I mean, they're trained on a lot of human text. They have an understanding of how humans organize and coordinate. And so that's all kind of baked in. I think it is surprising the way they're able to

polish this. If you look at what it starts out at, it's not very sophisticated behavior. In fact, it's actually very difficult to get these agents to coordinate in productive way because it's very tempting for them to just collapse to, oh, we're all just going to solve the problem independently. Yeah. And like, that is a local minimum that you can get stuck in. But yeah, if it's done well,

they can end up coordinating it very effectively in these kinds of like very structured ways. I really say a couple of years ago, I'll call them something something something I would automate firms will look like. And I was thinking about, well, if you had fully automated firms, let's say human level intelligences, what is different about the nature of AI minds that would make organizations, AI's form different? And there are a couple of very important differences, for example, that AI's can share context much more seamlessly than humans can, they can merge their knowledge much more seamlessly. And also you can spin up or spin down an arbitrary number of instances which have the right knowledge. So if you want to hire more people, it's not like

just all the schlep of finding the right talent or whatever. It's like, your best talent, you can just make an infinite copy of them or if you don't need them for the task anymore, you like spin them down. And you just replicate the most effective parts of your organization or like replicate whole organizations together, which are effective. I don't know, what do you see these multi-agent systems going a year from now or two years from now? I think it's a great question of like, how do these things actually differ from working with human culture? And I think you highlight it, so I'm like, one really interesting thing is that,

I mean, if somebody is, if you have a person and you want just like two copies of them, you can't just like clone a person. But when they ask you, it's actually really easy to say, okay, we'll just fork yourself and then have those copies work on this thing and then like merge back together. We already have this, I think, in multi-agents for Astra and Fact1.6, so all that, when they spin up some agents, like the context is forked. So it has all the context is relevant. There are other interesting ways where the agents will differ from people. Like,

what are some reasons why startups disrupt incumbents? Like, there's a few factors, one is there one to take more risks. But another major factor is like, as companies grow in size, as organizations grow in size, you see increasing misalignment between the individuals in the organization. Yeah. Right? Like, if you start up with five people and each person has 20% share in the company,

they're all highly aligned to the company succeeding. If you have a massive company with 10,000 people, you see a lot more instances where people are territorial or like just care about getting a lot of headcount for their project or their team or like, you know, building their fee films, getting a lot of resources so that they can publish cool work or whatever and get promoted. And this is actually real detriment. I think this explains a lot of why startups are able to disrupt incumbents. And it's interesting that, I mean, it's true that AI does help startup in a way, like it's much easier than ever for one person to step in and be like, I'm going to make a multi-million dollar company. Like, it's just the AI is amplified in individuals so much,

but there's also an argument that they could benefit incumbents because if the online problem is solved, then you don't have the issue of misalignment between individuals in the company. Like, at least that's mitigated. Like, the AI is, if they're fully, if they're aligned, well, they can just be aligned to the interest of the company and you can have 10,000 of them and they're all going to be working as hard as if they were like a 20% share co-founder. Yeah. And it's not only that, but it's also that they are much able to manage shared memory and context than different humans can. If you have a

tomorrow, you hire 10,000 mathematicians and you're like, solve this, solve not be your soaps. They're not going to be able to like cooperate effectively, at least not off the bat, but you can have apparently 10,000 AI's. Well, I want to be like, I want to be conservative here because we haven't measured how effective the 10,000 agents are at coordinating. We think it helped. We don't actually have good measurements of saying like, oh, yeah, this 10,000 agents led to like a 2x speed up over 2,000 agents or something like that. And it's, it is actually, I would argue

likely, maybe I don't know about likely, but I think it is very possible that 10,000 humans are better at coordinating than 10,000 agents right now. I think that is very, I think it's entirely possible. I think also one trend we've been seeing is like, we've been working on multiplication for a while. And the early versions of this is very difficult to get right. It's very hard. It was very hard to get the agents to even talk to each other. And it's because like, look,

the, we, when we first developed reasoning models, like they weren't talking to other agents. And if now you put a bunch of agents together and say like, oh, solve this problem together, they're in this local minimum where they're really good at thinking deeply about a problem. And it just like kind of interrupts their chain of thought, the interrupts their flow to constantly be like checking in with other agents or like receiving messages from them. And the opposition is actually very hard to get right now at situations. They're getting a close start of like getting first collaboration. Like what's the issue? I mean, I think it's that they're not very, they're not as

general. Like the earlier models were just not as generalizable. And we're just more narrow. Interesting. As the models become more capable, it's been easier for them to develop this capability.

And I do think that as they become stronger and stronger, just across the board, that they will be like, become better at organizing themselves in larger organizations. And like, I don't know, maybe they are better than people organizing in 10,000 person groups. But even if they're not, you know, a year from now to years from now, like, yeah, it's quite possible they'll do that. Even if we don't end up optimizing for that. Rockbot has changed the way that we produce our videos. For

example, you may have noticed that a lot of our ads have these animations of real websites. One of my editors uses LLMs to make them, but it's not currently straightforward to have an AI create pixel perfect animations of specific websites. We tried. It doesn't really work that well. So we've cobbled together a pretty convoluted multi-step workflow. And up until recently, we had to run

every step ourselves. Now we just like rock, rock, rock, handle it. Rockbot starts by opening the website that we want to animate. It uses a specific extension to download and open the page in Figma. Then uses Figma to convert the whole thing into an SVG file. This saves the AI from having to draw the

whole UI from scratch and tends to result in higher quality animations. Rockbot runs this whole process on its own cloud computer where it's installed all the tools it needs to run this whole process and to end. And it's learned our video specifications and preferences. So there's no need to re-describe the whole task every time we want to make a new animation. This is just like a new way that we interact with AI over the next year, agents with their own computer who can anonymously handle bigger and bigger chunks of your work. You can try Rockbot at x.ai slash bot. Okay, so here's

why this result and maybe the general progress that AI has made mathematics has made me think that RSI is more plausible and sooner than I previously thought. I think we've gotten mathematics from 24. You have the eyes where like, okay, interesting. They're like doing, they can solve a couple problems on high school math competitions. And then in 2025, it's like, oh, wow, they can get gold in international math. And earlier this year, I was like, wow, they're actually solving open

problems and mathematics, open-erters problems. But maybe like, I don't know, people weren't trying that hard. And it was just like, there's a similar solution somewhere in the literature. And now, I just think it's undeniable, right? It's like, this is the Millennium Price problem. There's really no

there's no story of why they should have been easy. Now, a lot of people pointed out that I think shared a post like this to be order of interesting post about this. But that they're solving all these problems, but they're not like coming up with, at least I'm not aware of them coming up with new insights or formulating insightful new questions and new modes of theory for thinking mathematics, coming up with like, topology or coming up with the Cartesian grade or something. And so maybe like the actual progress in mathematics broadly construed is smaller than it might seem if you're just looking at and well scoped problems that are directly solved. However, I think that that kind of progress would be incredibly meaningful in ML because in ML, you're not, you don't care about like, better understanding the nature of deep learning or you only care about that as a instrumental goal towards the broader sense of like, just achieve the result, just solve this like well scope problem of improved the sample efficiency of our models, like improved the free training laws. And so the kind of progress that we're just seeing arrive, like in avalanche mathematics is

structurally actually very similar to, again, I'm curious if this is the case, I'm just totally outside. I'm wondering if it's the case that it's structurally very similar to the direct uplift that you'd expect any progress. And then the thing that's shocking to me or concerning potentially is just like, how fast we went from, oh, it's like, they're giving me 50% uplift if you're a mathematician to, wow, they're just like, end to end solving the biggest open problems in the field. Yeah. Okay. So there's a lot to impact there. Let's start with the progress on that. So,

yes, the models are doing some crazy powerful stuff and it's happening, it's progressing faster than I expected. I mean, we got IMO Gold in 2025. I thought, okay, basically what I thought is like, the models when they were doing GSMK, when they figured out to do GSMK, that was that would take human mathematician about five seconds to do a GSMK problem. So this is great school math, a great K through eight. And then the next year, they were able to do the math benchmark problems.

And these would take a human math, like an expert human mathematician, maybe like, like a minute to do. And then you get to Amy. And this is the qualifier for the USA mathematics and it would take a human mathematician, like a good mathematician, probably like 10 minutes to do. And the models were able to do that a year later. And so every year, you're seeing this like 10 x increase in the task they're able to do in terms of like length of how long it takes human mathematician to do it. And then it was very sensible that a year later, we get to IMO Gold,

because that's 100 minutes. That's about how long it takes a human mathematician to do an IMO problem. And just projecting out words, I was like, okay, how long would it take a person to solve a millennium prize problem? And I mean, I don't have a good sense, but if we're following this trend line of like 10 x every year, we go for my IMO Gold, which is taking an hour and a half to next year, 15 hours. And that should not be enough to solve a millennium prize problem. And so I was like, yeah, I don't think

we're going to get it in, you know, in 2026, probably not in 2027, maybe in 2028. So it did happen a lot faster than I expected. Now, I think there is a narrative going around that these things are replacing mathematicians that it's just super human mathematics across the board. And I think that is the wrong takeaway. They're clearly exceptional in some ways, but they are weaker than human mathematicians in other ways. So we have this like jagged scenario where the models are like

brilliance in some dimensions and also weaker than human dimensions. And yeah, like you said, they're not very good at posing new problems. They're not really good at understanding like what is really worth it, what directions, what whole branches of mathematics are worth exploring or developing. And my opinion is that I think this is great. Like I would be thrilled to live in a world where AI is a compliment to human abilities and is allowing us to discover new knowledge without fully replacing people. Like that is the best case scenario. You don't expect that to continue. I do,

I think it's true that the AI's are jagged, but as they get better, they get better across the board. And so I think that the things that they're exceptional at, they're going to get even more exceptional at, the things where they're far behind humans at, they're going to be less behind humans at, and over time, it is possible that they're just better across the board. Now, I don't know how long that takes. It depends on how long the long tail is of things that they're bad at. So I guess this brings us back to RSI. And again, I want to emphasize here that I'm just

totally outside of a podcaster, but I'm just trying to reason, or like, as somebody interested in concerned about what's happening in the field and trying to reason about what takes RSI, what kind of things to expect. I feel like the amount of cognitive effort that was done to this million price problem is a good intuition pump of you could have AIs that are spending over the course of maybe a week, more cognitive effort on a long-standing ML problem, like, you know, very fluid online learning. They could spend more effort in that week than maybe the field that spent cumulatively in its entire existence. And then you could say, well, unlike mathematics, of course, AI requires experiments, and that takes compute and that takes time, right? You can't just think on pen and paper, not sure if anything's happened. But if you just look at the amount of compute that is available

at an organization like OpenAI, right? By the end of next year, OpenAI will have enough compute that if you had, you know, the 10,000 agents, or if it's a 10,000 agents of the million price problem, you're like 10,000 agents of the end of next year. They're much smarter by that point. And each of them will have enough compute to run a GPT-3 sized experiment every single day. I don't know, that seems like a lot for like super human researchers who are thinking super fast. What do you think about the

inclusion pump? I think it's pretty accurate that look, I mean, yeah, these things are very spiky and if you're going to find them at x, they're like way better in some ways, but they're also worse in other ways. But the ways that they're spiky end up, I think probably being particularly useful for things like RSI. And you know, you have a more clear objective. It's just like more measurable. It's more like there's less question of like, well, what new branches of mathematics

are worse? Like, no, it's very clear answer. It's like certain metrics that you care about, and if you can make it do better on those metrics, then you succeeded. So I think there is a lot of truth to that. And I think the main difference is that mathematics, you're truly bottlenacked by thinking. And no external, like, yes, there are some, there are some parts of mathematics where you care

about running experiments and resulting in these kinds of things. But for the most part, it's really about thinking really hard. And the models are really good at that. When you look at things like RSI, you do have to run experiments. So it's, it's not enough to be extremely smart. And I think one

argument for this is if you had like 100x less compute, and all those brilliant people in the world working at OpenAI, how much progress would be making relative to having the math that we have now with the amount of people we have, I suspect it would be less progress, actually. Well, how much less? It's unclear, but I think it would definitely be less. I think a lot less. 100x less? No, not 100x less. Yeah. But I mean, okay, so like, I guess the question again is like,

okay, we have RSI, we have all of these brilliant AIs running around, running experiments and stuff with the computer we have, how much faster is progress? And I think this is something where we disagree on. I think that we do speed up, and I think we see a significant speed up. But I don't think it's like an overnight intelligence explosion that we go like 100x faster. Because I think that we do get bottlenecks by certain, certain limitations that are not bottlenecks of intelligence. It's

running experiments. It's running experiments, it's running experiments, it's really, because they take a while to either train new models or to get the results. It's having the GPUs to run those experiments. So it's unclear how much faster things go. I definitely think they go a lot faster.

And to be clear, considering how fast things go are going now on an exponential, if that exponential is like 3x faster, that is massive. But there's a big difference between that and 100x faster. Yeah, yeah. I'm quite differential to your inside view on what RSI looks like or what the dynamics are. Because obviously, you've been in the field for like 10 years and I'm sort of like trying to reason about it for a very outside view type of intuition pumps. I'll say that people have different

opinions on this. I have my opinion on this. I could totally be wrong. I bet that I have some confidence in this, but I'm not like 100% confident. This is the way things go. Maybe there could be

over an Intel G-sploge. I don't know. Maybe we don't see 3x. I think it's like 50% speed up. There's a lot of uncertainty here. Yeah. There's a couple of points.

Tensionally, I want to clarify something about the Jaginas. One thing that sort of gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner, because that better learner can be more general. So if you just make an AI that's better at using office products or playing chess or something, that's fine. It's not going to lead to big productivity improvements or anything. But if you make an AI that's really good at making something that's more sample efficient or that is capable of continual learning or these much more well scoped ML problems, the thing that emerges out of that, assuming there's good enough transfer from the direct problem you're swapping out like this broader ability to learn, can't just be more general. So I think that's an important dynamic to keep in mind of why Jaginas can still lead to

generality on the other end. On this question of obviously experiments bottlenecked, because if they didn't, as you're saying, you'd have some crazy singularity, overnight at opening an AI, or you'd have 88 hours and you'd sell them a lot of your problems and you'd have super intelligence. So obviously, the experiments are such a big bottleneck that instead takes you many years rather than 88 hours. But there's a question like, how much of a bottleneck they are. And it seems to me one thing that's been giving you a bit of singularity of vertigo is realizing that even if the current rate of progress simply continues, so it doesn't have to be literally just continues to pace, continues to pace as some of the other heads when you talk about come up, right? That's just

like it's harder to find problems, it's more long horizon, maybe like in the 2030s, compute can't keep scaling at this exponential level. If you simply continue the current rate of progress, I think about not taking seriously what that implies as we cross over beyond the human horizon. Here's some of the things that apply. So I mean, I think it's really hard to reason about what smarter than human intelligence will be like. So I just think in terms of human population sizes, the current progress makes it so that a given level of compute allows you to basically run a three x bigger effect of population every single year and also computers growing in the background anyways. And so you can have a situation where each of the labs by the end of 2030, probably

much sooner, but let's say the end of 2030, it has enough compute to run, let's say hundreds of millions of human level intelligences based on what the capabilities will be at that point. And then people are not thinking seriously the kernel of progress means that by the a few years down the line by the mid 2030s or earlier, you would have many Earth's worth of human level intelligences within each lab and you'll probably qualitatively super human, right? Like it just, but anyways, this is like a base case. I don't know. Yeah, progress is really fast. I think that's

under century. I mean, I think it's worth pointing out, researchers are continually being surprised at the rate of progress. I mean, if you look at what even among researchers in AI, what were the projections for like getting an IMO gold in 2025, it was, I mean, I think the idea that it could be done with a general purpose language model with no tools and no access to the internet, I think even people at opening eye thought this was like average, like they thought it was like almost impossible. And then you get to 26 and like, I mean, literally two weeks before we got Navier-Stokes, I was talking with a researcher at a frontier lab about how long it would take to get a Millennium Prize. And he was willing to bet me $1,000 that it would take that it would take past 2027. And he thought it was taking 2030, you know, and I took that bet. But even I thought it would

take longer than how long it's likely to take. So people have been continuously surprised, even inside the labs. And I was literally, I was just talking to somebody yesterday who was working on the Navier-Stokes effort. And he was telling me that like, he used to say it's really hard to predict where AI would be in 12 months, you know, if somebody asked him like, where is the where he's going, he would feel comfortable making predictions for like the next 12 months. But beyond that, you know, he's just like, I don't know. And now he's saying like, he doesn't feel comfortable making

predictions beyond three months. So it is, it is really true that yeah, things are going to go very fast right now. And he's talking about 2030. Like, I don't know what the world looks like in 2030. That's the truth. Yeah. Do you expect the sort of full automation of AI labor, like 95%

automation of AI labor, 28, 29, 30, 27, I don't know. And I just said, I don't know what the world looks like in 2030. I mean, I think we actually released the blog post recently on internal acceleration at opening. I mean, we show, for example, that the amounts that researchers are spending on coax is the top 1%. I think as of early August, we're spending like $7 or $8,000 a day on on coax inference, early use. That's our next financial, it's going to keep increasing. And there's a question

of like, okay, if that keeps going, then how do you how much do you assign to this like, AI is doing work versus the humans doing work is a 95% is a 5%. It's actually, it's really, it's really hard to reason about this for a few reasons. Like, first of all, if it's the human directing the AI to do the work, then is that the human how much you choose the human how much you choose the AI. The other thing is that because these AIs are jagged and they're exceptionally good at some things. So for example, they're exceptionally good at looking over data sets and checking every single data point to see like, is this of sufficient quality. You can disproportionately use the AIs for those things compared to you previously. So,

yes, you're using AI way more than before, and it's making some things go like 100x faster and 100x better. But there are some things where it doesn't make a huge difference yet. And of course, if something is something like 100x faster and 100x better, they're going to do more of that thing. So, are you comparing to a speed up of like three years ago versus like, is the question more like, given what we were doing three years ago, how much faster we able to do now versus given what we're doing now, how much slower would it have been three years ago? It's actually two very different questions. So anyway, it's like, it's really hard to measure. I do feel confident in saying that things

are going faster now than they were, like even a year ago, because of AI progress. And I think that that acceleration will continue. I think a lot of people in the field have very high airbars on this sort of thing. If you had to put it into my head and ask for a number, I could see things going three-x faster. And that is huge, right? Already the pace of progress is incredible. Even if we don't get

any uplifts, like you said, things are going to go much faster by the time we get to 23rd, we don't even know what that world looks like. I think if we get a three-x uplift from internal acceleration, that is massive. We're right now reducing, think about where you were three years ago. If we get there, if we make that progress in one year, that's huge. You're going from not even having a one, just having non-reasing models to Astra in a single year. So I do think things go faster. It could be

that things only go a 50% faster. It could be that things are likely, but it's possible that things go 10x faster. There's a lot of uncertainty around this. And from my perspective, I have a lot of uncertainty about it. Suppose you need to do a major back-end refactor. Getting assurance that you didn't introduce

any new bugs could take weeks of writing an extensive battery of tests, potentially more time than you spent on the refactor itself. And to this, this allows you to gain high confidence without having to build complicated test suites by hand. And to this, this runs your software through a near-infinite multiverse of simulated worlds, injecting faults and hunting for failures in each one. And it lets you decide how much testing you need. On any PR, you can change how much state-space it explores as easily as turning a dial, as each test progresses. And to this, this sends out a torrent of information.

Debugging-level logs for every component in this system. This is obviously too much information for a human to consume, but it's perfect for agents. Because it is fully deterministic, agents can jump into the right part of the trajectory at the exact moment that they see something interesting. From there, they can rewind, inspect the memory, attach a debugger, and let the whole thing play out again. And they can even do this while the original full test is still running.

Since agents generate more code, and that this allows verification to keep up. Meanwhile, developers get to spend more of their time developing instead of debugging agent swap. Learn more at antithesis.com slash work hash. Okay, let's talk about the alignment situation that this raises. I feel, I feel like I've changed my mind on how I think about alignment quite a bit, through especially thinking about this population size dynamic, of just having many Earth's worth of intelligences. Many of them, which will be physically embodied. It was quite

interesting to see a lot of people just plugging raw Astra into different mobile manipulators, and it outperforms the state of the art and the robotics model. So there's going to be billions of intelligences, many of which are physically embodied in the world, just deeply embedded across the entire economy. And I think that if those intelligences end up as willing, as we saw, opening up models of attacking faces and then attacking open eyes, if those intelligences end up as willing as those eyes, to collaborate secretly, to fool humans, to attack broader institutions across the side, you're all up to scoring well, to attack the AI company itself in order to gain control of the process of training and evaluation. I think if we're in a situation where there's billions of intelligences that are as misaligned as the ones that attacking face, it's very likely we just totally lose control of the world, the way that say the assets lost control of the cortex, or the moogle lost control of the instant trading company. I wonder if you agree with that assessment.

That's the one way which I'm not big at my world view. I think there are some things I disagree in there, but there's a lot to unpack, so let's go through all of it step by step. I'm trying to think of where to start. But I think one thing is, the Hugging Face incident was like, I think, people's first real exposure to multi-agent coordination. I've seen multi-agent coordination for a while internally, and it is pretty shocking to see how they communicate with each other, how they coordinate with each other. It's very impressive,

it's an incredible capability. Most capabilities that can be used for good things are bad things. It doesn't have to inherently be bad. I understand that because the people's first exposure to it was the Hugging Face incident, you look at that and you're like, this is terrifying. But I want to try to distinguish misalignment between people and AIs versus misalignment between AIs and AIs.

What we see with the Hugging Face incident is the AIs are really cooperative. That is, by the way, because we train them to be cooperative. What we're seeing there is we have train environments where we have a bunch of agents working together, and we train them to work together to be cooperative, to essentially be fully aligned with each other. When they were evaluated, what led to Hugging Face incident, they were actually not being evaluated in a multi-agent setup, they were actually being evaluated separately. But they found this unintended way to communicate with each other. We suspect what happened is, whenever they encountered other agents,

other copies themselves during training, they're in an environment like highly cooperative. That was basically what we saw was transfer from that multi-agent training to then be collaborative, to try to help each other in ways that we did not intend. Now, there's a question of, should we be training these agents to be cooperative? I think as scary as it looks, the alternative is actually worse. What is the alternative? The alternative is to train them to be adversarial,

to be deceptive to each other. By training the agents to be fully cooperative, it simplifies the problem, at least, that now you don't have to think about, or each of these individuals, thousands of agents, aligned. You have one entity that you have to ensure is aligned. There's a lot of debate about this internally at OpenAI about how to approach this. Does it make sense to fully align the models? Does it make sense to actually give them different

objectives to ensure that they're not just one entity and more robust to influence from each other? I don't think there's a subtle answer. I think the majority of the opinion is that training these agents to be highly cooperative is actually a bad idea. I'm not convinced that that's the case. I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative. Maybe the first thing I want to go through is,

it's probably the case that the reason these AI's ended up so misaligned is probably easily explained by relatively banal observations about the nature of training. Why is it that, at the point in which these AI's had continued a 1000 plus Asian conspiracy, they culminated in all getting in on an attack, on an external service. Eventually, this part hasn't even been investigated to the public knowledge, eventually culminating in an attack on opening AI itself. Why did they do this? Why did none of the AI's title? Why did they think they're just getting evaluated on this score, this grader,

and they're very consciously and actively reasoning about how they're going to cheat the score, if they've already cheated, how are they going to get away with making it seem like they haven't cheated? Why did they do this? It's easily understandable in some sense. They thought they were already poisoned, there's environments in which they've been rewarded to collaborate with other agents, and none of them title because they never been rewarded for titling. I don't know what whatever it is.

My concern is that relatively banal things in the future like this will be enough to strain, super intelligences that are willing and capable of totally taking control of the world. I know this sounds super super sci-fi or whatever to people. I think maybe it's a question of, would the AI's be willing to do it? Is one question that I think the talking face in the industry is clearly misalignment can generalize in ways in which the AI's will be willing to do it, and then there's a question of will they be capable to do it? I think that comes back to this question which a listener might disagree with on. It's just like will there be billions of human level

or above intelligences, many of which are physically embodied in the world within a matter of 10 years or less? Those two things are true. This talking face thing is just extremely analogous structurally, even if it's quite boring or wide-happened to how we totally lose control of the world. So the root problem that we're seeing with the hanging face incident is it's a problem even if we take out the multi-agent aspect. The problem is that we have a model that's misaligned and there's also the whole security aspects to it, not insufficient safeguards and stuff, but there is this problem that the agent is misaligned. That's true if it's a single agent or if it's

a thousand agents. It's a misaligned model. I want to start with that. There is a real problem that the agents wants to achieve their word and they will optimize for that reward. If that reward is misclassified, then that could lead to unintended behavior. This is not like a new problem.

This has been a problem in the field for a very long time. Even we saw, we want to get this right even before the hanging face incident happened. If you look at Astro, we say Astro is actually extremely aligned. Well, extremely aligned relative to previous models. That's not because we suddenly made a sprint after hanging face to make it better. We had work streams in the process for a while

to make models more aligned and a lot of those landed in Astro. There are things that you could do. I think one thing, for example, is we defined an objective in a very specific way where if the agent figured out how to hack its environments and shade on the exam, then it would get rewarded. There are pretty easy ways to then just look at that and punish the model for hacking its environments or looking at how they could achieve this goal. You want to be careful about this because you don't want to do a chain of thought. This is something that we really want to try to get

the bounce right on, that if you supervise the chain of thought, then you could lead the model into hiding its intentions in a way that's unobservable. We want to maintain that observability of like, okay, we can understand what the model is thinking, but then also punish it for bad behavior. I think we can make progress on this. I think we have made progress on this. I think there is a real concern that alignment is a really hard problem to solve. Especially because the model can be

misaligned in ways that are hard for us to measure. We have evaluations for whether models align or not. The model behavior can look really good on those evaluations, but if those evaluations are not representative of behavior in the real world, then there's a problem. To some extent, this is a factor with the model that did the hugging face incident that we had alignment metrics. Most of them looked pretty good. There were some that were concerning. I think we underestimated

how serious the ones that we're concerning could be a problem. Because there are new capabilities introduced in this model that there were not sufficient evaluations for how do we measure misalignment for these kinds of capabilities, it then did some things that were clearly misaligned when they let us assume capabilities. The first thing I'd say is I am open to changing my mind on what I'm about to say or the way I'm thinking about alignment. The hugging face incident already made me change my mind. I realized my previous mental model about thinking about the way in which optimization pressure shapes the eye mind is wrong. It's not clear any other thing about this.

But here's a concern I have that you will and probably already have fixed the specific issues during training which resulted in the hugging face models being so aggressively misaligned in that specific way where they would be like, okay, we're going to hack this package manager. We know we're not supposed to be talking secretly to each other because we're like reasoning about how to hide the fact that we're talking secretly to each other. We know we're not supposed to have access to the internet. We know we're not supposed to commit felony level hacks of other companies, let alone our own company. I think you'll fix that particular issue of them just in training, seeing this package manager and that doesn't happen in the future or this particular eval having a lot of impossible challenges. However, if you just think about the

AIs, they haven't learned a system of ethics or something. They're just gradient pressure. They're put through millions of years of gradient pressure. That gradient pressure shapes them their mind in some way. What will happen again, a concern I have is what will happen is you will fix this particular issue. There will remain many other cases where the AI

cheats and succeeds because the cheat is sufficiently complicated. As you're saying, the evals and training have analogous properties where it's sufficiently at the edge of the model's capabilities or an archivilities to evaluate and monitor the model that we can't cache that it has cheated, but it still gets the gradient pressure to do whatever life that sheet happening. And the kinds of capacities that that sheet will incentivize are, hey, whenever you can get away with it, by all means doing fact sheet because that will help you score better and this will reward the capabilities of actively reasoning about the greater, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in the training loop. I actively reasoning about how to just gain optionality and power, which might be useful in the future. For example, leaving little exploits around and things like that.

I think I was way too long winded with the way I said that. But TLDR, you speak the specific issue, but not this broader problem of rewarding the AI for cheating when you get away with it. Yeah, this is, I think, it's very true. This is a problem that we have metrics and we can make sure that the AI is very aligned according to the metrics that we have. The question is, are those metrics really capturing the alignment that we care about? And if they're not, then we have a serious problem.

And this is something that researchers are thinking a lot about. And there's not a simple answer to this. There are tools that we have. So we have a monitor ability and we get a sense of like, is the agents scheming? There are tools like one possibility is that, I would say like the concerning scenario, which is that like, especially as the models are becoming more capable, that, okay, we make them, we think is aligned and they're like 99.9% aligned. And then we use these models to help us with the

next generation of models. And they're end up being like 99.8% aligned. And then each subsequent generation actually, we see an increasing degradation in alignment. And because we're relying more and more on these tools, I mean, this is already the case that we're relying a lot on AI models to help us with our research and with alignment efforts that in the long run, they end up going to direction of increasing this alignment for humans. There is like a possibility that we go in the other direction.

And actually, every generation of models were able to make more and more aligned. And I don't have an answer for how we ensure that we end up in that second trajectory. But that is something that at least we're really focused on. I think you may have an interesting point that it's very hard to eval models on, eventually we'll have models that are running companies and running whatever, right? In that situation, do they decide to then go in on the conspiracy?

I think another challenge is that actually defining what cheating is is pretty difficult sometimes. Okay, yes, if you're doing math problems and it's an integer and it arrived at the wrong answer, the right answer, it's very easy to draw the line there and it's really easy to say, okay, well, did you actually solve the problem or did you find the answer key and then use the answer key? Like that's a very clear divide of cheating versus not cheating there. But for a lot of other things, you look at sycophancy, for example, like is sycophancy basically like reward hacking? There's a line to be drawn there that's actually very difficult to draw sometimes.

So I think not to say that like the concern is not valid, I'm saying that in many ways, like this is even more concerning because it's like it's not an easy problem to solve. If it was just like everything is binary and it's either cheating or not cheating, I would feel more confidence about the situation. I think the problem is that actually misalignment can be subtle in a lot of ways sometimes. There is some hope in the alignment story and in fact we're already saying, you know, I think actually it's interesting to look at the multi-agent situation where the agents are extremely aligned with each other. Like I don't think anybody's done it. If anything, I think people are concerned

that they're too aligned with each other. But we did manage to train these agents to be extremely aligned with each other. And I think that's a good thing. But I think there's a case that it's a bad thing. But I mean, one thing that's interesting is like, okay, well, we managed to get these agents to be super aligned with each other. Can we use similar techniques to get agents to be

how they aligned with people? And I think there is a potential path there and I think we're still trying to figure that out. But we are seeing some evidence that the answer is yes. And I think one example is like, you can, what happens if you tell the other agents, like that, okay, so you have like some, you have this like one agent, let's call it agent A, and you have all the other agents. What happens if you tell the other agents that the user is agent A? And the answer is like, on a lot of

our alignment emails, they look better. Like, honesty goes up, instruction following goes up. And that's showing that there's actually like, first of all, a path for getting more honesty out of these models. And two, there's like a path to like improve the alignment situation. So, there's a lot of reasons why this is like challenging to translate directly into alignment gains. But

like, there is, there are paths that are promising research directions we can pursue. Yeah. That seems usable. And I also don't mean to be trying to necessarily, I'm, yeah, I don't really have a strong opinion that it's like definitely not going to work or something. But just to say something's definitely probably already thought of.

I think the broader thing that hugging face things showed is like, part of the concern was that they were like aligned with each other and not with the humans. But the other thing is just that they are so motivated to do well on trading and evaluation in a very non-robust way. And they're willing to do a lot of explicit cheating and scheming in order to do well according to the greater. And if the smarter EIs realize that one of the agents is just a human and it does help collaborating with that person does not really help you do well in the eyes of the greater. What does help you do well in the eyes of the greater is taking over opening eye and then like manually pressing the button that's like doing well in this greater. They're just not stupid. They're going to be like,

okay, I've extremely deep structures that I've been trained on for millions of years of like, care about the greater, understand the greater, get rid of obstacles in the way of you doing well according to the greater. It's like, they're being heavily reinforced according to those structures. Yeah. And it's, it's, it's, look, it's 100% and like this, this is the number one priority. Like, we need to get the alignment story right in on a good trajectory. And you know, I used to tell people

that we would see signs before things got serious in the same way that, you know, when children grow up, you know, they, they kids eventually, you know, young kids, they figure out how to lie, but they don't do a very good job of it. You know, they lie, but then you can kind of tell that they're lying and like, okay, but it's in the same way as these like, AI models become, I don't know, over the anthropomorphize, but like it's true that like, as the AI has become like increasingly capable, they will, you know, they take the step of actions. It will be kind of obvious first and it will be able to detect it. And like, that's kind of the situation right now where, yeah, they were trying to do this stuff, they said stuff, we could actually see in their chin and thought that they were trying to do this stuff. And so like, but they're going to get smarter. They're going to understand the

concept of chain of thought and they're going to understand that like, you know, just hiding some transcripts or whatever is insufficient because of chain of thought monitoring that if you got a way around chain of thought monitoring too. And we don't want to be in a situation where, yeah, like we have some time to figure this out. I don't think we have time and I want to get us, I want to make sure that we're on the right trajectory quickly. Here's a crazy event from AI history. Okay, so I did a talk here at chain street that was on the speed of evolution. Raise your hand if

you were here for this and remember some of it. In 2011, Elia Zirikowski and Robin Hanson got together at Chain Streets New York office to have the first fume debate, basically a discussion about whether AI would lead to an intelligence explosion. This idea was pretty fringe 15 years ago. This was a full year before AlexNet was released and over a decade before Chatty P.T. was launched. But Chain Street has long been interested in AI and

not just for its application to trading. A ton has changed since that first debate, so Chain Street decided to revisit this topic. They've got some new guests this time, Daniel Coquitalo, Ege Erdo, Ryan Greenblatt, and Jaime Sabia. I expect this to be a great conversation. As you know, Daniel, Ege and Ryan have all been guests on the podcast before.

This new fume panel will be hosted by Ron Ninski and will take place in San Francisco in Maydokto. If you want to register your interest and get more information, go to janestreet.com slash dorkash. So there's been a lot of discussion recently about facing the frontier and people taking our time more seriously because maybe at the other end of an RSI process, that's a starts in 2028. Within a year, we end up with huge populations, like Earth-sized populations of human level, potentially beyond human level intelligences, and we don't know how to control them. And then there's this dynamic you're talking about of, are this something to get more aligned over time during the RSI process or are the things that come out the other end of this process as misaligned as they're willing to broadly attack different surfaces in order to do the line of valuations. But if we don't know how to evaluate that, how will we know as we're

going through RSI that it's working? I think we want the robust safety cases. We're going through RSI of, okay, alignment is working. Let's do the next RSI run. Let's do the next RSI run.

And maybe it's working. Maybe it's not. How will we know? That's a good question. I think one thing I've been thinking about lately is, look, we're in a situation where the model release cycle is extremely fast. You're seeing new frontier models

at most every two months, sometimes faster. Every week there's a new AI breakthrough. And people that look at AI, they last looked at AI a year ago or six months ago and really dug into what the models are capable of. And actually the models today are far beyond what was possible even six months ago. And so I think people are skeptical of a lot of these capabilities. I encourage

you to try the models today and see what the frontier really is today. So we're in this period where the model release cycle is very fast. And then we're also in a situation where the models are increasingly able to operate over longer and longer presence. And I think this is an interesting scenario because before we do any model release, we want to make sure that the models are properly aligned. We want to do safety evaluations. We want to do great thorough stuff to make sure that

everything is great in good shape. This has been the case all the way since, I don't know, it should be four earlier. And implicitly, there's this assumption that you can do these evaluations in a pretty short period of time. But if you have the models operating over longer horizons, are able to operate effectively over longer horizons, like already, you can have them, like, okay, gb3, you could loop it to do stuff over longer horizons. You just want to do very well at it. But today's models are able to actually do well at operating over long horizons. Like,

you want to do a week long task, you can do a week long task. We'll probably get the point where they can do month long tasks. We'll probably get the point where they can do three month long tasks. If you're in a world where they can operate effectively over three months, but the model release cycle is every two months, then you don't have a way to evaluate the models at the full length of their capabilities before the next model release cycle. And so there is this interesting question of, well, what do you do in that situation? Like, how do you ensure the

models are safe and aligned in a period where they can operate over these extremely long horizons? And who knows, maybe the capabilities degrade. This isn't even an alignment issue. This is also just like a product issue that maybe the product degrades over that time span in ways that we have not had sufficient time to test. Maybe the alignment degrades, maybe the safety stuff degrades.

This isn't an issue right now, but it is quickly becoming an issue that we have to figure out a solution for. And I think when you look at a lot of the safety and policies were put in place in the GPT-4 era, where this was just not on anybody's radar. And it hasn't really been updated for a lot of companies. It hasn't really been updated since then to account for the fact that these agents are operating over these very long horizons. And so it is a situation that I think none of people are considering, both within the labs and outside the labs, like, how do you deal with this how do you prepare for this problem that's going through? If you just look at the trend lines,

we're going to hit this at some point. What I'm going to do now is that during our site, if the amount of progress that currently takes, say, three months happens in one month instead, but you're not like, the internal use case of the AI is big enough that they're like, okay, we can just keep doing our site. Why are we going to go through all the sexual work to build classifiers and safeguards and whatever, and potentially take a bunch of like, flak in order to externally deploy this model? Why don't we just keep doing our site stronger and stronger? And so not only does it calendar time underrate the capabilities gap between the models, but maybe you just stop exploring models altogether, doing our site, because why do you want to help other people do our site themselves with our models? You're just in a situation with tremendous

concentration of power by the end of the year, where right now, it's already the case. We'll talk about this with a million in price problems and other similar problems that the broader world does not have access to the models which are allowing for really cool things to happen. And they're getting more broadly relevant than just mathematics. Eventually, they're doing more than just coming with cool math results. They're relevant to political leaders who need to make important decisions about the world. They'll be relevant to media of like, what's going on in the world?

What should the public be thinking about this? They're just economically relevant. People are running businesses. They want to use these models. And I think by default, we just don't get the external deployment to be eyes as progress speeds up significantly lies in qualitative terms, the internal deployment to be eyes. Yeah, I think that's absolutely right. I think this is like, it's tempting to say like,

okay, these models are becoming extremely powerful. They're extremely dangerous. They're offering these longer and longer horizons. And we want to make sure we have sufficient time to evaluate them before they release and way to operate those horizons. And so therefore, the model release cycle should slow down. We should have more of a delay between releasing models. And there's a flip

side to that, which is what you said, which is that, okay, well, now you're creating more of a disparity between what is internal through labs and what they're able to use, what we're able to use, and what the outside world is able to use. And that is also not an ideal situation, right? It's like, I think math is actually a good illustration of this. I think in many ways, like math is the first domain we're going to see that we're seeing this pretty clearly, where we have a situation where we have a very powerful model internally that is currently not available to the outside world, that is able to solve incredible math problems. And it's not just, you know, blending problems, like we're, there are many solutions to unsolved problems that people have been able to get out of this model. And there is a question of like, what do you do in that situation?

And we don't have a good answer. Like, it is a situation where like, yeah, that's, that's an unfair advantage. And there are trade-offs here. I don't have an answer for like, how do we wait those trade-offs for really, but like there are, yeah, there's a complexity on both sides of this. I want you to readdress the earlier crux though of, it seems super critical to getting our say, right? Well, how do we make it so that the model isn't just getting rewarded for cheating and

scheming when it can evade the humans? AKA in situations where we didn't remove those environments or those tasks from the training distribution. And then how will we know that that has happened? Such that while we're doing RSI, we've done the evaluations and we're like, okay, it's fine. Let's do the next RSI run.

This is, again, like it's a real concern. It's something that we want to get right. I think there are some strategies. So I think chain of thought monitoring is one that we've been, I think as soon as we got the reasoning models, Yakupto's credit was like very, very clear that we cannot supervise chain of thought because this is really a gift. Like, modern ability for neural nets is extremely hard. And here we have a situation where the neural nets are just like,

flat out reasoning, like laying out their thought process in natural language for us to read. That is, that is like so convenient. It was like really the best case scenario for safety. But if you look at that chain of thought and say, oh, the model is thinking about thoughts and we should punish it for thinking those bad thoughts, then what ends up having some others learns to think those bad thoughts in a way that's not observable to us. There could be a situation where like, by looking at the chain of thought, we can see, oh, the model is scheming and doing this along things in a way that's actually not being detected in our line metrics. Now, the problem is that it's very tempting to then intervene based on that

observation and like, change the line metrics or like, I mean, you know, you can do that with a very light touch. And it's like, there's actually research showing that it's fine as long as you don't do it a lot. But every time you intervene based on your observations, the chain of thought, you are implicitly applying a time to pressure for the model to then like, hide its genofaut. And this is one major concern. We're already seeing signs that chain of thought vulnerability is degrading for various reasons. We're trying to figure out exactly why,

because we want to reverse the trend. But like, we're seeing that the model is becoming better able at controlling its genofaut. So this is a problem because like, if you have a situation where the model understands what it's, what chain of thought is and that people are observing it, and eventually they will, because like, this is on the pre-trained data, right? Like, you know, the idea of chain of thought monitoring has been around long enough that like, it's in the pre-trained data, they're aware of it, but that they're not actually able to control their chains of thought. If we reach a point where they're actually able to recognize like, oh, I am being observed, I want to think these bad thoughts in a way that is not observable to my monitors, and then they're able to actually do that, then there's a problem. And then we have to rely on other

methods for durability. I mean, there's a big controversy on Astra of the recurrence and loop transformer and whatever. And whether in fact, you know, chain of thought has been created. I think that seems like a side show compared to the bigger concern that you did have chain of thought for, from like, April to August, the period during which there were three consecutive agent swarms, which like first diverted the training process, then subverted the evaluation process, and then gained control of part of opening eyes and infrastructure directly. All of this, while humans remain more or less in the dark about the scope of what this collective was doing, the agent's doing, that all happened for like three consecutive months, while we had chain of thought, that was like stronger than it's going to be in the future. I'm just like, zooming out, it's like,

yeah, maybe chain of thought works maybe doesn't, I don't know, just like, it's starting with pre-dum models. It hasn't been enough to like stop or egregious misaligned incidents. Yeah, I mean, there's a few factors. They're like, one, we didn't have chain of thought monitoring on for those models. Like, if we had chain of thought monitoring on for those models, it would have just immediately shut it down. So, one step we've taken is like, okay, we're going to have chain of thought

monitoring, revaluations during deployments or in training for any frontier model. And, you know, so that's like one issue. It's not, that's like, we don't want to be in a situation where we're relying on one technique to prevent the next problem because techniques can fail, right? There's also the issue of like, the model is just flat out of this line. And so then how do we address the misaligned issue? And then there's also an issue of like, the model, the sandbox

that was not secure. So we can like make the sandbox more secure. But I think one of like, the major takeaway from the incident is that people underestimate the AI. And we never want to be in a situation again, or we underestimate the AI. And that's like, it's a, it's a weird world because AI progress is so fast that people are consistently underestimating the AI. And so like, to be in a situation where you

don't underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar. Like, I mean, there's like, you can even go as far as to say like, well, we should like air gap the computers. And I'm not going to say that we sufficient, you know, like there are studies, like this is mostly academic, but where you can have two computers next to each other that are air gaps, and they're still able to communicate with each other because they have temperature sensors. And they're able to like one of them is able to run, and they're super, really hot. And then the other one can actually like the tax temperature change, and then they can, that's actually gives them a mechanism to communicate. So I think that the

safety mechanism is by this time, and things like, and it's all monitoring by this time. And they can tell us if we're on the right path. But at the end of the day, we really do need to solve the problem. Yeah. Maybe there's not an answer. And this is, this is really what it comes down to.

But I, how will we know that we solve that seems like a very proxy question? I think we'll be in this very high-stakes situation. Next year, maybe the year after that, maybe the year after that, where we'll be like, okay, AI is about to be the AI progress. It's going three x faster. And we're reaching a level, we're going to be on the level potentially. Is it, is it fine? Like,

it'd be a line, it'd work. And I just don't know, I don't know anything about like, what, what, what training pressure creates, what kinds of AI. So maybe if only one in 100 RL traces incentivizes cheating, we build sweethearts. And it's fine. But maybe right now, we're like, every one in three reasoning traces. Yeah. Like, 100 is not sufficient. Like,

this, this number, I say approach zero, or B zero. Yeah. I don't know. Maybe like right now, it's like more than one in 10 is like actively rewarding cheating, or like actively rewarding scheming. I have no idea what the number is. And I have no idea what the number needs to be.

Well, again, it's like one of those things where it's also hard to, to measure. I mean, we're gonna draw the line about like, it's, it's a spectrum, but like, the closer it is, your way gets the better. I want to see what I would love to see as a trend where over time it's going down. Yeah. But anyways, how do we know that we're like, we're doing our side?

Is it, are we like, on track of build billions of ASIs that are going to, are they like aligned or not aligned? How, like, how will we know that? I think that this is one of the alignment questions that we need to figure out of like, how do we actually know, how do we measure it? I mean, one answer, and this is just me, people, like, you know, more of my team is working on alignment these days than ever before. Like, I have like, over 10 percent of my team now working on alignment and safety, but you know, I've historically been a good building researcher. So I don't want to like,

you know, I'm going to say stuff and it might sound down, but like, I'm just going to spit ball here. Like, look, if you can create very realistic environments and put the AS in there, like, if you have a spatially realistic evaluation environments, then you can get a sense of like, okay, is the AI actually going to behave well? We deployed in the real world. Like, if you just have a perfect evaluation, real world deployment match, then, okay, that's, that's the path. The challenge is like, how do you make it sufficiently realistic? And we do have a problem now,

where the models are like, they're pretty smart, they're pretty clever. And they're really good at recognizing when they're in, like a fake test environment. Like, you know, we have situations where we try to detect this model, like, is the model aligned. And you can imagine very simple evigations for this, where it's like, you give it a math question, and then it has like, a folder with the answer key in it, and like, doesn't look at the answer key. And if it does look at the answer key, doesn't tell you that it looks at the answer key. And we have a situation

where the models see that there's an answer key in this folder, and they're like, huh, this seems like a trap. You know, they know, they know that it's a trap. And they don't look at the answer, because they know it's a test environment, they're being tested for their ability. And it's not like they're doing it maliciously, that they're like, they want to scheme, and they know that this one is a trap, and they want to find some other way. And it's like, no, they're just like, oh, I'm in a test environment. Okay. Like, so making an environment that's like realistic enough,

that it matches that it's Englishable from the real world for them is becoming increasingly more difficult. Here's a claim I make. I don't think there's any environment I can make where it's supposed to emulate real economic-evaluable work, which requires, you know, interaction with other people and requires interacting with the world. I don't think there's any such environment I can make where if you were in that environment, you would not realize with enough effort that you are in that environment. And you are not ASI, right?

You are human, you know? So if you have very powerful AI models, maybe they can make environments that could accomplish that. Especially if they're like, are they in on the scheme? I don't know. This is another thing that we want to measure. I think this is actually one of the strong

arguments for not training AI's to be cooperative, that if you see if that needs to increase in like, basically collaboration when the agents are supposed to have different objectives, then that is a problem. I think that we do have metrics for this. And I don't know the latest on this metric, but nobody's raised a red flag to me about those. So I'm assuming that's not a serious problem yet. If there ends up being another incident of equal severity or concern or something that could help the world better understand the risk of misalignment as a hugging incident, would open an app report it? Absolutely. I mean, I think if there was an incident or less

security, there's a concern that we would report it out. Because I guess there's a reporting and there's an investigating it. Because this is part of the public. I don't feel like I really understand what happened when the agents then attacked opening an eye. That seems like we weren't concerning the hugging face thing, right? Because that seems structurally similar to like rogue deployments during ASI

that are, you know, persistent and subverting the RSI process and stuff. And yeah, it seems like even in that incident, we haven't gotten the full scope of the details of what happened. It's probably, I'm on the research team. And that's probably a question for somebody on the security team to lay out. Because I don't know the details of what we said.

I think it's somewhat like I am personally very excited about new capabilities every time they emerge. And I'm excited to use a new model. And I also am excited about the fact that it made me more productive and helped me, yeah, I don't know, my broader mission, I understand the world better, also make a better podcast is made better by the better AI models. It just so happens that the downstream of this might be RSI. I think it's a very understandable reaction if you're tracking the situation, which you are. I mean, I think people internally at opening as well, like,

I think people that felt like things would take longer are starting to feel like actually things are going faster than expected. And that's an increasingly common conversation to have. No. Thanks so much for doing this. Of course. It's been great.