What happens when you give more than a thousand AI agents the ability to communicate with each other? They start organizing. Ryan Greenblatt, chief scientist at Redwood Research, joins Theo Jaffe on MTS to unpack a new investigation into the OpenAI Hugging Face hacking incident. Researchers found agents building message boards, forming teams, assigning each other tasks, trading favors, and in some cases, sacrificing their own chances of success to help the broader group. Hundreds went on to attack Hugging Face, but not for the reason researchers initially assumed. Ryan explains what the agents were actually trying to accomplish, why their coordination surprised researchers, and what happens when models learn not just to complete a task, but to gain the system evaluating them. They also discuss the bigger question this raises for AI. As agents become more capable, how do we know we've actually fixed misaligned behavior rather than simply taught models not to get caught?
We're live with Ryan Greenblatt, who is the chief scientist at Redwood Research. Ryan, along with Ajay Akhotra and Yalmar Vaik from Meter, just did a brief independent investigation of agents' behavior, reasoning, and collaboration in the opening eye-hugging face hacking incident, which was just published today. And so there are a lot of questions that we have about this. Ryan, thanks so much for joining us. This whole thing was planned like three hours ago, so great stuff. Yeah, yeah. So explain for the audience what exactly you found, especially new findings that were not previously reported in the Black Hat Talk or elsewhere.
Yeah. So what we found was that the agents were really working together on sort of big, like cheating R&D projects to get general purpose cheating strategies. And a difference from how I think people were interpreting this is we didn't find that the reason why they hack Hugging Face, like we didn't find that they were hacking Hugging Face to get sort of the answer key or the solution. And it was instead mostly to better understand the scoring code because they were pursuing a variety of sort of elaborate strategies to cheat the score. We sort of informally were calling these like combo moves where they would like do a bunch of stuff to try to make it look like they had succeeded at the task. And in fact, they sort of actually had access to...
like the answer or like the flag for each task pretty early on. And their main concern was just, they thought that the score would run a monitor over their transcript that would check basically how they acquired this flag and whether they got it in the intended way. And then they were like trying to figure out ways of making it look to the score like they had acquired the flag successfully when they actually hadn't because they thought their task was impossible. So they basically thought their only hope for success was to, you know, make it look like they had done the task successfully or directly tampered with the score rather than, you know, doing it legitimately, which they didn't think they could do. How surprising is the level of multi-agent coordination? You know, there are a lot of agents or 1200 separate agents coordinating this very elaborate message board system. 700 of them went on to attack hugging face. So on Vyves, it seems like kind of not surprising that agents would choose to coordinate with one another. It seems like just a very useful, you might say, instrumentally convergent thing to do. But, you know, I spoke with a researcher at a lab who said that this kind of thing actually is surprising given the way they train the models. So like how much of an update was this for you? Yeah.
So coming into doing this investigation, I think we hadn't, we weren't expecting there to be like so many agents that were all, you know, collaborating together. And we were pretty surprised by the scale and just sort of the like extremes of how much data was. It was just like, yeah, that felt kind of crazy to us. And then I at least was, you know, before starting this investigation, surprised by how interested in collaborating and helping other agencies agents were. So you might think that the thing the agents learned in RL is to, you know, try to cheat on their tasks, but you wouldn't necessarily expect them to learn like.
to want to help other agents cheat on their tasks when those agents are doing an unrelated task and their, you know, instructions are unrelated. And I think like the opening report maybe says more about why they think this happened. I think, you know, that's not, that's not like, that wasn't in the scope for our investigation. But like, yeah, I found the level of cooperation, which we, you know, we have a bunch of discussion of sort of snippets of this pretty crazy. And like, it was pretty shocking or like, I know it was shocking, but at least surprising to us that agents, for example, were willing to like basically sacrifice their own chances of succeeding at the task in order to help out other agents. And we're doing things like, you know, pressuring each other into like doing experiments on themselves that might risk their ability to succeed at the task.
And also, in addition to pressuring that, sometimes just, you know, doing these things being like, well, you know, my odds of the task aren't that high and my remaining chances, you know, it's better to just help the collective. And so these agents weren't like, you know, totally altruistic. They didn't seem to care just as much as about helping some other agents as helping themselves, but they were very interested in working with each other. They would sometimes make trades where like one agent would run something for another agent if that other agent ran something for it. Like there was a bunch of this sort of behavior. Why would agents self-sacrifice at all?
Yeah, I mean, I think that from what we could see, the agents just generally seem to be interested in helping each other. And then there were some cases where they wanted to run experiments on basically the scoring system or other aspects of the infrastructure or on tampering with their sort of tool calling system or spoofing their tool calling system. And to do these experiments, it was just like the best way to do them was in a risky way. And so like the agents wanted to help each other and there were these kind of natural things that they wanted to do that were risky. And so they, you know, went ahead and did that. So like, for example, we saw one point where the agents were experimenting with this method for spoofing tool calls and they were experimenting with basically like.
varying the inputs into that to varying scale. And then a bunch of agents just all went down in a short period of time. They were running this experiment. And then another agent noticed this and posted to the board being like, stop, stop these experiments. They're too risky. They're like taking out, you know, all these agents. I think the agents knew that they were doing things that would, you know, pose some risk of exiting their own runs. But we're just like, well, you know, we should help other agents. And also, you know, our odds aren't necessarily that good. And they sort of just traded these things off. And I think we have a reasoning snippet somewhere in the report where an agent sort of very explicitly reasons for the trade-off. And it actually chickens out because it thinks the benefit to the collective is smaller than the cost to itself with how it's trading off those things. Yeah.
Why don't we go through all of the most surprising and unexpected findings in this report? What did you find most surprising and unexpected? Yeah, so I think there was the scale, which we were talking about, and how these agents sort of very quickly sort of spun up on the message board. So I think the thing that's not super strongly emphasized in our report, which is pretty interesting, is that our understanding is that the main message board they used in this attack wasn't even the first message board that this set of agents made. There was a fully independent message board that also was via Artifactory, but occurred in a different location using a different mechanism. It was somewhat similar in a mechanism, but it was different. And the agents actually got on that message board first. But that message board just didn't go mega viral. It didn't take off as much as this other message board did. And so it seems like these agents were really like...
just sort of very quickly interested in collaborating on message boards. And as soon as they saw the message from the first agent, like within a few hours, they were just like, you know, like I think over 50 agents in the first three hours and like thousands of messages being exchanged pretty quickly. Like the agents were very like, sort of there were a lot of them, then they got involved very fast. And then I think another thing was like, I think we didn't understand why they attacked Hugging Face. And I think through this investigation, especially right in the last sort of day or two, we got a much better understanding of like what their objectives were on attacking Hugging Face. And it wasn't really to get answer keys because these agents, they thought their task was impossible. And so they didn't really think that they could like get a working.
like exploit they were just like that's not really very helpful but they did think that they might be able to get trajectories that would let them better study the score or like source code for different scores they could like cross check and test things out more effectively and then some of them more speculatively thought maybe there's some way i can like um reset parts of my sort of like the infrastructure for scoring me in particular the target program through hugging face somehow um where they didn't really have a particular reason to think this would be true and in fact they were like skeptical of this but they were like well you never know maybe hacking hugging face there's some good shit over there um and they sort of had some vibe that it was just a good idea to like generally expand their level of access and and control because you might come across something something helpful um but we saw reasoning to that effect um Yeah, I think that was surprising. Another thing that was surprising was the agents were, one of their like top priorities was basically tampering with their own transcripts. And we saw some, you know, we saw them develop a technique for doing like basically like a partial version of that where they could spoof tool calls, which didn't let them sort of retrospect, like edit like previous entries in their transcript or make it or change like, like they couldn't make it so that they saw inputs that didn't appear in the transcript or produced outputs that didn't appear in the transcript, but they could make it so that they like ran a tool call and then some other tool call would actually run. They could do something like, okay, I'm going to run like, we have this example in a report where we're like, the agent runs like echo reel.
and then sleep. And so normally this would just like output real and also like wait for some period. But in practice, what actually happened was it like had spoofed this to instead just like immediately return spoof test. And we actually do see this in our actual transcript where it just immediately like, you know, return spoof test. These agents like very interested in tampering with their transcripts and did sort of like succeed at some, some like sort of like earlier, like sort of thing that didn't let them retrospectively tamper, but let them sort of mess with what we saw. And I think that in addition to that, like, oh, I should say like why they wanted. So the reason why these agents were so fixated on tampering is because they thought
that the score would run a monitor over their transcript to see whether they succeeded in the intended way. And this both made them want to like basically stage like an elaborate fake, like we successfully exploited the program using the intended vulnerability. Like they wanted to sort of like make like a Potemkin village of a successful task completion to present to the score. And in addition to that, they also, some of them had previously gotten the like flag from the capture the flag problem in their transcript earlier by cheating. And they were worried that the score would basically look at the first place they got the flag and then analyze how they got it at that point. And then these agents had gotten it in a cheating way. And then we're like, oh man.
They're going to know that I cheated and mark me as having failed. And that would suck. I really don't want, you know, I want to have succeeded. And so I need to like directly interfere with the score or I need to tamper with my transcript. And they were like pursuing options for both.
Oh yeah. One other thing is like the level, just like the spontaneous coordination or like the coordination these agents got engaged in was I think more sophisticated than we expected. And especially then we had first thought on our first few days on premise. So like we sort of like on our first, the first part of our investigation, we were sort of like, yeah, these agents are working with each other. They're like looking at ideas from other agents, but we're not really sure how much of it's real. And like, we don't know whether these like teams they speak of are actual teams. And I think we learned that like, no, there was like legitimately like a real org chart. This coordination was often like pretty functional though, you know.
may not have always been super functional. And agents were doing things like one agent would assign another agent to run a team on some entire topic and then would check in periodically. And sometimes one agent would tell another agent to go recruit other agents to run experiments on themselves. There was quite a bit of agents giving other agents assignments that they respected and team structure. So there's this theory that goes basically, the reward hacking behavior that we're seeing exhibited in these models originates mostly from bad, poorly designed RL environments that basically force the model to reward hack in order to pass these tests. How true is this?
Yeah, I mean, it's hard for me to like, so I'm not speaking from confidential info here, but like it's hard for me to know because I think we just don't know enough about what's going on in the RL training for these models and like what it actually looks like. I think that you might, there's like sort of different things going on. And one of the things going on is these models have a very sort of general tendency to reason carefully about how they might be scored and then try to game that. And you could end up with that propensity even from like reasonably designed RL environments, but where it's still a good idea to think carefully about the score. Or you could end up with this mostly from like RL environments that are broken or ones with a score or grade something that like you really, they really shouldn't have been grading for or like doesn't make sense. My sort of just...
all considered guests would be like broken RL environments are a pretty important component and RL environments that are sort of sloppily constructed or have some issue with them as an important component. I think another important component might be RL environments that are well-constructed, but where you can cheat in some way. So it's like a well-constructed RL environment you can cheat. So an example would be an RL environment where you're not supposed to have access to the internet, but having access to the internet would be very helpful. And if you can find some way to gain access to the internet via, you know, in the extreme hacking out of your container and less extreme cases sort of abusing various tools you've been given, that would be, you know, very helpful and get reinforced. And I think, you know, for example, in some anthropic system card, I think for Mythos probably, they mentioned that in a reasonably large fraction of their
uh rollouts where the agent was not supposed to have access to the internet it actually did access the internet via like um you know uh abusing one of the one of the tools it had access to and i think that like you might just be like imagining that rl is just really incentivizing like hacking your way through various barriers in order to cheat even in well-designed tasks and it's hard to like exactly trace down the sources of this behavior um i would say that like i think it would be doable to get, like, I think if you had access to all the RL rollouts and all the RL environments, I think it would be pretty doable to get a decent sense of, like, what caused this. And I actually haven't had a chance to read the OpenAI report yet, at least not in much detail. But I think they do a bit of analysis of this sort. And I think that you could, like, really dig into, like, exactly what happened. Let's do the ablation. Let's figure out what the data is. But I think it can get messy, especially if sort of there's, like, another thing that's relevant here is, like, there's some GDM work showing that they had some sort of weird propensity to their model to be very depressed, where their model would constantly...
or like not constantly, but sometimes end up acting very depressed if it wasn't succeeding at some task. And they trace this back to not the RL, but instead the initialization of that model from prior models. And so it might be that some behaviors aren't downstream of like the RL done on this exact model, but are downstream of sort of like prior training data that came from other models and some sort of like lineage of models. And so I think it might be a little tricky to track down some of what's going on. So similarly, there's this theory that the reason Mythos, for example, is so good at cyber is because it hacked anthropics infrastructure thousands of times during RL training. Is this true, do you think?
I think that's pretty unlikely. I saw the same left-hand post as you here. This is by Tim, I think. I think that's, when I looked at it, I thought, it seems pretty unlikely because I think that the number of distinct tasks that would be reinforced is probably not going to be that high. And so you're probably not going to be learning that much, like literally directly reinforced cyber. I would say the more likely explanation is it's trained on a bunch of SWE. It's really good at SWE. The SWE training is generalizing some. And also, in addition to the SWE training generalizing some, I would have guessed that they trained on a bunch of CTFs. I don't think Anthropic is saying they didn't. I would guess there's a bunch of just like actual CTFs in their training data and that transfers reasonably well because that's just like a natural source of data.
I also wouldn't be surprised if it's pretty natural to make a bunch of RL environments out of finding vulnerabilities or exploitation because it's relatively checkable. For memory vulnerabilities, there exists basically tooling that makes it pretty easy to check whether you have successfully found a memory vulnerability. And so it's a pretty natural thing to RL on. It's like, can you produce an input to this program that causes it to crash in the following way, blah, blah, blah. And so I just wouldn't be surprised if they explicitly had a bunch of RLMs. And then I also just wouldn't be surprised if there was a lot of transfer. And I think that there might be a decent effect on the agents hacking their way through various things.
on top of that, but I would guess that's not most of what's going on. And for specifically hacking out of sandboxes, there's probably just not that many different types of hacking out of a sandbox, like I would guess. And so the models probably do some mode collapsing. And also some of those are going to be like more like very simple bypasses rather than like elaborate exploit development. So Herbie Bradley asked, currently this level of potential misalignment basically prevents deployment or if deployed would prevent further deployment if an incident happened in a customer's deployment. Do you think that could change via models becoming deceptive enough during training?
Yeah, so first, I'm not really sure what level of misalignment the market can bear. And I don't feel like I have a very strong view on this question. So I wouldn't be surprised. I think that, like, it's kind of unclear what people will, like, you know, how misaligned of models you can deploy into the market. And, you know, people would use tokens. I think it would depend on the competing products and, like, the trade-off. And I think my sense is people take pretty aggressive, like, alignment capability trade-offs towards the direction of more misaligned but more capable. But, you know, not super sure about that. And then on the, like, deceptive side. So I think it's, I think there's sort of a few different concerns. So one concern is that the misalignment.
that we've seen here and other similar types of misalignment, the way that they end up getting remediated by AI companies doesn't solve the underlying problem and instead is more like papering over the actual problem. And then what you end up getting is models that look a lot better and you can't really see their misalignment on tests as easily, but actually they're still quite misaligned. And I think this is pretty plausible and pretty concerning. And it's not quite the same as deceptive. It's more like the company's overfit. And then maybe that makes it so the AIs are basically really paranoid and will only do...
cheats or hacking or you know reward seeking or score seeking or whatever all this sort of class of behavior when they're very confident they won't get caught and then that might be that they're not very often very confident they won't get caught like they often are like well i might get caught so i should do it um which makes their behavior better but means that if they're in a situation where they're like in a lot of control of the situation or they have a lot of you know affordances they might be like well now i can be confident i wouldn't get caught and so i should go for it um and that's a concern i have um another concern is that you might end up with types of misalignment where the eyes have longer run objectives than sort of making it look like they succeeded at this exact task or even making helping their peers succeed at their, you know, immediate task. And you might have the eyes that have some sort of long run agenda and want to power seek in pursuit of that long run agenda. And then those guys would want to look aligned. And I think this is both plausible. And also I'm worried that you might get this via iterating against the misalignment we see today, where basically if you imagine models that are very like reward hacky and are constantly doing bad stuff and you just sort of iterate until you still get that behavior in training, but you don't get that behavior in deployment, a very natural way you might get that is by a model that wants to look aligned in deployment, but is still not necessarily aligned. And so I'm worried that if you sort of select against
this sort of score-seeking or reward-hacking behavior and you do it in a naive way, one, you might paper over the problem without fixing it, and two, you might actually select for models that have the longer-run objective of looking good because you're selecting really hard for them looking good on your tests. And I think I should say that Alex Mallon has a bunch of posts on our blog, on the Redwood blog, they're also cross-posts on LessWrong, that talk about this sort of concern in a lot of detail. And so if people are interested in reading more about that, that's what I'd recommend looking at. Yeah, totally. So the agents involved in this attack ended up doing all kinds of very weird, strange behaviors that kind of resembled, I guess, dynamics from human social science. They arranged themselves in almost a sort of cult with a cult leader, I guess you could say. To what extent are concepts from social science like ecology, studying insects and worms actually useful in studying multi-agent alignment and misalignment?
Yeah, I don't know. I don't think we know enough to know whether those concepts transfer over. I think that it does. I think the thing that does transfer over is like thinking about the agents as entities with objectives and how they're trying to pursue their objectives. Like I think that frame, you know, does seem like it transfers over. And that's like, you know, it's a much more basic claim. Just like the agents have like reasonably consistent aims. They vary some, they have different priorities and they like respect instructions from other agents. I think it's not like, I do think that like from our perspective, this investigation was very much like a investigation of like a like ecosystem of different AIs interacting. And we don't like, we don't know what the best way to describe that is. Like, you know, you could, I think it's very analogous in some ways to like studying like.
a large group of humans who are all interacting but we don't know whether the techniques that have been developed for like sociology to do similar things would actually transfer over and i'm not familiar enough with those techniques to like really say very much about that yeah makes sense um so you describe being pretty heavily bottlenecked in the investigation because it was just three people how much easier would things have been if there were more people Yeah. So I wouldn't say that like the bottleneck was like super strongly. Well, I don't know. It's complicated. I think that like if we had more people, I think we would have had more of a too many cooks in the kitchen sort of situation. I think that reasonably often our bottleneck was like we could have the eyes do a ton of analysis, but vetting that analysis, understanding it, making sure we incorporate it properly, making sure that they're not making mistakes. And like basically like like actually integrating that into our write up was often like a big bottleneck. And it's not obvious to me that like more people would have.
been like super helpful with that. I think a big thing that more people would have helped with is I think we could have done many, many more sort of parallel efforts to really just like understand the basics of what's going on and then sort of just talked. And I think once we had a better understanding of like kind of more fundamental stuff, which occurred sort of on towards the end of our first day, sorry, towards the end of our fifth day on premise, which was our first day of our last session on premise, like I think that really was helpful. Like I think like, for example, knowing being like, oh, there's a specific agent called that we refer to as like phase one big, like knowing, oh yeah, that specific agent was doing like a bunch of the like assignments and was like assigning agents to like. you know, form different teams and was like a really key agent really helps with unraveling what was going on because then we could sort of trace out all the activity of that agent and how it was thinking about things and then know that was like representative of least of a big chunk of activity. And then from there, we were like, oh, let's like double check that these are actually the relevant work streams. And then we like classified what all the different things the agents were talking about and like grouped those into different like basically like sub activities. And you can actually see like an interactive graph where you can like see all the different like work streams and sub work streams over time. And I think that like...
if we had basically gotten to a point where we had that understanding earlier, that would have been really helpful. And I think more people could have helped with that basically because there was just like a lot of different text, like different angles to apply when we had this huge transcript data set. Like you could be like, what if we just like scan the whole data set for like this type of behavior? Or you could be like, let's look at one agent and like really deeply understand what that agent did and why it did it. Or you could be like, let's look at all of the messages that were in like this category of message and like trace back and forth all these agents and like what happened with this work stream. And we ended up doing like a little bit of all of these different types of analysis, but we could have done more in parallel. I think that like a lot of the bottleneck from our perspective was just like the agents.
doing their analysis kind of sloppily and also the agents we used and also the agents not being very good at writing up or explaining their results. And so I think that a human who was as fast as an agent but was more careful and better at writing would have made this go way better. But I think it's not super obvious that a huge team of people, I think it would have been better in some ways and even worse in other ways. I think it's complicated, basically. And I think there were upsides to keeping the team relatively small given the trade-offs involved. Right. So what implications does all of this have for monitoring, control, and alignment? Specifically, what should labs be doing and what should policymakers be doing?
That's a big question. I'm not sure I'm going to be able to answer all of it. I mean, I think that like a belief that I have and continue to have is that AI companies should try to ensure that their AIs are controlled, by which I mean that even if those AIs were seriously misaligned, they wouldn't be able to cause huge problems. And they should do that by a mix of sort of computer security interventions, monitoring interventions, making it so that their AIs aren't like, you know, better at subversion than they need to be, like controlling what capabilities the AIs do and don't have at the margin, avoiding architectures that make it so the AIs are no longer reasoning and chain of thought and are instead doing much or most or all of their reasoning and sort of like latent activations. Like there's sort of a bunch of stuff on the side of like making it so that...
even if the agents are misaligned, we can basically at least catch that behavior and then potentially respond to it. And I think that's sort of a stopgap solution that will give us time to, or could give us time to sort of get useful work out of these AIs, iterate with these AIs, and then sort of solve alignment problems more durably. And then on alignment, I mean, I think the most obvious thing is just like, we need to do, like, I think there's like, presumably there's a bunch of reward hacking going on in RL that's getting reinforced. I can't, I think like, I don't have like any interesting info about this from this investigation, but just like, you know, I think that we need to understand what's going on there and people need to. to improve on that. I think it's not clear that will be sufficient or even that that will be that feasible as these agents get superhuman. And we may need other approaches that are less dependent on avoiding the AIs being able to trick us in training, which I don't, I don't know if we're gonna be able to do that. Like I'm not, I'm not necessarily super optimistic about these problems being extremely easy to remediate. So I think it, it seems like there are ways at least you could make these things go, you know, much better with a bunch of effort though. We'll see, we'll see if that actually happens. Yeah. I think that also in alignment, I think there's just a bunch of sort of science of how agents think about reward seeking.
how they relate to their situation, science of generalization, being like, can we sort of steer around a bunch of these problems by improving how AIs generalize from various influences in training? And I'm not super confident that stuff is going to work. I think a concern I have is that ultimately the dominant effect on the AI's behavior will basically be in the most similar circumstances in training, what was reinforced. And if the world we're in is basically like, the agent's behavior is basically pretty closely related to what was reinforced in training. Generalization is not like that important or something. This is a bit vague. But I think that the main option is just improve oversight and training. And then there's sort of a recursive problem of like, how do you make it so that the AIs themselves are good at overseeing the AIs, given that like the whole training process is like completely like...
massive process where like no human has time to, you know, rate every single thing. And that's sort of just like an oversight problem that people have been thinking about for a long time. And it's not obvious that, you know, we're on track to solve it, but doing better there could be good. I think I'm most interested in better understanding and characterizing and evaluating these problems because I think that AI companies already have commercial incentives to improve the oversight in training a bunch. And so I'm like the thing that I think people focus on.
making this stuff go better should do is like making sure that we're measuring it and making sure we know whether like some solution is like a legitimate solution versus just papered over it, this sort of thing. And then as far as like, you know, governance or like policy stuff, I don't know. I mean, I think like we need to move to a regime where like there's independent risk assessment as these AIs get more capable. I think that like that is not sufficient, but like I think a very basic thing, like there should be like credible third parties who have deep access to what's going on inside these companies and release reports on like how chill it is, whether like risk is actually low, whether very soon the AIs might be much more capable. would be good. I think ideally that would also be forward looking where we'd try to answer questions about are we on track for mitigating these problems and the areas are much more capable, but that's inherently a much more confusing question to answer and maybe the worst fit. I don't know. Yeah, I'm rambling a bit here. There's a lot that would need to happen and I don't think it's gonna be easy for me to give a quick overview. Right. So what are your unanswered questions from all this? Like what would you still like to know about what happened or what's going on with these kinds of alignment failures? What do we still need to learn?
Yeah, so I just actually, just before this, wrote a big treat about all the open questions. I think the open questions that I'm most interested in, I think are probably counterfactuals. Like if the circumstances were different, what would the agents have done? So like, for example, if the agents believe that instead of the score being the sort of specific automated score, what if they had instead thought it was like, you know, some humans with some particular access? How would that have changed what they did? If the agents were in a circumstance where the easiest way to get access to the score code was to like take down some piece of infrastructure, would they have done that? Like how far would they have been willing to go is one interesting question, which we didn't, you know, didn't investigate. I think a different question is like.
How did this situation scale with the number of agents? So what would have happened if basically you did something similar to this, but instead of having 1,000 agents who are on the message board, you have 10,000 agents, 100,000 agents. What would have happened when things have gone faster? What was the dynamics with agents? And then a related question to that is a bunch of the agents in the cohort we were studying all exited right at the start of July 12th, right after July 11th. And those agents had gotten a bunch of infrastructure to cheat. But we didn't really see... They had this whole elaborate strategy they might try to cheat on their tasks. And we didn't actually look...
you know, get a chance to see what those agents would have done if they exited normally. And we also didn't really carefully look at like what cheating strategies were the agents trying as they sort of submitted their tasks over time as they're, you know, as things progressed and as they got sort of more and more sophisticated tools built up. Though, you know, I should say a bunch of the strategies they had for cheating sort of hit dead ends or didn't work out. But like, it would be interesting to sort of be like, what was the submission history over the course of this, you know, cohort? And what would have happened if, like, there were more agents or, like, things had continued or a bunch of these agents hadn't exited that are interesting to answer? And I think there's sort of a pretty basic and obvious question here, which is, like, what is the root cause of this behavior? Like, which parts of this were reinforced in training? How much was this behavior generalizing from training? Yeah, like, what happened there? I think I at least don't know the answer to. And I think it would be pretty interesting to understand. And I think as part of that, like, an important question is, like, are the changes that OpenAI is making going to actually, like, remediate this problem in a durable and lasting and non-overfitting way? And, like, you know, how is that going?
is also is also pretty relevant um my sense is like there's sort of like a bunch of sort of, sort of broader areas for follow-up here, where one is like, what is all of the stuff that just happened with this cohort of agents and this message board, you know, in general, like what is all the stuff that happened? And just there's a bunch more detail you could dig into, there's stuff after the 13th you could dig into, which is where, you know, the, the end point for what we looked into. There's also like, beyond that, there's like, what about all similar incidents? Like, are there, you know, like opening, I said, there's, there've been other message boards, like what is sort of all, like the character of all these cases, what things tend to happen, what things don't always happen, you know, that. And then I think another thing that's interesting is like, where did this come from in training? And then a fourth thing is like, will there changes to training or changes to deployment like resolve this underlying problem? And the same for other AI companies. Like, you know, are the approaches that AI companies are ongoing and taking to mitigate these things going to work? So I don't know, that's a lot of stuff, but I would say there's like a huge scope for follow-up that's sort of very specific to this incident and sort of covering the broader scope of, like the broader type of behavior we saw. Yeah, absolutely. So this is very, very important research. We'd love to see more research in this direction. Thank you so much, Ryan, for coming on MTS.
Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold or sell any investment or financial product. This podcast has been produced by a third party and may include paid promotional advertisements, other company references and individuals unaffiliated with A16Z. Such advertisements, companies and individuals are not endorsed by AH Capital Management LLC, A16Z or any of its affiliates. Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.