One of the big surprises the last few years has been the generalizability of coding agents across different domains. And now frontier researchers are showing that this includes controlling robots. This led MIT professor Philip Isola to suggest in a recent viral essay that we may be entering the era of robot-use agents, where general purpose models can make different robots more capable. So today, Francois and I invited the founders of Water Labs and RoboCurve, two groups of startups that are working at the very frontier of making robots more capable with LLMs.
Maybe you guys want to briefly introduce yourselves and just say a little bit about what each of your companies focus on. I'm Humme, I'm from Water Labs together with Vincent. We work on building LLMs like into robots, and we do this by doing two things. Building a harness that allows the LMs to do this very effectively, so I can data and I use that data to train better LLMs.
Hi, I'm Jay, co-founder of RoboCurve. We are an e-vowice company for physical AI. We measure everything, any robot, any model, including LMs and also classical approaches such as VLAs, vision-language action models and world action models. And we evaluate all kinds of our important ones, hands, rivers, arms, human rights, all kinds of stuff.
So in the last few weeks, videos from both of you guys went quite viral on Twitter and I think inspired. Both of them are actually quoted in that Philip Isola essay talking about robot-use agents. The videos showed things like LLMs being able to unscrew caps, being able to communicate between multiple robots, being able to do tasks like uncapping a pen, for example. And so I thought maybe one would be fun, is for a Francois and I to dig in with you guys on some of the research that led to this moment to begin, and then maybe we can also show some demonstrations of what this can actually do.
To start, why don't we talk about some of the early research around transfer between language models and robot control? So I want to talk about some of the early work on pre-training language models for decision making, and then also the RT2 paper, which established the VLA. So maybe Jay, you want to tell us a little bit about some of this early work involving being able to use language models to do any kind of robot control? One of the earliest successful approaches of using AI on robots is the RT2 paper, where they use a pre-trained language model on web text and images and use that to control robots.
And the interesting thing about that is that it's a fine-tuned version of a language model, and instead of outputting English, for example, they just output what we call an anti-defective pose, which is the coordinates that you can then translate into joint commands that can control the robots. In some cases, it's very similar to what we see now with large language models, just that instead of fine-tuning, the models are going to just do it out the box. And this is actually quite similar to the mapping that I have in my head is kind of like the COT moment. And so when we did GSM 8K and you said, Suzy had $8, she spent $5, how much does she have now?
It had to output four hash marks, and then the answer, and then EOS. There was no ability for it to chain a thought. And so then we allowed it to actually have like, okay, let me go 8 minus 5 is 3, and so like, hashtag, hashtag, 3. And so you allowed it to do this kind of thinking before it actually gave an action or an answer.
And similarly now, VLA is where basically in RT2 is basically like, it has to output an action. I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do. And I can say, okay, even if you were using the example where you're not actually outputting code, you're just giving it to Astra and let it think, think, think, think, think, and then output an action.
Similarly now, and then with code, it's a more common-gob complexity or lower-comical complexity code length to just use code and really compact code to just like, here's code. Yeah, I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less. If you give the agents or you give the AI model more autonomy, and you if you unshackle it a bit more and give it more resources, you can actually do a lot of the things that we find to in it to do.
I'm just going to hop in here as well. I think the really interesting thing about the bits and lesson here is that, like VLA is by design architecturally, they're built on top of language models as well, right? They can potentially reason they can write code. So perhaps the bits and lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful, right?
Like people have been sort of nagging at this VLA data botanect for years now, and we're seeing very, very slow progress. And so the lesson here is maybe we have a modality that we know works very well, and this idea of transferring across different modalities, which having I are super excited about, like, how do you take something that's traditionally out of distribution, a lot of robotics, I think that's an institution, like maybe the way of harnessing a bit less than a sink, that's picket data for which we know this mentality is pretty blessed. There's a lot of data, these language, what these models understand well, and then use this as a way of unlocking all the domains as well. Yeah, and to add to that, the reason why R2-RT2 is such a good model is because it is using a language model that taps into the modality of all the data that a language model is trained on.
So all those web images and web tags actually improves the VLA compared to just training a specific robotics foundation model without any pre-training. So you're benefiting in the case of R2-RT2, these early VLA approaches from pre-training. I guess what's the limitation in the action taking fine-tuning approach that you can now get around if you can directly write code? You're referring to a difference in data there.
What exactly is that difference in data you're referring to? There's a few things that models have gotten much better at since R2-RT2. Like one, I mean, they're much better tool use, for example. So one kind of data is now these models can, the right code much better.
So now they can write complex policies as code. They're also much better using tools to explore the kinds of environments they're in, like, arms they have access to. And a lot of this comes from in-context learning, like the key difference between the VLAs and where, like, GBD6 or LLM isn't necessarily the architecture, but kind of the approach you take towards training. We want to be a bit less impaled, right?
We want to benefit from all kinds of data. We want to pour in computer use data into our models. We want to put in coding data into our models. But when you do that, you just end up getting what we think of as, like, these general-purpose LLM agents.
And so to me, it's like, why not just build a really good foundational LLM and use that control robots, rather than training a model that's specifically for robotics, and that's more dependent on just robot data. On that, maybe one of the things that I think, if we could run the clock a couple of years, we may have been a good sign that we were training in a good direction to this, is research on code as policies, right? Do you guys want to talk a little bit about when the research community refers to code as policies? What exactly that means?
And I think it's worth putting this in the context of when this paper came out, which was coding agents just starting to work. I think this was still in the era in which people were putting comments and auto-filling Python code blocks, which now we think of as ancient history, but this was like two years ago. So how does code as policies work, and how did that inspire some of what you guys are now seeing as possible? I'd probably be mind back to Voyager, where, like, Voyager was one of the first, what does a coding agent mean?
Coding agent requires good tool use and on-the-fly tool creation. That's called code. Okay, you have Python built-ins. Let's just say I have only Python built-ins.
Those are tools, and I have sort, and I have if, and I have for, and I have while, and I have all these things, and I need to use those tools to assemble a new tool called a new Francois.py, right? And now that's a new tool, and I get to use that. That was Voyager. And yeah, I don't like how to say Voyager was the first one to do this.
Voyager was the most popular one to do this for Minecraft. And then they created tools that they could invoke on the fly to help them play the game better, and compressed thinking and experience into a new tool that I can later invoke. And then I think because everyone in 2024 had this insight, it was like, okay, if we just pour, you know, Darian Sam both all realized if I pour all of my heart and soul and resources and compute and intelligence into getting better at coding, we will have liftoff. Then I can automate the ML engineer, then I can have liftoff, and I'll have all the things, also solve all the things.
And so then that's what everyone did, and I don't think that they had the insight. I don't think they were so insightful to know that, oh, if I did this, then we will have LMs that we good enough to write policies for robotics, and I can displace all of robotics with actual coding agents. And I can displace artists with like, you know, coding agents that are writing JavaScript to make beautiful like art. Like, I'm not sure that they had that insight, in the honest.
Can we talk a little bit about what some of these early code as policy methods were even able to do? So even, you know, this is well ahead of us having coding agents that are widely available, and people are too totally understanding how they work. It's well ahead of people using our own methods to actually train those coding agents to be really good at tool calling. What did some of those early ones, maybe even before Voyager, like, cospossies, which is the paper came out in 2022 at the end, right around what's actually taking out.
What was the capability of those compared to what you can see now? Yeah, those code as policy papers, actually, once we go deep on team, we're incredible, because what they did is they created these kind of functions, like pick up an object, lift up, or like move to a certain post, like they provided this list of functions to a coding agent, in a form of like, they require functions, and then decoding agent will be able to write code involving these functions to then control the robot to do very complex hacks. And what was very surprising, what was very surprising was that coding agents can do this very one shot. They did not need additional robot data to in order to work with this code, because they're already trained on so much coding data.
They don't have a sense of like, how to, what to do first, what to do second in order to like, move a block into a bowl, for example. And I think it's this one shot ability, this in-context exploration ability, that really motivated a lot of data work to continue exploring, including us, to continue exploring how we can apply LLMs to robotics. I know, Francois, you have this framework we've talked about of, you know, the various ways that learning can happen. There's going to be learning that's embedded into the weights versus being kind of starting and so on.
Do you want to quickly talk about that? And maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going. I mean, I've been working on this experiment, and actually, I haven't really crystallized it into a paper yet, but basically it's like, what is the most efficient way from an intelligence sample perspective to input a learning, let's say, in this context, a state action result or state action reward tuple back into the policy. There's ICL, there is- The context learning.
We're like, I just append it. This is how most people are using LLMs. They're just like, oh, no, don't do it like this. Do it like this.
And then it's still in the context. And then I can remember it. I can iterate. And that really only works if you've trained the LLM or at least post-trained LLM to be able to learn and improve, and that self-refines a reflection literature from way back that actually allowed multi-turn into the train set.
If you don't have that, it doesn't learn. It doesn't, it doesn't, any kind of, learn. But even then, there's a limit to how good ICL can go. So I've done this experiment where you take an LLM that we've trained on and I've held out a task, let's just say, just to make a for simplicity.
And then ICL it and I want to measure on the Val set how much it improves on a per sample basis. And it improves greatly, very cheaply. It doesn't cost much. I don't have to, there's not SGD.
So I don't have any flops. And so very quickly, I can adapt and improve on my Val set example example example. One, it's non-monotonic improvement, which is wild. So it gets worse, gets better, gets worse, gets better, like pretty graphically.
Number two is it caps out very quickly. And so after like 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context is not improved. You're constrained by the models ability to intelligently use all of its context. Exactly.
From the post training, how many multi turns did it actually get to be able to improve and do self reflection? And then the most important one for sure, for sure, we can't do is after it hits the context window, context length of the model that it was trained on. If it was trained on 100,000, really that means you have a context window of like 50,000. Once you exceed 50,000, you don't improve anymore.
You actually just get worse because the model can't attend over everything. There's a really example you said, which I didn't really think about is like, well, if I can rag or I can like, I can load in my app into active memory, and this is very prime agent, continual harness, kind of like thinking, right, and pull stuff in on the fly example, similar examples, then it's like me, like, I have an exam or I have an exam with a textbook. I'm going to do better if I have the textbook to look up as reference. And so that is, you know, definitely going to work.
And it's still, and it also scales what I would imagine much better. And then the last two paradigms would be Laura with rank one or two or whatever, Laura with rank 10 or rank 100, and then all the way to full SFT RL. And if you're Tesla, and you have all the data, infinite data, and you're learning, doing such a car by ICL, what are you doing? Like you can, it's like, clearly we wouldn't do that, right?
And if you're figure something like that, but it's amazing how good in the in low data regimes, you can do with ICL, and then even cooler, we can compress the ICL into tool use, into tools, or like, compression into learnings. And that's, that's the hierarchy that we haven't really figured out. You guys are probably on the forefront of, you guys want to elaborate a little bit, because I mentioned that's a simple part of how you guys think about building a model. There's a few things that are quite interesting here, I think to unpack.
Yeah, the first thing is, we've been sort of thinking about this idea of a harness, almost as a form of, like, domain specificity. Like, when you, for example, deploy a robot in a new environment, maybe it's in a wet lab, and it just needs to do a lot of tests you picking up, for example, you can learn a lot of this in context. But then one way of consolidating this context for future agents is sort of packaging each of these skills that you learn into specific programs, for example. Like, writing out these skills, and writing out these memories is a form of consolidating.
It's like a form of distillation for past experience for your future agents to use. So this is quite interesting. The other thing that's quite interesting is, it almost seems like there's some relationship to the broader meta-learning literature throughout in machine learning history. Like, you have this sort of bigger model, and then programs a smaller model to do things.
And one of the very interesting things is that, yes, these smaller models are going to learn in context, they're sort of wrapped inside the harness and they're doing the task. You can make them, maybe smaller, you can make them run faster, as long as the bigger model can sort of do this domain specialization of your harness quite well. So, this is something that we're pretty interested in right now. And last point as well, I think the really interesting thread question is how far in context we want to get you.
And I think it's really anyone's guess as to that. Like, there's papers that show that these models can approximate gradient descent during in-context learning as well. So, it's this line between weight space and symbolic space. I think it's a little bit blurry.
YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com slash apply. It's never too early and filling out the app will level up your idea.
Okay, back to the video. So what exactly are we watching here? We're watching Ashra controlling the arms to pick a block of the table and put it into the ball. So here, Ashra is using just these camera inputs and presumably it's aware of the robot that it's going to go like that.
It's going to write code that controls the robot as individual joints. Yeah, so it does look at all the cameras. It has all the camera feeds. Instead of code, it's more like a tool call.
So it sends commands to the robot to control and put the block into the ball. So in this situation, we're using Ashra directly. But if we were using say, wattle harness, what's the kind of difference between directly using Ashra control versus doing this through wattle API? Sometimes directly, like having after that we command the post the robot should go to is not directly the optimal tool to use.
If it's repetitive task, you don't want to ask for it in the loop. If you want to write code, it can run repeatedly very, very fast. Or if it's a task you've done a similar task before, you should be able to call it skill that has compiled and use that skill to do the task faster and it end up edge cases better. So here we can see it seems to be approaching in.
Yeah, you can look at you can notice that the latency is a bit slow because we're bottlenecked by the latency of Ashra. But if you look at the trends, the latency of these models are improving very rapidly. So one thing that we saw is that for Babel class LMs, the latency is improving by around 2x per month, which is very, very fast. If the trends continue, we could get real time control by end of the year.
So now that we saw this task go from end to end, why don't we walk through what that actually entails? So there's a coding agent in the loop here running this. What are the steps that the coding agent would have taken? Let's contrast the direct communication version versus the wattle harness in the loop.
Yes. So what just happened was that in a series of turns, Ashra received the images from the cameras and it output the and effect of pose what a robot to go to. And it does it over multiple turns to complete the task. This is less of core responses, but more of a full calls.
The onscreen part is pretty repetitive. So you can actually automate a lot of that using code as well as actually approaching the model, picking it up. A lot of these things are pretty deterministic when you're doing the task quite few times. The interesting part is where you build in variation into the course policy graph.
And so we have a few points of variation. This can come from, for example, when you detect an object, maybe use a VLM as far as a tool call. Or when something fails, for example, how do you check how do you potentially do something else to pay on the failure? There's more flexible responses.
We tend to build a VLM inside of the loop to ensure that while the coding graph is deterministic, there's points of variation that allows generalize. The biggest insight I think I've had a change in world view on how to perform machine learning and how to get the rest of the distance on AGI was like a lot of conversations I've had with Fronts-O-Cholay. And in 2020, maybe even 2018 when he did on the measure of intelligence, he talked a lot about transduction versus just wrong versus program induction. And transduction just means I'm learning a function theta that maps from exits to wise.
And why is that wrong? It's just slow and it's information inefficient. And so to go from exit to wise, I need a lot of pairs of x to y's. And if I have a very small amount, then you need a lot of inductive bias.
And what you need to act is a good generator to map from exit to y. So now my theta takes in three and end pairs of x's and y's and he mits the function that maps from exit to y's. And that's what code is. And so that's like, if you give me a coding interview, a whiteboard, you say, okay, here's a coding problem, here's some examples, okay, go right the function.
And we are emitting an F that will map from exit to y. And this is quite interesting, right? Because I think, I mean, this was a lot of the traditional program synthesis literature. And I feel like a lot of reasons why those methods didn't work as well was because that inductive bias, as you said, right, it's your shitting difficulty of the problem from finding that mapping to find the right inductive biases to map it to this like smaller code to then do the mapping.
But finding that sort of inductive biases is super hard. But maybe that's what Asra is buying us. Like even in Cox, there's a giant shift in the literature from, you know, specific, you're symbolic methods to just using Asra to write code to map. And so maybe that's a cool way to talk about that is not some other way.
We have runs and then they're emitting symbols. Yeah, I think it's actually a pretty good segue to final topic, which is around inspired by this paper also from the titles, the Platonic Representation Hypothesis, right, where Platonic because is a reference to Plato's cave. And the idea that, you know, there are shadows of various realities. And I think here the point he's making is that there's extensive evidence to show that language models and various representations of different data is actually learn these distance mappings between similar things under different train policies that are overlapping.
And so like the idea there would be that as you train these systems on increasingly large amounts of data, they converge to a sort of consistent mapping of the world. And maybe this implies that we expect language models to get better and better at things over time that would make them useful for new tasks like robot control. But maybe I imagine this representation hypothesis is actually pretty central to all of your guys's worldview or your view of your companies. And so maybe one of you guys tell me a little about how you think about this.
And also maybe we can use that to make some guesses as to why we think models like Asra seem to be so much better at robot control than previous models and where it makes some predictions for where we might be going here. Yeah, I think one way to make this concrete for the robotics models with language models debate is that the very, very strong language models will have very similar representations of the world with very strong robotics models. And if that's the case, then if you have a really strong language model, you also have a really strong robotics model using if we believe in a platonic representation hypothesis. And if that's true, that's then that is the best manifestation of the data lesson, right?
Because you just need one really strong model regardless of architecture. And they would be outperforming any specific models that is slightly weaker. The interesting thing that that's maybe not entirely intuitive and maybe be curious to hear you guys all think about is why specifically it seems like these newest models specifically Asra seems to be so much better at spatial intelligence. We've all seen the demos online of controlling blender and making great 3D images.
Now it seems like you show J and Benchmark that is a meaningful step function improvement on certain tasks. Where do you guys think the pre-training and post-training of those models, what has likely changed about opening eyes approach there that made them so much better compared to models even six months earlier? That still were benefiting from their lesson. We're still extremely good at code and we're still solving math problems but having quite cracked this measure on the calendar here.
I think what Asra does incredibly well is it's like vision capabilities. It was probably pre-trained on way more computer use data than every before. It started pre-trained on so much like CAD data. And it's like all of these kinds of data probably teach the model a similar understanding of physical world as a lot of robot data might.
So the computer use data is an interesting point because it's not totally intuitive I think why computer use data is useful for understanding the physical world. It tends to be a computer where you're clicking around. The same more about why you think that is a big one because I totally buy the trained on way more computer use data than ever before. I think immediately if you were just training on a model for computer use that would not be a way for robotics.
But if you feed a computer use data into a big model like Asra and by computer use data I mean you drag a cursor around on a screen to orbit some CAD object and we're designing blender. This tells you how to reason about this. It tells you about top down left right all these concepts that you need to control the robot. I think that's why this data helps so much for making these LM so much better at robot use.
I think from the other direction a lot of people in the robot community we're trying more and more different kinds of data as well. Eosentric videos kind of went from just tell the operation to a broader range of this data because it's not just robot data that can teach a model how to use a robot. So if you take this to an extreme it's like why not feed every kind of data coding computer use Eosentric into the same model. I think that's how we get to the most capable robot use agent.
I think that's so funny that if you go to 1980s or X-Park or whatever we literally made the graphical user interface to be more like the physical world so that we could interface with it and like we ended up doing it building an environment that was actually helpful for robotics to learn how to use the physical world. We have the IEL systems, we have like Biles, we have Bolders, we have the gooies for like SOLIDWORKS and Autodesk to spin around stuff to make it similar to the physical world. And then we couldn't get robots to work in the physical world so we just trained on that and now it works. Exactly.
There were incredible papers I think from Princeton and also called Play's Robotics touches on this is like if you design a right harness where you make the tools that interfaces the robot look like computer use tools like you have a drag cursor around to control what a robot goes. That improves how well the LLM is able to perform on these physical tasks. Where do you guys see given like reasonable guesses as to where the base models are going to continue improving and your guys own investments in either evaluating these models or building harnesses around them? What do you think is going to be capabilities that we maybe now see as challenging to do but that are going to be increasingly possible or even trivial very few months from now?
Like six months ago even the demonstrations that I've seen you guys post on your Twitter I think would have been kind of mind willing to imagine coming from the lab. There's some consensus within the frontier labs and also in the robotics foundation models companies that we will have general purpose robots within the next two years or even earlier and this is something that society is probably unaware of or even unprepared for and when we say general purpose robots we mean something like if you give any natural language instruction it can do what a competent teenager could do with their hands. It's kind of like the chat gpt moment but for robotics in terms of capabilities where you can generalize to unseen tasks and unseen environments. For you guys what does that mean for what you guys are building?
I think we're going to take steps to get there in about two years time. I think there's so many challenges that are very visible. For example, when I left the online talk about latency this is a big problem. If you just have Astra being in the loop thinking at every step this is really slow.
It's not going to be economically quite useful. So how do you consolidate kind of first pass by Astra that's this contextual learning into like a faster scale or policy that you can then run repeatedly at incredibly high throughput? I think the mappings to humans will get more and more like this where like the optimal thing to do maybe to put a new SAR state action reward back into context is very quick but then there needs to be some like of go to sleep for a while like almost everything that is intelligent sleeps like tell me an intelligent system that doesn't sleep right in some way and then during sleep compression happens. The weird thing that happens from your hippocampus and short-wave ripples to both lobes and like there's weird you know pass from memories that were compressed throughout the day to train the weights and similarly maybe what the right thing to do is similar to Dagger data segregation framework in our classic RL where you're going you're collecting a bunch of data and then you're somewhat reflecting on it and then you're using it to update your wait file and then you have this elation in from those experiences back into an updated wait file that maybe isn't as sure that maybe is your own models.
Sounds like DreamCoder. Yes, but a lot of those specific tools I think that people use to build to take DreamCoder for example right it had this library of skills and during the sleep phase it basically refracted everything into what compact representation and so on I mean we're seeing some of things just happen not as rigidly as before in the space of programs but now it's maybe like maybe you want to refactor traces maybe you want to refactor skills a lot of these robotic codos policy papers right they have this growing library of skills and as anyone's gets really how you print them how you organize them and things like that so yeah I think as as one of those forward like how you manage our growing context of skills of the planet data all that is going to be quite interesting. I think with that I think this was an excellent discussion thank you so much Vincent coming and Jay from being here I'm very excited for all the incredible robotics advancements I think we're going to see over the next few years and I think you're totally right Jay that I don't think broader society is totally aware of how much is coming and it's going to be a very incredible few years to come so thank you so much.