When I think about model progress, it comes down to different forms of capital, and that is compute data in talent. How do you see that playing out in terms of like who's gonna win in the next couple of years? The Chinese are so good at building models. To say that China is anything but equal in talent, you're gonna have a hard time. What they don't have is compute.
Most people, regardless of relation, ask for like the main lever for them to do this. People distilled my products every single day. They take it, they strip it down to its raw parts, and that's called competition. Previous post-training was much more reliant on the supervised fine tuning. Now this is more debate into environments where it's all real heavy.
But it's just like hard to map this type of post-training, which is you get the completions, and the score goes up to what is going on in reasoning today. The discourse now is just so much fear on like how much the frontier is just hidden and also breaking away. The AI capability plot doesn't necessarily worry me, but the AI capability plot in the pretty much non-existing preparation across public cybersecurity and all these things. The Frontier AI labs are spending extraordinary amounts of money to build better models. So how does open-source keep catching up?
In this episode, A16Z partner, Lish Li, sits down with AI researcher Nathan Lambert, founder of the new nonprofit research lab, Trillium, to unpack the race between open- and closed AI. They discuss why open AI and anthropic still have major advantages in compute, capital, and speed. But why Nathan believes the technical talent behind open models, particularly in China, is often underestimated. They also separate the reality of model of the solution from the rhetoric around it, and look at how post-training, data, and our health environments are becoming increasingly important sources of progress. And finally, they turn to recursive self-improvement. Nathan explains why Trillium plans to document how AI is actually being used to improve AI,
from the tasks models can already automate to the places they still get stuck, and why making that progress visible could lead to a much more grounded debate about how quickly the frontier is really moving. So Nathan, really happy to have you here. Thank you for joining me, person. Super timely, given your launching, Trillium, which I'd love to talk about, but more on a macro level, which I had a lot of open source, and that's a lot of what you care about. And what we've been hearing a lot in the public discourse is like, hey, what to pace the frontier? The frontier does the frontier that.
And I think an arc of the stuff that I'm really trying to ask experts about is what isn't the frontier? What are we pacing? What's there? And then more importantly, kind of threading with open source and open weight models, like, can they keep up with the frontier? And I think this matters very much, and it's something that you're next to an on.
So maybe to sort of start on that arc, how you would break down what's gotten us to the frontier for both closed labs and open source models? How does it break down in your mind? Maybe first at a high level of pre-training, mid-training, post-training, how post-training is evolved, which is something you've done a great amount of education to the public on. When I think about model progress, it comes down to different forms of capital that you put into them. And that is compute data and talent. And all of the labs have various amounts of this.
And the closed frontier labs of opening eyes and propping schools have clearly had the most of these. And what we see is the different abilities of the labs to organize these forms of capital. And making the org-tart work for building models is really hard, which is why you have companies have to do complete resets from the bottom up. And then it actually works sometimes. The company is totally reset, and they're on a positive trajectory again.
I'm very interested in these human aspects of why competition drives the frontier and why the org-tart is so important, because we see these things. And this comes into play in the open and closed dynamic as well, where it's the closed companies, the true frontier labs, which if somebody wants to work in the frontier of AI, most of them to work in the open and open AI, which is the easiest bet. I think meta-Google is basically AI. I think you still might be there, but it's harder to see from the outside. And these companies have just taken the most ambitious bets that really is in order to get there.
And that's why OpenAI has done things like Kickstarter, the reasoning model revolution. And I think that kind of breakthrough, that beautiful plot with the inference scaling labs is peak opening. It's like they're really good at getting this plot that just changes how everyone thinks about everything. And another side is like in PROPIC has been very focused on the types of things the models are going to be able to do. And then their reward was this cloud code explosion with their focus on coding for so long.
And there's these suddenly different cultures. And I think those two companies in particular have been the most forward-looking and the most willing to spend egregious amounts to make a step towards the direction that people think is impossible. The Chinese companies came into this because they want to have a company that matters. And if they heard to build their models and try to sell an API from China, nobody would know about them. Nobody would know about Moonshot AI. There are people with no bunch of people who know a bunch of people who know a bunch of people who know a bunch of people.
And there's a very long list after this. They're very practical decision by them. They're following the discourse. They know there's going to be a huge market. They know there's a huge demand in the US for paying for these things.
And it's very practical. So they're really releasing open models as a way to shortcut this and get into the conversation. And I think there's a lot of complexity in why do they do this as a government involved? And they're just trying to build a startup. It's not to say that they will not get there.
But the Chinese government does things that would blow people's minds in the US. And I think that makes me think that the Chinese lab is having a much more serious approach to security. Things happen in China that are much more drastic responses to companies doing things in the US. I don't need to go down the rabbit hole. But between the balance of open and closed, these companies are trying to catch up.
This is a very different dynamic than what I said about OpenAI and Tropic. They're trying to create the new paradigms that are trying to make true RSI happen so that their company wins with best models forever if you get there first. They just boom ideas. And you just have to fill the market where there is void. And the open weight models are trying to do this.
And I think over time, there will be these tiers of open models where there's the best closed models which you can demand the best margins for. And then there's the near front tier models, which is a step behind this. And those will be open because there's just some sort of supply and demand there. If you get access to a different market, that might be less profitable, for example, like if you're private enterprise data or so on. And these tiers are kind of natural evolutions of companies that are trying to do things.
And I think we're talking about the frontier. And if you go back to capital, they have less capital, they're going to make less money. And that's okay. But I think this drive to catch up is very strong. And once you know, once a drop in opening, I show you the type of model you need to build, it's a lot easier to then be like, oh, we just need to do this thing.
And they buy similar data sets at Circle Studio Data Economy where OpenAI and Tropic spend way more to build out this function of a data industry. And then those data companies a year later might sell them to everybody. They might sell them to everybody. They might need to maximize their return on their investment. And they had some contractual period.
And I think there's a lot of dynamics where fast-following is going to be very doable. And we've seen the cycle proceed for a while longer than many people would expect. I think when tragedy was released, we talked about consolidation industry and predicting it. And I think we're just starting to see a much more active mergers and acquisitions market. But I thought this could happen two years ago.
And the open model companies have lasted much longer than I thought. You've gotten so many interesting threads to pull on here. I mean, starting with actually your commentary on the organizational differences between the top two frontier labs. We can maybe table that or go on through that thread. But then of course also kind of explaining sort of how the open labs in China have really just chosen that as much more of a distribution slash practical commercialization strategy rather than, hey, obviously out of altruism.
Though it's great that the researchers maybe for better for us were motivated from sharing the beginning. But at the end of the day, it perpetuated because it was an excellent commercialization strategy. So the other thread maybe to pull there is sort of, I think most people regard to installation as like the main lever for them to do this. So well and kind of increasingly well because the lag between the open source models and the frontier models have only shortened with time in the last few years. And it's like maybe only two to three months.
And this has also pushed the frontier labs to maybe want to close off the APIs, close off their best models to do first party services. One probably to capture more value because it's always possible that way. But then two also because it's harder to distill not possible to much harder to distill. Even those two threads because they're both very interesting. So we'll try to touch on it.
Let's start with is the distillation story really more of a caricature in the Chinese labs. Since you've actually met with them and you've studied this a lot more. Is it sort of like they're all rounded like great at pre-training also have their own good post training recipes that they may or may not have revealed. And where do you see the levers kind of accelerating? Yeah. So the Chinese labs are very technically excellent.
They are more compute start. Yes. Without a doubt they have a comparable talent pool to draw on. I think there might be like a slight edge based on who was where when AI really got going. But to say that China is anything but equal on talent you're going to have a hard time because they just have so many people being trained at such a high level.
And what they don't have is compute. You talk to people there and you're like, oh I heard an average open AI researcher has 5,000 H100 equivalents that they can access at any time. And like their jaw just falls in the floor. They're just like I can't comprehend. And this is like a top pre-training researcher at one of the labs that you would know.
And then you have this interaction a few times. And the compute thing I think is the biggest difference in distillation is a way to effectively spend more money to get what is compute equivalent. Like outputs from a language model are a function of compute. And if you know the outputs you want, it's the same as having compute to generate with a model. And I've seen the stages of distillation and how I see it is that previous post training was much more reliant on the supervised fine tuning and other to like early Jeff era, which is where post training started.
And you can get most of your performance just by you do this next to a prediction loss on outputs of a very strong model. It was very established that you take the best model you can. You generate collisions and you train on it and your models are going to be pretty good. So long as you access to the right prompts. Proms have always been a key to post training.
I remember at AI too, like with Hannah who I co-led post training there with. We would always say like prompts are the key unit of post training. So you need to get the prompts and you can do different things with them and different stages of training. And now this is morphed a bit into environments where it's all heavy. But it's just like hard to map this type of post training, which is you get the completions and the score goes up to what is going on in reasoning today.
And based on the public literature, it's very hard for me to be like distillation is a major factor because for example, in the fully open literature of people trying to do open post training recipes, all the open model builders are stuck using like a really outdated Quen model because you can't figure out how to get the teacher right. My hypothesis is that there's some like mid training and something in the common pre training data that's open that makes it hard to do this thing, which is get reasonable. Which is get reasoning traces from the absolute best model and do SFT and the numbers go up. So like if I were to have access to Claude's reasoning trace when I was building all the I'm actually like Claude Opus, whatever, I'm not sure that it would have actually made our model better. So this is like my evidence grounded in the literature.
Well, I think distillation is hard. And then like how do you take the outputs model and build in RL environment to do a big RL run? Because then you have the Xiaomi's of the world doing this week long open RL run and like here's what we did. And then they've dropped the model and it's like where exactly does distillation go into this? They're clever.
They know how to do marketing well. There's some questions like did they do this ahead of time because they did their RL run and then they dropped the model and then their tech reports that they had an extra post training stage after RL. So some questions. But like mapping the established distillation literature to that is very hard. But there are question marks that make me support the labs a bit, which is like these Claude proxies and these model proxies in China where there's a ton of tokens that are used in their intermediaries and they have tons of accounts and even cases where Kimmy was serving Claude in their products to users.
And I just like don't really understand how this fits into distillation as a common technical tool in post training. And like yes, we should watch it to like there's other weird stuff with Chinese companies using outputs from opening iron and drop its models at a vast scale. So I think we have a lot more questions we need to answer. And I think most reasonable people and I was the gap from distillation are like very big air bars. But when I say that distillation helps them by a few months, it's based on the scientific literature that we just like best information we have.
We should do more open research and and able this so that we can study it. But like I don't know how you can come to a different conclusion that it's like not a huge affecting current post training based on this. And that's just like the Chinese are so good at building models. And I think that the middle coke where people are like, only we are good at building models. And it's like there are so many companies. There's like 20 to 30 companies that are building good models.
And that's like you just, that's hard for me to get over. And I think that there's this very prominent paper that's like stealing reasoning traces from coz models. And I know that the Chinese companies are way ahead on doing this since the AI safety community later published it. Like I think every Chinese lab has a team that knows those sorts of things. And this goes back to what you're saying about like they don't need to offer the API.
And I think there's some characteristic of reasoning models that generate these thinking tokens where it's easy to just knock them out of the API. So you see the reasoning token and then you can use that to train your model. And so long as they offer these APIs, there's going to be no way to stop that type of distillation. But if they do make it so it's only in products and things where you have to sign enterprise agreements or whatever, it'll be way harder. And I support the labs doing that. I'm like, I'm just a curate.
But it's like they have to secure their products if they want to. Like I don't think knowing your customer would be good for anybody. I don't think it would be good for their business. But also I don't know why you should get the government involved in terms of service violation. It's like not even, like I don't even think terms of service are assigned contract.
It's just like a, I don't think you can sue them in the same way you just can cancel their access. It does not mean that I'm not that sympathetic. But I am like, I like to listen to the national security community is loud. But it's like there is a directional truth to that with where AI is going and how influential it is. You don't have to balance many, many things in your head.
I wonder what your response is. So I heard Jensen said in some literally either Twitter or YouTube. But it didn't put it in the show. People distilled my products every single day. They take it and they strip it down to its raw parts.
They test everything. That's called competition. Do you think that's fair game? You think distillation is not robbery. That's called competition.
I'm very simple. Because I think there's a lot of calls from the AI safety community that we want to have like US trying to dialogue as we're getting to such capability levels. And if you're going to ask the company to just like start hitting their AI industry with slaps on the wrist and more downstream of distillation. You won't regulate distillation with policy from Congress. You will probably just do federal action to make their life annoying.
And at the same time when we're talking about how do we manage frontier risks or you have close models and training going all over the web and accessing things that they shouldn't. And we have valid worries about what will happen with open weight models where this like hasn't been harmed yet. But it's like, why aren't people using open weight models? Why do they go to close models and get around the safeguards? And like, we want to be able to communicate with the Chinese tech companies and everything about this.
And I think like you're going to cause more harm than good if you do anything about distillation because they're not going to stop. It's like they're going to keep doing it. Right. And then the open source community benefits and therefore the startup community benefits a lot. And it's not just like, hey China, USA, actually like there's, you know, and we can actually talk about like, I'm sympathetic obviously or try to steal man the frontier lab position, which is like you have just unprecedented amounts of capital that's required to build these frontier models.
And it's kind of alarming slash not very conducive to this flywheel continuing if the through distillation or other kind of distillation related methods open source and catch up so quickly. And so, you know, and what I mean by this flywheel is just kind of a very simple picture of, you know, you train the frontier model by putting in a lot of money to get compute and then they park this computer course opening in and frothick or very guilty slashing is successful. This whichever way you want to like look at and there is a huge compute, you know, kind of sort of lack of compute available because it's actually like not just compute and chips but there's also power, etc. So, you know, the scale of this scarcity is kind of, you know, bottle neck until 2028. So this is all to say that it's a successful strategy to park that which leads to actually a lot of revenue because once they get to the frontier because they understand, you know, scaling is the one thing that kind of gives them or the science of scaling is what gives them a handle over how well that can translate into a predictable revenue stream.
So they get the revenue and because of that revenue and they're able to raise more money is just unprecedented like beautiful flywheel and this happens for both open and profit. Of course, we need exponential resources and so we've gone to the point where like, okay, we need IPO, we need to go to the public markets. And so anything that threatens this flywheel from functioning well is a huge threat to just the entire like being at the frontier. And so, and so, and that's exactly that actually. So like, you know, maybe not just kind of being sympathetic to their version of like trying to, you know, eke out capitalism themselves.
It's also just like, hey, if they don't do well in the IPO, that's actually bad for everyone else. And so I want everyone, you know, I want them to do well. On the other hand, you know, it's like the Chinese open source models are a huge benefit to the startup ecosystem. And we see this all the time, just kind of metrics I remember from like deckagon and Harvey was trying to do a lot of fine tuning. There's like a huge amount of startups doing post training as a service of various degrees and trying to productize that and it actually works because you can get to the pre to frontier with these, you know, kimmy K3 or pick your favorite like gel M open source model.
So I guess like weighing these two things, how do you. I guess there's so many questions involved from that, but I guess one. How do you see that playing out in terms of like who's going to who's going to win in the next couple of years versus, you know, what what does and also relate that to, you know, why you're doing obviously an open source. So I think open and close models are in different timescales of exponentials and the closed labs are so integrated where they have efficient inference. They can serve the model at scale when they're ready to because they have all these resources they can plan and they get to these markets first that are very high value.
We have seen this coding and this is when the story of the year where the revenue has just got zero 100 billion likely to grow, but the open model labs partially this is like an economic issue being the fast. All right, is that you don't have this flow of information and you don't have this flow of revenue. So the open model, you have an open model builder, then you have the open model inference companies and you have the open model post training companies and like there's information from those customers. Which are like things you need the models to do and it's like you can't like the information doesn't just flow back to the model builder very easily. And they also don't have the economy to scale to serve as many users to serve as efficiently to have as much margin on inference.
And these are just like, it's going to take a lot longer. But to like send the things can be commoditized. It seems like companies really want to have private data. There's a lot of things that they just cannot send over an API and we can talk about the like zero data retention or not and so on. But like these it takes a long time to figure out if company has internal expertise or they figure out how to build an internal evaluation that actually matches to revenue.
And then they figure out how to fine tune or then they go find they go find thinking machines or prime intellect. And like we can make money in mass if we figure out how to fine tune a model that solves this. And they have these customers but they're just so many more companies involved that it takes so much longer. But at the end of this process, they have real ownership which is model weights, no terms and conditions. So they can run on their hardware and they figure out they can make an ROI on.
So this is right isn't the open model economy. It's just it takes way longer to build and I'm less concerned about this being kind of bubbly where it's like this could eat into the market of the really fast take off as people figure out. What is the cheapest model that accomplishes a task and how do you compete on margins. And I don't really know. I don't really know what's going to happen. I think it's too early to know.
It's so hard. I mean, it's definitely something that's always top of mind thinking about like there's things where scaling up general reasoning just as a general model is so it's going to beat out any specialized model. And we see obviously examples in math and that's probably the most pure domain that you can see this. But also, you know, even in say robotics, you know, with Astra and I for maybe however degrees of like a true see the signal is it doesn't sound like they've actually tried to send in robotics. I think it was a cycle of like six to eight months ago. They started hiring people and buying a lot of data and robotics.
They did. And then the model, magically six months later is really good at it. But I think the other other other, you know, more contrary things I've heard is that it's actually coming out of computer use because it's, you know, because computer use is like some level of spatial reasoning and like getting to have good blender. You know, is another extension of that. So, you know, we can believe some version of that. But either way, I guess it's like maybe if it's like really specialized, you see it's the capability is really from like trying to get better in a certain area.
But like in math, I don't even know if that fell out of trying to specialize in math. I mean, I think the whole thesis there was just like, let's just scale up reasoning and actually got better in natural language about better than any specialized model. And I mean, probably was even trying to be good at math, you know, opening eye, you know, certainly, I think, you know, they hired a bunch of mathematicians, but they also were trying to like use specialized, you know, kind of like lean data set to get better at it. Or at least that's not. So anyway, this is all kind of fodder for the view that like it's kind of hard to compete with the frontier models because eventually they start ascending so much that you know you get good enough in respective categories.
With that said, I think actually there's also very competing, like a counter force of data that you're seeing, which is like the open source models that are ready, good enough, allow for fine tunes, you know, as your point of like these slow percolation diffusion of these very specific benchmarks and data sets are only available in these companies that are being applied to. And it's just, it's just good enough. And it's actually we're starting to see people adopt that. I mean, in the most maybe pure regime where you expect frontier models are good as I think cursor published something over the summer where like, hey, we just, you know, we had this long horizon task of writing, rewriting SQL light, and turns out using two frontier models, you know, one for planner, one for execution. Execution is not as good or is equivalently as good, but far more costly than just using composer with maybe opus or, you know, faster, whatever as planner.
And I think that's actually great evidence that the open source fine tunings are very good or good enough and has surpassed intelligence hungry tasks. There's a long way of asking what evidence do you find is compelling or pieces of evidence for various versions of this stable state of where the frontier wins versus open source. I'm a big proponent of a lot of the frontier progress and domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act and he'll climb on. And I think there's two, there's two things. We have the math and code thing. I mean, we don't know how big the coding market will be.
And then there's kind of everything else, which is still their targeting this knowledge work, this biology, healthcare, legal, and all of these things are not as verifiable. And I think it is much closer to the story of opening. I bought a lot of robotics data. They're probably going to tell us it's generalization magic poof. People love poof generalization.
But realistically, it's probably some of a some and B, but a lot of it comes down to data. When I talk to people in frontier labs, there's some people that have amazing jobs. There's some people that contribute by doing a very narrow thing, which is you make a dataset and you figure out what the behavior should be for your model in that area and you figure out how to make the data that helps get that data. You get that behavior and you figure out what data you need to buy. And that's going to take, I think we're going to have years of this across different sectors.
And that's very capital intensive to buy all of this data. But that's not just like poof magic. RSI happens where I do see in this math and code in some like distributed ML engineering. It's very verifiable. You know what your cost is and models are going to drive that down at a really magical rate.
And that's going to just have a lot of surplus benefits and we can do research faster. You can look. I'm very interested in like, I think efficiency models is going to drop multiplicatively over years. So we've seen opening I have drastic price cuts. And it's like, what does it mean for GPT soul or astra and these bigger models to reduce in price to X every eight months for the next three years?
You start getting, you start doing this math and it's like, it's 1% of the cost in one of really four years. And it's like, are they going to have higher margins due to Jeven's paradox there or their revenue is going to decrease because the market fills up. There's just like all these really weird dynamics but fundamentally AI progress is about data and about writing better software to serve your models. And those are very different worlds. Yeah, I mean, at least definitely I agree with you that observing from the outside, there's a lot of data, our environment companies that are doing very, very well selling both to the top two frontier labs but increasing and longer tail of labs as well.
So to your point, I think a lot of the capability gain has really come from just scaling that up. And it's not an easy task. Maybe there's actually a curious on your opinions between the data collection and environments that are built by hand versus the ones that are done in some, there seems to be different versions of what can be, you know, what kind of price they command. They all have good things and bad things. And I think there's a lot of talkably. We've talked about distillation, but another side is like the Chinese company buys our own environment.
They use six months later. And I thought this now there's a premium. The good data will do this. But the data that people are buying is pretty flawed. I would say that method is you buy a mass and you figure out what works and is not 100% hit rate.
And I think I personally think that this our environment explosion where there's just so much opportunity to make money is having a lot of people like cut corners and this can induce things like reward hacking. So it's like the models are so smart that if you put them in a world and they're like, my job is to solve this task, but they didn't give me the, like you have a test file and the test file actually has an empty file instead of the instead of the maze that you have to navigate. And they're like, the model is like, well, I should still figure out how to get a high reward anyways. And I think a lot of this behavior is downstream of this supply chain. And I think it all fix it, but it's interesting to see the demand from the labs tell him the neighbor working and all these researchers spending so much time on it.
And then all the data companies like by our data, I think proportionally spend less time on model behavior and more time on world building and revenue. And somehow these two parties need to meet together to actually make higher quality data. Yeah. Well, a lot of the labs are actually taking it in house partially because they don't want data leaking. Not that literally and figuring out who we speak is like, you know, signals leaking of what they're doing, but they're trying to send on.
But despite that, I think there's a tremendous amount of growth in how much they're purchasing. Do you think that's probably one of the strongest levers? Can you kind of qualify how much that will kind of increase? Do you expect that to increase based on spending? Or are there other, you know, because for instance, if it can be done well in STEM, I think maybe that isn't more of an argument to that being absorbed by labs in particular.
And then all the companies are doing a mix of like all the companies are selling synthetic data where they're using language models to modify or build the world or generate permutations of it. And like if a human is guiding this process, it's still synthetic data. So like there's very few people that are going to like sit there and lender and we're going to make a little world for our model to act in. Like all the companies are moving far faster than that. And they buy them because the models are getting better from them.
I still have denial of that. I'm like, these numbers are crazy. The billions of dollars of research on environments are like, they're only doing this because it is helping the models and they see the signal. And I think like my general observation is that the data industry has a lot of low hanging on improving quality and improving just like objects. So like if you look into the details, you can find weird things with like data companies that use contractors overseas.
Those people are through an intermediary. They don't know they're selling it. They just uploaded to GitHub. There's a lot of stuff out there that if you poke around the data industry, you find weird ref edges. And it's just very early.
And that's like, it's the one thing is like the numbers are too big, the quality is bad. But on the other hand, the labs are improving very fast and know how to onboard it. So I think this will take a few years to transition of like really pulling the winners into higher quality, higher ROI. You just take lots of resources when you have these systems for doing better. And then the labs keep working with them because they get even more training signal and they have to do less internal filtering.
I think this is going to just take years. How do you think about them using environments and like your purchasing decisions as part of the nonprofit that you're starting? I think most of our work will be making open source environments for tasks that are very high leverage. So I think it was this work that I advise you, you, W, A, I, two, state and hamish, I just sit on this teamax work, which is you use a few. It's for terminal tasks, which are like super popular and terminal bench, whatever number we're on.
And this is fully synthetic data where you prompt a frontier model to generate a type of problem and you use a few different prompts to generate diversity. And then you use model rollouts and other things to find a bugs and eliminate the ones with bugs. And this process is not magic. It's like you read every data point your data will get better and you can do this. But there's just not that many institutions that are investing what it takes thousands or tens of thousands of dollars to make these environments well and then release it.
Because the profit motive is so high. I was joking with all of your money. I probably sell this data for a million dollars in that release of paper and I'm like, no, we care about doing the other thing. But this is like these open resources are the fun foundation of so much study. And it's like you release some of our environments, some of our all our people can do research on say like a valuarness.
You have the generation from the model. People can look for different behaviors. If you've done RL on it, they can see how it goes over time. And this is just the units of infrastructure that PhD students spend a year looking at and produce increased knowledge. And the tech industry is the place where value is captured from CS research.
And all CS research and the benefit of careers to the tech industry in like 10 years. And I think like this is why more people supporting open better. They're gonna work in the industry in 10 years. Like Europeans are gonna be the ones that benefit from it. More fossil fuel.
It's not to play down. But it's just like the opportunity to be a short term win is so high. The opportunity to make what is $100 million like nine figure returns for your business within five months of founding a startup with your friends. It's just like these are people who may have done more open source work or these are people who left their PhD or something. And it's just like pulling back, which is why we came up with this name Trillium for our lab, which is like Trillium is a spring ephemeral flower, which it blooms before the canopy has grown its leaves.
So it's like winter is a tree leaves on the trees. These Trillium flowers bloom and they're like essential for spreading nutrients through the cycle of growth that comes through spring and summer. And this is where you feel like the industry is doing so much right. It's like so much is going right. But it's just like we just need a little bit more life to like spread this out and make things so new people can come in like there's so much attention and just like do research, do education.
And then as the scales become more impactful is like you have these open things which become the standards of study for such important problems. So just like how do you understand sandboxes? Like how do you define what loss of control is? And just see how this changes in a RL round that is a week on 1000 GPUs or 100,000 GPUs and it's just like make these places so that people have more trust. And I think this pays so much dividends to everybody that's a profit more than I do.
And it's just like hard to get people to take that leap. And like I've been rewarded so much for kind of stumbling into this area. And now I get so much more to do things like this and just like now is the best job because I try to create space for more people to like grow their independent thought and build this infrastructure. And maybe one like a naive way that it kind of perpetuates is just because it's open source people get to you might get more of a distributed contribution from people who are just so you know it's like they're motivated not for just trying to sell to laughs themselves but because hey it's out there they can share their work and then you actually have a very high caliber amount of contributors to the project. I mean that would probably be the name of how that.
Yeah, it's a new puzzle with agents where people just start things like I have some recent random researcher positories for research papers and I'm like people in the most crazy get how issues now. So what do you mean like fake research results they open an issue and they're like oh my God I found this thing in your data say I'm like well if you look at the paper we said that like sex. And it's just like a lot of language. And that's a new thing for the open community to go through this is a solvable problem. Like there are people out there that want to contribute to this stuff because they right now just cloud chasing in another dimension but like actually hey it's out there.
I mean I would just hope because it is published and open or you know and available you can actually get like PhD students or other researchers like I do this on my day job but maybe I can still be able to contribute. Yeah. I see people like this and I was like great somebody from like Eastern Europe was helping me improve the code for my book that should right here. And it's just like doing it for the right thing and like there are people that do it and I think a lot of people just take like they don't have the time or the stability to take the long time it takes because when you're trying to get into a new field you know nothing. You have to spend like weeks reading and knowing nothing and then you send an email and nobody replies and you're just like it takes you weeks to know the context like ask a good question.
And now with language models most people just push a button and they're like oh interesting I read this I'll send it off and it's just like this. Sitting in it is the thing that people don't do as much and that is the key to like becoming into AI is that you have to focus on it. You have to write some code by hand and you have to do this for months and then you have to wait six to 18 months for you to get notice. It just takes a long time and like AI agents I think just like frazzle that and makes it harder for people. So the mathematician is something about too.
It's like it's a hard time to be a PhD student but people are doing great work in academia still and but it's hard. I agree with you I think it's like solvable as the agents can perpetuate some version of the problem it can also help solve and identify an quality filter. So environments are going to release open source versions of that help precipitate maybe more people to contribute and reveal that. I think that's like really give me especially given how little it's to release this instead of just get fast revenue from the labs. Another area I think I read on your announcement post is RSI which is quite I think I mean we can first talk about what is actually recursive about it or how scary it is or whether we're going to have take off it if you have a very grounded opinion on that.
But you know another thing I want to talk about is I think RSI related to how much it's kind of hyped is a reason why we are afraid the frontier labs will pull away. In fact it's also a reason why I've heard not just once but several times that people are one convinced that their own jobs are going to be automated as a researcher. Yeah. I've gone to last year some of my best friends I'm like you are some of the best people in the world of this and then they go to the work and they're like Nathan I'm struggling. Like what am I going to do? Because you're like a lot of what they do is verifiable right?
But because of that you know there's actually quite a lot of other non-verifiable or harder out of distribution tasks such as alignment and things like that. And so actually because I mean one argument I heard is because there's such a difference between how well RSI is doing for model capabilities versus alignment which is much more out of distribution and creative work that gives people an ease on how maybe there is actual dangers etc. So anyways because of this I'm very interested in hearing from you like how because I think it's really important that we do have an open source kind of front in showing how RSI is being done. How much maybe then we can understand the capabilities of like if in within a lab they have a better model where the chances they actually pull away just better understanding calibrating.
So yeah I'd love to hear more about that. I think this we won't be the only people doing this but essentially like RSI has so much mystique like we're using models for everything and like what we can do is a open lab and a nonprofit and I think other companies like Luther Hugging Pays or Open Athena which is Marin and Stanford. They will get into this good marketing line for them is and we can share what tools we're using we're going to build an open source stack for SFT for mid training we're going to use Skyr-L Miles all these things you know and we have our task we have our configs we know what type of compute we're running we can work with cloud and GPT and all these models and we'll just be like we can store the trajectories for what we did we can tell you what our tasks will never use linear or whatever popular things are.
Say what our tasks were why we chose them and whether models work on them or not and you just do this for a year of a while but you're like here is what we did and just check in every six months and like has it changed and it's just like to be able to have the full stack and not need to go through these classifiers and like try to just like say what the task is vaguely and so on it's just like people that can share what automation is happening to do this and we have a different distribution of tasks but I think that the research is similar. The AI research that I've done at AI too and we'll do it trillion versus what people do in some sex of academia and the labs it's not that different you have more resources you scale it up and it's just like so good for the world when everybody's talking about this to show people.
This is what Nathan asked his model to do and this is what it got stuck on and I think that will help a lot of literal policymakers like oh okay. That's why it's like that's what it got stuck on that's the bug that put in its code. Like the bugs and the catch ups will reduce over time but I think the jaggedness and some of these fundamental things of the model is like some of the things that we're going to do. I think more things will take off like I said efficiency earlier like I do think the models will be really good at lowering serving costs these companies have billions of dollars on the line and the models are good at it. They're going to make a lot of money on being serving more efficient and tweaking architecture in order to go into that but it's not necessarily like how do you solve knowledge or cause.
I'm not going to be like Claude me tell yourself to make better power points like what what is the environment. It's gonna take a while. Yeah I love the specified goal of like let's just justify this because I think a lot of why the rhetoric gets cheated around our size is like it sounds. It's hard for academia to be like as a professor 15% of my time and 40% time my grad student is going to be documenting what we work on. There's like a great like policy and community service of like here are tasks people can use this data to make evaluations or whatever it's just like just like build things that have rare information so that when people put more high focus work into it it just has more value for all sorts of things that you can't predict in the future.
It's also just very timely because the kind of you know it's not from one person but several people that I know from the labs are like hey I'm going to get automated and I need to get to your point to your friends like I will definitely as a person or so I will get automated in a year and so that probably means it's going to be less people who actually are exposed to what's being done on that particular part of the org chart and so. There's an academic there's an open RSI thing which is a bunch of academic institutions I'm not affiliated with it they just launched like last week so I was like yeah I think I may see emails I'm like open our size thing they're trying to do this as well it doesn't need to just be one group. Different approaches doing this you need the different ways of looking at it and different results. The other thing I also heard was from more kind of like frontier lab costs is just like the actual methods that are being used in the labs are not that it's not that they're unknown it's just sort of like what are the proportions that need to be used that are successful and as a part of the capital scale it to experiment.
So I think that's one heartening to know because like okay it's not like there's a big secret of what's going on to people actually know the techniques but on the other hand you know capital is the one true barrier still because it's hard to actually know which of the techniques recipes are going to be used. That's why I'm not pre-training I'm happy to be free I can use any base model I'm now going to benefit from competition. I do think that there's like a chance for any pre-training open source effort I know the capital requirements are just like orders of magnitude larger so that is what's difficult but maybe one of the values of that and then maybe we can discuss potential commercial strategies. Pre-training science is pretty like the tools are pretty accessible and you have what is a pre-training ladder which is like the decision making tool you have to test if your idea scale and the pre-training data isn't growing at like the environment seem like it's essentially it's a person approximation you train on the internet and like there's a lot of other little details and secret things but like that's not changing that much the state of the architecture is and the GPUs are and that's like that changes and but it's not astronomically expensive to make a pre-trained model that is solid so I think like you don't have to spend a billion dollars to get a very good base model the more you spend the better you will get but it's like the people keep making progress and I'm constantly surprised by a new lab just goes from scratch because I can and some of this is on open tools so like you can go off Megatron like a lot of new startups they're like oh what would we train to train off the open source of the system.
They train off the open source tool and they fork it and then they never really sit again this is what all the all the like even prominent open model apps have done they take the open source code they make it their own thing and they just want to push upstream so I've always been surprised by this. So I'm like not getting against new entrance because like it's not that you need 10 billion dollars to have a relevant model the hard thing is like get a model is good enough for people to use and to get a story where if you release it you can do your next funders and just like you have to get past this lock of like we need our first model to land and like your whole company is based on models are the best and you're not in the ballpark and you're not releasing it.
You're on the clock and it's like I think of like thinking machines getting inkling out where it's like they were very transparent like this wasn't the best model we're proud of it has some cool things. And like they're now they released their second model and they're getting going and they have Tinker has this like that's very good for them. It's like I don't think that's what they were doing like they were building models open models wasn't their first plan. I'm like out there and there's rumors that like meta said they'll release a new open weight model again like that'll come. A lot of companies out there like into four models.
Yeah yeah I'm definitely glad that the machines is and it makes so much sense for their business model because they do post training. And you know with meta he mentioned this as well and in some previous writings it's for them it's such a good strategy. I think it was also just kind of obvious when they do it's like it's complimentary to my stream. The best blog post on this like mom and three came out here like the best like why open this top for it. Guy got a little distracted seems like he's coming back to the line.
So I hope they keep on being motivated to do that. Getting to the frontier with pre-training you know I think there's a lot of help given that there's such good open source models to maybe you know somehow still kind of be a strap from. I keep on getting kind of conflicting signals as to like whether pre-training is actually as saturated or well understood. I mean maybe to your point like there's a lot of like well known techniques that work but what's hard is actually having access to the 20,000. People are turning the crank on pre-training and pre-training became adjusted when super long context generation happened.
So there's just way more return if you're if you can for example and get a sparse attention architecture to work as well and then your 100,000 token generation is 40% faster. So I think there's a lot of co-evolution of the whole stack where they have their pre-training stack. They realize that imprint scaling is so much higher but even the fundamentals of pre-training keep improving. You just have a marginally better architecture that returns on investment and then it kind of blurs into post-training where this mid-training where you start the reasoning behaviors you take them from your older models you do our own them to see behaviors like this cycle is much more complicated than pre-training. Like if you don't have the right people it's a lot harder to get that off the ground than I think pre-training was.
Like I think they're like I think like to do a controlled like let's just start pre-training our models seems a lot more approachable to me than I don't really know much about language models. I'm gonna figure out this whole post-training thing where it's just like the wild west. Yeah. And that's why I was like when I left AI2 I felt like our post-training was honestly falling behind just because our cycle time was not high enough and we're going to these things and these are still some of the leading open source post-training recipes and that's just like that's why a lot of things is like it doesn't take as much capital to be very big compounding gains on the state of the art and the open source because the bar is very low. It's like it's not like it's not closed.
Yeah. No, that makes sense. It's like a kind of efficiency where you can put capital and make a huge difference. Yeah. Yeah.
Kind of related to the whole you know post-training side maybe you know pulling away from closed and open is the fact that the agents you know the type of harnesses that are being used in the closed labs are very different and slash you know actually trained and post-trained on to you know it to be specifically good for their models are specifically good for those harnesses and so that increases the mistake even more makes it very hard. How do you I guess want to agree in the shoe like how are you sort of hoping specifically to kind of address that and what things I mean are really examples like hey spin up like thousands of ages and tens of thousands of agents and just like a very different workflow than what people are obviously exposed to. I think it'll be okay for open because I thought years ago that say the closed labs would be like Apple would be integrated company where they have the model designed for the harness that should involve they give some kind of a kind of a scale efficiency but it turned out that so many people have different harnesses for different things that they like to do that the models companies have been pressurized to make it so that their model can work in all sorts of different hearteses and I thought it would be a weakness of open models that they have to be designed to train on all sorts of places and harnesses but it ended up being pretty equally related like it's at equilibrium where they all have to do it to some capacity and then what we were seeing with this like agents forms is that opening iron the proper going to be earlier to it they're going to know how to design their models they're going to know how to design what do we call what do we call to call a harness first form I don't know like I don't know but there's going to be a word for it it's not a harness it's like a coordination system but then eventually other people are going to want to use them and I think it's going to be like these cycles where the frontier labs like they push this out and they have this
to date they've had huge returns on being first but like what is the returns on Navier's jokes and all these things like we don't really know like we don't know they push the frontier in the assumption that there will be high returns and that this marginal intelligence has like disproportionate returns and we don't know if that's always true and I kind of expect these cycles like once you know the shape of how do you control 10,000 agents in one harness then the open model companies book oh we can do this too but it's not like Timmy is going to do that first or one they don't have a compute It's an astronomical amount of compute but these things go through cycles and videos chips will get way faster. There's more compute built out So you've heard like convinced that like there's not too much hidden there that you can probably guess what they're doing and then when it becomes relevant let the open source models actually try to also post strain on similar meta harnesses and I also think like the security at these labs in terms of information as high as I think like Silicon Valley is famous for like ideas traveling what is it like one beer or something like everybody chaps after work Which is why I so impressive. Yeah, that's what's going on. That's only passed but it's like we're all scientists if they made a breakthrough they're gonna want to tell their friends
They might hold it they might hold it for a few months. I don't think the amount of luck like information security you would need is just so high that Knowledge spreads it's hard to keep the secret like what you what they can win on is putting more inference into their infrastructure Which lets them serve at a lower price and make higher margins and then continue this over a longer time So I'm very confident in the frontier labs being able to serve the cheapest intelligence per Per watt per score on a bow They are going to be able to go to the places with the highest demand and have the like the cheapest models And that will be revenue creative for like I'll be good business for that The question is like they can't do everything and that's where the rest of the economy fills in is like is it just coding?
That is profitable then definitely not a good case for future models because they will help them like they have integration They'll do very well, but is it the whole economy of knowledge work is the market for things And then it might not be good for them because can they do everything at once? But we don't need to I think there's gonna be a lot of winners. Yeah That's what I have in the frontier models and startups Not because I want to hedge because I actually do think you know to gonna cooperate your view that there's enough Harder tasks at the fringe that requires a specialized data. That's kind of out of the splash So forgetting to AGI or whatever you want to call it now these days That makes it possible to have many many winners. I do like your RSI point is we talked a little bit about it Which is just kind of say practically hey, it's exponential cost and resource consumption and therefore There's a very obvious kind of limit to how much that the French your labs can pull away If that's not too oversimplification of your view Anything else you kind of want to add there because I do feel like the discourse now It's just so so much fear on like how much the frontier is just hidden and also breaking away I think there's so much spend on inference and that in the short term we are learning how to
Corral inference into performance gains Yeah, and this is like a thing that we just had and like this is what we're seeing now is that like people are spending so much and All the employees at the frontier labs are spending exponential amounts relative to the improvements But like we're figuring out how to do this and there'll be a lot of gains But I don't think that it's necessarily going to be this like fast takeoff when the models build themselves human human bottlenecks that all the scaling laws are For linear increase in performance and I would do say that like we have Navier stokes and I'm of opinion that math is exceptionally well-suited maybe like the scientific field that is best suited to this shape of intelligence and like intelligence can take many different forms And that's not to say that like if they do like if an AI model does say discover something that's very different than the transformer That gives very different properties and unlocks a new type of intelligence like it is worth updating on this and worth being like okay Like that is different but until that happens. I'm like pretty happy with my evidence The companies are gonna do well
They're spending exponentially more to keep this remarkable pace of progress that some of the Claude System cards say this they're like it would take say like 10x more progress to make this a fast takeoff And it's like for now they're helping us sustain the pace of progress And I think it's a good It's a good you go around but I do think that like the labs have people saying this the the science is like it's hard to overcome that It's okay, but it's like we don't know exactly what I'll go from here Yeah, no, I think that's kind of comforting because we know what the inputs and lovers are here Just there is a very successful autoregressive regime that we've been optimizing and that one just you know asks for a certain amount of like Exponential resources which are clearly capped and therefore open source hopefully you know can keep it strong as well It's just what I mean there is dramatic amount of work to do in preparing for structure Like there's the AI capability plot which doesn't necessarily worry me But the AI capability plot in the pretty much non-existing preparation of cross-public cybersecurity and all these things like there needs to be more Preparation in the assumption that this is coming Then there is and then that's kind of a thing that worries me more than say like our side is coming in unlocking new risks
Like there are risks that are here and that are gonna continue to diffuse and many times of close lives get there And then the open lives get there is like we're in this process for a lot of things that are gonna cause some problems It's just like figuring out how to get society to prepare is very hard Yeah, and I come of the opinion that we need more eyes we need more science So that society feels informed yeah, but there are other valid opinions and like I do think that Not fearing fast takeoff does not mean that you don't worry about risks I don't think you should maybe I'll send in some people your way that are affected from the lab I'm sorry before precisely about this. Well, thank you so much Nathan This is a lot of fun and yeah, I'm really excited to see what Trillium is up to you now that it's finally launched Yeah, thanks so much for having me Thanks for listening to this episode of the a 16z podcast if you like this episode be sure to like comment subscribe leave us a rating We're review and share it with your friends and family for more episodes go to YouTube Apple podcast and Spotify Follow us on X a 16 Z and subscribe to our sub stack at a 16 Z dot sub stack calm Thanks again for listening and I'll see you in the next episode
As a reminder the content here is for informational purposes only should not be taken as legal business tax or investment advice Or be used to evaluate any investment or security and is not directed at any investors or potential investors any a 16 Z Please note that a 16 Z and its affiliates may also maintain investments in the companies discussed in this podcast for more details including a link to our investments Please see a 16 Z dot com forward slash disclosures