I'm confused that people have super short timelines, yet at the same time are published on scaling up reinforcement learning on top LLMs. If we're actually close to a human-like learner, then this whole approach of training on bare-fived outcomes is doomed. Now, currently the labs are trying to bake in a bunch of skills into these models through mid-training. There's an entire supply chain of companies that are building our environments, which teach the model how to navigate a web browser or use Excel to build financial models. Now, either of these models will soon learn on the job in a self-directed way, which will make all this pretty making pointless, or they won't, which means that AGI is not even it.
Humans don't have to go through the special training phase where they need to rehearse every single piece of software that they might ever need to use on the job. It's a great interesting point about this in a recent blog by C-Rote. He writes, quote, when we see frontier models improving at various benchmarks, we should think not just about the increased scale and the clever ML research ideas, but the billions of dollars that are paid to PhDs, MDs, and other experts to write questions and provide example answers and reasoning targeting these precise capabilities. You can see this tension most vividly in robotics. In some fundamental sense, robotics is an algorithms problem, not a hardware or a data problem.
With very little training, a human can learn how to tell or operate current hardware to do use for work. So if we actually had a human-like learner, robotics would be, in large part, a solved problem. But the fact that we don't have such a learner makes it necessary to go out into a thousand different homes and practice a million times on how to pick up dishes or fold laundry. Now, one corner I recommend I've heard from the people who think we're going to have a take off within the next five years is that we have to do all this cludgy RL in service of building a superhuman AI researcher. And then the million copies of this automated Iliad can go figure out how to solve robust and efficient learning from experience.
This just gives me the vibes of that old joke. We're losing money on every sale, but we'll make it up in volume. Somehow this automated researcher is going to figure out the algorithm for AGI, which is a problem that humans have been banging their head against for the better half of a century. While not having the basic learning capabilities that children have, I find it super impossible. Besides, even if that's what you believe, it doesn't describe how the labs are approaching reinforcement learning from their final reward.
You don't need to pre-bacon a consultant skill at crafting PowerPoint slides in order to automate Iliad. So clearly, the labs actions hint at a worldview where these models will continue to fear poorly at generalization and on the job learning. That's making it necessary to build in the skills that we hope will be economically useful beforehand into these models. Another common argument you can make is that even if the model could learn these skills on the job, it's just so much more efficient to build in these skills once during trading rather than again and again for each user and each company. And look, it makes a ton of sense to just bake influencing the common tools like browsers, users, and terminals.
And indeed, one of the key advantages that AGIs will have is this greater capacity to share knowledge across copies. But people are really under rating how much company and context-specific skills are required to do most jobs. And there just isn't currently a robust, efficient way for AIs to pick up these skills. I was recently at a dinner with an AI researcher and a biologist, and it turned out the biologist had long timelines, and so we were asking about why she had these long timelines. And then she said, one part of work recently in the lab is involved looking at slides and deciding if the dot in that slide is actually a macrofage or just looks like a macrofage.
And the AI researcher, as you might anticipate, responded, look, image classification is a text with the learning problem. This is that center and the kind of thing that we could train these models to do. And I thought this was a very interesting exchange because it illustrated a key crux between me and the people who expect transformative economic impact in the next few years. Human workers are valuable precisely because we don't need to build in the schleppy trading groups for every single small part of their job. It's not net productive to build a custom training pipeline to identify what macrofage is look like, given the specific way that this lab prepares slides.
And then another training loop for the next lab-specific microtask and so on, where you actually need is an AI that can learn from semantic feedback or from self-directed experience. And then generalize the way a human does. Every day, you have to do 100 things that require judgment, situational awareness, and skills and context that are learned on the job. These tasks differ not just across different people, but even from one day to the next for the same person. It is not possible to automate even a single job by just baking in a predefined set of skills, let alone all the jobs.
In fact, I think people are really underestimating how big a deal actual AGI will be because they are just imagining more of this current regime. They're not thinking about billions of human-like intelligences on a server, which can copy and merge all the learnings. And to be clear, I expect this, which is to say, I expect actual brain-like intelligences within the next decade or two, which is pretty fucking crazy. Sometimes people will say that the reason that AI's are more widely deployed right now across firms and already providing lots of value outside of coding is that technology takes a long time to diffuse. And I think this is Cope.
And people are using this Cope to gloss over the fact that these models just lack the capabilities that are necessary for broad economic value. If these models actually were like humans on a server, they'd diffuse incredibly quickly. In fact, they'd be so much easier to integrate in onboard than a normal human employee is. They could read your entire slack and drive within minutes, and they could immediately distill all the skills that your other AI employees have. Plus, the hiring market for humans is very much like a lemons market, where it's hard to tell who the good people are beforehand.
And then obviously hiring somebody who turns out to be bad is very costly. This is just not a dynamic that you would have to face or worry about if you're just spinning up another instance of a vetted HGI model. So for these reasons, I expect it's going to be much easier to diffuse AI labor into firms than it is a higher person. And companies hire people all the time. If the capabilities were actually at HGI level, people would be willing to spend trillions of dollars a year buying tokens that these models produce.
Knowledge workers across the world cumulatively earn tens of trillions of dollars a year in wages. And the reason that labs are orders of magnitude off the figure right now is that the models are nowhere near as capable as human knowledge workers. Now you might be like, look, how can the standard have certainly become labs after tens of trillions of dollars a revenue a year, right? Until recently people were saying, can these models reason? Do these models have common sense?
Are they just doing pan and recognition? And obviously, AI bulls are right to criticize AI bears for repeatedly moving these goalposts. And this is very often fair. It's easy to underestimate the progress that AI has made over the last decade. But some amount of goalpost shifting is actually justified.
If you showed me Gemini theory in 2020, I would have been certain that it could automate half of knowledge. And so we keep solving what we thought were the sufficient bottlenecks to HGI. We have models that have general understanding, they have future learning, they have reasoning. And yet we still don't have HGI. So what is a rational response to observing this?
I think it's totally reasonable to look at this and say, oh, actually, there's much more to intelligence and labor than I previously realized. And while we're really close, and in many ways, have surpassed what I would have previously defined as HGI in the past. The fact that model companies are not making the trillions of dollars in revenue that would be implied by HGI clearly reveals that my previous definition of HGI was too narrow. And I expect this to keep happening into the future. I expect that by 2030, labs will have made significant progress on my hobby horse of continual learning.
And the models will be earning hundreds of billions of dollars in revenue a year. But they won't have automated all knowledge work. And I'll be like, look, we made a lot of progress, but we haven't hit HGI yet. We also need these other capabilities. We need X, Y, and Z capabilities in these models.
Models keep getting more impressive at the rate that the short timelines people predict, but more useful at the rate that the long timelines people predict. It's worth asking, what are we scaling? With pre-training, we had this extremely clean and general trend in improvement in loss across multiple orders of magnitude and compute. I'll be at this was on a power law, which is as weak as exponential growth is strong. But people are trying to wander the prestige that pre-training scaling has, which is almost as predictable as a physical law of the universe, to justify bullish predictions about reinforcement learning from verifyable reward, for which we have no well but publicly known trend.
And when intrepid researchers do try to piece together the implications from scarce public data points, they get pretty bearish results. For example, Toby Burd has a great post where he cleverly connects the dots between the different O-series benchmarks. And this suggested to him that, quote, we need something like a million X scale up in total RLL compute to give a boost similar to a single GPT level. End quote. So people have spent a lot of time talking about the possibility of a software and singularity, where AI models will write the code that generate the smarter success system, or a software plus hardware singularity, where AI's also improve their success or is computing hardware.
However, all these scenarios neglect what I think will be the main driver of further improvements atop AGI, continual learning. Again, think about how humans become more capable of anything. It's mostly from experience in the relevant domain. Over conversation, Baron Milligemay, this interesting suggestion that the future might look like continual learning agents who are all going out and they're doing different jobs and they're generating value. And then they're bringing back all their learnings to the hive mind model, which does some kind of bash distillation on all of these agents.
The agents themselves can be quite specialized, containing what Carpathi called the cognitive core plus knowledge and skills relevant to the job they're being deployed to do. Solving continual learning won't be a singular one in done achievement. Instead, it will feel like solving incontinence learning. Now, GPT-3 already demonstrated incontinence learning can be very powerful in 2020. It's incontinence learning can be least where so remarkable the title of the GPT-3 paper was, language models are a few shop learners.
But of course, we didn't solve incontinence learning when GPT-3 came out. And indeed, there's still plenty of progress that still has to be made from comprehension to context length. I tried a similar progression with continual learning. Labs will probably release something next year which they call continual learning, and which will in fact count as progress to words continual learning. But human level on the job learning may take another five to 10 years to iron out.
This is why I don't expect some kind of runaway gains from the first model that cracks continual learning that's getting more and more widely deployed and capable. If you had fully solved continual learning, drop out of nowhere. Then sure, it might be game set match as Satya put it on the podcast when I asked him about this by the way. But that's probably not what's gonna happen. Instead, some lab is gonna figure out how to get some initial traction on this problem.
And then playing around with this feature will make it clear how it was implemented, and then other labs will soon replicate the breakthrough and improve it slightly. Besides, I just asked them prior that the competition will stay pretty fierce between all these model companies. And it's informed by the observation that all these previous supposed flywheels, whether that's user engagement on chat or synthetic data or whatever, have been very little to diminish the greater and greater competition between model companies. Every month or so, the big three model companies will rotate around the podium, and the other competitors are not that far behind. There seems to be some force, and this is potentially talent poaching, it's potentially the rumor mill in SF, or just normal reverse engineering, which is so far neutralized any runaway advantage that a single lab might have had.
This was an aeration of an essay that I originally released on my blog at Work Catch.com. I'm gonna be publishing a lot more essays. I found it actually quite helpful in ironing out my thoughts before interviews. If you wanna stay up to date with those, you can subscribe at Work Catch.com. Otherwise, I'll see you for the next podcast.
Cheers.