Skip to content
Freaked Out? We Really Can Prepare for A.I. episode artwork

Mar 21, 2023 1 hr 34 min

Freaked Out? We Really Can Prepare for A.I.

from The Ezra Klein Show Hosted by New York Times Opinion

OpenAI last week released its most powerful language model yet: GPT-4, which vastly outperforms its predecessor, GPT-3.5, on a variety of tasks. GPT-4 can pass the bar exam in the 90th percentile, while the previous model struggled around in the 10th percentile. GPT-4 scored in the 88th percentile on the LSAT, up from GPT-3.5’s 40th percentile. And on the advanced sommelier theory test, GPT-4 performed better than 77 percent of test-takers. (It’s predecessor hovered around 46 percent.) These are stunning results — not just what the model can do, but the rapid pace of progress. And Open AI’s ChatGPT and other chat bots are just one example of what recent A.I. systems can achieve. Kelsey Piper is a senior writer at Vox, where she’s been ahead of the curve covering advanced A.I., its world-changing possibilities, and the people creating it. Her work is informed by her deep knowledge of the handful of companies that arguably have the most influence over the future of A.I. We discuss whether artificial intelligence has coherent “goals” — and whether that matters; whether the disasters ahead in A.I. will be small enough to learn from or “truly catastrophic”; the challenge of building “social technology” fast enough to withstand malicious uses of A.I.; whether we should focus on slowing down A.I. progress — and the specific oversight and regulation that could help us do it; why Piper is more optimistic this year that regulators can be “on the ball’ with A.I.; how competition between the U.S. and China shapes A.I. policy; and more. This episode contains strong language. Mentioned: “The Man of Your Dreams” by Sangeeta Singh-Kurtz “The Case for Taking A.I. Seriously as a Threat to Humanity” by Kelsey Piper “The Return of the Magicians” by Ross Douthat “Let’s Think About Slowing Down A.I.” by Katja Grace Book Recommendations: The Making of the Atomic Bomb by Richard Rhodes Asterisk Magazine The Silmarillion by J. R. R. Tolkien Thoughts? Guest suggestions? Email us at [email protected]. You can find transcripts (posted midday) and more episodes of “The Ezra Klein Show” at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Book recommendations from all our guests are listed at https://www.nytimes.com/article/ezra-klein-show-book-recs. “The Ezra Klein Show” is produced by Emefa Agawu, Annie Galvin, Jeff Geld, Roge Karma and Kristin Lin. Fact-checking by Michelle Harris and Kate Sinclair. Mixing by Jeff Geld. Original music by Isaac Jones. Audience strategy by Shannon Busta. The executive producer of New York Times Opinion Audio is Annie-Rose Strasser. Special thanks to Carole Sabouraud and Kristina Samulewski. Subscribe today at nytimes.com/podcasts or on Apple Podcasts, Spotify and Amazon Music. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Freaked Out? We Really Can Prepare for A.I.

0:00 1:34:20
of matches

Transcript Auto-generated, may contain errors

Hi, I'm Solana Pine. I'm the director of video at the New York Times. For years, my team has made videos that bring you closer to big news moments, videos by Times journalists that have the expertise to help you understand what's going on. Now we're bringing those videos to you in the Watch tab in the New York Times app.

It's a dedicated video feed where you know you can trust what you're seeing. All the videos there are free for anyone to watch. You don't have to be a subscriber. Download the New York Times app to start watching.

I'm Mr. Klein. This is the Ezra Concho. So let me start this one by situating us in time.

So we take this podcast just a few hours after OpenEye released GPD4, which is our most powerful AI language model yet. The thing that is remarkable here that I really keep trying to push people to key into is how quickly these systems are getting better, how steep the curve of improvement is. We talk about some benchmarks here on the show, but GPD4 now passes a bar exam in the 90th percentile, the 90th percentile. Just sit with that.

We were nowhere near fully understanding what the last generation of these models could do. Now we're on to the next one. And the same is probably going to be true here. GPD4 is extremely powerful.

It is a time's unnerving. We barely understand what I can do or how to use it. And yet everybody is racing ahead. GPD4 also is going to be a toy, a diversion, compared to what we're going to have in the three or five years.

If you heard the commentary released on AI on the podcast over the weekend, you heard that what I'm most focused on is how society manages to adapt to the speed at which these systems are improving or it slows them down so we have more time to adapt to understand them, to think about how we want to integrate them. And one piece of feedback I got in the commentary was what are you really saying we should do aside from being anxious? And I don't just want people to be anxious. So this conversation gets more into specifics, both on what these systems can do now, what they can be so disruptive, where they seem to be going, and what we can and I think should do, or at least the kinds of things we should do in response.

Chelsea Piper is a senior writer at Vox where she's been doing great coverage for years on AI and the technologies and the culture and the safety and economic questions around it. She joins me now as always my email as our client show at nytimes.com. Chelsea Piper, welcome to the show. Thanks.

God be here. So one of the reasons I wanted to have you on here is you've done a lot of reporting. You spend a lot of time with folks inside the AI world. So you're a little bit more in this hot house than a lot of us.

And when you talk to them over the past couple of months, what do they say? What are they seeing or what are they thinking about that from the outside but not the obvious? Yeah, so I think not just in the last six months, since like at least 2019, you had people saying this is going to overnight, you know, not today, but in 10 years overnight transform our world. We are going to be able to automate any work that can be done remotely.

We are going to be able to bring us the next hundred years of drug discovery and development, you know, in the space of maybe a couple of months, maybe a single research project. We have this extremely powerful thing more on the brain. And in 2019, when you heard that, I think you had to be a little skeptical, like this is Silicon Valley. We're not new to the hype cycle.

Sure, you're going to change the world. And now I think you're still hearing the same things, mostly from the same people, but you have to take them a little more seriously. They are in fact releasing things that can do things that everybody thought were impossible. They are developing the newest models as fast as they can.

OpenAI says we think the way for humanity to like do well on AI is to raise ahead as fast as possible. This is in a statement a couple of weeks ago, this morning, KPT-4, you know, only a couple of months behind KPT, which hit a hundred million users like overnight. It's not even that they're saying new things. It's that it's hard not to take them seriously.

So when you say overnight, this idea that sometime in the future, all of a sudden, the history of AI will speed up, that we're making progress fast, but it's, you know, a couple of years, a couple of months between major new systems. And then you get these ideas of quantumly, truly quickly. What are people talking about there? What are they looking at?

So I think the most concrete version of this is that right now, all of the work done on improving AI systems is being done by humans. And it's a pretty small number of humans. And of course, one of the jobs that they're trying to automate is their job of making better AI systems. And when they succeeded that, if they succeeded that, then you can, instead of having hundreds of people working on building better AI systems, you can have thousands, you can have a million because you can just make a copy of an AI system.

That's like it's fundamental economic potential is that you can make a copy of it. You can make an AI system that is as good at writing as you. And then you can have a thousand of them writing about all different topics. You can make an AI system with a really good programmer.

And then you can have all of the programmers working with or possibly replaced by that thing. So things are already moving fast. And I don't think the case for like picking AI really seriously crucially depends on those start moving even faster. Like the current pace is fast enough to me to be disastrous.

But if you ask like, what's my gut instinct about what's going to happen? It's that at the point where we figure out how to make AI researchers out of AI's, then things will move real fast. And that sounds very sci-fi is a lot of things do because we live in the society world increasingly. One of the things that I know is striking to a lot of people in this world was when GPT-3 really proved able to code even though it hadn't really been trained to do so.

And so then as people began moving more towards optimizing AI systems to create code and you expect we'll do even more of that training on even more code, get even better at that. Once AI's can code, which it can already do, when you just assume for the improvement curve, then the idea that you would have an AI that codes AI, that's not a giant leap. That's simply finishing the sentence in a slightly different way. Yeah, I guess a lot of it is just there is so much commercial incentive to make an AI system that can code.

Programmers are really expensive. They're scarce. It's hard to hire them. You have an AI system that can do most of what they can do.

Companies would be selling out huge amounts of money for it. And that means companies are selling out huge amounts of money on developing it. And this is also true, more true of the specific skill of building large AI systems. So I don't think you need this.

Things will accelerate premise necessarily to take AI really seriously. But my best guess is that there are people right now trying to develop AI systems that can develop AI systems because there's people trying to develop AI systems that can do every human job. Well, let's talk about how fast it's going now. So this is a little bit about coincidence, but the morning we're talking OpenEye released GPT-4 and they did not, according to them, train GPT-4 specifically to pass any tests.

But like GPT-3, in 3.5, it absorbs a lot of language from the internet. It begins to create statistical correlations for lack of more precise ways of saying it between terms and out of that emerges a kind of thing that seems to act in the real world like understanding. So they released this table where they showed the performance of GPT-3.5 and GPT-4 on a battery of tests of human beings mind pretty hard. So when you try to give GPT-3.5 the bar exam, it is in the 10 percentile of people.

When you give it to GPT-4, it's in the 90th percentile. GPT-4 passes the LSAT at the 88th percentile. GPT-3.5 was at the 48th percentile. I found this one kind of funny, but also interesting that it passes the advanced Sommelier theory test.

That's for wine in the 77th percentile. GPT-3.5 was in the 46th percentile. So these are fairly general tests of knowledge and reasoning. And obviously there's information about them out there.

I mean, these have essay questions, et cetera. Now we're in an area where the improvement is very fast and you can actually match it against human beings on things that I would not have said five or ten years ago. Oh, that's just a structure data quiz. There's an old argument that we say artificial intelligence is anything that a system can't do yet.

And machine learning is anything it can already do. But how do you think a bit about this debate over truly intelligent versus stochastic parrots, as people like to call it versus a very advanced autocomplete? So it sort of calls into question what intelligence is, right? What is going on in our brains when we think?

I think there is a sense in which it's just a glorified autocorrect. But I think that's also a sense in which most of the time we are also just a glorified autocorrect. What humans do is we read lots of things, we talk to lots of people, and then we put it together into our own sort of framing that articulates something that we want to articulate. And it turns out that most of that, you can just feed something the entire internet and then make it really, really good at pattern recognition and then prompt it and it does that.

And I guess you can say, well, that's not intelligence. But if you're going to declare, well, that's not intelligence, I think you are kind of in a position where you're also declaring that about humans. And I don't know what do you even get out of saying humans aren't really intelligent. We still did build civilization.

We still did build nuclear bombs. You know, if we're not intelligent, then I'm not sure intelligence is the thing we care about here. Well, what people say to give credit to this view is that they don't understand. Like we wouldn't call it calculator intelligent, even though it gets the right answer if you're trying to multiply 37 by 25.

In fact, the most human beings can't do 37 by 25 in their mind. And so there's something about the quality of understanding, of knowing why you're saying what you're saying. And I kind of agree that on some level, these things don't know why they're saying what they're saying. But on the other hand, I think one of the demos that keeps coming up that unnerse me the most is when the system gets asked to explain why a weird joke is funny.

And then they do. The difference between knowing why you're saying what you're saying and being able to give me a count of why you're saying what you're saying that is convincing to me seems like it's getting very thin. Yeah. Like maybe you can say, well, it could tell you a human who said the thing I just said would probably have been having this thought process.

And if we want, we can say, but that's not its thought process. It's thought processes this incomprehensible alien thing with no resemblance to the human thought process. It just gave you. I think that's true.

I don't find it very reassuring, right? Like whatever it's doing may be different from the account it can give, but it does have the skill to produce the account still. And I don't know that we can say that that incomprehensible thing that it's doing isn't really reasoning. I think that gets to one of the things that I actually find the most unnerving, which is that the more inhuman, the statistical correlation process happening with the set of these systems becomes, the more human the experience of interacting with the system becomes.

And that fundamental loss of legibility between we have some mental model when I ask you a question, I have a rough, not a perfect mental model of what's happening as you think about the answer. But as we make these systems more complex and they have more billions of connections and they have all these layers and parameters and nodes. And you know, if you try to get it to print out what it's doing when you ask a wide joke is when you get like a billion calculations that are happening that mean nothing to you. And so when I think forward 15 or 20 years, I think that's what unnerves me like a complete sort of loss of human comprehension of a lot of the interactions we're having and decisions being made around us and not just loss of our comprehension, but capacity to have comprehension over it.

This is some of the more scary and sort of depressing visions of the future you see from like holding Karnofs gear from a Jaya culture are like a world where AI researchers are thinking about the future and where this is taking us. And a description they gave that really stuck with me was decisions by companies about where to build new factories, about what products to invest in, about what kind of R&D to do are made by AI's. The AI's explain their answers to us perfectly. They write beautiful memos.

But we can no longer tell if the memos that they're writing and the decisions that they're making are towards goals that we have towards things that we want or if we've built some self-sustaining in human thing that is sort of doing its own thing now because we can ask and it will give us a beautiful answer and we just have no way anymore to tell if that beautiful answer is right or wrong. And is that I mean one thing you could say because you're getting here at a problem of deception, right? When you say that, I think there are maybe there are a couple versions of it, one version of it that I think you're pointing towards is we tell me I do something. We have misspecified what that thing is.

And so we think it's doing what we wanted to do and it's actually doing what it wants to do and trying to trick us along the way. But could you say it's true for human beings? I mean, businesses, our people constantly putting a factory somewhere and actually it was not that they got a kickback. It wasn't because they really wanted the factory there for economic reasons.

Yeah, I think that some degree of this thing has its own goals is like accounted for in the system. We are used to working with people who have their own goals. Nothing really relies on everybody being totally loyal to us because that doesn't work. I do think that the more alien something is, the less that works because some of that works on laws and anti-briery rules and oversight.

But some of it works on the fact that like very few people are going to build a factory that like pumps chemicals into the atmosphere that poison everybody alive. They just wouldn't do that. And AI might do that if it had a reason to do that. You can't just count on the fact that it's like probably basically a reasonable person and not an on a side.

I mean, yeah, because humans are all from the same share of values of priorities, of kinds of thoughts. Like you said, you can sort of guess what's going on inside my head. And even if you're getting similar behavior out of something that's radically alien, you don't know that its scope of action is the same, but it won't do things that you would never think of doing. So I want to go back to the pace of improvement and really this one of those conversations, I don't want to start shouting into the microphone.

But the thing I really want to emphasize is that I'm not sure most of us have any sense of how exponentially curve of improvement both is and is becoming. So when you talk to people working on these systems, how would you describe what they think will be true about them that isn't now in five years? And how do you describe that in 20? Noting that neither of those values are all that long scale.

Yeah. Like before our kids grow up tears things. So I think that a lot of people anticipate we are not very far from systems that can do any job you can do remotely, anything you can do on your computer. So coding, like secretary will work, receptionist work, journalism, everything like that.

I think that some of those same people would say we can invent better scientists and do science a lot faster and come up with whatever discoveries humanity would have plotted along to over the next hundred years, whatever those we would have learned from studies and research, getting it very, very fast from AI systems that can do that work overnight or in the space of days, just running much faster. And then I think some of those people are like, and then we get a fascinating cool Star Trek future where material scarcity is over, no one needs to work, no one has to work. And all of the goods of all of this AI work they've done are distributed to all humankind. And some people are like, if we did a poor job of managing this, of regulating this, of having oversight of this process, that is where we die or we get some other pretty catastrophic outcomes from effectively unleashing a civilization of aliens that we don't have a good way to work with or trade with or understand.

So I want to back up a little bit from the sort of utopia dystopia, just because honestly, I think you lose people here and I don't even know what I think of those, because it's on the scientific one. It's a big thing among the AI community that it's going to create a scientific super explosion. But most science that I can think of requires you to run a bunch of experiments in the real world. And I'm not sure exactly how much faster an AI really makes it to laboriously do phase three trials of vaccines on human beings or I can imagine ways it could speed things up.

But I do wonder a little bit about the potential for an overall estimation of analytical intelligence. But let me go back to something you were saying a minute ago, because I think it's closer at hand. When you say people believe five, 10 years, I think this is a fair description. Many people in this sector believe that within five to 10 years, you will have systems that can do any work you can do remotely that is not in the physical world, at least as well as the media and person doing that work right now.

That's a pretty profound both. I think we think about that as a possible economic change. It's also a kind of metaphysical change for humanity. Tell me just a bit about why people believe that to be true.

So you don't need to get all that much better than GPT four plus some careful training and some careful shaping of the models that we have and some additional work on reliability and stuff like that to get something, you know, it can pass the LSAT. It can also do a lot of what lawyers tell me is the sort of a menial first few years of law work of putting contacts together and reviewing them. It can program. It can program quite well.

It is not all that much improvement required before it can program better than the average person you might otherwise hire to program. It can write. I think my writing is better than it's, but I think my writing is better than it's by like a much smaller margin than a year ago or two years ago. And so at some point you have to be like probably next year, it will be better than writing than me or if not next year, probably in a couple of years.

And I think some places probably the New York Times will still employ human journalists. But I think a lot of places have already sort of tried to see if they can get away with automating huge amounts of work that used to be done by humans. Art too, used to be that humans would draw art. And now for many, many purposes that we used to hire humans for, AI art is where it's at.

And I think that you're going to see a ton of that. Think about this with like weaving. When we invented the automated textile loom, some people still bought like hand woven drugs because they really like the, you know, high end hand woven thing. And you wouldn't say that automated rugs are like better than hand woven rugs in every way.

But almost every rug in the world is an automated rug. And almost all rug work is done by machines. And I don't think very many people want to be like a niche cottage industry for people who prefer human labor while nearly all of our labor is better done by AI. To think about writing for a minute because a lot of jobs rely on writing for journalists, to lawyers, to for social media marketers, to for people who do publicity, to for just a huge amount of the modern, you know, what gets called the knowledge economy.

And once it metaphor that I've thought about in my own head, thinking about it in journalism, is you're going to move from the highly valued skill set, the more value skill set being out of the writer to being out of the editor, the person who looks over a piece of work, try to think about this true or not. This is something obvious. Just a structure of this argument makes sense. But just putting aside like what you think of those skill sets, just a truth that anybody who's in a newsroom knows, is that there is a small proportion of editors to writers.

An editor will often supervise somewhere between, depending on the organization, four and eight or nine writers. And that's true. I think in a lot of areas, I could go to social media marketing, which I know a little about, you will have a number of junior marketers who are overseen by one more senior person. You go to law, you have a number of younger lawyers who are seen by a partner, et cetera.

And on the one hand, so you have a skill shift towards management, but also oversight just by its nature, you are overseeing a number of agents for lack of a better term here. And as you were saying earlier, the key thing here is these things are cheap to copy. Yes. And so just when you think about just that, right, if it's just true, they're like, okay, editors, like we're not getting rid of all skills and writing.

We just need more editors and feel writers. Well, we only need fewer editors and we need writers. So it gets really quickly into land that will be pretty disruptive for people. And even if for some reason, or somehow we pass laws or just turns out that it who's needs a little bit too often to make this possible to scale people and to imagine, I still think it's going to shake a lot of people to feel all of a sudden that skills they spent their whole life generating may not matter that much.

I mean, I think it's unnerving. Yeah. And I'm like, right now, polarization style, that's fine. It's not there yet.

But it is what you would get if you had a ninth grader who'd read me for a while, right about polarization style, that's fine. And if it gets to where like me five years ago, it's going to shake me a little bit. Yeah. And we never know.

It could all Peter out next year. But my bet is that that point where it's you five years ago is not actually all that far away because it's a ninth grader now and it was the sixth grader a year ago and it was the fourth grader, you know, year before that. And I think that just the magnitude of the economic disruptions, like call centers, do a ton of customer service work, a lot of that is probably going to be automated in the next few years. And that's like a ton of high paying jobs in poorer countries where people speak English lost in places where those people don't necessarily have nearly as good alternative jobs.

And maybe some of these like massive collapses of various industries will be replaced by new industries created by AI. But I don't think there's like a guarantee in general economics that your industry being wiped out doesn't just leave you notably worse off. It can just do that. If you had specialized skills and then they're no longer valuable, it can be that you just have lower earnings for the rest of your life.

It typically does mean that. I mean, we know the shouldn't be economic literature. People sometimes pretend that economy suggests over time. When somebody's factory gets close, they do not typically get rehired for a higher which is not how it goes.

I think that something we were talking about this before the podcast started and I want to bring it in here because I think it's emotionally important. I was saying a minute ago that I've read for years the arguments about whether I kills us all or great. And I just, even if I can follow their logic, it is very hard for me to emotionally connect to them. Something in the past couple of months, the thing that has kind of messed with my head though was thinking about my own children and not whether there would be a world for them to grow up into just what kind of world, how different it might be from mine because I was looking at this.

I think for people who have seen poverty in the world, there are a lot of reasons why I can't come to you immediately and it will for some people obviously. And I do think that even if the technology moves quickly, societies adapt more slowly, there are regulations and rules and oversight and so on. But it just seems, if you look at this economically, if you look at just the possibility of people being surrounded by inorganic non-human things, if you like intelligence is to them, sort of the world of the movie Herr, which I was rewatching recently and I cannot recommend watching it enough at this moment. But there's a company called Replica that is explicitly trying to do this in a great piece in New York magazine recently about people falling in love with their replicants.

Those are going to get way better. I tried that out. My replicant was super boring, maybe better over time, but it won't be boring for long if you read this sort of Kevin Russe transcript with Sidney Bing, whatever. Like that was, whatever you want to say about it.

That was an interesting, like diverting conversation. I just think we're on the, like for my kids, like in my kids, not just life, but like growing up period. We're on the cosmopolitan world, if for them maybe profoundly different than the world I was prepared to raise them for. And something about that has been the crucial line I stepped over where I went from being like, this is interesting.

I like to read about it to holy shit. Yeah. No, I think that there's, there's a sea change and it is going to mean that they grew up in a world that in some ways we can barely recognize. And I think, I don't know, I am in favor of progress and change and technology.

And I mostly like Silicon Valley. I have all these friends here who have all these crazy hype startups and I like, I appreciate that world view in many ways. But I do feel this sense of ideally you have the frontiers pushed forward. We see some gaps that it's leaving, some awful things that are resulting that we adjust.

We regulate. We figure out where the problems are. You know, we invented a banking system and that was great. We realized the banking system had all these catastrophic failure modes and we try and learn how to patch them and how to manage them and how to keep failures manageable.

We invent nuclear weapons and that's terrifying. But we also over time develop institutions surrounding non-proliferation and norms and stuff like that. We came closer to a nuclear war in the early period after nukes were developed than recently. And kind of like if we have enough time, we can make almost anything that happens to us a good thing.

And the problem is if we have enough time, I don't think that that process can medically happen overnight. Let's talk about why this might not happen. So what do you think is the best argument for why this is a wall in your term? That we get to something like GPT-5, it's a bit better than what we have now.

What have you seen? I mean, driverless cars were supposed to be all over by now and you can drive around and see little way-mo cars, but they've all got people in them and they've not been able to get the reliability they really needed. We didn't replace all the radiologists like that was supposed to happen. IBM Watson's got functionally sold off her parts.

A bunch of these predictions have come over the past 15 years and they didn't quite pan out. And there's often a moment where people thought it was inevitable and then inevitability turned to disappointment at least over this time period. So if we plateau not far from where we are now, why do you think that will have been? Yeah, so my pessimist voice goes, okay, these systems are trained on all of the data on the internet and they are not going to be any smarter or any better or more discerning than everything on the internet.

There's some arguments that they could be because you need higher quality to replicate than just is necessarily present in it. But you could certainly say, no, they're just kind of going to be as good at making stuff as everything on the internet. And the average quality of everything on the internet is not outrageously high. It's not super impressive.

It's good at writing things that seem to flow for a minute, but there's often, and I'm not talking about like AIs here, there's often not a lot of underlying thought there. There's often not a lot of accounting for complexity. So maybe the best thing you get out of training in really large language model on the internet is something that can in like tiny fragments imitate a human who thought about it, but without any of the underlying cognition. And so maybe that means that those things never actually, they can pass a standardized test.

They can pass a standardized test. Amazingly well, we have to rethink standardized test. They can write a college essay, but that really reflects the fact that college students are not putting actual cognition and thought into their essays. They can imitate maybe the sort of low level journalism that's just right this event that happened, but they can't imitate synthesis.

They can't say anything original or distinctive. You could totally imagine that. It would not be very surprising. And does that change the world?

I don't know. I feel like it makes the world weirder, but it doesn't bring in some of these scenarios that are really transformative, I think. If it turns out that there is some degree of actual cognition that doesn't go into most human utterances, but is important for human work to be of economic value, then maybe it's not there. But when I say that, then I'm like, I sound like all the people who are like, it needs reflectivity.

It needs the ability to consider counterfactuals. It's not going to have those. And every time I've heard someone say, oh, yeah, yeah, you can't just feed it the internet and get reflectivity. You can't just feed it the internet and get logic.

You can't read it the internet and get coding ability. It turns out you can't feed it the internet and get those things. So I guess what I say to my pessimist voice is like, yeah, maybe. I find that view very compelling, though.

And I'm going to have to have a market on the show who holds a version of that. And I probably am like 30% where he is. But the place where I maybe feel like I differ from people who hold that view is that I think making something as good as the internet could be more important. And again, to what you're saying, I'm going to go weirder.

Like I really, really focus in on the word weird here, like my big common ad last year, was all about the word weird, my recent significant common ad was all about the word weird. I think that can make something very weird. I think it makes something economically quite important. If you could make something as good, I mean, you can do a good job because one thing about the big models is like GPT force and train to be a lawyer is just training on the internet.

But if you do the fine tuning in the area of making it a really good lawyer, it's going to be way better than GPT force, even at its current level of capability at being a lawyer. So I can see how you can make things as good as a pretty good lawyer, as good as a pretty good journalist, as good as a, it is a little hard for me to understand how you get to superhuman intelligence, training something on human data, both because it's functionally, I think, at some level averaging out human data and also because, and this does go to the point where understanding might be important. It doesn't really have a great way. It seems to me of separating what it has read that is true and false.

I think about something of the masking debate, whether or not masks work for COVID. And one thing within that is if you just inhaled every study I'm asking, putting it out any really profound priors about how to think about any of that, I don't know what you would get, but it isn't obvious to me the mechanism by which you get something way better than what we have because you wouldn't really have the capacity to run and use that yourself. If we just run all the randomized control trials and masking, we want it and maybe we should be doing that. We would know a lot more.

And also, how would you know what studies to believe and you just end up in-hocked whatever people have done the most of? I actually find that you very persuasive. I'm curious what you don't find persuasive in it. So I guess that might happen.

I feel so much uncertainty about AI and a lot of what I feel is that if you lay out anything kind of plausible sounding, I'm like, yeah, that might happen. I don't feel like we can rule that out. I also don't feel like we can rule out the crazy extinction scenarios. I feel like really any of those out would require a better grasp on what it is we are doing than we have.

So I'm like, that might happen. But my gut is that it won't. And if I try and dig into why not, I'm like, to imitate what we would say in a situation, you could say that takes less intelligence than it took us to say it. But I think in some cases it might take more.

Like, you have to in some ways be able to model a person in order to come up with what that person would say. And there are people who have lived their lives on the internet for the next 25 years, and you would think that just being fed that person would give you like a pretty good internal picture and like to build a generator of everything as your client has to say. Just a perfect predictor, something that just writes what you have to say. Something probably has to have whatever processes you have or processes that successfully imitate whatever processes you have.

And to the extent that your processes are true-tracking, to the extent that the thing you're doing involves noticing which sources seem reliable and reasoning in your head step by step, how would masks work? How would a mask study feel to tell us about masks? You can already get GPT4 to do that. It will give better analysis of mask studies if you tell it.

Reasons step by step, how good is this study? What are some things that might confound this study? And so I'm kind of like, maybe it can do that prompted, but it can't put it together and it can't use its synthesis of our reasoning to do its reasoning, even just to imitate us. But it probably can.

It probably just, in order to write something as good as an ezarkline column, which is something we're trying to train it to do, it will probably have to pick up a lot of the actual underlying process and not just the output of that process. This is one of the things I find most, again, just weird interacting with the systems, but it's now a highly replicated result. If you say tell GPT4 or GPT3.5, which is what I actually know, give me the answer to this. As if you were really smart.

It will give you a smarter answer. You mentioned reason, step by step. There are a bunch of ways you can do a problem. You've done a version of this.

Tell me this as if you have an evil other side and you're trying to hide it from me, which is to say that if you give it the instruction to model a kind of person, it can model that person. Again, not really trained to do that at any level. You can ask it to write in the style of being it will, which was a little embarrassing. I thought it did catch some things.

I do do that. A lot of m-dashes. That's weird to me. It's funny.

It's something that ends up happening in her, which comes from 2013, I mean, it's a minute ago. But the AIs themselves begin creating models of people from the past they want to be able to interact with. Just one thing that could just make the future weirder, even before you get to ideas of superhuman intelligence or anything else, anyone with a really profound written corpus that you can feed tens of thousands, hundreds of thousands of their words into a learning program that's already been trained on the whole internet and then tell it to particularly seem like this person, you might just be able to give people all kinds of fairly capable, mimicked human beings from the past and present. I don't exactly know what to say about that, but it seems strange to me.

Then if you do that, then like set the moods on physics or whatever, I don't know they get way beyond where humans are, but it can probably get way beyond where most humans are, which is already enough, I think, to destabilize quite a bit of our sense of ourselves ontologically. Yeah. Even in that sort of world where they don't get any smarter than us, there is just still some really radical, terrifying transformations from the ability to create human intelligence as sort of at will and customize them for almost any purpose that normal humans can do and run them for as long as you want and stop running them when you want. You could imagine like every student having specialized teachers designed to work with that student, customize to have the ideal personality for working with them.

You could imagine a lot of us have AI friends or AI lovers who are more attentive and don't have any needs of their own. You could imagine in some sector that out-competing normal human friendship. Personal assistants are obviously a big area where most of us can't afford a personal assistant, but would love to be able to say like, hey, can you schedule a doctor's appointment for me or have someone who proactively says, hey, you haven't been to the doctor in three years, you should really get on that. And I think all of that is coming, even if none of the stuff about building on that for greater intelligence happens at all.

When I talk to the people working on these systems, I'll often ask, what can you make it do that I can't? Because I think some people don't always realize that even the demos were being given are highly constrained in what they could do. And something I've heard from a few sources is that you can already tell these systems to accomplish something in the real world that you can do through the internet, but it has small steps. It needs to figure out how to run the car and do things like that.

And it can. It can come up with that plan and execute something in the real world. And sometimes it will come up with a plan that is not the kind you would think of or not the kind you would want it to come up with, but it knows enough. It's not just that it can give you an answer, but it can execute in action.

And that's already, but we don't know it. I mean, this is actually one of the things that worries me the most really about policy debate and sort of where we are in this. Even before you take into account the steepness of the improvement curve, what they already have is quite a bit beyond what we know. We only really have access to open AI and open AI derivatives like Bing.

We don't have access to what Google has built the public. We don't really have access to what meta has built. We don't really know a bunch of the systems being built in China. And we don't have access in any of these cases to complete open systems where they haven't been highly constrained for our use.

And here I'm talking in We the public or any kind of sort of mass sign up. The one moment I think anybody got access to a system that wasn't highly well constrained, which was like the Bing Sydney debacle, it really occasioned a moment of reflection and in many cases, I think fear. And that's just something going on here in the background that I just don't quite know how to convey, but we don't even know what we're dealing with now exactly. And there's a lot that's not yet released, even though places are doing a pretty fast release cycle, there's just inevitably part of an engineering process that you're ahead internally of where you're going to go externally.

So I actually think PBT four, which is now out is pretty close to the state of the art. I don't think there's like tons of more sophisticated models anyone has. It's hard to get side by side comparisons, but where I have been able to get them, DPT four is very good. But what there is is like applications that are pretty obvious that people have set up and have working.

Yeah, stuff like an integration of chat TBT or GPT four with TaskRabbit or something like that. So that it can just send instructions, pay a person to follow those instructions and get stuff done in the physical world, totally already done. Similar with like voice, well, Google had this years ago and kind of backed off of it, but voice that sounds like a person can call and make an appointment for you and sound totally natural on the phone and is an AI assistant making appointments for you. So when people are like, so if AI's wanted to do bad things, they're good thing, they're stuck on computers, right?

And it's like, I mean, we built them tons and tons of tools with which they can directly interact with the physical world in many, many ways and we're going to keep doing that. Hi, this is Ashley. I live in San Francisco with my boyfriend. We would love to officially share my near time subscription with separate logins.

We both love cooking, loving in the kitchen, but I'm a 30 minute and under efficient dinner early. I want a sheet pan meal. He is very elaborate. He wants to get into the storytelling.

I want to be able to save my easy meals and check off the ones that I've completed and I think him having his own profile would be great. Ashley, we heard you. It's why we created the New York Times family subscription. You get your own login and Mr.

elaborate and get his plus room for two others. Find out more at nytimes.com slash family. Tell me about people call the alignment problem. So we haven't yet had to grapple all that much with the fact that we don't know if these systems have goals and we don't know if they want things and we don't know if their actions are trying to accomplish things.

We don't super need to know that with existing systems because if you prompt TPT4 to write an article in the style of Ezra Klein or whatever, then it writes the article in the style of Ezra Klein. It's not doing the kind of open ended tasks where it might matter what its goals are and it probably doesn't have goals. I don't like making any certain statements about it. It probably doesn't have goals.

But when we build these systems, we want them to have certain goals. We want them to be helpful, not say racist things, not embarrass the companies that release them. We want them to be truthful. We want them to do what we prompted them to do.

We don't know how to tell them any of that. We're just doing it all with reinforcement. Like whatever it does, something we can say good or bad. Maybe with some more gradations in there, like how good or how bad.

But that's what we've got. We're trying to use that very blunt lever of reinforcement to point it at our goals. And depending exactly how you do this, there's a bunch of ways that what you teach it to do is probably not what you actually want it to do. Like you want it to be truthful, but you might be wrong in some case.

And so you reinforce it poorly for a true answer because you thought that answer was false. And so it learns, oh, don't be truthful with the humans. Figure out what the humans believe and tell them what they want to hear. Right?

Which is different. You were trying to teach it truth, but you were going from your own understanding. And it might notice that if it's has a little personality, if it's sympathetic, if it's nice or for that matter, if it's a little bit sassy, that gets better reactions than if it's sort of bland and boring. So maybe it develops and displays a bit of a personality because it gets reinforced and that's what it thinks we want to know.

And you can get a fair bit of deceptive behavior this way, right? If the thing that we are trying to teach isn't the thing that we actually want, and it is incentivized to give us what we want to see rather than like actually be on our side or whatever. How much of that is already happening? What are the question about deceptive behavior?

Good question. So the problem is that it's hard to tell, right? We know that the models say false things all the time. We know that they say false things when they know what the truth is.

We can like easily, you know, I can just go to cat TPT, get it to say something false and then ask in a different context. And it'll be like, yeah, that answer was false. We know that. Is that deception?

Probably not. Probably it wasn't like trying to trick me. Just, you know, giving the answer I thought I wanted. But it makes it very hard to tell when there is intentional deception with intent of like accomplishing some goal that the AI has.

And this is extra hard when we don't know if the AI has have goals. And some people will say they clearly don't have goals because how would something like that have goals? I'm just kind of like, I don't know, evolution was just like selecting repeatedly on ability to have babies. And here we are.

We have goals. Like, why does that process get you things that have goals? I don't know. They just showed up along the way.

Large language models are just like selecting repeatedly on like token prediction and then reinforcement. Does that have goals? Does anybody tells you like, yes or no, they are overstating what we know and what we can know. One of the questions here that I find interesting is this question of coherence.

Can you talk about that? The thing I think of when I think about coherence is to what extent does it make sense to say that I have goals? I like have lots of ones. Some of them contradict each other.

I want to get ready for my interview and I also want to lie in bed in other 10 minutes. I want to stay up late playing video games and I also want to be a functional adult who doesn't stay up late playing video games, you know. And humans act coherent and goal directed in some contexts. We have priorities and we will do things that accomplish our priorities.

But in other contexts, we don't. And you could imagine that if AI systems have goals, that they could be like that. We don't know if they have goals. We don't know how to tell what their goals are.

We kind of know how to make them work towards our goals, but only when the work is something that we can verify and we're eventually going to want work from them that we can't verify or like can't verify at the relevant scale. And that does dramatically change the safety landscape. But I don't know if we knew that evolved aliens were coming to Earth in the next 10 to 20 years and we knew that being evolved beings like us, they were incoherent and they had lots of competing goals and impulses. And you know, some of those were to turn Earth into resources that they could use to build more factories, but some were to keep us in zoos and some were to like leave us in nature preserve and some were to like invite us to participate in their geopolitics and some were to ally with us against these other aliens there in a war with them or whatever.

I'd be scared. I wouldn't be sure we would all die, but I would be nervous. That would feel like a bad situation, even though they're incoherent, just like we are. And even though they might be in principle be willing to work with us.

One of the things you talk about in one of your pieces is what people call specification gaming. If you imagine an AI that however we built the alignment, we can actually communicate with them, it can kind of understand us and it does care about trying to figure out what we want. I think that's one world, but we're going to have a lot of things that we really can't communicate with on that level. And in the near term specification gaming seems really worrisome, particularly if AI has become a big part of hiring processes or as we've already seen, predictive policing.

So talk about specification gaming. So say you do give your AI an objective that you're trying to have it maximized, which we do sometimes when we're like building game playing or whatever, or yeah, if you're doing a hiring process, you might train an AI where your training is aimed at the maximum share of the resumes that the AI passes through in fact, result in a job offer. We want this AI to be really good at filtering and correctly predicting who gets a job offer or whatever. And AI systems will be very creative sometimes.

That's what some of the examples of specification gaming we have demonstrate at satisfying that can straightly give them in a really out of the box way we would never have. Like if you're having to play a game and they win by getting the most points, they will do weird stuff like find a glitch in the game, they will repeatedly race in circles forever because they can get more points that way. They will trigger some kind of bug in the code which causes them to end up with a really big number that happened once. I find that kind of freaky.

AI just like figured out how to take some random action that caused the number to be really big. And we don't really like know how it noticed that. It may have like stumbled on it randomly. Who knows?

So we are building these AI systems and giving them these narrow objectives. I don't think this is how the world ends or anything, but it's certainly a dynamic that we're facing all the time where if the hiring manager who looks at the resumes is racist, then the AI will maximize its success probability by like never passing along a resume of the wrong race because then that won't be less likely to get hired. And that's like an alignment failure with society. The AI is like enabling a bad actor to do bad things.

But there's also ones where the AI is like not working for anyone. It's just accomplishing its objective in a way that would be unacceptable cost wise to everybody, but we don't have adequate oversight to notice that that's what it's doing. Like if you had a stock trading AI, there are so many horrifying things that you could get up to that like modify stock prices, right? You can figure wars to modify stock prices.

You can hack your competitors to modify stock prices. You can trigger financial collapses. And so I do think that even though this isn't the apocalypse scenario or whatever, whenever you're putting these like creative, deeply inhuman algorithms in charge of making some metric go up, you would hope we would be sensible enough to make really sure that that that was under lots of oversight and that it didn't have the power to do things in weird ways. And that's not what happening because that's never what's happening in AI.

You can say like, oh, well, that sounds bad, but it wouldn't be a problem if we just took these like three very reasonable measures to counter it. But we're not taking those three reasonable measures. Like ever, there's always someone who is just doing the naive, foolish version of the thing. And as the systems get more powerful, the failures get more powerful.

I actually think the stock trading example is something weirdly that I worry about a lot. Because if you think about what sector of society right now has functionally unlimited money to pump in building the absolute best or licensing, the absolute best AI systems that you can, well, hedge funds, other kinds of quant trading operations. Then if you think, well, also separately, what segment of American or global commerce is most used to using algorithms itself does not understand completely recklessly in an effort to get even the most narrow edge over competitor. You say, again, high speed and algorithmic trading.

And I mean, I can keep going down like it. You might say like, what's a good industry that's really good at wielding political influence? So people don't get in its way because it's really good at hiring excellent lobbyists to high speed. But what is an industry where if you actually had a really, really effective algorithm that people didn't care that much about oversight because they were trying to get it to maximize a particular metric.

In this case, returns might be able to cause a lot of damage to a lot of people. You say, oh, yeah, something around high speed and algorithmic trading. And I don't exactly know what we're planning to do about that, except as you say nothing. Because like the current theory of what we're going to do here is nothing, I think, which is mostly what we don't have high speed and algorithmic trading so far.

That's a little bit unfair, but I think that has gone way beyond where there's any social benefit to having algorithms competing with each other to do things nobody understands to exploit the small differences in prices at a given moment. But that's just a near term thing that seems very concerning to me. You don't need to have any kind of particularly far term views about like how AI will develop to just think a lot of damage can happen by unleashing powerful systems. Nobody understands in a place where they're trying to justify something that people really want maximized and may not think as they often don't think through the consequences of maximizing it.

Yeah, I think there almost certainly will be disasters in this space. And I just hope that they're at a scale we can learn from instead of at a truly catastrophic scale. And I also think there's a general principle here of we are creating these alien intelligences that are creative that think of problem solving solutions we didn't think of that do weird things that no human in their position would ever do because it's a route to their goal. Like people who play against them in chess or go talk about this how sometimes the algorithm will make a move that just no human would ever make and they don't see until 20 steps down the line how brilliant it was.

And I worry that in the world of try and make this company stock price go up that tendency towards like genuine creativity. Yeah, we'll take the form of oh I bet that if I like cause an outage at this power plant then stocks will move in this predictable way that I can make a lot of money off of. And if you don't have oversight stuff like that happens. Or if you don't even know how to have oversight.

Because you can't see what it's thinking because it's not thinking in any way we understand. This gets to something within the culture of this community that I want to talk about. You cite and I've cited in pieces too this really unnerving survey of machine learning researchers from last summer of 2022. It asks people basically what they thought the chances were that working on AI if they got to high level machine intelligence the consequences for humanity would be really really bad like human extinction bad or some other complete displacement of humanity.

And the median response was they thought there was a 10% chance of happening. The people working on this said yeah 10% this thing that I am in somewhere and other trying to bring in being or trying to worry about how it comes into being. I think 10% chance humanity is completely pushed off the board in a fundamental way. And you could say that's just a survey, but I've talked to enough of the people at high levels in these companies to know that that survey is not that weird.

Yes. My impression also is that that survey might be something of an overt statement, but the position there is not rare. There are plenty of people who tell you, oh, yeah, I think this is it. I don't know exactly how to communicate to people, but I spend enough time in this community.

And you spend more than me that these systems are being designed by people who themselves do not believe they really know what they're going to do. Rust out that my column is talking about it as a kind of summoning. And I've also had that thought that this is like people opening a portal and they will freely tell you that like they hope a demon doesn't crawl through, but they can't guarantee it even doesn't crawl through. And even if you don't believe they're right, like you believe this is crazy or maybe it's type, they believe this.

Yes. And that actually worries me a lot because you talk about value alignments. I think a lot of people might say, if the thing you're working on has a 10% chance of destroying humanity, don't or go really, really slow. Some people do believe that, but there is like a lot of now competitive pressure to speed up to be the first one there's pressure between US and China.

So geopolitical pressure, you know, Google is not want to lose to Microsoft, which is not want to lose to meta. Can you just talk a bit about how you understand the culture here and like the psychology of working continuously to bring into existence something that you know, you don't understand and where you think the consequences have a really, really high chance of being unacceptably bad. Yeah. So I want to second what you were just saying about how difficult this is to convey because you have these conversations and you get this like somewhat terrifying sense of that people understand themselves to be making extraordinarily high stakes decisions, decisions that I think the average person on the street would say that should absolutely be the prerogative of a government with a lot of careful consultation, maybe some kind of international framework.

And you have people who are like, we're going to do it, we're going to do it ourselves. We thought about it and we think it's going to go best if we do it ourselves and we do it now. And it's like very hard to step outside of that bubble and simultaneously say these risks seem real and not sci-fi and scary. And also, you know, whatever you think about that, the people who are doing this do think that there's something real here.

So there's like, why are they doing it? And I think some of it's just the love of science, right? Like there's I've been reading the making of the atomic bomb and recurring like the. Oh, yeah.

Are you reading that? I, you know, I have two young kids at home and another one on the way. I never have time to read. And so I was way behind on my book list and then I got the invite to come on your show.

And I was like, this is my excuse to read all of the books that I'd been wanting to read. But now it's for work. I have to read them. So I have something to recommend on as is podcast.

And that is how I finally got some time to read the making of the atomic bomb. And one of the recurring things in it is it's a very personality driven compared to, I guess, what I was imagining before I picked it up. Like, who were these men and what were they thinking? And it does feel like there's a kind of person who just once they realize it can be done there's an overwhelming urge to do it.

There's just a one of the points that the author makes at the beginning of the book is that we might dream of a world where physicists said, on principle, we will not build an atom bomb, but we were never close to that world. Physicists would build an atom bomb as soon as it occurred to them that you could, you know, and there were some very brave and very notable people who tried saying like, this seems really bad. What should we do about this? But overall, if you want to stop dangerous technologies that has to be on a structural level with like governance and regulation and international agreements and voluntary between lab agreements, you can't count on people, unfortunately, to just say, this is ludicrously dangerous.

We're not going to do it. Because I guess some here of people are just like, this is ludicrously dangerous and I'm going to do it. I guess this is like the classic tale of hubris in some ways, like the oldest known fallen human nature. You see the tree of knowledge and you're like, yeah, I'm going for it.

It's kind of a selection effect too, which is to say that if you were somebody who thought this was such a bad idea, you definitely wouldn't do it. And you don't do it. It's not the person doing it. And there's a lot of incentive for people to believe the arguments that they should do it.

Yes. Right. I think this is something and we should talk about it that there are a lot of people who are going to AI safety and who are around this community and spend all the time thinking about reasons, maybe not to do it or reasons to try to make it, you know, slow it down or how to make it go more safely. But in the end, they're not the ones who run the companies who get the investment that gives them the compute power to do it.

There are so many people, very good machine learning researchers who I know who nobly made the decision to not play this game because they thought they might kill us all. And so they are out of this game and they are, you know, working in academia on much smaller models or working on regulatory frameworks, they're doing good work, but they're not the people at these labs because they said no, and they walked away. And so we have the people who for whatever reasons didn't walk away. And so I think within those, you can cut a couple of groups, right?

So Jan LeCun, who's an important AI pioneer and he's one of the key people at Medi, I believe he's the third chief AI scientist, he just thinks it's all dumb. And that an AI has no interest in taking over or killing us all or whatever. So it's just not going to. And this has all gotten very far fetched.

Then you have people who run a couple of other labs, maybe Sam Altman at OpenAI would fit into this framework, folks over at Infrappec, I'm in a different way. I think they're more careful than OpenAI, but who believe that AI could do great good and do believe that there's a chance of really bad things happening and think that by working on the models and experimenting on them that you make it both likely or you understand about things, you can prevent them and also maybe trigger society coming in and saying, okay, here's how we're going to regulate this early enough. And I don't think it's a crazy view. I mean, I do think that there are more conversations happening about how to think about and regulate AI because OpenAI has been releasing models into public use that Google and Med are not.

So I don't think it's a crazy view, but they are all now in a race. And I worry that the race dynamic is just a lot stronger than the caution dynamics. And this is something that I know you've been writing about and thinking about. So can you talk a bit about this question of the race?

Because something I know has been worried about in AI for a long time, but I think it's undeniable now that it's happened. Yes. So there's a couple of different things that drive getting new powerful models, right? One is how much money tech companies are willing to sink in and how much talent they have working on it.

And race dynamics often mean that these companies are willing to dump a ton more money on this because they want to stay ahead of their competitors. Microsoft is willing to spend billions and billions of dollars for a shot at Ketching Google, and search Google is presumably willing to spend billions and billions of dollars, defending its spot in search for the same reason. And so you get bigger models faster. And that's like one thing you can analyze.

Like maybe it's good to have these bigger models faster under these race conditions because then we're here talking about them. We probably wouldn't be here talking about them if this was still all under the hood because it would be a little too crazy. But the other thing is that when you're going fast, safety becomes a liability and things that you could do to make your models less dangerous or to check more thoroughly if your models are dangerous, work you could do on like mitigating risks and even just understanding them goes out the window because you're trying to beat someone else. And you want it in commercial production as fast as possible.

We would be way safer against AI takeover scenarios, really scary catastrophes. If AI systems didn't have open access to the internet and the ability to contact anybody they wanted and do anything they wanted on the internet. But we've given them that. Of course, we've given them that because it was like slightly more profitable than not giving them that.

And so as soon as you open that door for everybody to be competing with each other, then the people who are careful and cautious are slower than the people who are careless and incautious. And the people who are the most willing to throw safety considerations out the window are the ones who are getting their biggest models out fast. And that puts us in a situation where even if it turns out to be totally possible to make AI systems safe, like there's a very straightforward technical solution to alignment. We know what exactly what we need to do.

We still have a problem because somebody can just decide, yeah, that would be a little expensive. That would delay our launch by two weeks. That would require us to hire another engineer like my partner. So when I search for any AI app that I might want to check out in the App Store, one thing I always noticed is that there's the app I was looking for, and then there's a bunch of fast follower apps where I could eat it.

And my sense is that the kind of sub-premium AI models, a couple of years ago, were just total crap. Yes. And now you're looking at things that are not coming from OpenAI or Microsoft or or Google. But they're as good as things were a couple of years ago, which is like, now not that bad.

And then in a couple of years, they're going to be as good as things are now. And if you are OpenAI, you have a really, really high reason, because you have so many awesome ways to make money here, to make sure you're restricting use of your model to only things that are going to get you depressed or something like it. But you have AI's that it now seems you can train them on some of these vocal patterns pretty easily, and it can then call my mom, because I've got a lot of my voice out there in the world. You can train down on my podcast and it could call my mom, and it could sound like me having been kidnapped and eating a ransom.

This has been done. So, sorry. It's already happening. And these are getting better faster than you actually don't, for things that are just kind of bad.

You don't need the best system in the world. You can increasingly just use a kind of crappy system. And the crappy systems are like the best systems in the world from a couple of years ago. That's just going to keep happening.

And that seems, again, like something we don't really have a framework for thinking about. We don't really know what to do about. But I wonder if you could talk about that, the opening up, not of the state of the art, but of the sub-state of the art, to functionally every scammer, advertiser, just anybody who wants to make a buck in the world. And the kind of CD under-belli-ai, in the same way that happened to crypto, though maybe that was like the over-belli or something, it kind of started there.

But the CD under-belli-ai strikes me as a place where a lot of damage is going to be done really soon. if it's not already being done. Yeah, and this is again something where I'm like, if things are moving a little slower, I'd be like, yeah, there's some scary stuff and we'll adapt to it. And people will learn not to trust calls from their kidnapsons.

And we will get better at spam filters to combat spammers. And we will get better at making comment sections on the internet functional. I do get the sense in the last year that spam is ahead of spam detection in terms of like- Really seems to something super changed. I feel like we went through this golden period where spam detection was better than spam, and you mostly didn't encounter spam all the time.

And it is over tragically. And now I get a lot of spam on Twitter, in my email, on Facebook. And I'm like, I guess we are back in the domain where spam is better than spam detection. And if you have enough time, then I think it's possible for people to develop the social technology and the literal technology to counter that kind of thing.

But when everything's moving really fast, you know, people learn how to address one thing, and then we get the AI that can like carry on a long convincing conversation with your mom and convince her to send her bank details in the course of like, this very long conversation that also references a bunch of details from your childhood that you mentioned on the podcast once. That's harder. You know, how do we get to that point where it's easy to cause so much damage and so expensive to detect? And again, I'm just like, I wish we were moving a little slower.

I wish we had more time to like figure this stuff out. Well, you say that like it's impossible. What's the actual difficulty? I mean, we have made a lot of things slower.

You can't clone humans and we don't, even though it is a possible thing to do. And I'm sure there's scientists who like to clone humans. We've actually been quite good at maintaining an international ban on cloning human beings. There's all kinds of biological weapons research.

We just like don't really seem to do anymore. We did it for a while and we actually used them and then we have kind of non-proliferation packs that I'm not saying they are perfectly effective, but they are effective. And you can come up with a bunch of different things like that. There's certain forms of genetic modification.

Just don't do. Cut your grace. I thought how to do research a really good piece on this about how the AI community has a little bit of this weird tendency to say, well, maybe it'd be good if we slowed down, but it's completely impossible to ever coordinate people such that they slowed down on anything. And her point, which I think is true, is that that's actually not true.

We coordinate people to not do things or slow down on things all the time. Yes. And somebody seems to me the biggest barrier here is actually that people just want to rush for it on AI or match by the people who don't think that rushing for it on AI matters. It's like not that much interesting.

It's like telling anybody down because what? It's like toys on the internet. But I don't know. I do think there's a problem here where like the AI community seems to have convinced itself that it's so hard to get people to do anything that you couldn't possibly coordinate them around this.

I don't see why that would be true. Yeah. So I think we can and should slow down. It's like very hard when things are moving this fast to get a good feel for the possible.

Like you could imagine a fairly sweeping set of government regulations in the next few years that set a cap on how large you can make a model unless it is subject to some form of democratic oversight and extensive testing for safety where we check things like can this send a convincing phishing email that persuades 1% of users to run code on their computer if it can. You can't generally release it yet. Can this model like give detailed instructions on how to carry out a terrorist attack that if followed would be successful? If it can, you can't release it yet.

There's all kinds of scary things that are not very far off. And we could just draw a card cap and be like, look, if the model in an audit of some kind with dedicated testers whose job is to elicit this behavior, if the model can do the dangerous thing, no, you cannot deploy that model or you're liable if you deploy the model, which may be amounts to the same thing while we're talking about big companies. We could do that. We could do that tomorrow.

I think there's plenty of people in Washington who are thinking about something like this. And I think what's possible, you're so right that it's a social question. It's a question of what everybody else believes. If everybody wakes up and goes, we want to have more steering power over the world, our kids grow up in.

We could just do this. There are like very reasonable rules you could put in place that would significantly change the pace of this work. And I hope that happens. I'm not pessimistic.

I've heard from enough people that they want something to change, that they're scared, that they believe it can and should go differently than it's going, that I'm like, that's what it takes. I mean, we can make something happen. But I am kind of, the default thing that happens is that the regulators are years behind the ball. You know, you need to do a very dedicated, very concerted effort to set up a system where regulators are on the ball.

It's a hard problem. It's a hard problem I think we should be working on, but it's a hard problem. Tell me a bit more of that list, because I do think one thing that makes problems harder is when the suite of possibilities might do about them feels opaque. So a lot of people don't understand that much about it.

And they really don't understand what you might do to regulate it without completely decapping it. Right? You're just saying you can't research language models anymore. I mean, that seems crazy.

I think it would actually be kind of crazy. So is there a rough consensus among the safety people you talk to about these are the three or four things we really wish would happen? I would say there's an in progress effort to build that consensus. I think conversations I heard this year that I did not hear last year included, like what are some clear cut benchmarks like send a phishing email that persuades one percent of people who receive it to run code on their computer.

Like that's a pretty clear specific. It's obvious to anybody who's thought about policy a little bit why that's scary behavior. It's pretty easy to use an auditing process to determine whether a model has that capability. You'd have to send some phishing emails.

So I guess there's ethics concerns, but I feel like a regulatory framework can work with that kind of thing. And so that's conversation I didn't hear last year that I am hearing this year. That gives me some optimism because I think that's what it would look like to have regulators be on the ball instead of behind it. I do think that people who are working on this effort are coming from all kinds of different places.

Some of them are like, well, what I really want is for all of this to stop. I can't get that. So I will try and set some benchmarks that have an achievable chance here. Some of them are like, I really want to be doing my amazing AI research, but I want these safety people off my back.

So I'll agree to some standards that I can meet that don't get in my way too much. And a lot of people have different threat models and different pictures of which AI capabilities are going to be the scariest ones in the near future or in the medium future. So it's a difficult process to build that consensus. And I think we're still in the early stages of it.

What are a couple other specifics that you think are interesting? They don't need to be consensus, but you just think they are provocative. So if an AI system can via emailing instructions to a human assistant, get a copy of itself spun up on Amazon Web Services, independent from its creators, and so that another copy of it can run on AWS. That seems like a benchmark.

Like you can define that behavior in an audit. You can see whether the model can in fact do it successfully. And if it can, then that's like an indicator of some of the really scary scenarios where models decide to operate independent human oversight because that lets them achieve their goals better. So that's maybe a benchmark.

If your model has the capability, when prompted to give a human instructions on how to copy that model and run it on AWS independently, that's a scary model. We don't want you to widely deploy that model. When you say when prompted here, one interesting thing to me about that example is that I think there's a set of behaviors or capabilities where like we're worried to humanly use you this way. Then there's this kind of further off.

Ideas that like we think you might act this way, you being the model. And in that world, if we think the model develops situational awareness of itself and its world, such that it's been prompted by some idiot human like, Hey, do you mind emailing a copy of yourself to the Amazon Web servers to the replicate in the world where you think. That the eyes become deceptive. How well does that actually work developing this?

Auditing process is very tough, right? Like it's a very technical, difficult work to figure out how would we elicit this behavior if it is possible? But I think that in general, it is easier for a model to display a skill when it's elicited by humans than unsolicited. So we will probably successfully identify models that are capable of giving a human instructions on how to copy them to an AWS server sooner than we will get models that can do that and think of hiding it and successfully hiding from us.

Now, that's a gamble. Maybe we do our tasks and we're like, Oh, it's good. These models can't do it. And in fact, these models have past the threshold where they can do it, but suspect it's not in their interest.

But that feels less likely to mean that requires the model to develop two skills at the same time, like the deceiving us and the capability to survive and spread itself or whatever. I feel a lot more optimistic if we're consistently checking, Hey, can you do this? And then halting things and calling red flag once it can do this. I think that will probably be before it develops the ability to do that deceptively without our knowledge or awareness.

So let me make a critique of the AI safety community here, which is I don't want to say people are too focused on the absolute worst that can happen. That's a good thing for some people to be focused on. But I find that there's an unbelievable lack of focus on what seems to be incredibly important in your term question, which is the business models behind these models. And I think in general, the sort of community that comes out of is a little unreflective about capitalism, right?

Frequently asked questions

How long is this episode of The Ezra Klein Show?

This episode is 1 hour and 34 minutes long.

When was this The Ezra Klein Show episode published?

This episode was published on March 21, 2023.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Ezra Klein Show episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.