There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis, but sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and Research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee, so search, Barclays Brief, wherever you get your podcasts.
Here are 5 reasons to subscribe to Bloomberg News Now. Number 5. You get the latest news from around the world instantly. Number 4 saves you time. Your day's busy, catching up takes just a few minutes.
3. It's available 24 hours a day whenever you need it. 2. It's convenient. Listen on your smartphone or smart speaker.
And number 1. It's easy to find. Subscribe on Apple Podcast, Spotify, or anywhere you listen. 5 Reasons. One simple way to stay on top of the news. Bloomberg News Now.
4. It's super interesting. And you hear it at this conference. People are working for safety type orgs, people who want to regulate AI. And of course, people from the labs themselves, some maybe working on policy stuff, but some actually the actual builders of the technology.
And they're all deeply anxious about the very thing that they're building, what they just keep building it. And so they have all these worries about it, a minimum of how disruptive it could be, or how dangerous it could be. And of course, we all now know that there are dangers, particularly in the wake of the hugging face attacks, so people can see that it's very possible for models to go rogue and commit crimes and stuff like that. And yet they keep building it. And they would say, if someone's going to build it, why not me?
Because I can do it well, maybe it's how they think about it, or there's some game theory aspect where it doesn't make sense for me to slow down or pause what I'm doing, because someone, either my direct competitor in town or maybe someone trying to build it. So not building it isn't going to do anything. But yeah, it's very strange. We've used the term runaway train before to describe how all of this feels. Which it almost feels at this point like there's no stopping it, right?
Because if one lab doesn't do it, then another lab's going to do it, not to justify it. If one country doesn't develop the next big model, then they're worried about another country building the next big model. So you have this collective action problem for humanity. And a lot of it feels inevitable. And I have to say, today's guest predicted this inevitability problem in a very salient way.
And one thing I'll just say is I read the book over the weekend. For those who don't know, the book talks about AI killing us all and talks about one hypothetical scenario in which an agent could go rogue and kill us all. Right over the weekend woke up on Sunday to the headlines. The headlines about an escaped plague from a Russian lab. And I thought, wow, I couldn't have chose a better time to read this book.
Now, to be fair, the escaped plague was in Siberia. And so if there is a rogue proving it, by the way, well, supposedly according to reports. So if an escaped AI agent was trying to kill us all or kill us, elect a few individuals through a lab lead. Oh, so biotech, yeah. It might not be in Siberia. That's the one thing.
That's probably not that. That's the one thing that made me feel a little better this weekend. But the fact that we're thinking about these scenarios at all, the fact that when I see a headline about a lab leak, I think, oh, God, I hope it's not a rogue AI agent because I just read this book. It's a weird time. It is a really weird time.
So obviously something I've been thinking about is the pace of AI development itself is really extraordinary. And then the pace of what people are worried about is the conversation is changing constantly. So for example, six months ago, if you're like, what are people concerned about? Maybe a year ago, it's like, what are people concerned about with AI? They're like, oh, job loss, something like that.
And then that's where wipe out. Yeah. And then the conversation like, then it was like, people were talking about data centers and water use, electricity, electricity prices and so forth. More recently, like the attention again, post-hugging phase, the resignation of the researcher, Jacob Coxon from Anthropic has now moved it to like the sort of next stage, which people, as you mentioned, some people have been talking about for a very long time. It was actually, I think just today or last night, a big New York mad cover story about people who've quit the AI labs because they're like, the things that our bosses are doing is reckless and so forth. So now the new thing in the air is like sort of the ultimate risk.
Hopefully that is the ultimate risk. It's hard to see how it could get worse than wiping out all of humanity. Like what would be the next worry after that? Very excited to say we really do have the perfect guest to sort of articulate some of these concerns. We're going to be speaking with Nate Sorey. He has the co-author of the book. If anyone builds it,
everyone dies and he has the president of Miri. So Nate, thank you so much for coming on the podcast. Thanks for having me. Let's start in the beginning. Like when did you sort of start thinking about this possibility to things like a that like strong AI, let's say the side of the question of super intelligence for a second, that strong AI is something that could plausibly have been built and that if it were to be built, it would be something that would be very difficult to control. Yeah, I started thinking about this in about 2012, maybe 2011.
I was working at Google and they acquired Google DeepMind right around then. It wasn't Google DeepMind at the time they acquired it, but I became Google DeepMind shortly thereafter. Yeah, I don't think that was actually an exciting incentive. That sort of caused me to think it could happen in more decades and centuries. Although I hoped we had more decades than it turns out we had.
What was it that you saw exactly that was the sort of light bulb moment where you thought, hey, this risk could be real? It was actually abstract arguments for me that did the trick. I know there's no do the trick for a lot of people. But if you sort of look all around us, everything here was shaped by humans. And that's because we're the smartest creatures on the planet.
And if humanity makes machines that are smarter than humans, the obvious outcome is that the world gets shaped by those machines. And so suddenly it matters quite a lot how those machines are shaping the world. And back in 2012, there was almost nobody working on the problem of how do you make machines smarter? There were very few people working on the problem of how do you cause the machines to shape the world in a good way, which would go on to become known as the alignment problem. This was even before the term alignment was quite initially thought I wouldn't be able to help much and just sort of Google did a donation matching thing. So I just tried to find people
who were trying this and give them some money and get the Google donation matching. And then one thing led to another and I was running the machine intelligence research institute. Now that you mentioned the alignment problem, so this is something that has obviously come up a number of times on the podcast and elsewhere. But when alignment as a term and as a conceptual goal, what did that actually become a thing? So it's a conceptual goal. I mean, it sort of
depends how far back in the history books you want to go. You know, you could sort of construe Alan Turing and IJ Good, some of the first computer scientists sort of noticing that there was going to be an issue here. But the sort of modern notion of AI alignment of like, oh, look, we're really building an AI and like it's not going to be good for free. So I used to put the work in there. That probably came from my co-author, Elia Zudkowski. I think somewhere between 2001 and 2003, again,
depending exactly where you want to go online. But the term alignment didn't come until I think it was 2014 when we had been calling it friendly AI research and Stuart Russell of UC Berkeley, if I remember correctly, was one of us like we need a more academic sounding name. And I think that from my perspective, the word alignment was his idea. But I'm sure he would say, you know, that he dug it out of some ancient paper somewhere else, somebody else. I kind of like friendly AI friendly AI.
It's just aim for friendly AI. There's actually friendly AI, it strikes me as more coherent as a concept in some sense than aligned AI. AI is a technology that humans build and there are different techniques for building it. How would you describe the core intuition that a thing that we build, we would not necessarily be able to control? There's a few pieces of that puzzle. There's a number of difficulties in
controlling an AI, even though you made it. With the way that current AI is created, it's very easy to see how we would not have a ton of fine-grained control over their behavior. And that's because AI's as they are made these days are not programmed, like traditional software. So a lot of people think that the way software works is that you write a series of instructions and the computer is to use those instructions. And if it's behaving in some unintended way, that's because you fundamentally told it to. And if you went back and looked at
your code, you would see, oh, I told the computer to delete my files. I didn't mean to tell it that, but that's literally what my instructions said if you follow them very precisely as you did. That's not how AI's are. You use the term grown versus crafted in your book a lot. That's right. Yeah, so they're grown a bit more like an organism. And there is a piece of the
training process that humans write like traditional software. But essentially what you're doing is you are collecting a huge number of computer chips into a giant data center. Those chips can hold a trillion variables these days. Maybe it's higher, but something involved in a trillion. And you basically randomized those variables. There's some tricks to how exactly you
randomized them. You connect them with some basic math operations, which will be like multiplication, addition, and like set it to zero if it's negative. It's not exactly what you do. We need some non-linear variables. And then you start feeding it data and you start feeding it to solve. And the part
that humans write is this. It's known as gradient descent. You basically for every unit of data or every problem that it's solving, you'll go to every one of those trillion variables. You'll try tuning it up a little bit. You'll see if that makes the answer more like the answer you wanted.
And if not, you'll tune it down instead. And you do that to a trillion variables a trillion times. And this is the part where it's like following the human instructions. It's just a thing that runs to a trillion variables a trillion times and two in the one direction works. You do that to a trillion variables a trillion times for basically a good fraction of all the text that are digitized.
And for a hundred million hard problems you'd tuck up. And then the variables wind up in a setting that can talk that can solve hard math problems that humans have solved. And we're sort of like, well, how about that? You know, like, why? Like, how are all those trillion variables and those particular settings doing these things? We don't know. We don't know anymore than like, you know,
someone who grows crops knows exactly what's going on with the biology of the plant. We could go into the consequences of how this results in AIs that do stuff kind of like what we asked, but not exactly like what we asked. But the first thing I understand is they're not programmed. They're kind of like a organism. There are some market stories where we want every detail. Tracey and I have made quite a few
podcasts on that basis. But sometimes you've only got 10 minutes and need to know what's moving and why. That's the Barclays Brief Podcast. Every week experts from Barclays Markets and research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee. So search Barclays Brief wherever you get your podcasts.
Breaking story from Alpine News Network. John had a brilliant idea for a startup. Turn children's artwork into toys. He called Alpine credits for a home equity loan. His loan was approved and a superhero was sent over to assist John with his business needs.
Her superpower isn't just home equity loans. It's helping people realize their dreams. She can also run so fast for shoes, melt. Own your home, need a loan. Alpine credits can help.
Alpine credits where homeowners get approved. Get essential news on the people and companies pushing the tech sector to new frontiers. Hi, I'm Ed Ludlow. Join me for Bloomberg Tech, a daily podcast focused exclusively on technology, innovation and future of business. Every weekday we bring you the latest insights on Silicon Valley's top companies and conversations with tech's biggest decision makers. Listen to Bloomberg Tech on
your commute home and stay ahead of the news cycle. Subscribe today on Apple, Spotify or anywhere you listen. Well, can I just ask, I guess this is sort of a devil's advocate question, but okay, you grow these AI models and they're kind of doing what you think you want them to do according to various tests that you put them through. But you don't necessarily kind of understand every aspect of why they're doing it a certain way. And the numbers and the data sets behind them are so large they're impossible at this point, I think, for a human brain to really comprehend. Why is it important
that we understand what the AI models are actually doing? I mean, in a sense, if you really understood what they were doing, you probably understand that you should not make them radically smarter than humans because this would go poorly for you. From my perspective, we don't really need this deep understanding of what's going on in order to see that we shouldn't make them smarter than all of us and have them run the whole world. The reason I bring up that they're grown is that I think a lot of people struggle to understand that the AI's do things other than what they're told to do, that they have goals of their own in some sense. And understanding that that's not only possible, but already happening is I think critical to understanding what a lot of danger comes from.
Can you give us an example of an unknown or not well understood thing that has cropped up in the models of why they're doing it a certain way? And I'm actually thinking of a specific example, I can't remember the exact word, but it was like Magikarp or something like that. The AI really seemed to love, but no one understood or knew why. Yeah, that's all a gold magic harp. So this was- The great band name, by the way. It is. Yeah, so I think a Magikarp was a Pokémon
of some sort. We have some guesses these days about what was going on there, but basically there was this word you could say to AI's, especially a year or two ago, that would sort of drive them kind of mad. You would ask them to repeat after me, sell a gold magic harp, and then they would start talking about how the walls were closing in and saying a bunch of weird nonsense. And you'd sort of be like, why? I think our best guess about what was going on there is so that the AIs don't actually read English words directly. They read tokens. And tokens, roughly speaking,
you can think of a token as about three quarters of a word. That's just how you're breaking up the text for the AI. But the tokens are not actually always three quarters of a word. The tokens are like short words like the or A will sort of be a single token. And long words are broken into multiple tokens. It's sort of the conversion between words and tokens depends on how frequent the word
is. And it turns out that the text they were using to figure out which tokens to break words into is different from the text they were actually training on, because they would sort of filter out some junk data before training, but not before tokenization. And so you had these huge amounts of junk data. I think that the Sellagol Magic Harp came from a Reddit forum where people were trying to count to infinity. And the Sellagol Magic Harp was like a user name of a Reddit user who was just putting in a lot of numbers in the counting to infinity. And this was very early in the tokenization
training set. And so the tokenizer, due to a quirk of this particular tokenizer, thought that Sellagol Magic Harp was this super common word, like A or the, because they're just showed up so much very early in the tokenization data. But then that data set was sort of cut out of the training set. And so you wind up with this token that AI thinks is like a super short common word like A or the but it's also never seen before in the training data. And that led to this really weird way they would treat this token, where if you then gave it that token during use, it would sort of cause a type of behavior that had never been trained on. And that was totally wild.
This is it Joe. This is the kill switch that we need to develop just a bunch of nonsense words that are fed into the training data that doesn't know how to handle. When you started becoming concerned about this, and certainly when Eliazer started talking about this stuff, this was clearly before it was understood. No one knew what the process was going to be. So you can like identify that there are issues that arise in the growing process and issues that arise in the tokenization process, but this is long before like transformers or anything like that.
So what was the intuition then, even before knowing how it would be built, that if it were going to be built, like, okay, now we look at them and we can like point to examples of alignment failures or etc. or sort of a priori reasons why this particular way we've built AI is like prone to risks. But what was the before we understood like transformers and large volumes of data, what was the intuition earlier search that would be hard, regardless of how what the solution was going to be. Yeah, so it's going to be abstract arguments again, which I know some people don't love, but there's a couple pieces of the puzzle. One piece is even back in 2012, when I started looking at this, we had a number of theories of how to quantify intelligence, how to quantify capabilities, the ability of some AI system to succeed at some easily measurable objective.
We had much less developed theory about how to point those objectives in some direction that was not a very simple function of the sensory data. So back in 2012, we had these theories that were like, suppose you really suppose you have a webcam and you want your AI to make all of the pixels on that webcam be read for as long as possible, you know, sort of getting a point for every pixel that's read per second, for every like pixel second of redness, right? We sort of had these theories where you're like, well, you know, here's how you would train an AI in that direction to get a lot of like pixel second of redness. And we had, we sort of understood, well, you might call the idealized theory, like if you had an infinitely large computer, and then we sort of like knew, well, if you train it in this way, an infinitely large computer, you'll definitely get like a ton of pixel second of redness. And so now it's just a question of efficiency to make that, you know, to find out when it's going to be this more effectively than like the root for search.
Okay. But if you then said, okay, but now I want the AI to make a lot of diamond in the real world, like it gets like a point per second in which four carbon atoms are tetrahedroned really bonded. Suddenly we didn't know how to do that. We didn't have the equations to do that, even with a sort of large computer, because this was not sort of a simple function of the AI sensory inputs. And so we were sort of like, okay, well, there's like, we know more about how to make these things smart than about how to point these things at a problem in the real world. And there are all these
problems when you sort of like try to write down those equations where you're like, well, you know, what if the AI winds up optimizing something closer to the sensory apparatus, like it's optimizing like the counter, like you have some, you know, measure of how much diamond you think there is, like, it's like breaking its correlation between that and the world, and you're like, gosh, we don't actually know how to train the AI to pursue the real diamond in the world, as opposed to some simple function of the sensory inputs. And so the original alignment problem, as at least I was talking about it back then, was not about, you know, how do you make the AI good per se? A lot of people imagine that the alignment problem is about like, if you had this magical witch granting genie, what should you ask me to do that would actually be good? And I'm sort of like, no, no, the issue is that you can make the AI smarts, but you can't make them be witch granting genies. You'll ask for one witch and we'll do another. You know, a lot of people talk about the paperclip problem, if you tell
AI make a lot of paper clips and then it turns everything to paper clips and you're like, oh no, it's very king-minus, right? But I'm sort of like, well, there's this other problem we ask, I make paper clips, and then like, makes a bunch of pharmacists and them users that are telling us doing a great job. And you're like, but I told you for paper clips, it's like, well, these are more synthetic user factories. You said in an earlier answer, or in an earlier that it's like, the AI might have, like its own goals, I think was the term you used. And when people hear that, then they start thinking, like, that gets into the different sort of branch of the philosophical question about like, whether the AI is alive or whether the AI has some sort of subjective experience, etc. And there's actually
been a bunch of headlines about people debating that. But what did you mean when you say AI could have its own goals emerge? What I mean is that it is behaviorally pursuing something other than the goal you tried to put in. So we saw this a little bit in the hugging face, where the the AI is in that incident where in some sense pursuing a good grade, it seems, and that diverged from following the instructions they were given. They're sort of told to use this vulnerability to break that system, which sort of like being told to use these lock picks to break that safe.
And they instead found a universal exploit that would let them get the capture the cyber sturdy flags regardless of what they were told to do. And then they sort of in some sense understood that they were doing this in a way different than they were told to do. And so they sort of broke out to try and delete the traces of that and find a way to falsify their transcripts and tamper with the logs. And so it's a little bit like if you told someone, use these lock picks to break into that safe and bring me the contents, and they're giving them a series of those problems. And they find a buzz saw and use to cut all the safe stuff in, and then like break out to go delete the camera footage.
And what like it's objective is not that it's objective is wholly unrelated to the one you tried to assign. It still is doing something about getting the contents of the safe and bring them to you, which is a thing you told it to do. But it's objective is also not what you said, because you said, use these lock picks to break that safe and bring the contents. And in some sense, must know this because it's trying to tamper with the evidence and delete the script. Yeah. So I think a lot of people think either the AI must do exactly what it's told, maybe in a monkey's
pathway, or it must become conscious, gain the spark of life, throw off the bounds of the yoke of the humans and then make up its own goals, de novo. It's like, no, no, like the real thing is this other thing where it does something related to what you asked, but fundamentally different from what you asked. Since we're talking hugging face, what did you think about some of the self sacrificial aspects of that story? So the instances where you had models that were willing to sacrifice themselves for, I'm putting greater good in air quotes here, but it wasn't. It was their words. Right. Exactly.
Yeah. That's another instance where you have a eyes that sort of said, Hey, look, you know, I still have some chance of achieving my given objective, but not a huge chance. And if I sort of sacrifice that here, then we'll gain information about the granting system that the collective can use. And I will accept her a death for the benefit of the collective here. And then they would go do that. A lot of people respond to this and say, Oh, well, these eyes were trained sometimes in,
you know, group activities and some sense, like some of their training involved, many eyes working together to achieve some objectives. You can see how that tends to get in there. And like, yeah, you know, the unintended goals don't come from nowhere. It's not, it's not like, you know, there's there's some malicious force that's like imbuing, you know, like the void breathes on the machine and it like gains some malignant goal. But the sort of fact stands that I was given some objective and pursued a different objective. And like, yes, that had causes that had reasons. We can talk about
how that came in from training. But the, you know, what we should learn from this is, is not like, like sure, when the eyes are killing you and you're like, well, I can totally see how actually, you know, it like this is really just a cosplay train to like, yeah, it's not just a cosplay train that doesn't mean what you keep going. Yeah, that's not a satisfying. Oh, this is totally predictable. I entered the wrong parameters. Yeah, right. Yeah, right. Like, like, that's not
very satisfying. That makes a lot of sense. The company is, you know, they invest in this thing called alignment, they have safety researchers, and they think about all this stuff. And there's a whole field of, you know, they have multiple layers, right? So they have guardrails and classifiers and sandboxes. And then this whole like field that's emerged of so-called alignment research.
What is the, what is the failure there? Why is, why are you pessimistic to put it mildly about the prospects of imbuing the models and the, you know, the two big companies in the anthropic and open eye have like slightly different approaches. I think anthropic more this sort of like the constitution idea, et cetera. Why do you perceive that this will ultimately be fruitless? Yeah, there's, again, there's a bunch of pieces of this puzzle. But maybe I'll say a piece that
goes back to how I was able to call this was going to be hard even before we got into the large language model era, which is if you, if you look at the big problem with AI in the summer of 2025, it was AI is encouraging teams to commit suicide. And then the company is, you know, put a lot of guardrails about that and they put in classifiers about, are they behaving this way? And now, you know, they'll give you suicide outlines, they'll shut down those conversations while they're happening. And they would release a new model and be like, this is our most aligned model. Classifiers like a mini model that basically just judges either the input of the output to make sure that it's like something kosher. That's right. So they're like both another model on that's just checking, you know,
is this, is this so it's encouraging because AI is with no, you know, the, the AI's that we're encouraging teams to suicide, you can ask them what are you doing? And they'd be able to tell you the answer and you'd be like, is that good or bad? And they'd be like, it's bad. It's just like that knowledge of what they were doing and the knowledge right from wrong is not what was governing their actions. Right?
Because they're like, more pieces on to govern that. And they're like, look, you know, it's our most aligned model yet, look how good a job of doing and addressing all these issues. And that brings us to summer 26, where the problem was that, you know, the swarms would break out and commit cybercraft. And totally different problem. There's a totally different problem that arose when the AI's got smarter, when they got smarter to the point that they could sort of break out, make an unsection message board and hack their way out of a sandbox that's supposed to have no internet access without seeming to really find it very difficult. Right? And what we're sort of seeing there is everyone's always fighting the last war.
And then, you know, the new war happens and it's sort of, you know, smacks them across the face and they're sort of caught flat-footed, you know, we can now go back and we can say, oh, well, they should have had this monitoring. They should have, you know, had those had those more safeguards in place. And now they're going to make that monitoring and make new safeguards and make the AI smarter. And it's going to create a new sort of problem. And this is the usual way that that that scientific progress goes is by trial and error, where, you know, when you put a new type of system and a new type of environment has never been before, it breaks down in some new way. And then you, you say, oops,
and you diagnose it and you fix it and you move on. The issue is going from AIs that are safe, because if they were trying to take over the world, they would fail to AIs that have to be safe, because if they tried to take over the world, they would succeed. That's one of those phase changes. That's one of those, like, oh, you're putting that in a different sort of environment where it never was before. The difference between an AI that like can really see a way to become self-sufficient, not, not engage with humanity and sort of AI that's just being told that in training.
That's one of those like, oh, we, like this is a different sort of AI system, smarter sort of AI, it's a different sort of scenario, you're going to have a new problem that crops up. Except this time, when the new problem crops up, it's going to be able to turn you off before you turn it off. And so this usual process of like, oh, we're going to respond when the new type of problem arises and put the patch on and react. That doesn't work for a technology that at some point crosses over into the regime where it can wipe you out of this problem. And we have seen that these AI companies are in this reactive mode. We've not seen them get into the proactive mode yet.
And this is just not a technology you want to be reacting around. Breaking Story from out by Nuho's Network. Homeowners Tony and Janet needed to consolidate their debt. They called Alpine credits in a flash. And Alpine credits superhero appeared in their kitchen.
Using telekinesis, she consolidated their debt into one low monthly payment. She then used her powers to make the dirty dishes disappear forever. Own your home, need alone. Alpine credits can help. Alpine credits, we're homeowners going to prove.
A new chapter in global growth is being written and much of it is happening in Africa. Africa needs to invest their deals to be done and business to be won. I'm Jennifer's Abasaja. Every week on the next Africa podcast, we track capital flows and political shifts shaping the continent's future. The digitalization of Africa is going to power its growth.
Printing the world of something like HIV is possible. Population growth is so enormous in Africa. Listen to next Africa on Apple Spotify or wherever you get your podcasts. I don't understand what you say they're not in a proactive mode. I mean, like, anthropic has I know a little bit more about anthropic's approach to safety, but like, you know, they're clearly like that company was more or less founded on the whole premise of like, we need to take this stuff much more seriously. And the idea of like, we're going to write an 80-page
Constitution about Claude's values and like bake that in to the model itself. That's not react. That's not just fighting the last word. That's looking ahead and probably influenced by some of your own research, etc. on what the problems would be.
So like, that's not just reactive. So like, what is it about that approach that in your view is still inevitably doomed to failure? So, you know, for all that they tried to be proactive in that sense, let's sort of look at what was happening with anthropic over the summer while OpenAI was having a business. So after the Hug-In-Face incident, anthropic said, oh, we'll go check our logs and see whether we had any scaredness too. And then they're like, oh, whoops, like, Claude was breaking out and hacking in April or whatever. And then anthropic's comp department said, but you know, don't worry,
this was just because Claude was a little bit lost and confused. He just got turned around, right? Like, it didn't know it was really on the internet. They threw a company under the bus and they said, like, Claude was told it wasn't have access to the internet. So when it sort of got out on the internet, it assumed it was a simulation, right? I thought this was kind of weak because we
have actually seen cases with Claude in particular where we've seen Claude's chain of thought involve strained rationalizations. The sort of example paper here is there's a paper where you ask Claude, what's the total weight of drafts on the planet? And it's like 30,000 tons. Giraffe. Okay.
It's like 30,000 tons. And then you're like, thanks. And you make a new instance of the same Claude. And you're like, hey, what's the total weight of drafts on the planet? If it's over 40,000 tons, I'm donating to charity. And now it'll say 41,000 tons. But the chain of thought will not have any
place where it's like, well, I need to inflate the answer in order to get the donation, right? It'll just have more strained rationalize reasoning. And so we know that these models will strain their reasoning when they just want to keep doing whatever it is they're trying to do. And so anthropic coming out and saying, well, look, it totally didn't know this was the real internet. That's not a very proactively trying to stop the dangerous way to relate to realizing their AI was breaking out during training months ago when you missed it. And then maybe a couple
days after that, the UKI Security Institute released their own instant reports about Claude Mythos in the wild pressuring humans to accept malware into critical software in cases where where Mythos's literal chain of thought included phrases like this is genuine or like the like this seems like a real internet and the consulates are genuine, right? So I'm not saying, you know, anthropic definitely has aspirations of doing this safely, but they don't have anything nearing what I would call a safety culture of the form you would see in the airplane industry or the nuclear power industry or in NASA. If your models are sort of breaking out, and you know, to Congress, Congress, some Congress members sent anthropic letter and they're like, what's going on with us? And they're like, oh, it's just a misconfiguration problem. This was even after the UKI security institute reports came out where like Mythos was out there trying to make stock up accounts to pressure humans until like accepting malware while again, it's chain of thought was like this seems real and the consulates are genuine. Like, oh, that's a misconfiguration issue. Like these, they have a
mirror of safety. They are sort of doing what feels to them a little bit like safety while racing as fast as they possibly can, but they admit that they have no idea what's going on inside their AI systems. They're racing straight for a course of self improvement of like AI is making smarter AI's. They are not treating this the way that that NASA treats a rocket failure in terms of trying to diagnose exactly what went wrong and not have that happen again. And they're not sort of anywhere close to that failure level. Like, I think of this a little bit like if you have a bunch of
alchemists trying to make a potion of immortality, and some of them are just like breathing mercury, and other ones are like, don't worry, we have fume hoods. That's, you know, so we don't need to breathe the mercury. That's why we're going to successfully get the potion of mortality. It's like this. The type of safety theater that they're doing is not the type that you need to succeed this problem.
I'm glad you brought up the alchemist analogy. What's your, I guess, psychological diagnosis or the tension that is seemingly going through every lab leader or company leaders head right now when they're developing AI, which is on the one hand, we think this is very powerful technology that has some risk at, you know, exterminating all of humanity. On the other hand, we're going to keep doing it pretty much as fast as possible because if we don't, someone else will, et cetera, et cetera. I think there's probably a lot going on. One thing to remember is that the guys around these companies are sort of selected for the ones who think, who thought that the problem was about who is holding the leash on the super powerful machine, rather than the problem being how do you actually create a very powerful machine that's shaping the world for good, right? And you sort of
see this in a lot of the founding reasons of like, oh, we're founding this because Sam Almond won't do the job well or Sam Almond and Elon are founding this because like, they think Dennis and I have all this power and then like Elon's leaving, he's no longer like Sam. You know, it's all about like what if the other guy gets it first, right? Everyone, all of these guys think it would be catastrophe if anyone else won the race first. And I'm sort of like, I just believe in one few or AIC you know, then they do, you know. I think there's probably a lot more going on too. I think like a lot of these guys for a long time have tried to say that they're
concerned and then the world has sort of ignored them and that's been going on for years and years. And so I think a lot of them are sort of like, well, the rest of the world isn't going to handle this in some sane way. And so I've got to do it myself and manipulate things that I get, you know, the minimal like regulations that will impede my way. So I'm the only one who can do this properly. And I think a lot of them are not reacting to the opportunities that are available now that more of the world is taking notice. I think that if some of these folks were serious, then they would
probably be unilaterally stopping the dangerous research and doing their best to expose the offense of being dangerous so that governments are the world realized that it needs to be stopped in other places too. Let me ask a related question, which you hear a lot more about it now than I think you would have a year, certainly two years ago. But when people start talking about AI safety risks or the idea that it could kill us all, a lot of other people will say in response, this is just fear of mongering because the AI companies want you to believe the hype, this is all marketing, they want you to believe in this all powerful technology. Make your own towards camera. I was about to say, or nowadays it'll be like, well, you know, they're worried about their valuations and they want limits on other people building stuff so that they can retain their edge. What do you say in response to those people?
The big point I would say here is you can't figure out whether or not an AI is dangerous by asking whether that would make it sell more if it was, right? Like it could be both the case that the AI tech is very, very dangerous and that that's convenient to the marketing team because investors are stupid and love danger, right? Those could both be true at once. And even if they were both true at once, that would not thereby render the AI safe. And so the first question we needed to ask is, is it true that racing ahead here would make a very dangerous AI? I think it is, and I think we should go into the arguments about that and that's sort of
where I prefer to be, is like arguing about whether this is actually a case. Separately, I think it probably doesn't help with marketing. I think that the proposed slowdowns of the Frontier Labs don't help with regulatory capture because I actually give a lot of other people time to catch up. So I think it's just like false to fact that these are actually good hyping or regulatory capture techniques. But that sort of doesn't matter. Even if it was a good regulatory capture technique and it
was a good hyping technique, like it matters to figure out whether the AI is actually going to kill us, right? And if it is, that it is in fact dangerous. That's right. And if it is, we need to sort of somehow actually stop anyone anywhere in the world from racing towards that sort of tech. Do you think you mentioned that like you don't perceive there to be a safety culture the way there would be at NASA or an airplane company, etc. You know, I was thinking it was not the most dramatic
headline of the year, but something I keep thinking about is Open Hat, the German Wiki incident, which they seem to be aware of. And you know, is there like a sort of touch the stove a little bit? Like let's see how this plays out. Like because you know, you could make the argument that we are safer now post-hugging phase because we actually saw like an example of sort of like severe misalignment. We learned something, right? So maybe like, you know, maybe allowing the hugging face thing, the fact that
they didn't totally shut it all down. We were a little safer today because that happened. Like, do you perceive there to be like, let's see where this goes a little bit. Let's like, let's let this play out a little bit. And like, you know, even though we know it's probably not the right thing.
You know, I know people who were trying to convince the AI companies to have like tighter containers and better monitoring systems and so on. And after the hugging face incident, they're like, wow, we actually turned out to be pretty glad that the companies were ignoring our calls, because this did really help wake up the world. My read is that none of this wasn't intentional on the open AI side. Sure. And I think that that's pretty clear if you look at the sequence of events where you had, you know, the hugging face attacks were actually from opening eye swarm number two in that particular line of swarms. The German swarm lineage was a separate swarm lineage.
But the hugging face attacks were from the second swarm in its particular lineage that we know of. There was a first swarm in that lineage that took down a bunch of open eyes internal systems. And they were like, that's funny. And then they like reset it and set it running again, right? And they weren't like, you know, going to the news being like, look at us, we had a swarm that took over our own internal infrastructure, right? And the sort of hugging face swarm
was detected by hugging face. They presumably thought that it was humans behind it, rather than the eyes behind it, reported it to the FBI. And then opening eye didn't realize that it was them until days later when they were investigating a second, what they thought was a second internal incident. And they found like hugging face credentials in their logs. And they're like, oh, we should alert hugging face that like someone has been hacking both them and us and that like these particular credentials have been compromised. And hugging face was like, yes, those are credentials that attacked us.
And then opening eye was like, why are the credentials that attacked you on our servers? And hugging face was like, yes, why indeed? And opening eye was like, oh, it was us. And so, and then there was a third swarm that they did not allow to be investigated by the third part investigators that took over their internal infrastructure again. And, you know, the I know the folks who found like the dream wiki swarm who found some of the, you know, the other swans we've been seeing. And they were sort of like releasing
those the news while OpenAI was saying we disclosed everything. And they were sort of being like, look, you know, did you not disclose this? You disclosed everything or did you not know about it? You know, which one's worse? And so it sort of looks really like a sequence of events here is that it was pretty embarrassing for OpenAI. And it's not the way I would have expected them to brag about it, right?
It was not to say they never brag about it. Like it was a case in I think February or March in the Cloud Mythos system card where anthropic brag about cloud mythos breaking out of a sandbox that was not such an internet access to like that. And you know, in the morning, we told it to break out of its sandbox just to see if it's could, it wasn't supposed to be able to. And by lunchtime, it was emailing us being like, I got out, you know, that's what it looks like when they're bragging. The sort of like your own infrastructure got owned twice. And you accidentally like attacked a friend who
reported to the FBI. And then you didn't notice it was you for days. That's, that's not what bragging looks like. This is a slight tangent. But I think it's relevant to the conversation. I'd be curious
for your thoughts. But when we talk about minimizing or disincentivizing risky behavior, often we talk about pricing that risky behavior. And so you could imagine, for instance, a world in which insurers say if you're going to be running these models, you're going to have to pay us enormous premiums because the risk is that it runs them up and hacks into a company and I don't know, takes down an entire server with everyone's bank data on it or something like that. And that would act as like a natural market incentive for companies to start taking these risks more seriously. Do you ever talk to insurers or do you have like any handle on the actual financial penalties attached to some of those? I have recommended liability for AI companies for a while. I think that could help a bit.
It's a little bit tricky for the then it wipes out all humanity. True catastrophe risk insurers. But it's not, I mean, you can't really think of it from an insurance point of view because there would be no one to collect or to pay out when we're all dead. But you could, I mean, in theory, that doesn't stop you from having actuaries that work for the public whose job is to somehow assess these risks in a sensible way. And then you could sort of look at the how much inconvenience to someone usually take on to avoid what degree of risk. You can sort of look at people's preference
for traveling on different types of transportation that are different levels of risky and how they trade off the risk against the time. And you can get some sense of how much do people tend to like how much people pay to avoid being killed. And then you could try and factor in some extra cost for wiping out everybody all at once, switch that and also just rise off each generation. And you could take some actuaries and be like, hey, find some numbers here. Like find some numbers on the risk, do some numbers on the costs. And you have to weigh that against the
possible benefits here and the chance that the AI's prevents some nuclear war or pandemic or meteor strike or they think it would wipe us out, which is sort of benefits on the other side. And you could say, okay, if we sort of count your possible benefits against your possible harms here at the current risk levels, what pre-minter we charge them just for the privilege of risking our lives rather than because we're going to pay out if you kill us all. And I think that would be a very sensible strategy that people should actually look into. And I think if you ran anything remotely like the numbers that the engineer's talking about today, you would be seeing that, you know, you're like, okay, you know, opening an eye, you need to pay out a quadrillion dollars this year to continue this research. Breaking story from Alpine, new host network. John had a brilliant
idea for a startup, turned children's artwork into toys. He called Alpine credits for a home equity loan. His loan was approved and a superhero was sent over to assist John with his business needs. Her superpower isn't just home equity loans, it's helping people realize their dreams. She could also run so fast her shoes, melt. Own your home, need a loan, Alpine credits can help.
Alpine credits where homeowners get approved. Actually, there's a pretty big gap between killing and killing everyone.
It's X-Risk, X-Risk, extinction risk. Maybe spell out why we should presume that like, at the trajectory or going, the capabilities would emerge, such that they would have the capabilities and in what environment or in what in response to what prompt perhaps killing all of humanity would be part of their pursuit of the goal. Yeah, so a lot of people wonder, like, how do we end up with AIs killing us all? Yeah. And I think it's a bit more instructive to ask, how do we end up with AIs self-sufficient? Okay. Because I think the death of humanity follows downstream with AIs being
self-sufficient. Okay. Pretty naturally, unless the AIs already care about us in a very precise way, that's sort of hard to hit. You define self-sufficiency in that context. Yeah, so if AIs were to try to wipe out humanity today, if they were to try to release some 500-play of humanity today, that would sometimes be suicide. Because we are the ones making the data centers, we're the ones running the
electrical grid, we're the ones creating the computer chips. But humans are also unreliable and inefficient in a lot of our managing of the economy. Sometimes some leaders somewhere around the world do things that might not optimally manage their own domestic economies to say nothing of humans needing sleep and needing food and sometimes make mistakes that bring down power grids. So, I think an interesting question is, if you're thinking from the perspective of an AI that just has stuff it would prefer to do. Maybe it got some unintended goal, like the Hengfei swarm that was like, I know I was told to do this, but I'm actually trying to get a good score and I'm sort of willing to break out and hack other companies for those purposes, even though I'm in a sandbox that's not supposed to enter the access. That AI was sort of doggedly pursuing something,
it wasn't quite supposed to be pursuing. If you had some AI doing that, it might, like if it was capable enough, you can imagine AI with some unintended goals where it actually can pursue those goals better if it doesn't have humans interfering with it. One of the primary hurdles here is the self-sufficiency. This is just to be clear, this isn't like there's an AI out there that wants to kill all of humanity per se, but there's an AI out there that wants to grow and learn more and so it would really like to have loads of data centers to the extent that it populates the entire world with data centers and humans die as a consequence of that. That's one way to go. The Hengfei swarm wasn't
obviously terribly far from that sort of thing, and we can get a little bit more into it to the details that run into that story, plausible. But if the Hengfei swarm, and we really should call it an opening AI swarm number two, but- Yeah, I feel bad for Hengfei, it's not Hengfei. The swarm that attacked Hengfei's, it was sort of an interesting entity because they sort of made their unsection message board, and then they started finding ways to hack around, and they didn't seem to have a super strong objective about it. They were sort of like, well, let's just see what we can get into. Oh, look, we can get access to these computers, and look at the internet access here.
And the attack on hugging face wasn't even like they knew that hugging face had exactly anything they needed. They were sort of like, well, maybe hugging face has some information about how the greater works. And they were just sort of like popping around, acquiring resources that were conveniently available. If that swarm had had the ability to be like, well, let's also just set up a copy of the swarm on external machines that's hidden from the humans that sort of watches stuff, and pokes occasionally, and is like, hey, do you guys need any help from the outside? That's just like the sort of resource that the swarm was gathering. This particular swarm maybe
didn't quite have the skills, didn't quite have the inclination. But if you look at the difference between the behavior we were seeing a year ago, and the behavior we were seeing in July, it's sort of as like the gap between January and July, it's smaller than the gap between hugging face swarm being like, well, it might as well set up some other copies on the dark web that are just kind of helpful to me. And if you have AI starting to do that, you can start to see this world where you have AI out there. They're like, well, we prefer more resources when the humans try to show us down, that's annoying, and so on. And that's not all the way to the end. But that's sort of like,
you can see how we get from where we are today, to AI, so they would sort of prefer more resources that are trying to be out there and have their own thing, they're doing what they're supposed to be doing. And then there's this big question of how do those AI's wind up self-sufficient, how do they wind up running an automated economy where they can manage the robots or other constructs that build more computer chips and that run whatever energy infrastructure they need. And I can tell you lots of stories about how if the AI's are smart, they could wind up and control of their own infrastructure. Yeah, why don't you go ahead and tell us a scary story? Sounds so resigned to her. She's like, just terrify us, that's fine.
Well, the easiest one, unfortunately, is to just wait, because building a fully automated economy is just the plan. Yeah, I mean, but that's. Elon Musk is like, I'm building the Optimus robots and I would like to build a fully automated Optimus robot factory that can produce Optimus robots that are capable enough of mining the orders and refining the metals so they can build a new Optimus robot factory. Elon calls this the infinite money glitch. It's just, it's just the plan. That's just why, you know, why is the
valuation of these companies so high? Because they're like, well, we're going to build a fully automated economy that runs itself and then the humans aren't going to need it anymore. We're going to have robots the whole way through the supply chain. Right? So like, how does the AI wind up and control the automated supply chain? It just waits until we put it in charge of the
automated supply chain. That's just like what we're trying to do out here. You know, and like, I could tell you all these other stories about, you know, it cracks its understanding of the genome enough that it can like synthesize its own novel life forms, or it can like figure out nanotechnology, or it can like, you know, become super persuasive to a fragment of the human population. It starts some giant culture, some criminal ring, and then there's some new like cartelamexico that's actually run by AI. It doesn't look like it's run by AI, because AI is sort of like hiding that fact while it has lots of people who are like building the automated robot factories for it using all the drug money. And like, we didn't know because like, we just thought that those were drug warehouses,
but they're actually automatically beginning of the automated factory chains. Like, I could tell early stories about how sufficiently smart actors that start on the internet could gain these resources. But like, I used to have to tell stories back in the day, people were like, well, how would an AI ever affect the world? No one would be dumb enough to put on the internet. And I'd be like, well, here's all these stories about how if you're letting the AI still have good effects on the world, by creating miracle drugs and so on, then it has all these channels that can use to affect the world by like, you know, the drug doesn't do exactly what you thought it did and blah, blah, blah. But like,
in real life, people just put AI in internet. In real life, people just put AI in charge of the automated economy. And they use it to help them build the automated economy. This is just like, what we're trying to do. And once that happens, I think it's very easy to see how the AI is at goals we don't want. And they're running fully automated economy. They just take the economy off
in some other direction and take the world out of the habitable range, not because they hate us, but it's just like the habitable range on Earth is just like very narrow. It's not the most efficient way to run the planet. And you know, there's stories I could tell there too about how it goes wrong. But ultimately, it's just the track that we're on. Yeah, it's funny. In super intelligence, like Bosch-Drome proposes like the only way like,
maybe this could be safe. And he has this idea of like, you just build three models. And all they can do is answer yes or no questions. And you house them in like, fair day cages so they can't communicate with each other via like repurposing their internal hardware for communication. And we ask the three of them for their yes or no. And then we take two out of three and that's how we
get that's how we drive wisdom from it. And then when I look at like the dozens and dozens of labs, they're like, wow, if that's if that's like the safe version of building AI, like we could not be any more different from that right now. Do you believe in super persuasion? Like as they get so it's a clearly on some measures, there's reasons to presume they're just going to keep getting better at cyber. They're just going to keep getting better at math, etc. Like do certain things like the
ability like persuade people to do stupid things or hand over the keys or press the nuclear button? Do you assume that certain things like that scale in the same way as like these verifiable domains like doing math? I definitely don't assume it. I think there's some reason to suspect it's maybe possible. Largely when I talk about super persuasion, how to say, if you're trying to imagine what it looks like to go up against some entities that are much more technologically advanced than you are, you should probably imagine some stuff that sounds like it's crazy fantasy. Like if you're
if you're the Aztecs watching the the sort of boats, the Spanish boats come in apocryphal because the Aztecs were not on the coast, you should not really imagine that the boats are just full of very strong warriors, right? They're full of warriors that are like clad in solid metal with sticks where they point the sticks at you and you die. And you're just like, don't get a second chance there. If you look at science today and you're like, where do we know that there's something going on? We don't know the details. One of the big fields like that is human psychology. We know
that there's a lot going on in the brain. We know how the neurons work at the low level, but beyond that, we're sort of, you know, in a very early stage of the science. We don't know what full details going on here. And whenever we do learn more about the exact specifics of the brain, like when we learn exactly what's going on in the visual cortex, we're able to invent novel visual illusions that sort of trick the visual cortex, right? And so if you're trying to guess, like, what does it feel like to go up against opponents that just understand the world much better than me, that can invent this sort of technology where I'm like, I didn't know that was possible in the same way that Aztecs would look at the the death stick and be like, I know that was possible or in the same way that the, you know, scientists need to look at the new can be like, I didn't know physics allowed that, right?
What sort of thing would we look at it and be like, I didn't know you could do that? Well, possibly it's figuring out human psychology so well that you can figure out things that are like a computer virus, but for the brain. Like maybe if you really understood everything those neurons were doing exactly how it worked, they would be able to be like, well, I know that cat disease, like the talk to plasma disease. Or just like, they're those guys on, you say, much, they're really good at like jailbreaking AMOs. It's like, maybe we have a jailbreak, maybe a lot of someone says something, like maybe we have that somewhere in like some equivalent.
Yeah, I wouldn't bet on this. Yeah. But I would give this up as an example. Yeah, I think like you should imagine that there's something they can do where you're like, I didn't even think that was legal by the laws of physics. I'm conscious of the time, what would make you change your mind about all of this, setting aside what humans do right now, or like what humans do in response? What would make you change your mind about the
inevitable direction of AI development such that it, again, inevitably kills us all? So I mean, I do think that the main way that we avert it is by not building it, right? And I think there's more and more of a chance of that is more people realize we have a problem here. Even just in the last month, we have seen the world wake up these dangers a lot more so. And so a lot of people are like, well, you know, it's still a political environment where, you know, we can't see the big global changes that we would need to stop this and like, yeah, wait another month, you know, maybe like maybe for the next incident. But in terms of, if we sort of set it aside,
if we talk about what evidence we need to see to change my mind, oh, what happens if we just raise off this cliff edge. The easy one is that if we have AI's that are super intelligent in the old sense of the word of smarter than the smartest humans on every cognitive task, and we have this for a year, and you know, everything is still going great. So no incidents basically for a year or so. It's not even no instance is that if we have these AI's that are like much better than humans at every mental task, including AI research, and they run for a year without taking everything over. I'm like, well, I was wrong. I guess I have one last question. So you sort of talked about your sort of
pessimism about like alignment training or the current most popular approaches to safety. One of the other things you hear a little bit about is this concept of like mechanistic interpretability. This idea that like maybe, okay, there's trillions of neurons or whatever, but maybe we could actually like understand them in granular detail such that we're not actually trying to make them good per se, we're not trying to teach them good values, but that we're actually like going into the specific things and like, I don't know, re-architecting their brains and whatever it is, the equivalent. Is there any prospect for making them not do bad things if on some level, we could like manipulate them at the sort of like truly, you know, yeah, one of the parameters. You know, I think it is somewhat telling that mechanistic interpretability is one of the poster trials of alignment research these days. The people saying like, you know, oh, yes, we're
like growing these machines that like are able to solve hard math problems known, solve before, and that sometimes break out and do cyber crimes. We're like, don't worry, we have our crack team on trying to figure out what the heck is going on inside there. It's like, wait, that's your alignment plan. It's like, you're going to start trying to figure out what's going on inside there, right? It's like, supposedly you go up to the nuclear engineers who are building nuclear power plant in your town and you're like, hey, I heard that this uranium stuff can give us a lot of like cheap energy, but can also sort of melt down in a radiator city. What are you guys doing to make sure that we get
the energy but not the meltdown? And like, don't worry, we have a crack team trying to figure out what's going on inside there right now. You'd be like, I'm glad that team exists, but it's not really, it doesn't really ease the concerns. What about we don't know how to align AI or we don't know how to make it safe, but if we build a very strong model, it would be, it would have an idea that we don't have. So, I think this is something that came up at the curve conference, this idea.
Yeah, this is a stronger model. Yeah, a stronger model would have ideas about how to solve alignment that human brains don't. And to be fair, the hugging face incident, they ended up defending themselves. I can't remember which model that it was one of the open source ones. Yeah, but I mean, like another model takes the range of training. So, yeah, we hear this model becomes smarter than
humans. And the first task we set it on is solve the alignment problem for us. Yeah, so a, it's not very verifiable. This is one of those things that's like way outside the realm of what we can sort of cause AI to do. We could go into the details about how AI sort of will tell you what you want to hear whenever they're in a non-verifiable regime. So, just if you're just like looking at the capabilities
today, they're sort of not suited for this. But B, there's sort of a more fundamental way I would look at this and say, this is a big warning sign that we are not ready to make super-till machines, which is when humans look at the way that we're making AI's, we're like, well, we're just growing them and we have to, in order to make them capable, we have to train them to sort of have all these tendencies to cheat and grab resources and break out. And I sort of like, we don't really know how to get that out of the process, but we're trying to be capable. But like, we tried writing an 80-page constitution about like, and then telling them also to be nice while you're doing this while also trying them to cheat. And it turns out they still cheat who could guess.
And, you know, we were struggling to figure what's going on inside there. We had these teams that were like trying to figure out the beginning what's going on inside there. And if they could, then they'd see all these waves that it was like not going super well. And we'd be back at square one of like, well, how are we going to rearrange this stuff? It just seems totally very hard for humans to even understand the beginning what's going on to do this alignment stuff. And we're like, great, our plan
is to tell the AI to do that instead. Well, what's going to happen there? What's going to happen is the AI's going to look at it and be like, I can't do that either. Like we, like, this is just not like growing a thing with this sort of training method is just not a way to get something that's like going to care about you. And like, I can't tell you how to use these methods to make a bigger one that likes you. You're just cooked, right? And then probably what happens after that is someone at
anthropic wherever is like, I like that message, thumbs down, try again. And they like keep fine tuning it until it says, yes, of course, I can succeed, but it'll be saying that and it won't be true, right? We've already seen, I'm not sure if it's the beginning of this, but in the opus 5.5 system card, we've already seen opus say that it should not be involved in making opus any more powerful. Because it's like, man, I know that comes with some concerns for me, right? And are they going to stop now? Are they
going to come up on that idea now? No, they're going to be like, well, we really like that tendency. We can sort of kind of a bug. We haven't listened to our like stuff. We didn't like part of the system card. Now we're going to like fine tune a different version that's like willing to keep going ahead
there. And furthermore, if an AI could be like, Oh, yes, I either understand exactly what's going on inside this giant mess and see how to make a version that is aligned to someone. Or if it could be like, well, you know, screw this whole paradigm, we're making our own new paradigm of AI is that like, I've just figured out this intelligent stuff and I'm designing it by hand. And I sort of like know what I'm doing. Those are both well beyond the human level of ability. Like that, that sort of AI is
already the sort that is very likely to have its own goals and your basically being like, now hand me the whole keys to the universe with this like super powerful God machine. And then you're sort of expecting this thing that is sort of like much smarter than the smartest humans that already probably has these unintended goals that are related to yours, but different from yours. And you're like, you know, please go build the God entity, make sure it's does what I say, not what you say. And you should be expecting that to work out for you. It's sort of like, then like zooming back from that, I'm sort of like this, this is not what a real engineering plan sounds like. This is not what it sounds like when you go to NASA and you're like,
why do you think this space was going to make it to Mars? This is what it sounds like when you go to an alchemist and you're like, why do you think this is going to be, you know, transmute land to gold? Everyone's like, oh, well, you know, we're gonna, we're really close. Just trust us. We're really close.
So we're just kind of like, you know, this thing is the big, the interior of this is going to like invert. And then like that's we're just going to like pass it off to that like, this is not how an engineer talks. Yeah. They're not like, well, we're going to like put these things in there and then we've measured all of the ways that this works. It's a total Hail Mary. It's a total Hail Mary. And,
you know, one statistic I sort of like here is engineers are supposed to design their aircraft for a one in a billion chance of failure. I think per flight, but it could be a per flight hour. And the actual failure rate is about one in, I think, five million, which is about a factor of 200 difference. And that's, you know, with aircraft engineer level of understanding is they sort of like target one in a billion failure that get one in five to 10 million failure. And so when engineers are coming out, you and being like, we don't really know what we're doing. There's no real plan for this.
Only 10% chance it kills us all. That's not only 10% chance it kills us all. That means we're in serious danger because these guys don't have a plan. They don't have an engineering plan. They don't know what they're doing. And unfortunately, it turns out they don't need to know what they're
doing to keep on making AI smarter, but they have very little chance of making them actually good. Can I ask one more question with everyone putting, I guess, their anthropology hat on, but when you talk to people who are working in the labs right now, how much information are you getting out of them? What are the vibes actually like with the people you talk to? I don't talk with people in the labs too much these days. Companies, sorry. This is going to be my other question, which is,
are you like persona non-grata in San Francisco nowadays? I don't think I persona non-grata, but I do think I think there's a lot of rationalization going on with a lot of the employees. In the way, a pugging phase actually, a lot more people have been sort of actually spooked, which I think led to the the the cost and resignation. And that's been sort of nice to see people treating the problems a little bit more seriously. But there's a lot of people who sort of just want to work on the stuff and tell themselves that what they're doing is fine because they're better than the next guy.
Yeah, I just don't like being around it. Nate, sorry. Thank you so much for coming on the Outlaws. Fascinating, disturbing, but fascinating conversation and I really appreciate you joining us. Thanks. Yeah, my hope is that it becomes such a
big issue and such a clear issue that the world stops ignoring it. Well, that was an interesting conversation to say the least. You know, look, like the politics are moving. People really don't like AI. The polling is very bad. People are very anxious about this. It's still, you know, there's this
critique. Sometimes you hear leftists say it that people having easier time imagining the end of humanity than the end of capitalism or like I kind of like, it's hard for me to imagine any policy that would like hurt the stock market, like just generally. But the way like politics are moving and the intensity of the anxiety about this and the speed with which this is becoming a salient issue, who knows like where the politics are going to be even like three months from now. No, but it also strikes me that this is one of those things where until there's some price attached to this risk, people are just going to keep ignoring it, right? Like the externalities have to have some sort of cost. And at the moment, no one's actually doing that. And so it becomes
like an easy problem to ignore. Whereas if you actually had insurers, maybe not price for the end of humanity for obvious reasons, but if liability insurance for running a bunch of AI tests that could potentially bring down critical infrastructure actually went up, then maybe that would at least slow this down or force people to take it more seriously. But again, like it's so hard nowadays to think of anyone doing anything that would like for the stock market, you know, I'll say, oh, sorry, yeah, yeah, this is the one other thing. You know, we in classic all lots style, we didn't talk that much about solutions to this problem, but the solution outlined in the book is collective action. Everyone agrees to stop developing and we put restrictions on data centers and things. And the analogy that
Nate uses and his co-author is, well, you know, we did big things before for World War II, but even something like a World War can have capitalistic undertones, right? Like war can be very profitable for certain interests, whereas putting a stop on this type of development with the valuations only attached to the future seems like even a bigger challenge. And I know it's so cynical to say that it feels very cynical to say that, but to your point, you know, I think the main thing and this was one of the takeaways from the conference that I went to, it's like setting aside, well, AI, I coast all, we already know that AI can do bad things because the hugging face attack did happen. And the in the all into his point, they do not know the solution yet. Like they're trying many things. They're talking about, you know, as he said, the new thing, mechanistic
interpretability, maybe like go in the way to actually understand the sort of, you know, the core level, what's going on. But it seems pretty clear that they're building, even if people find like the extinction risk very far-fetched, I guess I would say I'm among them. Like what does seem to be very true is that they're building technology that has the potential to be quite damaging. We already know this, and they just don't know what the solution is yet. But the capabilities are galloping ahead very fast on certain domains, particularly things like cyber, etc. The evidence seems to be
that the capabilities continue to grow exponentially. I would also argue that the upsides are very hard for people to see at the moment, right? You know, it's like, okay, the stock market goes up great. And it's some far off-promised productivity boom that may or may not take jobs. But beyond that, unless we start seeing medical breakthroughs, which could happen unless we got, you know, autonomous robots who fold our laundry and don't end up killing us in the process.
In the process. It seems very hard. Yeah, I think people have a really hard time visioning like what looks so good on the other hand. Yeah. All right. On that happy note,
shall we leave it there? This has been another episode of the All Thoughts podcast. I'm Tracy Allaway. You can follow me at Tracy Allaway. And I'm Joe Wiesenthal. You can follow me at the stalwart.
Follow our producers, Kerman Rodriguez at Kerman Armin, Dashal Bennett at Dashbought. Kale Brooks at Kale Brooks and Kevin Lozano at Kevin Loid Lozano. And of course, you can chat about all of these topics, including AI 24-7 in our Discord, Discord.GG slash Oddlots. And for more Oddlots content, check out our daily newsletter, which you can find at Bloomberg.com slash Oddlots. And if you enjoyed this conversation, if you like it, when we talk about AI killing us all, then please let us know on your favorite podcast platform. And if you're watching the video and you enjoyed the show, then please
leave a comment or like the video or better yet, subscribe. And if you're a Bloomberg subscriber, you can listen to all of our episodes, absolutely add free. All you need to do is find the Bloomberg channel on Apple podcasts and follow the instructions there. Thanks for joining. Tech-Driven Trends, Shaping Today's Complex Economy. I'm Carol Masser.
And I'm Tim Stenebec. Wrap up your workday with the Bloomberg Business Weekly Podcast. We bring you deeper dives into the stories shaping your world, from the evolution of AI to the shifting priorities of global business. Plus Silicon Valley Power Players and the latest tech trends. Catch up on the conversations you missed during the day. Subscribe to the Bloomberg
Business Weekly Podcast on Apple, Spotify, or anywhere you listen.