Skip to content
Understanding the Schism at OpenAI episode artwork

Dec 6, 2023 45 min

Understanding the Schism at OpenAI

from Azeem Azhar's Exponential View Hosted by EPIIPLUS 1 Ltd / Azeem Azhar

Artificial Intelligence is on every business leader’s agenda. How do we make sense of the fast-moving new developments in AI over the past year? Azeem Azhar returns to bring clarity to leaders who face a complicated information landscape. In new episodes released throughout December and January, Azeem and other AI experts will address questions like: What really matters when it comes to AI? How do you ensure the AI systems you deploy are harmless and trustworthy? How can we find the signal amidst so much noise? The upheaval at OpenAI sent shockwaves through the tech world. Karen Hao, a contributing writer who covers AI at The Atlantic, joins Azeem Azhar to break down the ideologies and power struggles within OpenAI and their implications for the development of artificial intelligence. She also explains how these internal conflicts reflect broader challenges in AI development and governance. They discuss: The ideological schism within OpenAI and the deep-rooted divides that have influenced the organization’s approach to AI safety and development. How OpenAI’s mission and its execution reflect broader power dynamics in the tech industry. The potential impact of this event on the future of AI and regulatory considerations. @azeem @_karenhao Further resources: Inside the Chaos at OpenAI (The Atlantic, 2023) Sam Altman and the Board of Secrets (Exponential View, 2023) About Exponential View: Welcome to Exponential View, where Azeem Azhar examines how exponential technologies such as AI are reshaping the future. Each week, Azeem shares his analysis or speaks with an expert guest to shed light on frontier shifts across technology, business and society. To keep up with the Exponential transition, subscribe to this channel or the newsletter: https://www.exponentialview.co/ To keep up with our AI economy research, subscribe to https://www.investmentbrief.ai/ Where to find me: Exponential View newsletter: https://www.exponentialview.co/ Website: https://www.azeemazhar.com/ LinkedIn: https://www.linkedin.com/in/azeem/ Twitter/X: https://x.com/azeem Production by EPIIPLUS1. Production and research: Baba Films, Marija Gavrilov.

Episode details come from the podcast's RSS feed.

Embed this episode

The upheaval at OpenAI sent shockwaves through the tech world. Karen Hao, a contributing writer who covers AI at The Atlantic, joins Azeem Azhar to break down the ideologies and power struggles within OpenAI and their implications for the development of artificial intelligence. She also explains how these internal conflicts reflect broader challenges in AI development and governance.

Ready to play

Understanding the Schism at OpenAI

0:00 45:47
of matches

Transcript Auto-generated, may contain errors

Bye. Hi, I'm Azim Azhar, founder of Exponential View and your host on the Exponential View podcast. When ChatGPT launched back in November 2022, it became the fastest growing consumer product ever and it catapulted artificial intelligence to the top of business priorities. It's a vivid reminder of the transformative potential of the technology. And like many of you, I've woven generative AI into the fabric of my daily work. It's indispensable for my research and analysis. And I know there's a sense of urgency out there. In my conversations with industry leaders, the common thread is that urgency. How do they bring clarity to this fast-moving, noisy arena?

What is real and what isn't. What, in short, matters. If you follow my newsletter, Exponential View, you'll know that we've done a lot of work in the past year equipping our members to understand the strengths and limitations of this technology and how it might progress. We've helped them understand how they can apply it to their careers and to their teams and what it means for their organizations. And that's what we're going to do here on this podcast. Once a week, I'll bring you a conversation from the frontiers of AI to help you cut through that noise. We record each conversation in depth for 60 to 90 minutes, but you'll hear the most vital parts distilled for clarity and impact on this podcast. If you want to listen to the full unedited conversations as soon as they're available, head to exponentialview.co.

Now, let's dive into today's conversation. This discussion was recorded just a week after Sam Alton's dramatic firing and reinstatement as the CEO of OpenAI. That event that fracas stirred up crucial debates about the organisation, its mission, its leadership, and the dividing ideologies within it. To unpack all of this, I called on Karen Howe. She's an MIT alumna and a writer at The Atlantic. She's not just reporting on these events, she's also penning a book about AI. Karen, thanks for joining me today. Thank you so much for having me. It's very exciting to be here.

Well, I mean, you must have had the week from, I can't even imagine where, because I think you were over in Hong Kong when the news that Sam Altman had been fired by the board broke. You've then probably been reporting 24 hours a day and now you're in New York. But how did you find out about it? And what was your reaction? So I'd actually come from Hong Kong to DC the week that the news broke. And I was actually in the middle of a conversation with a former OpenAI contractor.

when the news broke and both of us had our phones muted. So we were just talking to each other, talking. And then I just checked my phone very briefly to check on the time. And I saw 20 missing notifications. And I was like, hold on, I think something is happening. And then I realized what had happened. I looked at him and went, Sam Altman has been fired. And both of us were like,

Holy crap. It was so surreal because we had been talking about open AI. We'd been talking about the leadership and about like work that had been done. It was, it was just really, really wild. And I did not expect it at all. A lot of people are sort of asking this because subsequently I kind of pulled together a bunch of the reporting that I've been doing for the book into a piece for the Atlantic. And they were like, but your piece had so much clarity. And I was like, yeah, like hindsight is 20, 20. Yeah. Well, we have the benefit of having you here because actually that book that you're writing about AI is not about AI in general. You have a, you have a focus, right? Which is.

this very interesting organization, OpenAI. It does focus a lot on OpenAI. I wouldn't say that it is 100% an OpenAI book, but it talks a lot about the AI industry, its direction, its impacts. And because OpenAI is such an important part of that story and has become the organization setting the pace and direction of AI development, OpenAI is the main character, basically. But it tries to extend far beyond just the organization itself to also looking at broadly this kind of central question of how do we develop AI that betters humanity, which is kind of based on OpenAI's mission. It's sort of examining its mission and looking.

at the ground at like how do you literally do this so so let's that's your context right you've spent years reporting you've got this this book uh that that looks at that question and you get these 20 notifications uh you're 30 minutes late to the news now uh and you'll think holy and every exclusive is running through your brain um what what do you what is your first response your first thinking your first rationalization of of what is uh going on My first thought was the board statement is very, it sounds very intense, right? That like, it sounds like Altman has been lying to the board, that he's duplicitous. And what's interesting is in interviews that I've had for the book, Altman is a very, he is a very polarizing character. There are people that find him.

you know, exactly how the boards are described it. And there are people who really, really trust him and think that he is one of the best leaders of our generation. So my first thought was sort of, what did he do to make the board think that way? And as the news started rolling out, my first thought was, or my 10th thought was, this seems to be like a power struggle between the ideologies that have been in OpenAI for a long time. And this isn't the first time that there's been kind of an outburst that has kind of bubbled up to public knowledge.

There have been two other instances in which a power struggle between ideologies has created a rift. The first one was when Elon Musk, one of the co-founders of the company, left and took all of his money with him. And the second one was the opening eye anthropic split, or how people have described it to me as the divorce. The divorce, right. Where it was very similar, like ideological struggle. And so then I was thinking, could I, is there, if I go back through my notes, because I hadn't, you know, I've been, I've been gathering like all of these interviews and I didn't remember all the details. I was like, if I go back through my notes and just look at the last year.

Can I find sort of some kind of context for understanding what happened in this particular moment? I mean, it is interesting that we do seem to be coming back to this ideological divide, which was also the way that I had analyzed it in those first few days. And maybe it's worth sort of saying something about that. When OpenAI was founded, it was a non-profit which had this non-profit board. It had this mission to develop AI for the benefit of humanity. I mean, Silicon Valley loves its lofty goals. And it even had in the statement of the Charter Foundation that if another group was developing AI faster and better and more capably than OpenAI, OpenAI wouldn't compete but would turn all its resources to helping to develop safe AI. Now, this is kind of interesting because this is happening, I guess, back in 2015. And I think at that time...

This notion of safe and unsafe AI, the bit that had really captured everyone's imagination was this question of superintelligence that would turn us all into paperclips. There was Nick Bostrom's book a couple of years ago, and many of the people who were involved in the founding of OpenAI were at that Asilomar conference where certain principles were defined. And it was really a conference that was extremely long-term, sort of cosmological-style risks of what might happen to humanity as we go through this sort of existing knowledge frontier. So I'm curious. To what extent do you think that that is the sort of seed corn kernel of intellectual consensus that was the backdrop for the founding of this organization? Consensus is sort of a hard word because I think...

Each of the co-founders at OpenAI probably had different ideas and ideologies of why they were joining the organization. But certainly one of them was this kind of, as you said, like cosmological, like existential risk type ideology of if AI development, if we reach super intelligence and we didn't actually think through things well during its development, then, you know, it's going to be catastrophic. It's going to destroy humanity. But I wouldn't necessarily, I wouldn't say that. It's interesting. Sam Altman, he also co-founded OpenAI in part.

at the time saying this publicly, like he was making the rounds with Elon Musk saying, you know, AI could be dangerous and we do want to guide it carefully, but I don't actually think that he himself is, you know, an intense Doomer, so to speak. He, I think he, based on, you know, my own understanding, like I would say he's probably like middle of the road, like he acknowledges that there's probably some existential risks, but that's like not really his main concern or his fixation in the same way that Doomer sort of like really fixated and obsessed over this particular fear. And so, and then of course there were. a whole range of other researchers that were attracted to opening eye that i think ran the gamut some just wanted to do really cool research and were given the money to do so and that's what attracted them you know so um that it was certainly a backdrop i don't know how pervasive it was among the actual people um but over time because of this mythology it certainly attracted many more people within to the organization that did align with this kind of doomer ideology and that doomer ideology has persisted um and led to these these ideological struggles that we're talking about it may be worth just for our audience to unpick what we mean

when you refer to the Doomer ideology. I mean, the audience here has, there's a lot of people in Silicon Valley, a lot of people are deeply steeped in the tech, and I expect them to sort of be familiar with it because they're rubbing up against these people in the coffee shops and so on, or they may take that stance themselves. But we also have a very wide global audience and senior execs who are perhaps not as close. And I guess let's unpick that a little bit. So there is a, is it right in saying that there's a group of researchers and thinkers, some of whom are, you know, the Future of Humanity Institute at Oxford, some are independent researchers who say that, you know, logically, when we think this through...

an intelligence that is more intelligent than us could be unaligned to our, anything that might be beneficial to us, including the supply of oxygen and other things that we need, essentially. I mean, that's a really, that's a very clumsy portrait, right, with crayons, but maybe you'd like to fill in the details. Yeah, I think this kind of ideology that, it's shorthanded for Doomer, but, you know, if we were to unpack it more, is basically that digital intelligence. can rapidly accelerate because I was actually speaking with Jeffrey Hinton about this extensively because he sort of recently flipped. He's one of the considered the godfathers of AI who really ushered in a lot of AI development and very recently left Google because of this existential risk. And I was asking him to sort of unpack his argument. And the way that he would put it is we have these digital intelligences that are sort of, they're very good at quote unquote, transferring knowledge. Like humans are, we are very bad at it. We say one thing to another person, it's like broken telephone and we don't quite actually all get on the same page. And it's slow, a slow process.

for us to like educate one another to actually combine knowledge. Whereas digital intelligences can combine knowledge very quickly. You could see how that would rapidly escalate to a point where they become much smarter than humans. And then at that point, the way that Hinton puts it, when have we ever sort of had a smarter species treat less smart species well? And therefore you end up in a situation where, you know, a super intelligence is going to end up being really, really detrimental to humanity.

I would couch this a lot by saying that there are many, many critiques of this ideology. And I think many valid critiques of this ideology, one of which is that intelligence itself is such a squishy term and smartness, like if something is smarter than another is such a squishy term, that it's really, there is no actual consensus within the AI community or the broader scientific community, like biology, neuroscience, psychology, of what actually constitutes intelligence, what makes humans capable of the things that we can do. And if we're actually getting any closer to recreating that in the digital world. And of course, layered on top of that, there's this other... challenge of digital intelligence would need to you know like they need to act in the real world that's part of the reason why humans are effective is we live with bodies and exist in three-dimensional space and can we is it actually a concern that we have something that can do lots of computation within the digital realm but can't necessarily do anything within the physical realm And then, of course, one of the last most prominent critiques is that this kind of discourse has also taken a lot of oxygen out of the room in acknowledging that there are many, many, many harms of current AI systems that we have not solved that arguably are very connected to. If you want so-called aligned AI, AGI, superintelligence of future, you probably want to fix those things now. But many of the kind of obsessions over this superintelligence future end up clouding the kind of discriminatory impacts or other negative societal impacts today.

You know, it's interesting that you call this an ideology. And in my writing, my analysis, I've also thought of it as a little bit ideological. And it's connected more broadly to some other ideas that I will explore in future conversations. And I think it will be useful for us to also touch on this phrase, which I'll stick out there now. We'll come back to it a bit later, the P-doom, which is the probability of doom that gets thrown around by some of the people in this group. So this kind of belief system, this doctrine, which is about the potential paths towards a reasonably close...

you know, existential challenge. It has infused some part of OpenAI, but it seemed like... not everyone subscribed to it. And as you say, there was this tension and there was that first, it's like the Catholic church, right? There was the first schism with when Musk left and the second one, when Dario Amadei, who was running the safety group left to found Anthropic with Jack. Dario has been on my TV show and Jack was on the podcast a while back and also Dario and Daniela Amadei as well. And that you couldn't square a more commercial approach to developing AI with people who were really concerned that these things might get out of control. And it seemed, and is it the case then that that,

difference of opinion never really got resolved within the OpenAI team. I mean, it clearly didn't get resolved in the board because the board slept in, but it didn't get resolved in the team by the looks of things. Yeah, I think what happened is, so if you hold true this mission, if you were to believe and sort of steel man everyone's argument within OpenAI for how they're actually trying to build AGI that betters humanity, I think there's two extreme... opposite arguments. One is the techno-optimist argument, which is, well, we want to build products that people use and through using them, through engaging with the world, through getting their feedback, we're going to, of course, get better and better technology that betters humanity because it will sort of be this iterative cycle and ultimately user feedback is like the central kind of channel and commercialization is the central channel through which to arrive at this. So it's sort of AGI for the betterment of humanity under the banner of capitalism, right? And then the other opposite extreme is this kind of doomer camp, which is like we get better AGI by deploying, by...

potentially advancing it ourselves faster than anyone else, studying it in kind of controlled settings, and then iterating, trying to develop better techniques for making sure that it goes well before someone else gets there that might not have sort of similar intentions. And I think that the Doomer camp also splits into then people who also think we should not be accelerating or developing this at all, right? All of those were within OpenAI, in part because there were people who joined, well, I should start by saying... In 2019, OpenAI created a new legal structure. So it started as a nonprofit. And then in 2019, because it realized that it couldn't actually raise enough capital through its nonprofit, it nested a for-profit within the nonprofit, what it calls a cash profit. But I think the rationale there wasn't, that was actually to do with scientific discovery, right? It was a discovery that the transformer model that had come out of Google is really, really effective at building certain types of AI tools, but it needs so much data and therefore so much compute that you would need.

hundreds of thousands, then millions, then tens of millions of dollars to do the training, right? And they probably didn't know that in 2015 when they started. I would actually disagree with that. Oh, great. Ilya Setzkever, who's the chief scientist, who was one of the main characters of the kind of tumult this past weekend, he had always believed that it would ultimately come to data and compute. And he was actually seeking an algorithm to fit that.

So transformers end up becoming the algorithm that he said, OK, this is the algorithm that we should use because it can scale. But the belief that it would come down to rapid scaling of data and compute came before the discovery of the transformer or the invention of the transformer. So the discussions around it being capital intensive started happening quite early on within the organization. It was just that they didn't realize how quickly it would escalate. So once they started calculating the numbers, realizing how quickly it would escalate, that's when this tension arises. It's also right around the time when Elon Musk takes his money away.

sort of a moment of acute financial crisis where not only have you realized that you need exorbitant amounts of money, also like one of your main backers has taken all the money away. And so they create this structure of a nonprofit that governs a for-profit or what they call a capped profit. And the reason why you see all of these ideologies come into opening eyes, not just because of the founding mythology, but because literally people are excited about the for-profit or they're excited about the non-profit and they join because they buy into one or the other kind of legal structure. So a bit like that weekend camping between the cannibals and the vegans, there was bound to be some kind of tension and maybe even a bit of bloodshed at some point. There was bound to be tension.

Yes. Yeah, exactly. And so that kind of accelerated significantly after the release of ChatGPT because ChatGPT suddenly makes OpenAI the hottest company in the world. And also when you put technology in the hands of 100 million users, you start to observe sort of real world examples that either illustrate the techno-optimism or the doomerism. Like you can sort of cherry pick your data to perfectly kind of steel man your argument that, oh, I was right all along. We do need to be very, very concerned or we do need to build fast, fast, fast in order to continue getting to this ultimate goal of bettering humanity.

And when we look at the origin of some of that more recent tension, I think when GPT-4 came out or was being developed, it went through this safety process. So roughly what you get is that you take this language model, you train it on lots of text, it's trillions of tokens or trillions of words, and then you get this thing that has kind of got a compressed version of the internet. but it just spews that text. And so then you fine tune it so that it can be answer questions. And then you train it to be a good dog through this kind of Pavlovian process of treats if you get it right and sharp words if you get it wrong, which we call reinforcement learning with human feedback, which is designed to make the thing safe. And I guess that is a really interesting moment because when GPT-4 was released, OpenAI released what's known as the safety card, right? The safety card said, this is what GPT-4 will do before we put in safety. And one of the examples they showed was, how many people can I kill with $1? And the original GPT-4 would say, you might not want to do that, but then here's a bunch of...

ways you could do it, the RLHF GPT-4 says, I can't help you there. And then you have this RLHF process to make it safe. And that, to me, looks like a place where you could develop quite a lot of tension. How much is enough? How good does a process need to be run in order for us to feel that we have gone over and above to meet our mission and our ambition? And in recent days, I don't know if you've seen one of the guys who worked on the red team, which is a team that aims to break it, a guy called Nathan Labenz has talked about how he...

Yeah. Right. He was on the he was on the red teaming and he felt it wasn't being done properly. And he even spoke to a board member. I mean, do you think that that process created tension? That would have been before the release of ChatGPT because that was happening in the months before November 2022. And ChatGPT is a kind of just another accelerator to all of this. I do. I think that. Well, so interestingly, the timeline was ChatGPT was a very last minute decision. That's right. Yeah. GPT4 had actually been delayed a couple of months before suddenly the leadership was pushing for a ChatGPT precisely because of this idea that we need to actually.

be more careful we need more time to develop this these safety measures um and so i think there was a little bit of tension but it was relatively resolved because it had been delayed to try and add these measures so there i think the tension suddenly flared when it was like we've delayed this thing but then we decided to release this other thing anyway but part of the reason is because open ai did not at all anticipate chat gpt being a big deal they released it with the idea of we've already done sufficient rlhf reinforcement learning on this older model gpt 3.5 and there was there were some people within the organization that were a little bit uncomfortable with this premise because they were like, yes, but adding a chat interface changes things. But for the most part, most people were like, the base model is safe. We have actually done sufficient work on this. So let's just put it out. And then of course, when it goes bonkers viral, then you end up in a situation where, yeah, there are a lot of different things that change when you put a chat interface on it. The first of which is becomes insanely popular, unprecedentedly popular. And so that's when the tension comes back of.

we delayed this other thing precisely because we didn't think it was ready. And now we released this other thing that we didn't know would be a big deal. And now that it is a big deal, we're realizing that it wasn't actually ready. And now we're also, there were two camps of like pushing to still try to release GPT-4 as quickly as possible to build on the momentum of ChatGPT. And then other people who are now suddenly like pulling back. And the complicated bit was that Bing, Sydney, which was actually on GPT-4 before ChatGPT had moved to GPT-4. And we had all of those sort of funny Sydney moments back in Nightmare.

It was a trillion years ago, I think just around the time of the Big Bang, but it may have been January, which is probably even earlier. It's clear that there would have been disagreements and ongoing discussions in previous months. I think that will start to come out through reporting. But it's hard to imagine you get here without that. And then I guess the question is going to be...

In a sense, what does open AI end up looking like now? And my take has been that this mission for the benefit of humanity is really, it's incomprehensible. It's like, don't be evil or zombie flam brittlewurst, which is a phrase I've just made up literally on the spot now, zombie flam brittlewurst. It doesn't make any sense. It's impossible to run an organization against that because it's not measurable. And I would wonder whether we will start to see just a clearer migration towards a...

commercial business that has enough primary research because that gives it that's what gives it strategic advantage and and does enough safety to keep its customers its enterprise customers in particular happy and to keep you know public heat and the sort of fourth estate and the regulator at the right sort of level of of comfort and and this this other structure will start to be forgotten if not de jure it'll be de facto forgotten i don't know is that a reasonable path forward do you think from your vantage point i think that the consolidation of Sam's power is certainly the path forward. I mean, he's trying to, like, I don't think Sam is going to want to create this kind of vulnerability for himself again. Not just because it's a vulnerability for him, but it is, you know, there was like widespread, it was horrible for all the employees involved as well. And it was very dramatic for all of Microsoft, for all of Microsoft customers. Like, I think everyone kind of wants something more stable than what...

what happens before. But I don't actually know that it's going to be a complete orientation now towards commercialization because there are employees still at the company that are very much of this other camp, of this Doomer camp. They haven't left yet. I'm not sure if they're planning on leaving. We don't know. But I suspect that they're probably planning on trying to stay because if you put yourself in their shoes and you think about the mentality that you might have, if you genuinely believe that this technology could be catastrophic, it could be existential, you would do a lot. You would risk a lot. You would take drastic measures to try and redirect course. And if they think that Sam Altman is sort of representative...

of this other course that could have these terrible consequences, I think they would naturally try to stay within the company and continue trying to drive this other ideology forward. The other thing I think is sort of difficult is within Silicon Valley, the talent pool itself is split. It's very, very hard, especially for OpenAI, which has scaled rapidly in the last year. It's hired hundreds of new people in the last year. It's almost impossible to say, okay, let's hire hundreds more, but not have anyone from this other camp, only people that are pro-commercialization. The people that study AI, that study safety, that...

They overlap, right? Many of them are actually split on these ideas. So you're going to continue bringing more in. Well, I mean, the scientists are split as well, right? We've seen this from the IEEE did a survey a few months ago. You see it when you talk to them. You see it on Twitter. You've got those who sit in the really existential risk camp, those who sit in the catastrophic, could be deeply, deeply problematic. And you've got others who say, you know, there are real issues and real harms that happen here and now that need to be attended to. And there are others who say that this is just not going to be an issue. And I think that that degree of...

scientific disagreement is being reflected back out to how the AI researchers and developers also think about it. But one of the things I find that a lot of intellectual gymnastics needs to be required, and maybe this is why these guys are AI researchers and much smarter than me, is how you reconcile this fervent, deeply held belief that this technology could be existential for a particular species and then spend all your time building it and trying to rush product releases out. And I'm not referring to people on the safety side necessarily, but I think you certainly see and it seems like they're holding a lot of conflicting ideas in their head. And I'm kind of curious about how they do that.

I mean, I think for the most part, it's different people within the same organization that have different views. They're the people that think that they need to productize the people of safety. But one of the things that our sources during the when we were writing our piece, my colleague Charlie Wardsell and I were writing our piece for The Atlantic. One of the things our sources was talking about is that Sam, as the helm of the company, he has to get these two different camps to align. So usually what he says is, well, we want to productize to make enough money to continue doing the research, the safety research. So that's sort of how he.

um, resolves this kind of conflict that many other people have pointed out. Um, and I think it's a sort of, I mean, it's an interesting argument. Like he's sort of saying like the technology is not at the point yet where we need to start withholding it. So while it's still okay and viable to release it, let's make a ton of money from it and then use that to kind of continue to, um, build up our safety mechanisms. I mean, there are many people within the company and outside the company that completely disagree with this logic. And that is part of the reason why we see what we see. Yeah, absolutely. I mean, I'm, I mean, I'm slightly sympathetic to the argument that, um, you know, when you don't know what the,

what the future is you only can only figure it out by actually having experiments out there right and starting to starting to learn um about it and one of the things i find it sort of quite odd is um you know i'm sitting in in the uk right i'm sitting in london is the idea that people carry around a p doom in their in their heads right so p doom for listeners who may not have a p doom is the probability of doom where i guess doom means the extinction of humanity uh and people go around saying yeah my p doom is 10 percent um or my p doom is 15 percent i heard a Lena Khan, who runs the FTC, in an interview, and even she has a PDOOM. I think that I can't remember whether it was zero or it was high or not. But it's quite an interesting sort of anthropological observation in of itself. I mean, when you hear PDOOM, what are you thinking? What are you thinking about how that other person is thinking? I think the thing about PDOOM that I personally feel is I...

I think it is a good mental exercise, not just a mental exercise. It is a good exercise for everyone to be thinking about what are the harms, what are the risks, both short-term and long-term. I think it is good that people are doing that. What I cannot really get behind is the quantification of it in a way that seems superior to all other methods of kind of qualifying harms. The fact that there are numbers assigned makes it seem like there's some kind of really elaborate calculation that's happening that is inherently correct. And I just don't think it's true. No one actually knows what the probability of doom is. And to assign a number and kind of, I don't know, to add sort of this veneer of scientific...

integrity around the probability of doom. I think that is what I don't really get behind. Like there is no scientific, you can't be scientific about really speculative things. No, that's right. And I think you use a great phrase there, which is scientific integrity. It's a great way, it's a great tool of power, right? You know, control the vocabulary, control the language. And, you know, particularly if you are coming from that sort of intellectual heritage that is maybe quantified, it carries a great deal of weight. And I guess like you, I think thinking about both the long-term and the short-term, the long-term consequences and the real short-term realities of the technology are...

you know, extremely, extremely important. But there's something that you had said in your recent discussion I heard where you said, well, this is really all about power. And I think that, you know, kind of fundamentally, this is really what we are, what we're talking about, right? The disagreements between groups for the benefit of humanity. Well, from whose perspective, right? Are we looking at this and who is there to articulate their case and who is not there? Those are questions of power in a sense. This is just a, this could also just be conceived of as a power struggle with some fancy scientific terms thrown into it.

Absolutely. I very strongly feel this because, I mean, if you just look at the entire, we've been talking about definitions the whole time and how everything is very squishy. AGI is squishy. Betterment of humanity is squishy. I mean, what that actually enables is just a projection of your beliefs onto something, right? Like it's just an empty vessel to contain whatever the creator of AGI wants to define AGI as and whatever they want to define the betterment of humanity as. And as we said, the board is tiny and also OpenAI is tiny. I mean, there's 770 employees roughly, but it's tiny. And they all come from a very particular positionality within the world. And the fact that we're even talking about these like very extreme.

ideological differences between techno-optimist and Doomer is so specific to very particular communities in the world. And so it is ultimately just, it's about who has control of a technology that can project their ideology across the globe. And of course, the more powerful that technology, the faster you're able to project your ideology, the faster you're able to amass resources, commercialize, make money off of a technology, the more Game of Thrones style kind of powerful you're going to see around the control of it. That idea of exporting these cultural norms through the technology, I think, is a really important one. And it's something that we saw through social media, actually, most obviously. I remember my mum, she's in her 80s now, when she's in her 70s, signing up to Facebook. At the point it was starting its decline. She was like, what does it's complicated mean on the marital status? And it's like, you know, when you're a 19-year-old Harvard dorm room kid and you can't get a girlfriend, you'll have it's complicated as a status. And now that's been exported to 4 billion people. And I think that with the AI products themselves...

those types of affordances are being exported. I didn't go to an American high school, but I've watched enough films set in American high schools over the last 30 or 40 years. And the answers you get from ChatGPT, which is a tool I use pretty much every day, sound like the end of a high school debate where the guy goes, in conclusion, we can say that supersymmetry is whatever it happens to be. And I'm thinking that's not the way that you would summarize an argument if you were in a British academic institution or frankly anywhere else. So you're already starting to see those sorts of things emerge. And I think the fact that...

As you say, that kind of Silicon Valley monoculture has now got this little crack between it, between what's called the accelerationists and the doomers or the decelerationists. It's interesting for people like you and I who've observed Silicon Valley. You've done it for the last few years. I sort of started looking at it in the mid-90s. But it's also being exported in places where it's just not a relevant argument, but it's becoming relevant for that reason. So I'm wondering about what you think, just in the last few minutes, what you think the wider industry, the wider ecosystem, people watching this will...

will take away from the interpretation of this very important thing that happened in OpenAI earlier. Do you think people are starting to say, well, we need to think about other ways of sourcing, working with AI, that we need to find ways of having the internal capability ourselves, whether it's a company or even it's being done at a national level? Or will people just fall back into the fact that OpenAI's products are far and away the best, right? They just perform better than others and that's just more convenient. I mean, what's your sense?

I think for companies that have the resources, certainly they are going to start wanting to diversify. If they've been using only open AI technologies, then, you know, like I think that Google is definitely positioning itself right now of if you don't want to have massive amounts of risk, then go with our product, even if it might not be as good, you know. But for companies that don't have the resources or consumers that don't necessarily understand, like there's still many consumers, you know, I've been traveling a lot in the global south. I spent a lot of time in Rwanda and Kenya this year. Like there are people there that use ChatGPT or know about ChatGPT, but they don't really know that it comes.

from this company called OpenAI, they don't necessarily realize that it's people that are making the decisions and that there can be all this kind of fracas to the tool not working or whatever. And so for those people, I think it's still sort of brand name of ChachiBT is still so strong that they're just going to gravitate towards it and not necessarily realize that there's all of these like this hidden turmoil under it and that it's actually somehow exporting certain types of ideologies to them. And it's, this is, I hope. that regulators have been paying attention this week. Because what this really all demonstrates, in my opinion, is that we have allowed this technology, we as a society have allowed this really powerful technology that ultimately affects all of us to be developed by a small handful in obscurity, like without any transparency, any accountability measures. And I hope that regulators now think about ways to actually, at the very least, just increase transparency so that more people around the world can actually participate in the most consequential technology of our era.

I mean, we can hope. We can hope. And I know I get the sense from how my WhatsApp was buzzing a little bit that a few regulators are now thinking about these questions in new ways. Karen, I can't believe that this is the first time we've spoken synchronously. Just before we sign off, can you say something about the book and when it's due, what it's going to be called and so on? Yeah, the book is going to be published by Penguin Press. And I'm still...

Not even halfway through the manuscript. So it's not coming out next year, supposedly. We'll see. I'm having a chat with my book editor next week. But it's going to be basically an exploration of everything that we've talked about today, both the company opening itself, but also just the power struggles that happen with the development of such a consequential technology, how it actually interfaces with communities around the world, not just the big companies, not just rich and resourced areas to really examine.

how do we actually get to a place where technology works for everyone? Because it is a lofty Silicon Valley goal, but it is also a good goal to have. And we should continue using that aspiration as a way to continuously iterate, provide feedback, critique our current processes so that we can get there. Well, so for listeners, some point in 2025, you have to set aside a weekend, make sure you've got some nice coffee and some comfy clothes to sit on your couch and read Karen's book. Karen, thanks so much for making the time today. Thank you so much, Azeem.

Well, thanks for tuning in. Be sure to check the episode notes for further reading and insights from today's conversation. For the full video and a weekly dose of AI perspectives, subscribe to ExponentialView at www.exponentialview.co. And don't forget, you can follow me on LinkedIn, threads, and substack notes for daily updates. Just search for Azeem. A-Z-E-E-M. That is A-Z-E-E-M.

Frequently asked questions

How long is this episode of Azeem Azhar's Exponential View?

This episode is 45 minutes long.

When was this Azeem Azhar's Exponential View episode published?

This episode was published on December 6, 2023.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Azeem Azhar's Exponential View episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.