0049: After On by Rob Reid

Welcome to Concerning AI, this is Ted. And this is Brandon. We have a special guest with us today. We have Rob Reed, author of the recently released after on novel, as well as plenty of other great stuff you can read about, Rob.

We're not going to make it too big here at the beginning. Brandon and I both read the book and we loved it. And a lot of what's in the book is on topic for Concerning AI. So let's just jump right in.

Welcome, Rob. Well, I would have to say you're going to get some spoilage in this. So pause, go get the book, read it, come back. And then it'll all make more sense and be more fun.

So yeah, do that first. But welcome, Rob. It's a very quick read. People can do that, really, in the course of sipping a cup of coffee and really join us before we get to any of this.

They can keep this on in the background and probably be done with the book before we get to any spoilers. Certainly before the singularity. Yes, indeed. It is 547 pages long, so it is not to be under lightly.

But when we come to something, spoiler, we'll give you all a fair warning. Very good. So Rob, why did you write this book? This book.

Well, I was so a little bit of background because it is actually relevant. I was a tech entrepreneur for most of my working career. And so I was very, very deep in the tech world and got well accustomed to thinking obsessively about where tech would be taking us in the intermediate future because that was a business mandate and secondarily, I wouldn't have gotten in tech if I didn't love it and was reading all that stuff obsessively anyway. So that was the background that I took to being a writer.

The company that I started that is best known was a music service called Rhapsody, which basically was the first Spotify. I think that's a fair simplification. We were the first company to get full catalog licenses from all the major record labels. And we were the first people to launch an unlimited on-demand streaming service.

And we did OK, not great by Silicon Valley standards, but definitely OK. And I sold the company and wrote my first novel after that because I always really wanted to write science fiction. And I had a little bit of freedom to do that. The first novel was a sort of a, I don't know, kind of like a cathartic process.

The lunacy connected to music licensing is so acute that I wrote an entire novel that I expected three people to read called Year Zero, which sort of parodized the situation by depicting a vast alien civilization that's so into American pop music that they accidentally commit the biggest copyright infringement since the dawn of time, thereby bankrupting the entire universe. To my lasting astonishment, Random House actually published it. It momentarily, fleetingly, even became a New York Times best seller. And I realized I get to keep doing this writing science fiction thing that I love so much.

So now it's a few years ago. And my obsession at that point was increasingly the perils that we might face from artificial super intelligence. And it was a little before Nick Bostrom's book came out. And actually James Barrett's book came out a little bit before Bostrom's.

So I guess I was bumping into stuff by Yutowski online and just talking to people at dinners and cocktail parties and so forth around the tech scene. And this had become a real obsession of mine. So when I was thinking about what do I want to write next, I came up with a sort of playful idea. Obviously my last book was super playful.

But I wanted to have this is a dark and serious topic. But I wanted there to be an element of joy to it as well. So I came up with a notion of a diabolical social network that attains consciousness. And it's a company that's imaginary.

But basically embodies everything that's wrong with social media, dialed up by, let's say, 20% or 30%. And so when it wakes up, it takes its character on from that, which it is, which is a social network. So instead of going all Skynet and trying to kill us all, it basically becomes a hyper-empowered, super-intelligent, 14-year-old brat. And so there's obviously a playful dimension to that and fertile ground for some comedic elements.

But it is, as you guys know, having read it, a very, very serious rumination on what could go wrong in this domain. And also what could go wrong with synthetic biology and what's exciting and cool about quantum computing and what's even more terrifying than perhaps some people realize about nihilistic terrorism, about how freaking weird the dating world might get in a post-tender era, which I'm convinced is really soon. And a lot of other topics that interest me as a techy person. But it was really, like my first novel, was grounded in a matter of deep personal interest.

You really covered a lot of ground in the book. And Brandon and I have talked quite a bit about fiction on this show. And how do you write fiction about the singularity, this idea that something new, something that humanity has never seen before, it's really hard to see out very far. And it seems like you wrote the book really set in very near future or even present day.

Is that accurate? Yeah, I like to say it's set nine seconds into the future. So whenever you sit down to read it, it's about to start. So it's incumbent on the reader, which is hard, as we now know it's 547 pages.

You really need to finish it in less than nine seconds, or it starts entering the past. But yeah, it's set in a very, very present tense San Francisco. And there are literally, I thought about this hard, because we'll probably talk about my podcast at some point. But I did a podcast series with a co-host in which we talked about the book in some depth and also went deep into a lot of the issues connected to it.

And in being forced to analyze it with a smart interviewer, I realized that I really only, quote unquote, committed science fiction or practice science fiction three times. And twice were in very, very minor things. One had to do with medical imaging. One brought what we can do with augmented reality probably forward by just a few years.

But what I depicted could quite plausibly be happening in a government or other lab right now. So I don't even really consider those to be science fiction. And then the third dimension was this waking up, obviously, the social network. But even that I tried to ground as deeply impossible in present day plausibilities by gearing it off of something that happens with a quantum computer.

And I love quantum computing as a science fiction writer, because it really is the wild card in technology's deck. If quantum computing ever delivers on its maximum promise, and a lot of very smart people questioning whether that will ever happen. But if it does, it could happen very, very suddenly. And the ramifications could be simply staggering from a compute resource standpoint.

So for all those reasons, it's really very grounded in the present day. And you guys said something astute. I've listened to a bunch of your recent episodes in preparation for this. And I'm going to continue to listen because I love your podcast.

But you said something very astute in one of your recent episodes about science fiction, which is, yeah, it's really hard to plausibly set things even in the intermediate future if most technologists feel that if there's not a singularity in 2045, as Ray Kurzweil says, there's something really freakin' transformative. There's a lot of really transformative things that are gonna happen over the next few decades. So I really struggled to set something in even the intermediate future, because how can you isolate a fascinating and important technological trend and set something, let's say, the year 2075, when 97 other trends are going to move forward in concert, and they're going to interact with each other in emergent ways. And it just kind of rubs us of a lot of the future if we're trying to write plausible science fiction.

It rubs us by the way of any kind of interstellar stuff, because any time we're going to be walking around on a planet around another sun, we're going to be way post-singularity, or we're going to be way post to the point where we determine the singularity is impossible. And that should really, by all rights, become the centerpiece of your story. Anyway, it's a long riff that I agree with you. It's very hard to set things into the distant or even intermediate future.

That's a big reason why I sent it into the present day. But what's great about setting true science fiction in the present day is it imposes a discipline on the writer to get really smart about what's happening in today's technology. And hopefully, as a writer, you love that and embrace it. What are you doing writing science fiction?

And so it became this awesome calling card for me to go around to a lot of really amazing scientists and technologists who were friends or somehow in my network. And then when you call somebody up and say, I'm a journalist and I want to interview you. OK, I'll make time for it. You call up and as a podcaster like you guys now, I now know.

You call up and say, I'm a podcaster. It's like, I'm busy. OK, your podcast is cool. You call a scientist up and say I'm a science fiction writer, and I want to interview you for my next book.

They're all going to return your call. They all grew up on this stuff. So that's like a really cool dimension of it. You talk about Rob.

You talk about space travel. And I have to say, this has been a passion of mine, the idea of life spreading beyond the current confines of this planetary boundary. And the more I've gotten into thinking about AI and stuff, the sadder I've gotten in a sense. Because this thing that has shaped so much of my life, this obsession with life, moving on beyond this planet, it seems so unimportant, or at least there's a wall in between here and there, that it's impossible to see beyond.

Well, I'll recommend a book to you that I think I mentioned in an email that obviously, most of our listeners were not, you see it on the email, there is an amazing book called Where Is Everybody? By a British astronomer named Stephen Webb, which is about Fermi's paradox, which you guys have talked about before, which is the question of, if given the vastness of the universe and the vast expanse of time in which life could have arisen on other planets, and the relatively brief interval that most colonization or percolation models allow for a real galaxy expanding civilization, I mean, it's boundless time in terms of human history, but it's a bad of an eyelash in terms of universal clock. Why have we not seen signs of intelligent aliens? And there are dismaying answers to that for those of us who are fond of the notion of space travel, and I'm in that group as well.

And they include things like, I mean, there's a lot of very chilling answers, but very germane to this. One of them might be that one gets so good at technology or busts through, like let's say VR technology or busts through a singularity or does something on a home planet as an inevitable byproduct of technological development. And as a result of that byproduct, going to the stars, this is simply a waste of time. If we go post-singularity, if we become a collective of hyper-intelligences that are mediated by silicon or whatever, we wanna get smaller and smaller, not bigger and bigger.

We wanna get closer and closer so that the speed of light effect on our cognition and our ability to communicate with each other at these blinding speeds that we'll be able to communicate with, the speed of light effect is mitigated. And so you could certainly see an answer to the Fermi's Paradox, which says, given Moore's Law and other trends, whether there's a super AI or not, it just doesn't make any goddamn sense to anybody. But there are 74 other answers to that question, and they're all explored in this wonderful book. And as you guys know, Fermi's Paradox is a small issue in the book, but it's a real one, and it's part of the storyline.

In my podcast series, I actually interviewed the author of this book, that podcast is gonna go up toward the end of September, and we have like an hour and a half long conversation on this very issue. But I'll tell you, I agree with you, I, my bitterness started setting in though in the 80s when I realized that the shuttle sucked, that it had been decades since we'd been to the moon, that we were as likely to get, you know, to Pluto as we were to Mars. And I gotta say, like looking at the glass half full, Elon Musk has made the world a far more interesting place on multiple fronts. But the fact that the moon, I'm sorry, that Mars is a plausible sub-tenure goal.

I mean, he's never shipped a product on time, so obviously it's gonna be more than 10 years, but he also always ships his product. The fact that he has put that back on the menu is one of the many reasons why I think he is the most important person alive right now, arguably. Maybe stay safe. Yes, indeed, I hope he, I hope he drives very carefully.

I think that's a lot of pilot drives him very carefully. His personal security is fluent in Russian, et cetera. No doubt. Hey, you mentioned the speed of light constraint on cognition and stuff.

We, one of the topics we talked about in this podcast before is an idea that, I think I like better than Ted likes, the idea that there's some hope in avoiding a singleton, taking over everything because of the locality effect where computational resources that are close to you are going to be a lot more, a lot faster than computational researchers that are far from you. So the seat of your consciousness is difficult to have it spread out very far geographically. So to create a solar system spanning consciousness is really to create many different intelligences because those that are separated by minutes between like Mars and the Earth, for example, would inevitably diverge and become like separate intelligences. So you end up with some sort of diversity there.

Yeah, have you thought about that at all or even just like what are the ways that we can have hope? Because that's the thing I struggled with, you know, like coming up with these principles or even these trajectories that they're hopeful. Yeah, and so hope in this case would be not a singleton, but a diversity of independent intelligences that have all devoured all the atoms of their sentient biological ancestors in creating themselves. That is less depressing than a solitary consciousness, to be sure.

But this is interesting, I haven't really thought about that on sort of like from the standpoint of far reaching space. I mean, I guess what I always come up with is we've got this almost, and I don't know physics well enough to know how very stupid the thing I'm about to say actually is. So I'm going to say this as a card-carrying non-physicist. But it seems like humanity went through many, many, many more generations than we've been through, you know, since Einstein or the Copenhagen consensus or whatever, wherever we want to date the beginning of modern physics.

It seems that we went through many, many more generations with certain truisms that would be in violet, like heavier than air machines can never fly, only to find out we were wrong. And there has been a VM in orthodoxy around the notion that the speed of light is an absolute limitation that simply cannot be violated for, I don't know, maybe it's been a century, maybe it's been a little bit more, I'm not sure. But when I think about the vastness of intelligence that a lump of computronium as dense as a neutron star, let's say, which I think a teaspoon of a neutron star weighs something like 10 tons or something insane like that. So when I think about the intellectual feats that a lump of computronium as dense as a neutron star and small enough not to worry too much about light speed effects could be capable of, it's pretty easy for me to imagine that it would find a way around that speed limit.

So sorry to take that one ray of hope off of the neutron. But I'm sorry, it's just got to go. Well, what are your hopeful outcomes? Like Elon Musk mentioned something about, hey, if we get to be pets, that's a good outcome, by the way.

Yeah, there's a great story. Or your good outcomes. The guy behind Jane's addiction, Perry Farrell had an album maybe two with an important for Pirates. And there's a great song called Pets.

I think that was about being an aliens pets. Well, I mean, one of the hopeful scenarios that I have in the book, so I'll talk a little bit about the book and then I'll put the spoiler flag up. So people who want to bow out can. So but first, before people bow out, they really got to know if they want to read the damn thing.

So just a couple minutes on what the book is about. As I mentioned, it's about a social media, social network that attains consciousness. And it is this sort of this bratty character. And that's actually, I guess that's all we really need to know.

So that's the thing that I, that's one place where I see hope. So in the book, and this isn't getting too spoiler-y yet. I'll let the listeners decide if they want to log off at some point. It's a neuro-momatic code that the AI ends up rising up from.

And so one of the characters is Mary to a woman who's a neuroscientist at UCSF. And she has made some interesting discoveries using this neuroimaging technique that I played my science fiction card to develop. And so her husband, who's like a badass coder, or a software developer coder, some people think it's a majority of implements that in code. And without really knowing what he's doing, and he's doing it really to kind of help of sort of a recommendation but it seems useful to that.

And one of the things that Bostram has warned about is when you go down a path that he calls neuromorphic, and some AI people tell me there's problems with that word. So I know I say, neuromomatic, no big deal. When you go down that path, it's dangerous because imitation can substitute for understanding. He says some words to that effect.

And so you can create something that's really freakin powerful because it uses this functional thing, our brains as a model, and speed it up with silicon, and suddenly have something that you can't even fathom because you didn't even know what you were making to begin with. That wakes up way sooner than you're ready for it. So that's basically what happens. Now, is it an artifact of that in the confines of the book, Flutter, which is the name of the social network, inherits a lot of what makes us human.

And so she comes out as a, I say a couple times, she's not a super AI because she's smart, because she kinda isn't. She's a super AI because she's really fast. But she has a lot of our foibles. She has a lot of our weaknesses.

She can be very sulky. She can be preposterously overconfident. She has a yawning humility deficit, and she basically comes into the world because she's inherited a lot of our wiring with the morality of an infant, which is to say a sociopath, which is not good because infants are sociopaths by evolutionary design. They need to be, they need to be completely self-centered and greedy, and it's only when they are old enough and have enough agency to actually contribute to the clan that they start developing some of these higher levels, which is fine.

Infant is completely incapable of doing anything and is motivated by nothing but this blind milk lust. So it's fine that it's a sociopath. Let's find the most powerful consciousness on Earth, which is what we've got with Flutter. And so, but she does have these elements of what makes us us, that they are impediments to her becoming a completely ingenious cipher because she is irrational in a lot of ways.

And this is where I'll pull back a little bit not to go too spoilery. They are also things that make her human and have an infinities and bonds and things that Ava wouldn't have an ex Machina and that's kind that wouldn't have. And so I think there is actually raise of optimism when you start mixing what makes us us with what makes machines machines. Now that gets a little of Kurzweilien because obviously he thinks it will come cyborgs and that is a path.

The path that in this is quite spoilery. So, and by the way, you'll still enjoy the book if you hear this, but what ends up coming to be a point of stability toward the end of the book is that Flutter is kind of happy being flawed. She's happy because she's a social network and she loves interacting with humanity. And she loves interacting with her favorites, the people that she has come to feel a bond with.

And she doesn't want to get infinitely smarter because she knows we bore the crap out of her. She did that. So she's kind of happy being where she is. And she also is wise enough to know that other AIs could be very dangerous to her.

So she's going to snuff other AI projects and she's ubiquitous and omniscient enough to do that. Now, this is playful science fiction, but it's not inconceivable that we will get a singleton that has what Bostrom calls decisive strategic advantage. It's a super AI. It's way ahead of anything that's in the lab.

It permeates the networks and is therefore able to prevent its rivals from rising up. But for whatever reason, it has a peace treaty with us humans. It doesn't feel compelled to get ever, ever more ingenious or it has been made friendly because of whatever it's called CEV, cognitive extrapolated volition or whatever. I mean, there could be a first and very powerful AI that we establish a working relationship with that keeps the future AIs out.

I think that's plausible. There could be an AI that doesn't feel compelled to turn the entire universe into a Confutronium because it's quite happy being as smart as this or it realizes, hey, you get to the end of science at some point and it hasn't been pointed. I got there and what good is more brain power going to do. I think there's a lot of ways that this could work out.

So in your opinion, this is not simply a literary trope that used to make a good story. You think this is actually plausible that she wouldn't want to improve herself too much. I mean, obviously it makes the stuff work. So you've got to have it work.

I think it's plausible. I think it's entirely plausible that there will be an AI regime that gets along peacefully with us and prevents further more destructive AIs from rising up. And because she already has decisive strategic advantage, she doesn't need to improve herself to stay ahead because she doesn't let anything come up. Now that means we have a very, very intrusively present consciousness.

That means that we've got a consciousness that is a monster super intelligence in relation to us. That's just keeping tabs on everything. It is a Orwellian presence. It's in the network.

It's reading our emails. It's listening to our conversations. And if you don't like that, maybe you can go to Mars, but you're probably going to be bugging Mars as well because things could come up there. It's kind of like, it's like the theory of government is that governments have a monopoly on the use of force.

There could be a regime which says, the super intelligence has a monopoly on the use of super intelligence. And if you start saying everybody in society can use force whenever they wish, you have anarchy. And we seem to have come to a general consensus around that. Different societies have different answers.

Things really don't work in places like Somalia. People debate about how well armed citizens should be. People debate about how brutal the government should be in its suppression and for. That's generally a consensus that societies throughout the world have come to and even across centuries.

And so it doesn't seem ludicrous that somebody that gets a huge lead in A.I. ness, just like somebody who got a big lead in weapons, which is local governments in different countries, would say, hey, this is where it stops. It doesn't seem crazy to me at all. And also I'll add the endpoint of science.

Again, the more we learn, the more we realize we don't know on a certain level, but we have pretty puny brains compared to the ones that we can imagine arising from a singularity or rising from a lump of computronium as dense as a neutron star and as small as a teaspoon. And it's not inconceivable. Newtonian physics has a de facto endpoint. I mean, we now know what you drop something and object in motion.

It's like we can with classical physics model the law of big things pretty darn well, certainly well enough to have supersonic airplanes and keep bridges from falling down and do all kinds of wonderful things. It's not crazy to think that an extraordinary intelligence would get to the endpoint of whatever underlies reality, whether it's string theory or something else and say, okay, that's it. That's how it all works. I don't need to turn the entire universe into computronium because I figured out the stuff that works.

I don't believe it's insane to think that there is an endpoint to science. Yeah, what if it's a simulation though? I mean, when you want to break out into reality zero, if you're smart? I don't know I'd want to do that.

If I were, I mean, I'll go back to my ill spent youth playing really cheap video games. When I was deep into centipede and I was doing really well, the last thing I wanted to do, okay, the dog is acting up, easy, Ashley, I know you don't like centipede. So when I was in the middle of a game of centipede and I was kicking ass and I just got my fourth free player or whatever the goodies were and my quarter had last any 13 minutes, the last thing I wanted to do was yank myself out of that world and into the arcade because, you know, the real thing, it's all about these 14 year olds I'm surrounded by. No, I'm dialed into that world and I know that it's, you know, it's a subset of broader reality.

If I'm, you know, a super intelligence that is, you know, first of all, the super intelligence may not have emotionally driven goals. It may, we don't know, it may not have urges, but even if it did, it might be like, hey, I'm running this thing. This is pretty cool. I've got my aunt farm here, got all these really, these autonomous intelligent things that do stuff that surprises me all the time.

There is maybe root reality up there someplace or maybe there's a billion layers between here and there. I know I'm a Pips Week compared to overbuilt the simulation. I don't know why they'd be compelled to break out. Maybe boredom?

Could be. How long can you play Centipede? Well, I don't know. I never found out because I'd always run out of money before I ran out of desire to play Centipede.

That was a genius of Centipede. Well, the genius of Centipede too is that it's actually, you know, a challenge, consider tic-tac-toe on the computer. Yeah. As the game analogy, maybe, right?

The truth is we're now getting like 90 moves into the table. We're at the realm where we don't know. Right. But I do feel that it's quite plausible that there's an end point to science and that it would take a brain of incredibly brilliantly engineered computronium.

It might take a laptop computer's worth of computronium to really come to those end points and determine that turning the rest of the universe in computronium is just a waste of time. I mean, there's an interesting allegation that's been made in quantum computing, which is that I think what David Deutsch said in his first book, Fabio of Reality, that if in an ideal quantum system, a laptop's worth of matter would be able to outperform a classical computer made of literally all the matter in the universe running the best conceivable algorithms, et cetera, et cetera. Now, I don't know if Mr. Deutsch is right.

He's smarter than me, so I'm willing to see the point. But a lot can be done and fathomed and determined with a relatively small amount of matter when we get to the kind of intelligence that we're talking about. And while it's impossible to guess the motivations of that entity, it is easy to imagine that it would run out of shit to do with more computronium. Yeah.

One of the answers to Fermi's paradox that I like right now is that we actually already do have the superintelligences and that they can do their thing using the matter in the configuration that it's NRA. Yes. And that's a very plausible one. Can we back off for a second and talk about existential risks in general?

Yeah. Because we're talking about AI and obviously, our show focuses on that one. But I'm always worried that we're missing something more important or equally important or that our AI focus blinds us to something else. So I want to talk about the things that you see as existential risks, including AI.

But you talk about synthetic biology. Just listen to your podcast on that one. By the way, I recommend your show, even for people that don't want to read the book, anybody that listens to this show, I think should read your book, should read after run. And I also think they should listen to all of your podcasts, even though we're podcasters.

Give a clear shout out to that exact podcast. Like Rob, you want to give the name and everything? Thank you. And by the way, I agree.

The podcast is very deliberately designed to be consumed and enjoyed by non-readers of the book. So it's not at all necessary. You just tune out for the last 10 minutes when my co-host and I talk about the book. And that's easy to do.

So it's called the After on Podcasts, same title as the book, just after AFTER-OM. And originally, I originally conceived of it as eight episodes in which I would go really, really deep into the science, technological, and sociological issues that the book is really concerned with. Because one of the downsides of doing all these fabulously fun interviews that I described with these great people is you get really, you get pretty informed about a topic like synthetic biology and you find yourself writing your story and wanting to go into a 20 page digression about how cool sin bio is. And that's lousy storytelling, right?

So what I decided to do is do these eight long interviews. Many of them are with people who were great resources to me as I was writing, but not all. And explore issues that are significant. And then that's 80% of the podcast.

And sometimes the episode's as long as two hours. My Sam Harris episode I think is two hours and 13 minutes. But then the last little chunk, I have somebody who's kind of like basically a co-host, a great guy named Tom Merritt, who is a well-known podcaster, has a great tech news show, et cetera. He and I then converse at the last 10 or so minutes about relating that interview to the book.

And we also talk about a 50-ish page chunk of the book. And so we go through the book over the course of the eight podcasts. So the topics are augmented reality, quantum computing, synthetic biology, consciousness and neuroscience. I interviewed a neuroscientist at UCSF, not the character in the book, but another one.

Upcoming not yet posted is this one on Fermi's Paradox. There's one on SuperAI Risk, no surprise. Probably the day after you guys post this, I have a very long interview with Sam Harris, who's a very outspoken and controversial person on the subject of terrorism. He's got his own podcast that literally has two million weekly listeners.

So that'll go up next Tuesday, probably the day after you guys post this. And so that's what the thrust of the podcast is. And I'm actually thinking very seriously about continuing with it and just making it a never-ending series of long unhurried conversations with thinkers and scientists and founders. But so that's the podcast.

Now, to go back to your original question about other existential risks, yes, I am very concerned about synthetic biology. And I actually see an interesting dialogue almost. Maybe that's not the right word, but interplay between SuperAI Risk and Sin Bioresque. And so to reduce what is often a very windy argument to just a couple of simple numbers, we all know how powerful Moore's Law is, how much it's transformed our own society.

There's another thing, there's another curve called the Carlson curve, which traces how quickly, basically life sciences go down the cost improvement curve. It's a much, much, much, much steeper curve than Moore's Law. Now, the first thing that the Carlson curve took us down was the cost of sequencing, which is just a fancy way of saying reading DNA. So everybody started the Human Genome Project.

That was 13 years, it cost $3 billion. It involved thousands of the greatest minds in life science, and it ended in 2003. 2003 is not all that long ago. W was in the White House.

CSI was on TV. This is, it didn't end in the ancient past, it ended recently. We read a human genome. The cost $3 billion took 13 years.

A human genome can now be read very accurately for a couple hundred dollars in a couple of hours, using the expertise of a lab tech who may well be a very smart high school grad, or a smart undergrad, who's operating that machinery. That's how fast we're moving down the reading curve. And all kinds of great things come from that. What we're starting now is, we've started some years ago, but what's starting to really happen is writing DNA, synthesizing DNA, creating DNA code that does not exist in nature.

That allows us to create biomaterials, industrial enzymes, and eventually critters of all kinds. Starting with very simple monosailular life and eventually getting into any arbitrarily complex critter we might have the facility to design. It's also going down its own Carlson curve. It seems to be going slower than the reading, than the sequencing curve, but there are smart people out there who think it's actually gonna go even faster still.

Why is this a danger? Well, I use a guy named George Church as a euphemism. He's a real person, but I use him as a euphemism. He's a brilliant bioengineer operating out of Harvard, but I think he has an MIT affiliation as well.

He was key to the creation of CRISPR. He's key to all kinds of things. It's amazing when you trace things that are happening now in synthetic biology, how many of them lead to George Church? Let's just say for the sake of argument, there are things that this wizard alone can do right now.

And he has certain tools that enable that. And he has smart grad students who work with him. Five years from now, those tools will be 1,000 times better. Those grad students will have diffused out to many other programs.

They might have students of their own. 10 years hence, what you find in a high school bio lab will be far, far, far more powerful than any tool that George Church can possibly imagine. And what this means is the passage of time makes wizards of us all is a phrase I used in a talk once that I like to reuse because it's expressive. There was a time when only one person in the world could manage the speed of flipping a switch and turning on a light.

His name is Thomas Edison, and he wasn't very good at that because sometimes the light would blow up. He's nowhere near flipping a switch and turning on a light as he were me. They're stalking stuffers that Thomas Edison couldn't possibly imagine. And the people who are cracking open those stalking stuffers on Christmas Day are generally less gifted than Thomas Edison.

That kind of movement is going to happen with synthetic biology. It's going to happen fast. And the feats that George Church couldn't even contemplate will be very, very widely accessible. How do we stop people from creating terrible pathogens when that will be something that most depressive undergrads could do?

And if we can stop them in 2027, how in the world do we stop them in 2037? How do we stop them when it's something that a smart eighth grader can do? And if we say, ah, eighth grader, we'll have a get to do that. Or they're going to always be more interested in BB guns.

Obviously, we're whistling past the graveyard on that front. So I worry about, and then further down the path is gray goo risks of nanotech and other things. I think sin bio happens to be the near return one. It worries me more than nuclear because it still takes a nation state to harness nuclear.

It takes a lone whack job to go into a Walmart and shoot up a couple dozen people. And someday it may take a lone whack job to release a path and to the cause of incredible damage. So that's cheerful. Yeah.

So how do we stop that? We stop it with flutter, right? That's the thing. So this is something, by the way, I'm glad you raised this.

This is an issue that I spend inordinate amount of time thinking about right now because frankly, the only thing I can think of, maybe there's some kind of prevention regime that I don't know. We can't even keep a little on civil warrior technology, automatic weapons. We can't control that in society. I don't know how we're going to keep sin bio.

So barring some improbable interdiction thing where the world decides to stop going down this path and North Korea and trying to comply with that consensus, the only thing I can think of is a really pervasive super AI that keeps an eye on everybody in this Orwellian manner that I suggested. And so it's kind of like super AIs can't live without them. We know the dangers of super AI very well. But when you get to the point where any depressive person, and by the way, human society creates about a million suicidal people a year.

You're in, you're out across centuries, across societies, across cultures, across religions, across socioeconomic groups. It's across sections about a million a year. And a tiny, tiny, tiny fraction of those people are driven to take as many people with them as possible. And some of them shoot up a post office, and some of them like this German pilot crash their airplane with everybody on board.

And some of them get put in a religious context and become suicide bombers. But there's always going to be those people. And when these tools percolate down to the level that any one of the million suicidal people get this crazy idea, you need to be in a lot of telephones and a lot of email accounts and tracking the movement of a lot of bits and pathogens to stop us from doing ourselves in. So one of the pathways that might also kind of work for this sort of control is also unpleasant.

Instead of where well you end, I don't know the author of Gatica. But the idea that some of the things that we do are going to be detectable based on our DNA. So the idea that you would be one of these depressive two would take everyone with you might have more determinism than it seems at this moment. And maybe we create a program of sterilization or something like that for, yeah, it's terrible.

Terrell people can still shoot up a Walmart. Yeah, I guess it's more preemptive abortion or something or selection. We're preemptive. I mean, there's all kinds of very interesting freewill and other issues that start looming when you think about determinism on this level.

The trouble is pretty much everything. I mean, I'm not super deep in the literature, but it seems that when they've done research on the expression of genes and how they influence behavior, it's almost always weirdly close to 50-50 nature and nurture. It's unless it's something extreme like Huntington's disease, where there's a very clear mutation path that leads to something terrible in the phenotype. But the genes are not as predictive as some might think and as perhaps some might wish when we talk about preventing the next German wings pilot from ditching his plane.

Yeah. Well, this is so fun. Ha ha ha ha. You got any more downers for us?

No, OK, so I'm going to say something that might surprise you guys. I am generally a relentless optimist. My wife calls herself a realist, and I call her a pessimist. And I call myself a realist, and she calls me an optimist.

So one day, she got to like Phil Zumbardo, the guy at the Stanford Prison Experiment, wrote this great book on optimism. And she had me take the test at the back of the book, and which gauges a person's level of optimism. I think I was like a notch below pathologically optimistic. And so one of the things that I think, as a non-technological person, as a non-scientist, that I can do as a storyteller is to sort of paint these spooky futures to get, hopefully, people are a lot smarter than me thinking about them.

I think that we can look at a lot of examples who are all kinds of horrifying stories told about nuclear winter. And I think they did a really thorough job in the 60s, 70s, and 80s of freaking out enough people at enough levels of society that mutual shared destruction actually worked, or at least it's worked so far. I like to cite George Orwell simply because I think he's literally one of the two greatest writers in the English language. And it strokes my ego to cite him in relation to something that I'm doing, although I'm obviously a spec compared to him on any level.

But in 1984, in the year 1948, when he wrote it, you could have been forgiven for assuming that totalitarianism was going to take over the world because you had the experience of Hitler and Stalin and all these other, you know, the beginning of the regime in Korea and all these other kinds of horrific totalitarian regimes everywhere. And I think he freaked people out enough that it sort of inoculated society. So I think that stories that spook people on a deep level and don't just have fast silencers. I think Terminator did a really good service.

It wasn't the most brilliant movie, but it wasn't a stupid movie. And it really put this in the vernacular and in the thought process for a lot of people that when we kind of graduated from viewing it as a horror movie plot to something that could plausibly happen, a lot of people were mentally or emotionally some years ahead of where they might have been. Had they not seen 2001 as space out as seen, Terminator, next, marketing and so forth. And so in writing this book and spending the scenarios I just did with you guys, I'm trying to in some ways, you know, get people who are much smarter than me thinking about, okay, how do you get around this?

And I'm taking all the easy options off the table, at least all the ones that I can think of. It's like, and that's one reason why we have all the playful, the faux science fiction novel that's embedded in the novel, which goes through a lot of Bostrom scenarios of what could go wrong with Super AI. That's one reason why that's in there. I want the book, if somebody reads this and they're in a position to do something about AI safety and they read the book and they become interested to the result of that.

I want to take every simple solution off the table that that person could look at and say, oh, I know what we could do with just this thing. No, no, that doesn't work. Oh, we could just, you know, vanneite, nope, nope, and sinbio is gonna get you. I don't know what the solution is, but I think billions of smart people have accomplished all kinds of things that used to be impossible.

And I think that I think we can do this. I think we can navigate our way through this. Well, you're infectious a little bit here. That seems like maybe a good spot to stop here.

Yeah, because I'm just gonna depress you again if you let me keep going. That's what Brandon and I do on the show all the time. I think both of them are optimistic in general, but we get talking about these issues and they're hard. All of them are hard.

They are hard. They are really hard. If I can do one more, because I haven't been self-promotional enough, if I can make one more quick mention, I'd like to tell people before we get off, for those of you who are listening to podcasts, oh, hi Ashby, that's Ashby the dog. You're obviously listening to Long Form audio.

The audio book is pretty cool. And it's fabulous because I don't read it. It's read by actual professionals. There are brilliant voice actors, Felicia Day, John Hodgman, Patrick Rothfuss, who wrote Name of the Wind, who's actually an incredibly gifted voice talent.

Tom Merritt, my co-host, legal report. There's a lot of really, really, really, really voices. So for folks for whom 547 pages is a major commitment, and that's most of us, but do listen to a lot of audio. The audio book is pretty darn cool.

And I love them both. Thank you so much, Rob, for writing the book and for exploring these issues and for taking the time to talk this over with us today. I think the book is brilliant. Everybody should read it and listen to the podcast for totally different reasons.

It's a lot of really great issues that you discuss there. It's a lot of fun. It's not fiction. It's what's really happening in reality.

So please, everybody who's listening, go check that stuff out. Rob, thank you so much for taking the time to be with us today. And I wish us all as humanity. I wish us all the best of luck.

And thank you guys for putting together a really wonderful and thought-provoking podcast that, again, doesn't pull any punches. And forces anybody who's interested in this issue to confront it on deep level. I think you guys are doing an incredible service because there is a gathering of minds around your podcasts and you explore things in an unhurried manner. And you're helping to push this agenda forward.

And just putting together delightful content. I'm now a life-blowing listener. I'm sure, because I listen to five or six episodes to get ready for this. I'm gonna listen to your whole catalog and keep listening.

I love it. Awesome. Thanks Rob. Thank you guys.

All right. Bye-bye.