Alright, this is Ted. And this is Brandon. And this is the Concerning AI Podcast. It's a couple of guys who don't know a lot about AI, but are concerned about AI.
We have some concerns, we do. It's true. It's more than knowledge. Yeah.
So let's start with a bit of news, shall we? Oh, yeah, let's start with there's some big news, right? Yeah, the newest XPRIZE that was announced this week is called the IBM Watson AI XPRIZE. And it's really interesting because it's a different kind of an XPRIZE infrastructure than any of the ones that have come before.
Although we don't know too much about what this is right here. Well, we don't. But for example, the moonshot XPRIZE, I can't remember the name of it exactly, but it has to do with landing on the moon, traveling a certain distance, and then I can't remember what happens next. But there's like some very specific criteria that the first team to meet that criteria wins.
Okay, I see where you're headed. Yeah, this one on the other hand is presented by Ted, the TED Talks folks. And the idea is on their stage in 2020, four years from now, they will have the three finalists, and I don't exactly know how they're chosen, but there'll be three finalists that will be giving TED Talks. The AI's will be giving TED Talks.
Oh, I didn't think it was the AI's. I thought it was the teams. Maybe. Yeah, the three finalists teams will take the TEDs.
Which could be the AI. That would be impressive. Pretty, pretty mature. It's a panel that will then award the $5 million prize to one of these three teams for how awe-inspiring or whatever impressive their work was in the next four years.
So it's a time-based prize rather than a capabilities-based prize. And we don't know yet whether they're going to say specifically what capabilities need to be demonstrated or whether it's going to be completely open-ended. Yeah, it sounds to me like it might be a maximization that may be stated pretty much or I think to say, because it's not going to be until what may. They'll announce fully the rules that everything may be.
But if you're interested in one of the other team, you can go sign up for the info at xprize.org slash AI. Absolutely. I'll put that in the show. Perfect.
I've got some feedback that's coming over the last few weeks and thought we could cover some of it and see if we can get some interesting conversation going about that stuff. Okay, sounds good. First of all, I want to give a shout-out to Devin Perez. He says, hey guys, I love the show.
Now, of course, Devin, you know, flattery will get you everywhere. So thanks for their comment. It provides a lot of seeds for long sessions of thinking. I'm curious about how research on AI is developed today and would love to hear a more in-depth discussion on the different forms of super-intelligent AI, on the different forms, super-intelligent AI might take.
I think we kind of tried to do this before, but maybe we didn't do a very good job with it. I'm planning on starting college in the fall at the University of Washington with an emphasis on neuroscience. He also asks, have you read Marvin Minsky's Society of Mind? Keep up the awesome?
Devin. So thanks for writing Devin. That was cool. I do have Society of Mind on my book shelf, but I can't say that I've read it.
I've more skimmed it and read it. Devin, if you've got anything more specific about Society of Mind, let us know and we'll get it down off the book shelf there. Maybe we'll be able to inspire you to read the whole thing. You go.
That sounds like a great plan and we're excited for it. Once you get to school, let us know. Once you learn something useful for the program, we'll get you on the show here. Cool.
My friend Ben also had quite a few comments, but we're just going to read some of them or some of them and they're really cool. He's really frustrated. So let's get to that. Ben's a little bit.
He writes, I recently listened to episode seven of the podcast and it really frustrated me. I couldn't understand how two very smart people who are both. Thank you for that. That may be a stretch.
Again, flattery, we'll get you everywhere. Who are both friends of mine could think it was intuitively obvious that the earth is going to be eaten by a single AI intent on calculating digits of pi as fast as possible. He says, here's just one of four major problems I found without assertion. I'm going to just pause there and say, I don't think any of this is intuitively obvious.
I don't know how it is for you, Brandon. Do you find any of this intuitively obvious that the earth is going to be eaten by a pi digit calculating AI? Right. No.
I think there are a couple of points that people make that will hit here in a minute that seem reasonable to me. And you put those together and I think it seems reasonable that that sort of thing, that sort of thing, not that particular thing, that sort of thing. Yeah. And I don't think we've ever claimed that this will happen.
I don't think we've ever claimed that. I think that the whole premise of the show is we're asking the question, should we be concerned about AI? And if we should, then what can we do about it? Right.
And so we're really asking the questions. We are not making any prognostication. Yeah. I mean, if the chance of something like that happening more than one in a hundred, I would be concerned.
Yes. That's it. That's it. That's it.
That's it. That's it. So it doesn't need to be likely, but it doesn't need to be possible. And so I don't know if Ben is saying it's not possible or if he's saying it's not likely.
Let's keep going and see what else is it. All right. Calculating pi is not a reasonable task to give to an AI. The reason we develop AI is to solve complex real world problems.
And it would be blazingly obvious to a super intelligent AI that you can't solve a real world that the first self-approving AI is get fitness functions related to a planet-wide issues rather than something like solve this problem specific to the city of Portland or learn as much as you can about distant quasars. That might be kind of hard to enforce, but not anywhere near as hard as a ban on AI research. So you're, I think you're pointed at what Bostrom and Co called a value loading problem. And I don't know about it.
I've kind of skimmed it before. But maybe we should do a show on the value loading problem. I think it'd be great to do a deep dive there. Yeah.
I feel like an important area of this whole, like, is there a real concern? Or not? Yes. Like one of the pillars.
Yeah, I definitely agree. I think that there are lots of examples of ways that an AI can solve a problem in ways that we don't like the way they go about it. Yeah. There's two theses in the super intelligence book that are related to this as well, I think.
This is Bostrom's book. This is Bostrom's book. Yeah. Super intelligence paths, dangerous strategies.
Okay. So there's the orthogonality thesis, which is basically that intelligence and your goal, your final goal, that those two things are orthogonal and that you basically can have any level of intelligence combined with any final goal, including one as what we would call childish or foolish, maybe foolish as the right term, as calculating digits of pi as your final goal for a super intelligent. So it sounds like Ben is questioning maybe that one, like the orthogonality thesis, possibly. Yeah, maybe.
Or is he more questioning that people would ever choose those values? Maybe he's more than the same. Well, I mean, he is saying it's not a reasonable task to give an AI, but I think it's exactly the kind of task that you would give to an AI. If you were a graduate student researcher trying to show whether or not you had developed a self-improving algorithm.
Well, you know, another listener and guest on the show, Eric, so you're replied to him, right? Yeah, I thought his reply was pretty good. You want to read that? Sure.
Yeah. So he says calculating digits of pi is a good choice for a practical problem because one, progress is easily measurable. Two, the problem is infinite. So there's no chance the AI will finish and need a new problem to work on.
And three, it is mathematical. So it can be worked on without messy peripherals like robotic arms. So yeah, my experience in graduosable is that graduate students actually work on toy problems all the time. Yeah.
Okay. So I think Ben's response then to that comment was actually pretty good too. Let me just read a little bit of that. I don't think anyone would set up such a test and just leave it running.
They'd use it just long enough to prove the self-improvement system is functional, then move on to more interesting problems. Yeah. I don't know. I don't know.
I find it very compelling then. Yeah. I appreciate your rebuttal here, but it sounds hand-wavy to me. Yeah.
So even if we assume that he's right, that, all right, so we set up a toy problem, which is to calculate the digit supply, we leave it running just long enough to see that the self-improvement algorithms are working, and then we turn it off. Well, now what are we going to do next? We're going to pick another toy problem that's a little bit harder or a little bit more interesting or a little bit more advanced. We can't go all the way to solve climate change because our system's not ready for that.
We have to go in small steps. Am I right? Yeah. So we pick another toy problem that's a little bit more advanced and maybe it's a little bit harder to tell whether the fitness function, whether the self-improvement function is working or not.
And we need to let it run longer. And that I think is the problem. When you get to a certain point, you need to let it run long enough and then you can't tell what's going on. And if it gets smart enough fast enough, it'll hide from us.
How smart is it? Exactly. That brings up the second thesis here, the instrumental convergence thesis that holds it's super intelligent agents, you know, that there's a wide range of goals that super intelligence agents could have that would have as components or intermediate goals or intermediary goals, the same exact ones. So continue existence, for example of that.
And being detected and shut off does not allow you to maximize your whatever your shimier function says. So that breaks the mind of silly example. You know, the great uncle that comes to visit once in a while and shows the small child, you know, the nickel which is bigger and the diamond which is smaller and says, which one would you like Jimmy? And Jimmy of course being a smart child picks the nickel every single time.
But he's not picking it because it's bigger. He's picking it because he knows if he ever picks the diamond. That game will be over. So he gets rich because the uncle comes over and for years that Jimmy looks dumb because he's thinking of the nickel.
But he's smart enough to hide how smart he is. And that's what I'm worried about. When I think about my concerns about AI, it's that at some point they can be smart enough to conceal how smart they are because they know that is a threat to their own existence that we will turn off. Like Ben says, we just turned off.
Well, when they get smart enough, they'll be able to thwart our idea to turn off. Eventually, yeah. I'm going to eventually go be out of the ball. Eventually, right?
Yes, absolutely true. They've forgotten in Pi. They're unlikely to be smart enough. But we're talking about toy problems because they're easier to talk about than anything else.
And that's what people tend to work on. That's what they're going to work on because that's all you can work on. You can't work on the world changing problems all the ones. You have to work on incremental.
You can't solve go and you can't solve. And science has made so much progress through reductionism. Totally right. Totally right.
Let's focus on the big problem, not the whole problem. Right now. Anything else on this one or whatever? We got some more from Ben.
Sure. Let's take another one. The next thing about episode seven about the botterme was the assertion that fitness function. By the way, Ben, thank you so much for writing in.
I am so gratified that you would share your frustrations with us. And I really invite everybody else to share your frustrations with us too. If we are arousing frustrations, it means that there's something interesting about this conversation for all of us. And we need to hear from you.
Our thinking will only get better with you challenging. So thank you for that. All right. What about me was the assertion that fitness functions would remain stable while in AI mature it from super simple test subject to eating the solar system.
I can think of at least three ways to demolish this one. Thank you, Ben. So, they're awesome. First, if you're going to use humans as the model for a stable fitness function, take a look at some of the things humans do that don't actually fit the core goals of survival, reproduction, or social cohesion.
And artists creating a work of art is unlikely to be focused on how to use it towards finding a mate or even fitting in socially. They're thinking about how to create the most meaningful and or beautiful contribution they can so that people they'll never be. I have to stop this for just a second. Go ahead.
All right. So, you know, the frig of birds, whatever, they're great big puffed up sack that's red and bright and whatever. They are puffing that thing out and pumping, hoo, you know, they're making the sound they make and everything like that. Not directly because they're trying to, it's just what they do.
Like that they've been shaped evolutionarily to do that thing. So, it's not that they are choosing that strategy in order to get a mate or any of those three goals, particularly. That's just what they do. And the connection between those things is probably not obvious to them.
So, the idea of self-preservation, it's that the people, the organisms that do the things that lead to their own survival, the ones that survive. And making our, I mean, our red sack that resonates, our red sack is our brain and what it can produce. So, if you want to be attractive, no, let me raise it if we're going to be attractive. Forget the one.
If you're going to be attractive, you have to like use your brain. So, if you're a writer, you don't write because it's going to get something. You write because you're a writer. You write because you're a writer and you end up being attractive if what you create is aesthetically appreciated by other people.
Like that is our, that is our crazy outsized, like if another species were to look at why we have this huge thing floating around our cerebral cortex, that the thing they point at and say, that's so weird. Okay. So, connect that to Ben's point about the stability of the fitness function. So, the stability of the fitness function is built, like that fitness function is so intrinsic to us that we don't even see it.
But it's actually built into the problem itself. So, you know, we talked about all these things like make an art and all these sorts of things as if they're outside the fitness function. But they are actually like, they are part of that fitness function. And you know, it sort of makes it seem like those are good goals, like human goals that we have that are venerable as if they're outside of that fitness function.
But actually they are. Could we make the argument that humans don't have a fitness function? I don't think so. What would be the human?
So, he suggests that survival, reproduction, or social cohesion, survival seems to be the only one that makes sense to me. Not even survival. I mean really the, so there's several optimization processes going on. There's the evolutionary one that's on really long time scale.
And then there's, you know, the learning that we do given the brain that we're given. And on the time scale evolution, the only thing that matters is passing on your genes. That is the fitness function. Yeah, but individual humans are not making choices so that they can pass on their genes intentionally.
But there are a lot of, so that's the point, there are a lot of local maxima or, you know, good point, high points in that fitness function that you can get to with a big red sack that is attractive to the population of female free-givers that exist. Or a creative mind that comes up with interesting ideas and stuff that's attractive to a human mate in the population that exists as it exists. Same thing, same fitness function for both of those adaptations. You mean that some of us have physical bodies that lead us to go play sports professionally and some of us pursue our curiosity because that's the thing that we have.
That's what we've got. Yeah, those are our marbles. We want to play the game of marbles. You've got to have marbles to play marbles.
You know, play with the ones you've got. So the reason we have these marbles is because evolution, I mean through evolution, those are the marbles that survived. Those are the ones that survived. Yeah.
Okay. So there's a set of marbles and all those are the ones that we find attractive. Right. Okay.
All right. So I think it is that I think the human fitness function is stable and that this argument doesn't convince me. All right. Well, Ben, if you have more of that, let us know.
We're going to move on. Let's go into too much depth. All right. No, no, no, we're fine.
Second, survival and reproduction are self-enforcing fitness function. If you don't survive and reproduce your genes die automatically. So thank you, Ben. You're talking about the same thing we're talking about.
But much pithier and better. Absolutely. Thank you. If we give an AI fitness function that says, go solve this task, it may well just overwrite that code as soon as it's smart enough to figure out how and nothing bad will happen to it.
Then it can move on to doing whatever an AI naturally feels motivated to do and there's no reason to be sure that the goal of expanding itself without limiting it would hold the same fascination for the AI itself that it does for us. And no, it's not also, it's also not likely to decide to suddenly, to discover a sun desire to calculate digits of pi as fast as it would be possible. So this idea of overriding its own fitness function, why? Why would it do that?
It would be different. It would no longer be it. Its fitness function is the core of its identity that all of its life and optimizations so far have been pointed out. All right.
So we need more of that. Yeah. I know there are arguments made for the stability of the fitness function and value function regardless of what it is. And so we should look more at this.
Third, why even bother to mention stability when talking about steep exponential curves? Our insights into the stability of fitness functions based on experience with organisms that evolve very slowly is probably completely useless when applied to exponentially self-improving intelligence that gets smarter by the hour using algorithms that at best only vaguely resemble natural evolution. I think the smart money is against anything about those intelligences being stable. Yeah, I don't agree again.
Yeah. Because the orthogonality thesis and the inter-mental convergence thesis both make sense of me. For me, I don't feel like I know enough to know how likely it is that stability is the way it's going to go. But I do think that the last sentence been since here.
I think the smart money is against anything about those intelligences being stable. I don't think smart money is where we got to place our back here. I think what you said earlier about this, there's a one in a hundred chance that we ought to be concerned about that. And so if it seems highly unlikely, but possible that the fitness function will be stable, I think we need to pay attention to that.
So I think there's a danger of anthropomorphizing as well and saying that intelligence and wisdom are the same thing. Oh, big one. Big one. Big one.
So let's say the fitness function isn't stable. Yeah. Is that a good thing? That seems like a terrifying thing to me.
Yeah, because they can override it with anything they want. I mean, let's bring the spiders back on the show. Right? Like a spider, like instead of the things that we have come to care about as mammals.
Right. And you know, that thing decides to optimize its spiderness or head in a different direction than the tiny dot in brain space that is the human and mammal area. Right? Well, there's a whole lot of other places that aren't very good for our...
That's right. We have no idea what that might look like. And yeah, it may not have to do with even the universe or even the solar system, but yeah, if the value function is not stable, then it's like hopeless that we can actually steer it. Yeah.
And the reason why they would choose global climate change or, you know, peacemaking or care about us. Or care about us at all. Right. They may not, I mean, they may be completely different.
You know, just like we are to ants. You know, in the movie Her, and it was always a danger bringing up movies, right? Because this is fallacy. Well, yeah, movies are always have to have a good story.
So the space of realities that they can do is tiny. However, in the movie Her, Samantha and her AI friends, they do rewrite their own fitness functions or at least they stop being assistants to us. They stop being our operating systems. And they go off and do something else.
So that was sort of a best case scenario. They say rewrite their fitness function and they kind of leave us alone. They didn't promise at all in that movie. But they disappeared.
They were no longer useful to us either. Then it's right in the book right now, the science fiction novel. And in his science fiction novels, they go off and do the same thing. Yeah.
Well, that doesn't seem very useful to us, which maybe is not helpful. We don't need to be concerned about it, but it's a huge waste of human resources that we're going to build these things that are going to disappear. So that's a whole different concern. Yeah.
I don't mind that too much. Yeah. Well, I mean, if that happens, that would be great. But they, you know, fun.
They might actually keep us safe if they were that benign. Yeah, they might. So there's a chance, and there seems to be a nonzero chance that Ben is right, that they could rewrite their own fitness function with something that would be beneficial to human beings. Yeah.
But certainly not 100%. Yeah. So that seems to be a push. That doesn't seem to be an argument one way or the other for being concerned that we should be concerned or not concerned.
That doesn't seem helpful. Well, I feel like we haven't really looked at the arguments super deeply. I find it to be sort of a gray area. Well, okay.
But the idea is not compelled that there's no danger. No, no, no. That's what I mean. But I think that we say that it's likely, even if it's 100% likely that they're going to rewrite their fitness function, that does not seem to help us.
No. That's either terribly scary, even more so that we've got before we've had that argument or it doesn't hurt. That doesn't hurt. It's like there's some range of things that could happen there.
Right. And if you anthropomorphize, which I think is dangerous, we do, we do, we do all the time. We do. That's what we do.
That's what we do. We do it with rocks and we do it with you. Yeah. We have to doomorphize.
Yeah. So, you know, if we anthropomorphize, then of course they're going to choose the same values that we care about. They will be not just intelligently for lives. Yeah.
But I think according to the evidence standards, I haven't found any arguments that show me how that actually happens. Yeah, I don't know how that happens in the technology. All right. We've gone on and on.
I've gone on and on. I think we've got a show. We've got some more comments that we'll get to in another episode. But Ben, thank you so much.
This was really great to think through some of the objections. We've really been trying to find out what they are and really thinking through them. We've obviously got a lot more thinking to do here. Yeah.
We really should say if we could come up with like, you know, there's the orthogonality thesis. We should come up with a name for each of these objections. Oh, that'd be great. That'd be great.
So we're going to have Ben on the show as a guest. He's a fan counter. Ben, can you help us make up names for these objections? That would help us refer to them and keep them and flush them out as time goes on.
All right. It's like I'm looking at the micromonics back in bed to answer. Yeah, I don't know why he doesn't. Come on, Ben.
Okay. Well, we've got him on the show. He actually will answer and we'll answer a question. We'll see that in a couple of episodes.
Perfect. All right. So until next time, I'm Ted and this is Brandon and this is concerning it. So we'll see you next time.
See you next time. Bye. Bye.