Skip to content
Stephen Hawking, Artificial Intelligence, and the UN Summit episode artwork

Nov 13, 2017 9 min

Stephen Hawking, Artificial Intelligence, and the UN Summit

from CNAS Podcasts Hosted by Center for a New American Security

This week, countries discuss lethal autonomous weapons at the United Nations, while Stephen Hawking warns of the dangers of artificial intelligence.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Stephen Hawking, Artificial Intelligence, and the UN Summit

0:00 9:57
of matches

Transcript Auto-generated, may contain errors

This is Neil Irwin, we're here today with Paul Shari, the Director of our Technology and National Security Program, to talk about AI, and one of our favorite people to talk about Stephen Hawking. Paul, thanks for joining us. Thanks, Adam. Paul, you've talked about Stephen Hawking's concerns about AI.

You're in the unenviable position of perhaps correcting the smartest person in the history of the world. So would you care to have at it again, can you explain what he said and why he might be a little off base? Sure, so Stephen Hawking recently made way of talking about the deep artificial intelligence, how the micro-static humanity. And it's not the first time that he's raised these concerns.

He's repeatedly talked about artificial intelligence as an existential threat. You know, it's interesting because this is a really hot topic of debate in the tech community and the scientific community. And it's one that we chatted about before a couple of times and you know about people like Elon Musk, Mark Zuckerberg, we just said Eric Schia, and our AI Summit a couple of weeks ago, and his topic came up and she was in a very different place, saying that he thought AI would be very beneficial and useful, and was fairly dismissive of some of these risks. But it's kind of hard to dismiss someone like Stephen Hawking.

He's a bright guy. So what is he seeing that many other people in the tech community are not? I think one of them is an issue of timescale. Hawking is looking at things in sort of a cosmic sense, right?

It's looking at things with long timescales. It's a brief history of the universe. Yeah, like things with millions and billions of years, right? So he's in some ways thinking about looking at the broader artificial intelligence technology, not just today, but where this might be going on in the long term.

And what would it mean for a world where machines eclipse humans in terms of their intelligence? You know, who knows? One more time frame that. Some of that might occur in decades could be centuries.

But a third number of researchers in the AI think that it's at least theoretically possible. I wouldn't say everyone. There's debate. But many of you.

Let me key you off of something you just said there. You said that AI and whether it's grossly oversimplified, good or bad is a hot topic of discussion in the tech community and the science community, you pointedly left out the policy maker community there, or notably left that out. That is, I understand the reason that we had this entire AI summit last week. Can you talk about what the learning curve is here for the policy maker community and what they really need to understand and get on?

Sure. So there have been some discrete conversations underway in the policy and national security community. There's a fairly robust debate underway on the rule of automation and machine intelligence in the use of force on the battlefield, autonomous weapons, or if you like the more sensational term, killer robots, there are discussions going on this week at the United Nations and Geneva among countries on this topic. But I think that there's, in this past couple years, the countries have been debating autonomous weapons.

This will be now the fourth year of conversations. We've seen AI technology on really advancing leaps and bounds. And so that's a sort of broader conversation where I do think the national security community is still kind of playing catch up, in part because the technology can be so rapidly. And we're just trying to understand, what is it capable of?

And how do we think about adapting and competing in this space? Not just in terms of civil automation, but some of the more advanced techniques today, things like deep reinforcement learning that we're saying, be very effective at things like AlphaGoZ, where they come out a couple weeks ago from DeFi. Let me piggyback then on these UN meetings that you'll actually be joining very soon. What do they hope to accomplish?

And is it possible for them, for the policymakers to catch up with the technology, or is this a tortoise and the hare, I guess? Yeah, and certainly the pace of technology advancement versus the pace of policy is very different. It's very much a tortoise and hare kind of phrase. You know, of course, that race, right?

It's tortoise, because I don't take the technology to cap on us. Yeah, that analogy. So, well, no, it's hard because technology's not slowing down. And so a couple of years ago, when international discussion started, it's not actually pretty proactive and a little bit ahead of the curve.

I don't know if that's the case anymore. And you've seen, where do people want this to go? What depends on who they is? There's a consortium of non-governmental organizations who want a legally-bodied ban on autonomous weapons.

There have been a couple of different letters from scientists and leaders in the tech community arguing for some kind of international solution, one of these letters is called for a ban on offensive autonomous weapons beyond meaningful human control. That was the sort of phrase they used. I don't know if anyone knows what that means. The more recent letter didn't call for a ban, it sort of asked that you went to save us from these dangers, which was a little bit vague or terms of what the solution should be.

I think there's a lot of debate about what policy approach, if any, is really needed. Because there's also, I think, a strong community of people who say, look, the laws of war protect us in principle or this will have all of the things that you might be worried about, right? Well, weapons, right in the market, and killing civilians, those things would be illegal. And so we should just trust the laws of war, which don't depend on specific technologies.

And that's, I think, also really valid point of view. So I think we'll see what the conversations go. It's been a tricky issue because, in part because the technology is moving, but in part because there are so many different perspectives that the table, there are many, of course, countries that don't want to see a ban or don't want the technology to be restricted. Putin and Russia, for example.

Yes, so for example, you know, Vladimir Putin, earlier this fall, talked about how whoever controls AI will rule the world. It's hard to see convincing him to therefore sign up for some treaty that bans the technology. Or even if you wanted to, how that would even be feasible, given how diffuse it is. And that's one of the objections that people have made.

At the same time, I think, you know, most reasonable people could agree that there are probably limits on what we might want machines to do in warfare and other things. You know, it's one thing when there's a clear right answer. You probably never want your car to crash into another car. This is a great function for automation to avoid accidents.

But what about a pregnant woman or a child? Which one would you rather it has to hit one? Yeah, then I'm always sort of more than that. I think like this concept of time in the case of automobile, do people say what the car can, you know, sort of here and hit this or go there, and that person, what's the right thing?

I tend to see those as, you know, either case the car is going to be better than a human because it's not like people in the midst of accidents are making these like complex moral judgments, right? In many cases, they're not paying attention to the current or something which is what's the answer. So the car is going to be better, in any case, just reducing harm. But there are places where we might want to say, you know, we want humans to be in control or to make these decisions.

Think about maybe sentencing in the criminal justice system or might think about things like medical treatments and what types of interventions make the most sense. You know, diagnoses make sense for where you want to use this technology, right? Like you don't have cancer or you don't. I want another right answer.

When you think about the intervention that's appropriate, that might depend on a whole variety of factors, you know, for some quality of life or other things where we're probably going to humans making those decisions. So last question and kind of the sum up one, can we expect anything to really come out of these UN meetings or is this just going to be a furthering of the conversation? Well, see, there've been, you know, two significant developments in the last kind of round. In the meetings of 2016, there seem to be some sort of nascent consensus forming around this idea of some degree of human involvement in a use of force.

There are different ways of formulating that. Some people like the term meaningful human control. Others don't like the term, they use other variations, like appropriate levels of human judgment, which the US government uses. There are different ways of formulating this, but there seems to be some gravitating in the discussion towards some, in the principle of blank human blank in the use of force.

This idea that there is some irreducible human element in the use of force that's really important. So you can see more conversations in that regard. You know, at a structural level, the conversations that evolved to a more formal body, like a group of government experts, it's called a GGE, what that means in terms of the substance, what that will be able to begin to hammer out, for example, a working definition, which is kind of one of the things on the to-do list we'll see. Well, great, Paul, thanks so much.

And when you get back, tell us about it. Thanks.

Frequently asked questions

How long is this episode of CNAS Podcasts?

This episode is 9 minutes long.

When was this CNAS Podcasts episode published?

This episode was published on November 13, 2017.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this CNAS Podcasts episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.