Hi, I'm Paul Shawry, Director of the Technology and National Security Program here at CMAS, and we're here to welcome you to the first of our 2018 podcast series on artificial intelligence and national security. I'm joined today by Helen Tomer, Senior Research Analyst at Open Philanthropy Project, and Jack Clark, Strategy and Communications Director at OpenAI. Helen, Jack, thanks for joining us. Thanks for having us.
Good to be here. You both work in some interesting spaces thinking about artificial intelligence and some of the transformative effects it's bringing to society. I want to start with a question on how people should think about AI as a whole and the types of changes that we're seeing in AI. Let's start with you first, Jack.
How do you think about the types of changes that AI is bringing to society? I think one of the initial effects is going to be stuff that we use every day will get cheaper in the background. So to the point of view of people, not much is going to change, but from the point of view of the companies operating many services or utilities, things are going to get dramatically cheaper. When people will become aware of this is when you have the first major products in their hands.
This will initially be stuff like self-driving cars, followed by robots, at which point I expect there to be a much larger conversation nationally and internationally around first things like automation, and then basically how this technology develops, issues of safety and progression. Helen, what about you from when you said it up before the project? How big is AI? How significant is this to you?
Yeah, I mean, I think the innovations that we're seeing is that this is going to be a really big deal for the rest of this century. We're already seeing a lot of progress in kind of the basic science of AI, so machine learning research, particularly deep learning research, deep reinforcement learning, that are getting results that seem really likely to transform a lot of the different sectors, a lot of different things that humans interact with every day. And I think, you know, obviously, what I'm kind of saying is making predictions as hard, especially about the future, but I do think we're seeing a lot of other evidence that this is going to be maybe the most significant sort of theme of this century, if that's not too grandiose to say. You know, Jack, OpenAI is doing, he's really one of the groups at the cutting edge of this research and putting out some very surprising results regularly.
How much are you surprised by the pace of progress? Internally, we regularly surprise ourselves. We attempt to be conservative in some of our forecasts, and in some of our projects we've overshot those, as in stuff that's happened earlier than we expected. However, that's traditionally where you're dealing only with software.
So if you're dealing with software, software progression does be faster than the intuitions of experts. But the moment you go to hardware, like real robots, our intuitions internally haven't been great. We've tended to forecast something will arrive and it arrives a bit later. And I think this is a general trend that you see when you talk to AI researchers, they're more optimistic than they should be about hardware and less optimistic than they should be about the ways of software progress.
Yeah, I think this is a great theme, and I think it's something that is not reflected very well in media coverage of AI, which often focuses a lot on robots. You know, the dystopian sci-fi stories are often about robots going wrong in some way. But I think of the most influential AI systems that are currently being used every day as things like Google search algorithm or Facebook's new speed algorithm. I think those kinds of things, potentially, you know, increasingly, Siri, Alexa, Google Home, as they get more capable, as Jack says, the possibility to develop and roll things out very quickly in software, I think, is underestimated compared to the sort of glamour of robotics.
And just to tag on to what Helen said, you know, robots are obviously very, very glamorous, but a smartphone is kind of a robot. And so in the last decade, we produced, you know, tens, hundreds of millions of tiny little cameras, which are billions actually, which are embedded in the world now. And so in the background, kind of invisibly, all of this information is now open for analysis and open for insight. And I think that we, as kind of a community, aren't ready for the impacts this will have, and especially when it comes to issues like national security, there are these very large epochal changes, which are just going to start percolating through in the next few years.
So can you give us some examples of those kinds of changes we might anticipate in the national security space, or things that would be relevant to national security? So I have a few examples. I mean, one is obviously we, many nations now field far more systems for collecting visual information about the world, whether it be in the form of satellites, drones, high altitude and low altitude and embedded cameras on the ground, wherever they are. Many nations still use, you know, analog analysts.
They use humans who are powered by hamburgers and sleep to go and look at these photos and draw bounding boxes around them. I think in the future, you'll see a transition to partial automation there, or to augmentation of the human analysts to sort of massively increase the effectiveness of their labeling. And that's going to mean that the awareness every nation has about the world will increase dramatically, which I think will change the strategic calculus around engagement and around aspects of diplomacy, as everyone will have access to slightly more information in proportion to their investments in AI. Yeah, and I think going back to the theme of software being able to be able to make progress in software more quickly and then roll it out more quickly as well, I think, you know, cyber is obviously the national security domain that involves the least kind of hardware on physical machinery, and I think there's good reason to expect that current approaches in machine learning and reinforcement learning, experts that I know are far from expert in cyber operations, but experts that I've spoken to seem to expect that we will see much increased ability to automate these kinds of operations, and that's something where you don't need to, you know, spend a long time doing procurement and building machinery and figuring out how to get it into the field, you can just build a system and deploy the system relatively quickly and potentially with great effect.
How do you think about proliferation of some of these technologies? This is something that comes up all the time in the national security context. Now, on one hand, some of the software can proliferate fairly easily, and one of the most advanced systems require a lot of computing power to run them and train some of these neural networks. So how do you guys think about the requirements to use some of these technologies, and who might have access to that?
In my head, I break it into two distinct categories. If you're doing what I just call a classification task, then not only are the technologies widely proliferated, but the ability of people to perform that action is also sort of distributed. Those technologies tend to be well understood and require the sort of computational resources that most people can have access to on their own. Now, if you want to do larger things, like tasks that Helen referred to involving things like reinforcement learning, where you need to model the world, simulate it, and run games within it, that's where the large hardware requirements come in, and that may be a potential point of control for policymakers, because we'll know that you'll need certain huge amounts of hardware to accomplish things, and these will likely be the things that are thought to be potentially dangerous as well.
I agree with that. I just had a couple of points. I'm not really sure how these will play out, but I think they're relevant to be thinking about. One being the distinction between the computing power required to...
I agree with Jack that computing power is likely to be a big determinant in computing as well. So the distinction between the computing power required to train an algorithm and to run the algorithm once you have a trained model, and the computing power for running these models is often many orders of magnitude less than that required to train it. So that's one interesting factor. And then another thing we're keeping an eye on is the progress in hardware and the beginnings of deploying specific computer chips, which will potentially be deployed in smartphones and things like this, which I think on both of those elements, I'm not completely sure how they'll play out, but I think they're going to be relevant to be in place.
I think that this speaks to a really important point in security that we're already familiar with, which is that when it comes to why you deploy encryption services, we build our encryption far ahead of the computational envelope. So you know that at some point you're going to be able to crack a given encryption algorithm, sort of algorithm given in our computation, the difficulty of that encryption far, far ahead of what you model as potential likely progress in computation. I think that we're going to have similar issues in AI, where we need to figure out ahead of time what sort of dangerous things look like and what that computation might require so that we're ready to think about it. And as Helen said, hardware is getting better all the time.
So once you've got a thing out there which is expensive, potentially dangerous, and currently limited in the number of actors that can deploy it, check back in five years and everyone will be able to deploy it on a mobile phone or something. And that effect, we haven't really dealt with it as a community of researchers and scientists yet, and I think we need to somewhat urgently. So what about some of the vulnerabilities of these systems, whether they're vulnerable to spoofing attacks or different control problems that can come from them? How should we think about risk of these systems, and then it happens to have vulnerabilities?
So I think the way that I hope we will be able to think about these risks is something that I think will be very relevant to their use and something that we should be proactively tackling, and there is a potential for systems to be deployed without vulnerabilities or issues being noticed and then having really negative consequences on the ground. I think there's a lot of concern in national security communities about relative advantage and competitiveness, and I think that's all very valid. But I think I'm interested to see and hopeful that we can see work to help these systems be robust and work well and reduce the chance of unintended negative outcomes that can be shared even between competitors, whether that be states or private companies who are also in very tight competition. So I think I'm about to start an argument with Helen, so you can position yourself between us if that seems appropriate.
My view is that it's actually good that these technologies are risky today because I think that it puts a break on deployment and it puts a break on rollout. Now, yes, it does increase the chances, but a less careful actor will roll something out that's non-predictable. That can have very damaging consequences, but I think broadly the technology today is at such a point of immaturity, but it's good we have this essential risk in A2A, which is going to force us to slow down and think about it. And just to add to that point, Ian Goodfellow and his team with Google published, I believe this week, a paper on his work called Adversarial Examples.
Adversarial Examples is the best evidence we have that you can smooth these neural networks very, very easily. And his group has done some work to show that it seems like as you get to more and more complex domains or domains that we call higher-dimensional domains, adversarial examples persist. So this lack of surety seems likely to persist in the future with at least some of these components we have today. And again, I think that's a good thing because it forces us to sort of maturely study it and hopefully it allows policymakers to plan a thoughtful rollout program that takes that into account.
I would agree and disagree. I think it's a good thing as long as, to the extent that we have common knowledge about the immaturity of these systems and about their vulnerabilities, I think one thing that I worry about somewhat is a version of a security dilemma playing out in which maybe two parties both are aware of issues with these systems and reasons to be cautious in deployment, but they're concerned that their counterpart might not be thinking about those issues and therefore might be worried about things that their counterpart might do and might therefore be incentivized to pay more risk. So I agree with you to the extent that there is a sort of knowledge on both sides and ideally common knowledge about that and I think that's a good reason to talk about that. So I'm going to just present a quick counterpoint to that.
No, yeah, go ahead. Continue to get out of this. I think, I still believe that this ultimately nets out to most people not deploying this stuff and I'll give you a tangible example as to why. So there was recent work showing that you can take these adversarial examples and you can make them work in the real world.
And the way it worked as an MIT student got a small 3D printed turtle painted a pattern on its back and now a neural network classifies it as a rifle. So that's obviously somewhat disturbing. I spoke to this student and I did a write-up with Tim Wang for my review of NIPs and we were saying, if I am a party and I have a bunch of machines, like let's say tanks or Humvees and I don't want the automated classification systems running over satellite imagery from another government to accurately know what these things are, then I could potentially paint them with adversarial patterns. And I talked to him about this and he said, yes, this would work.
You'd need to know the distribution of sort of resolution of your aggressor or the person that you're going against but this is something that many states place a premium on knowing. And that suggests to me you might be in this interesting sort of game-theoretic situation where you will assume that if you roll something out people are going to try to compromise you constantly and if some of these flaws seem like they are for the moment unfixable that could at least some of the larger nations lead to a break. Though I agree with Helen that this doesn't deal with the problem of small malicious and or ignorant actors who could pose a threat. Yeah, I think you'd be excited that these vulnerabilities and you don't know about it.
I agree with you. Yeah, I think one observation I'll make and I'll make each of your reactions to this is when I think about digital systems today. We know that they have lots of vulnerabilities. We know that software have bugs and zero days and people get inside them.
I think it's fair to say it will hold you back from deploying it in critical circumstances where you're dealing with lives that are at stake. I also think it's fair to say that if you're making things for the consumer that don't risk the consumer then you're going to be much more cavalier about this because it's a kind of tragedy of the commons where we don't have systems to enforce costs on people for not paying attention to this. Well, good. Any last parting thoughts from each of you?
Let me ask you one final question. Each of your organizations do really interesting work thinking about AI in different respects. What are some of the top things on your agenda for next year that you're excited about in the field of artificial intelligence? So for OpenAI there are two things that we're really excited about.
One is safety and this relates to what Helen's been talking about. We think that making people aware of both the flaws in this technology and also the ways it can be made safe is important so you want all players in this sector to be aware of that. And the second is meta-learning. I think many AI research organizations learn to act effectively in a new simulator you haven't seen before.
So learning to acquire new skills. And the jury is somewhat out as to how effective this will be but we have many examples in small domains where it seems to be effective and I think if we can crack that then we will further inflect the AI progress curve in a deeper direction which will probably lead to more podcasts like this. Good. And one final question for you, Jack.
OpenAI recently beat Dota 2. AI companies are constantly attacking some of these games. What's next few shows or the next big game-brand challenges? We need a somewhat limited version of Dota 2 involving one player versus another player and we have a goal of having a team of five players operated by an AI go up against a team of five human players.
We think that's quite a big enough challenge to have taken a bite off but that's what we'll focus on for now. After that I expect you'll go to much higher resolution games with more moving parts but I can't predict for you what they are. I play games myself so it's in the back of my mind at Christmas when I download a new game. Excellent.
I look forward to that. Thank you both for being here.