Hi, I'm Paul Shari, Director of the Technology and National Security Program here at CNS, and I'm joined today by Dr. Andrew Moore, Dean of the School of Computer Science at Carnegie Mellon University. Andrew, thanks for joining us today. Happy to be here.
You're in a really interesting position in artificial intelligence, both yourself and Carnegie Mellon as a whole, doing lots of exciting and amazing things. Carnegie Mellon was the site of researchers who won a DARPA Grand Challenge recently, beat some of the top human poker players in artificial intelligence. I want to start by asking you, how much are you surprised by the progress we've seen in artificial intelligence in the past few years? I knew that it was going fast, but the last 12 months in particular have totally bewildered me.
My faculty keep bringing crazy new things to me that I had no idea we'd be doing this decade. What kinds of advances in AI that are coming down the pike soon are you most excited about? I think there's a lot of good in being able to understand the human emotional state and understand many aspects of us as real beings. So the idea that computers communicate with us just through typing or listening to our voice is not enough anymore.
We really can start to have proper discussions where they perceive how stressed, how happy, how excited we are. I think that is going to completely change how we interact. You know, I think the people who don't work with this technology, sometimes they can just seem like magic. You constantly hear of new problems that AI is solving.
What are some of the limitations of AI systems today? So it very much isn't mathematics, and a lot of it is just linear algebra. So the clever thing has been taking all these things that we thought were extremely fancy versions of intelligence and breaking them down into their mathematical components. There absolutely is no magic.
This means that there really are serious limitations. We, for instance, no one I know of in the world of artificial intelligence is making any real progress on concepts like general reasoning or reasoning by analogy in the way that humans do. The thing is, we are very, very good at generating artificial experts in lots of very narrow fields, and if we didn't even bother with anything like general intelligence, we could cruise on that for the next 20 years and completely transform the world anyway. What are some of the techniques or methods that you see as most promising towards getting some more of this general purpose of reasoning?
So the first thing which, of course, has been incredible is machine learning. The reason learning is so important is in the early days of artificial intelligence, if you wanted your robot to do something smart, it would have to keep on thinking about what is the repercussion of doing this, what is the repercussion of doing that. And back in those days, we had no way of predicting what's going to happen as a result of doing something. That's what machine learning gives us.
By seeing many, many examples of hours driving down a freeway or interacting with humans who are asking questions, you can automatically build up models so that the AI can predict what will happen next. So I'll start with your debates about deep learning in particular and sort of to what extent this is something that's going to make some headway towards more general purpose of reasoning versus others critiquing it and saying, look, there's a lot of limitations to deep learning. It's not going to get us there. Do you have any thoughts on some of this technique versus others?
I do think deep learning is incredibly powerful, but it is only part of the system. When you actually design a full AI solution, you have the module which does the learning, either learning to classify things or learning which actions to take. But that is a module in a larger system. For example, in many realistic systems, especially in the defense area, you have to incorporate game theory or strategic reasoning.
So that actually is a whole different class of computation. But if you build the optimal system, which always takes the optimal action in any scenario, then it's very easy for the opponents to predict what's going to happen and defeat it. So things like the old theory of Nash Equilibria, which are used in the Cold War for big state-level decisions, we're now programming that into the basic sensors and actuators of robots. So we're now in this interesting place with AI where the technology is mature enough that we're starting to see it apply in a variety of different industries out in the real world.
Talk me through some of the challenges as we start to move this technology from the lab into the real world. Very good. It's exciting, but there's a few big problems. One of them is if you've got a great idea for an AI solution, it will probably work, but you probably won't be able to hire the people to build it for you.
So the most important thing if you want to set up an AI shop is to figure out what is going to make the very limited pool of experts decide that it's you that they want to tie their flag to. So clearly universities like Carnegie Mellon, we are very aware of this and we're pushing extremely hard. For instance, we're just introducing a new undergraduate major in artificial intelligence specifically so that we can be training up more of these folks. But I can't emphasize this enough.
If you try to do an AI project and you don't have personnel around who have some experience of this, it's going to be very disappointing. That's one big problem. The next interesting thing is the availability of data by itself is not enough. I've heard people sometimes use the phrase data is the new oil.
And although that's amusing, the killer here is that what we call unindexed data, data which is just a bunch of sensory readings or containing a bunch of numerals which have no specific meaning associated with them, useless, in fact, to some extent worse than useless because they can delay what's going on. So what we need there is beyond having access to large amounts of data, we have to be able to link the data elements between many systems together so that the AI is able to solve big problems. I'm very concerned about the idea that AI is considered some magic which takes huge databases of raw data and creates something clever out of it because it doesn't. You have to have the indexes.
Talk to me a little bit about the challenges in human capital here and the competition for talent, whether it's among universities or industry or as governments begin to make sure that they're engaged in space, how they're going to think about human capital. Right, so this is something which keeps me awake at night. I shouldn't complain, though, because there could not be a better situation to be in than the fact that the whole world needs people with these skills. My problem is that I and many other folks in the education industry, we have to make sure we don't destroy the seed corn that the brilliant AI educators and the people who are thinking of inspiring research ideas which creates the next generation of big thinkers, they don't all disappear off into highly profitable short-term industries.
Because if we do that, if we hollow out the huge strength of the AI scholarship which has taken the United States into its current leading position, then we're going to quickly fall behind China. What is your sense of how the United States stacks up today versus other countries like China? So an important numerical score is that a decade ago about 5% of the papers in the big international AI conferences were from China. Now it's up to 50%.
So there's been an incredible ramp up there. And of course, the steep curve is continuing to grow. There are some areas like, for instance, the acceleration of deep learning so that you can make it democratized so that more people can try it out. I think China is doing very well and we're absolutely neck and neck.
Some of the mathematical foundations of, as I described earlier, linear algebra, I really respect so much of the work coming out of China. There are some other areas. In particular, there's an emerging area of game theory, safety, and sort of, if you like, the codification of the free laws of robotics, where I very much see the United States is still well ahead at the moment. There's no way that you can say the codification of the free laws of robotics and not explain what you're talking about there.
So what do you mean by that? What are people working on in trying to avoid this concept of different rules of robotics? Very good. So as you know, Isaac Asimov, a famous science fiction writer, postulated that robots would operate under three clear rules, which the first one is you shall not harm or cause harm through an action to any human.
And there were some other important similar rules. So this, while we were very inspired by that, that turns out to be not specific enough for the engineering of safety-critical systems. So if you're going to have an intelligent collision avoidance system in a vehicle or in an aircraft, you have to really specify things like, should I risk a one in a thousand chance of an injury to my occupant in order to reduce the potential accidental destruction of a piece of property at a one in a hundred chance? And those are the kind of things which we have to code up before we can proceed any further.
Another great example is an autonomous battleship built by Leidos with some of the autonomy components written by Carnegie Mellon, where the main part of the autonomy work was, one, making it be able to make decisions for itself, two, being able to show that they obey the doctrine of sea navigation and right away and all these other rules of the road. This is the Navy Darball Act of Sea Hunter Program. That's correct. So when we're talking about engineering AI systems that do, they're operating in a high-consequence environment, whether they're on the road, they're ships on the high seas, maybe they're national security cyber systems, how should we think about building these in a safe way, designing and testing them to make sure that we're confident in their behavior?
So I need leadership from the country, from legislators who are democratically elected in Congress to make sure we do the right thing here. I have my own personal view, which I will now tell you, but I want to emphasize this isn't for any one person or engineer to describe. My personal view is very conservative. Even if I knew that, for instance, launching a fleet of autonomous vehicles in a city would reduce deaths by 50%, that I wouldn't want to launch it until I came across some formal proofs of correctness which showed me that it was absolutely not going to be involved in unnecessary deaths.
There is a possible, very interesting cultural aspect of this. I do believe that the United States will follow this policy of really holding ourselves to a much higher standard than a simple utility measure would be. We're not going to be willing to give control to the robots just for, say, a 10% increase in safety. It's got to be super safe and we've got to be able to prove that it's safe.
Some other countries with different safety standards may well actually move initially ahead faster because they say, well, let's just get it out there, let's get these devices onto the roads, into people's homes. I'm sure there will be a few teething troubles but we'll keep on moving. I advise against operating that way because I think it will bite us when the population starts to distrust engineers after there have been a series of AI disasters. What types of things concern you in terms of disasters or risks or consequences if things went wrong?
So the biggest initial fear, again, I don't want to be over-alarmist about this. We have very responsible engineers working on this but if you have trained up a device by training it on lots and lots of different experiences so they go up, maybe as one of our faculty is doing, they're building robot arms which can be attached to wheelchairs to help folks with high spinal cord injuries and still be able to move objects around. That's really, really cool, right? That could be very, very good for the world.
If a system which is learning to improve itself all the time is looking very good and seems to be just getting better and better there's still a question of whether some completely new set of circumstances might crazily confuse it. This kind of phenomenon was implicated in the fatal Tesla crash last year. For instance, big internet companies often see this that they might build a huge system to predict what advertisements someone's going to click on tomorrow and it works very well for 18 months but then suddenly perhaps because of an external event everything changes. The thing that you've got to watch out for is do machine learning algorithms bad decisions.
I would strongly advocate for as much technology being put into the monitoring and diagnosis of behavior as machine learning systems as we put into the machine learning systems themselves. Are you seeing that right now in terms of either the research or how these systems are being implemented? I've had experience in industry and in academia and during my time in academia I've interacted very frequently with the government. I would say that industry is much better at this than either academia or the government.
There where there's millions of dollars an hour at stake if your system goes down there's been huge amounts of work put into not only building great machine learning systems but formally being able to diagnose their behavior. In academia we're much more forward thinking so we're just trying to make the system behave with less consideration of making sure it runs over many years. My exception to that is the work by Dr. Manwala Veloso at Carnegie Mellon where she has deliberately set up a project with robots that just continuously roam the halls of CMU and they've actually been running for years now.
The idea being that that way you can establish the engineering practices you need to not just succeed in a little YouTube video but to keep on and on succeeding. So here at CNAS we think about different technologies being used in a national security context in conflict and war when you might be faced with situations that you can't always replicate in testing. And in fact you have an active adversary who's trying to manipulate your system or trick them or trying to present you even people in a situation that they've never experienced before. How do we think about designing AI systems that are robust in those kinds of environments when the real environment we may not be able to test 100% ahead of time?
Very good question and my answer will differ from other experienced AI people's answers at least some of them. I do see a need for what's known as symbolic reasoning on top of the neural networks where you in fact if you've got a goal like get this injured person to this location within the next 20 minutes you are reasoning about how to do it which will include putting together inferences to try things that you've never tried before and actually kind of do lateral thinking to solve the problem. And that's okay if you've got the high level symbolic reasoning which is talking to lower level modules which are doing things like locomotion and healthcare then you can get this sort of almost unexpected high quality behavior from your AI system. If we ask a neural network to just do its magic there's no evidence that it will be able to jump out of the previous kinds of things it's done and solve a new problem.
In what is your sense of how far along we are in developing this kind of symbolic reasoning that you're describing? In some places we're in really good shape as you know if you use maps or Uber or other things like that geographical knowledge is very well represented and route finding does use a huge amount of machine learning to estimate costs of routes and the amount of traffic that we predict in the next 5 or 10 minutes but it still pieces together a lot of extra information about your destination and other information. So in that case it works well and the reason it works well is because anyone running those systems has got excellent databases of all of the locations that are involved where it's less successful at the moment to do symbolic stuff is if you're in an unmapped area and one example of that would be healthcare regulations. Oh I was going to say the trouble areas of Pakistan but yes.
No that's true but aside from physical mapping it's areas where we have not put in the effort to map out all the agents and activities and entities which are involved in a situation. Excellent. If there was one message you'd like to pass along to US policy makers as they begin to think through how AI will be incorporated into national security applications what would that be? I think that it's a national security issue that we could places like Carnegie Mellon producing the very strongest computer scientists in the world could be creating a very much a two-tiered society in the United States where we have the small technical elite running the country and a whole bunch of people who are just thrown into a secondary position and at some point I would see some sort of actual internal strife as a result of that.
I think it's incredibly important to design our AI strategies so that we're bringing the whole population along just like the United States did when we first went into industrialization and we decided we're going to educate the entire population up to 16 years of age because there will be a huge problem. if 75% of the population is left useless on the farm. So bringing along the whole population to engage in these conversations about how AI is being introduced to society, you think it's essential. Yes, if we're going to creatively use drones to save lives in a disaster scenario, before that, we're talking to the populations of cities about what would excite them, what they would rely on, and what would actually cause them more anxiety than benefit.
That sounds excellent. Okay, thank you very much for joining us. Thank you, I enjoyed the talk.