This is Neil Erwits at the Center for a New American Security. I'm here today with Paul Shari, who's leading our new artificial intelligence initiative. Thanks, Paul. Hey, thanks for having me.
So we're here to talk about one of the coolest yet least understandable stories that we're going to see in the next few months. Apparently two were an artificial intelligence program that Facebook had invented its own language at which point Facebook got scared and shut it off. What happened here? Yeah, so the tech news is full of stories that are a little bit hysterical about Facebook killing this AI because it invented its own language.
So if we set aside for a moment maybe the anthropomorphization of killing the AI, right? Well, what exactly was going on? Well, one of the things that AI researchers have seen again and again and again is as it develops agents, bots, if you will, that talk to one another, they'll often start with them using English, but they'll tend to drift off of English for learning bots that can sort of come up with novel ways of talking because English is not necessarily the most efficient way for them to communicate. If you've ever listened to me talk, you know that.
So, right, so just like, you know, for different settings, people come up with code words and slang and shortcuts to deal with specific problems, these bots often do something similar where they come up with kind of their own shorthand ways of communicating. In this case, these bots were negotiating and one of the things that they would do is they were repeating phrases as a way to negotiate over how many, like to do much quantity between the two bots. And so they would repeat, say the word the or some other phrase again and again and again. In a way that would seem very strange to a person but would make sense to the bots as a way for them to basically count instead of saying a number.
Well, that, I guess that sounds scary but manageable, I guess. Why was Facebook's reaction so quick and decisive? Yeah, so I think scary is probably an overblown idea. Like I think there's a lot of that in some of the press headlines of like, oh, well, maybe this is a precursor to bots rising up against us.
I wouldn't sleep over it. The more interesting thing is, you know, we need to be able to understand why the bots are doing these things. So the fact that bots come up with novel shortcuts and ways to communicate is a good thing. That's interesting.
Yeah, that's an interesting feature, right? We also want to understand how it works. We want to understand what they're communicating, right? So if they're using some other language, we want to be able to kind of peel back and make AI more understandable.
And that's a feature of these kinds of bots that are talking to one another and other types of AI. They just don't have to have this system that works. We want to understand how it works so we can understand kind of when it will be successful and when not, what settings it's appropriate to use it in. So then when do we, what are the kind of safeguards that we can put in, you know, figuring that a kind of binary system doesn't really work, that you can't just say, no, we're not going to allow this system in any way, shape or form and then go on to something else.
What are the ways you can put safeguards on, say, this program that Facebook just shut down? Yeah, so I think within the AI field, you hear a lot of researchers. We talked about this in the last podcast, and the discussion between Elon Musk and Mark Zuckerberg, pushing back against some of the pop culture images of AI is something dangerous and scary. Because there are such a lot of great potential for this technology, I think it's worth thinking about as we begin to incorporate this technology into different applications in healthcare, transportation, and other things, how do we do so in a kind of safe and reasonable way?
Because some of these applications will have, you know, meaningful uses, right? If it goes wrong, it could have harmful consequences, right? If we have an AI that's giving a medical diagnosis or driving the car, we want to make sure it doesn't write things. That doesn't mean that technology itself needs to be terrifying.
I mean, look, electricity can hurt you, right? If you stick your finger into a socket, we have electricity in our homes and in our smartphones and in our headphones. When you plug in, you know, put it, you know, set an earbuds, you're putting electricity into your ears and know about your buds. You know, there's ways to use it safely, right?
So similarly, for AI, one of them is really making sure that our systems are more understandable, that we understand how they're functioning, what they're doing, if they're learning from data, how they learn, that testing evaluation is really important. And then the last thing is really thinking about, as we get to integrate things into society, how much freedom do we want to give systems, how much intelligence something has, and how much autonomy we give it are not the same thing. And so how much freedom do we want to give a machine to maybe perform certain functions, and how much- Or invent its own language. Or invent its own language.
And how much you would oversight do we have, right? So in some settings, inventing your own language, then maybe we don't understand very well it might totally be okay. And in other settings that are maybe more high risk, we may not be okay with that. Well, to re-fect this, I'm sure we'll be hearing more and more about AI programs that are shut off or that are highlighted.
So thanks, Paul. Thanks so much.