Hi there, I'm Azim Azhar and you're listening to the Exponential View podcast. Now, every week I come together with a brilliant mind to explore how exponential technologies are shaping our near future. Before we get started, tiny reminder, my first book comes out in September this year, and I am sure that as listeners to this podcast, you are going to love it. So please go and find it at your local bookstore or online retailer and place a pre-order. today. The book is called The Exponential Age, if you're in the United States or Canada, and simply Exponential if you're in any other part of the world. Please look it up, The Exponential Age or Exponential by Azim Azhar. I would love you to be one of the first people to read it. Now, my guest this week is Bruce Schneier, a leading expert on all aspects of computer and internet security. He is a New York Times bestselling author of 14 books, including Click Here to Kill Everybody. He's a fellow at the Cyber Project in the Belfer Center of Science and International Affairs at the Harvard Kennedy School, and a fellow at the Berkman Klein Center for Internet and Society at Harvard University.
In between all of those commitments, Bruce has found some time to speak to me. Bruce, welcome to Exponential View. Thanks for having me. Now, on the subject of security, how do I know I'm really talking to you and not to a fake you? You kind of don't. So here we are on Zoom.
If you know what I look like, that's going to be a decent authentication system. It's really hard to do that kind of fake video live. We talked over email. It's relatively easy to spoof. My voice, it's easier. But again, doing it live is hard these days. You just have to accept that it's me. It's going to sound like me. I'm going to say me stuff. So kind of why do you care? We care because we've always been about authenticity on this podcast. I will give a short disclaimer to listeners.
I think I'm talking to Bruce Schneier, the world-renowned internet security expert. And how do I know you're actually an exponential podcast? It's a great point because we've never met before. So it could be a fake interview with a fake expert, or it could be the real thing. You recently published a really fascinating long essay on the coming AI hackers. I found it really interesting. It touched on so many questions of what AI technology really is, how it plays into our systems most broadly, what kind of vulnerabilities it creates.
I suppose one interesting observation is that you use hacking in a broader sense than perhaps the narrow mainstream media sense that people use when they say the word hacking. This is something I've been thinking about for a couple of years now, that I'm trying to broaden the definition of hack to more general social and technological systems. So we know about computer hacks, and that's where the term comes from. But I'm thinking of it as something more general. So it's something a system permits, but is unanticipated and unwanted by the designers. It's an exploitation that subverts the rules of the system. It's something that's allowed technically.
but not really allowed. So a good example is the tax code. The tax code isn't computer code, but it's code. It's a series of algorithms with inputs and outputs. It's a set of rules. It's deterministic.
And that code has vulnerabilities. We call them tax loopholes. It has exploits. We call them tax avoidance strategies and has an industry of black hat hackers whose job it is to find these exploits. We call them tax accountants and tax attorneys. So the notion of a hack easily extends to the tax code. And I think it also extends to financial regulations, to political systems, any place where there's a set of rules. And there are people who want to subvert those rules for their own benefit.
And this work on AIs and hackers comes out of this notion of these more general hacks. What's fascinating, of course, is that you've identified a question of the limits of knowledge when it comes to codification, which is that can we really capture our intent when we come up with a set of rules that govern a deterministic system? Or are there aspects of our intent that we can't access when we come to design them? Is it simply too... complex to actually make our intention explicit in the set of rules. I think back to an old artificial intelligence project, which you're probably familiar with by a researcher called Doug Lenat, called CYC, where they sought to encode all of common sense in a set of rules. And it was a three decade research project because it turns out that common sense is this vast and expansive landscape. We seem to often struggle to capture every possible use case, every possible intent in the codification of systems. I'm thinking about these systems that we design and let's say it's a system for a game.
right go is very simple you can codify all the rules to go in a way an ai can understand it and they're not going to hack the rules in in the way we might think of it they'll invent new strategies that's different So sports is actually full of hacks. In basketball, dunking was a hack. The rules didn't prohibit it, but no one thought of it. Formula One racing is full of hacks. As engine designers find things that aren't prohibited by the rules, but that the race committee's never thought of. I mean, one year, one group showed up with a six-wheeled car. That's right. The Braben. It was the Braben six-wheeler. And people said, what's that? And they looked in the rules. It never said how many wheels the car should have. Now, of course, the rules say you can't have more than four or actually more or less than four wheels on your car.
So there is that notion that rule sets we design will be incomplete because we just don't. Think of all the things. And the other is more pervasive. And you were getting to this, that rule sets are, I think, by definition, underspecified, or at least intentions and desires are underspecified. Now, some of this is cultural. And if we're different cultures, you might specify more about what you want. But there's this theorem from philosophy that you can never specify all of the avenues. And I think about genies. Genies are known to be pedantic about your wish. And they will always be able to grant you your wish in a way you wish they hadn't.
because there'll be something you didn't specify. So this reminds me of Michael Polanyi and the French philosopher and tacit knowledge, right? There are things that we know but cannot tell. And those things seem to be embedded actually in a lot of areas that relate to AI and automation. I mean, one of the challenges of automation within work is that a lot of work has implicit, learnt attributes, particularly when you interface with other humans, other people, but also when you connect to technical systems, that there are just ways in which things have evolved. And those things are not codified in the worker's manual. It's one of the reasons why automation within the workplace has taken a different path.
to the one that was perhaps predicted seven or eight years ago when people were saying, you know, there's going to be a robo-apocalypse of jobs. It turns out that actually there's a lot of tacit information that's not written down that's hard for a robot to come in and take on. And this is why you see the AI-human hybrids working best. AIs are good at boring and repetitive. In my field, they can look for anomalies and look for attackers, and maybe they can find vulnerabilities in source code. But right, that metahuman stuff is very hard for an AI to do. But having the AI do the boring and repetitive stuff frees...
So right now, AIs are finding vulnerabilities in computer code. So it's pattern recognition. And then humans look at it and say, yeah, that's a good one. No, that's not that great. So if we come back to this idea of AI hacking that you so beautifully exposed in your essay, what does an AI hack actually look like? And how is it different to a human hack? So it looks exactly the same. So let's take a good example, hedge funds. Hedge funds are full of hacks. I mean, the decades of hedge funds is the story of someone finding a loophole in the financial regulations and hedge funds exploiting it. And then either it's closed by a law.
or it's codified and becomes the new normal. That's what happens. So in this future I'm postulating, AIs would come up with the hacks. So an AI would pop up and say, look, here's a clever hack. And then a human, maybe working for some investment firm, would say, that's great, let's implement it. Now, there's no real difference between the person and the AI coming up with it. One is going to be faster. But there's a couple of hacks that spring to mind from some of DeepMind, the AI research lab's early work with reinforcement learning algorithms. The most famous one, I think, is Breakout. So Breakout is an old Atari-era video game where you...
have a little paddle and you bat a ball around against a set of bricks and you have to break your way through and it gets faster and faster. And Breakout figured this sort of superhuman strategy, which was if you built a tunnel on the left-hand side, you could fire the ball through it and then the ball would just stay sort of constrained, bouncing up and down and zip out the wall for you without you having to use the paddle again. And that felt to me like a kind of hack that would be really hard for a human to realize and figure out and execute because it required such even precision to get to that point where that strategy would work. So as someone who played Breakout as a child on those Atari games, we all knew about that. So humans did come up with that hack. What's great about the story is that the AI independently came up with the hack. It wasn't told about that strategy. And I think it was after 400 games, right? So here's an example of some kind of parallel construction. But I'm curious, if you hand an AI all the world's tax laws, would it figure out that it makes sense to incorporate your corporation in Delaware, to register your ship in Panama? Could an AI come up with that? And what would it invent that we haven't thought of yet?
So it really is AI as a creative agent. Because you don't tell the AI any rules except make money. Pay less tax. And it's going to optimize it in ways that we haven't thought of because AIs go down decision pathways that humans don't. And it's going to be fascinating. And of course, AIs don't fatigue when they go down those pathways and never gets bored. And also hold more variables in mind. Hold more variables in mind. It's going to be a very different decision process. And to me, that's interesting. So what's a hack? The filibuster is a hack. It was invented in Roman times. We actually know the senator who invented it, who realized that the rules of the Senate said that all business must be conducted by sundown. And if he just kept talking...
then sundown would come and no business would happen So there's an interesting hack that has become normalized in our political system. It's no longer thought of as a hack. It's the way things work. Could an AI come up with that? The point about the DeepMind Breakout example, and there were many others that came through reinforcement learning experiments over the last four or five years, is that AI systems are able to discover this without those being prior inputs into the system. And I guess as we move into a world that is not just merely more codified, because we've had codification since the Code of Hammurabi, but it's codified digitally. And those digital codes interact through APIs and integrated systems.
that you actually have a wider set of terrain and different rule sets where the kind of discovery process of an AI can actually play and roam. So not only is this tool extremely good at discovering potential hacks, it's got a larger landscape in which it can find them. That's right. And some examples are very far future. I mean, you're not going to have an AI discover a sports hack by curving a hockey stick until it understands not just the rules of the game, but the physiology of the players, the aerodynamics of the puck. I mean, so many real world things, you almost can't imagine an AI doing it. But financial regulations are much more codified. They're designed to be algorithmically tractable.
So there's going to be a lot more, a richer landscape for these AIs to traverse looking for hacks. And there are two different worries here. The first one we're talking about is what happens when a hedge fund or investment bankers hand an AI all the tax laws of the world, all the financial regulation of the world and say, find me new vulnerabilities, find me new hacks. The second is when this happens inadvertently and in secret, that an AI is just given an optimization task and it comes up with a hack. But because we can't inspect the code, because AI is a black box, we don't even know that it came up with a hack. And this reminds me a little bit of a story from the start of Max Tegmark's book, Live 3.0, where he has an AI which...
sort of games the business system to become very profitable and then the financial system to become even more profitable. But it's all done in secret. And it's very, very distinct to the example that Nick Bostrom uses in his book, Superintelligence, of the AI that's asked to make paperclips. And it's just going to grab every resource in the world to make paperclips. I mean, when you think about the thing that struck me about your idea of AI hacking is that it felt very present compared to those much more philosophical and science fiction scenarios that we've seen.
previously, but those ideas came out five or six years ago. So is it more present because the risk is now greater and the examples that we can put our hands around are more tangible? Or is there something else perhaps going on? I think it's because the systems are becoming more arguably tractable and it is more tangible. I mean, TechMark's example is an excellent because it really does talk about the system hacking financial systems. The example I like to use is Volkswagen. So this is 2015 and Volkswagen is cheating on a mission control test in its automobiles. Now, this isn't an AI story. It's a human story. It's humans cheating.
Human engineers program the engine to detect emission control testing and behave differently when being tested. But the cheat remained undiscovered for a decade because it's hard to figure out what software is doing. And it was a couple of researchers that found it. And if they didn't, we still wouldn't have known. So imagine an AI being told to design a car engine and their constraints are maximize performance while still passing emission control tests. You could easily imagine the software.
coming up with that cheat, not realizing it's a cheat. The human engineers knew they were cheating, but the AI produces this software. It hands it to Volkswagen. The engineers say, great, look at the performance. This is fantastic. The marketers think it's great. Nobody can look under the hood. So we don't know it's cheating. And now, of course, we've had this conversation. We know to program the goal and say, look, you have to behave the same on the road and during tests. You've opened up a Pandora's box of issues there. We're going to start with the question of explainability. We've talked a lot about explainability and AI systems on previous episodes of this podcast. And it's always been a...
a question of a sort of trade-off, right? That if you make your AI systems more explainable, perhaps they won't be as performant. And it strikes me that the cost of explainability are tangible, they're measurable now. We can see they're happening because the AI system is not predicting as well as one that is not explainable. But the costs of not being explainable are somehow uncertain. They're sort of optionalities in the dim and distant future. There's like an externality for a future generation to deal with. So it feels like we have a kind of like a human incentive alignment problem here, which is it's a bit like credit card debt, right? If I make my AI unexplainable...
When it blows up, that'll be at some point in the future, and maybe I'll be wealthier and more able to deal with it then. And the term used is technical debt here, that we are building these systems we can't explain, and we're building up this debt. I agree with you that explainability... has some trade-off with functionality. That giving an AI the constraint of make a decision, but make it a human explainable decision, reduces the decision space. I also think that human explanations are not as good as we think they are. What we know about human decision-making is that explanations are largely justifications. That we as humans make a decision, and then we produce an explanation after the fact.
but it doesn't really explain our decision. It just justifies it. So the benefit to me of an AI is that it will come up with decisions and decision processes that humans can't. And I want that. To what extent is there a deep seated belief within the culture of technology that performance matters above all else and everything else is a trade-off? Because there are lots of places where we have accepted that performance isn't.
the only thing that one needs. For example, lorries, large vehicles, trucks have speed limiters. For example, in the UK, many trucks are limited to 56 miles an hour, even though they could drive at 110. Putting on a seatbelt slows down your journey because you have to spend 10 seconds putting a seatbelt on. We've accepted that we can put constraints on performance in lots of other domains. But I look at the software domain and I think, I don't know if the people building this stuff believe there should be constraints. They sort of perhaps believe that constraints come from the technology and Moore's law will every year or two loosen that constraint slightly. I think there was a belief certainly in the early decades of tech, that tech is what tech is. And any constraints are...
these dumb human things. I mean, Google had this attitude, right? You know, their search algorithms have to be sexist or racist, not Google's fault. It's the algorithm, right? You know, we can't do anything. And I think we now know that we want our tech to reflect our values. And if it means that you have to force the Google search algorithm not to be racist, not to be anti-Semitic, that we're going to do that. And that's too bad on Google. I've always pushed out back as a notion that we should design our society for the near-term financial benefit of a bunch of tech billionaires. It just seems like a dumb way to organize society.
And AI needs to reflect that. It's going to be harder because it's more opaque. So we see some of this, the AI algorithms that recommend videos on YouTube, push people to extreme content. And no one asked the algorithms to, they just realized that extreme content produces more engagement and the algorithms are optimized for engagement. And that's not really a hack, but it definitely is an unexpected outcome. Yeah, it's an unexpected outcome. But there's a question there about what is testable from an engineering perspective, right? Some of these problematic areas that you've raised,
arise as almost like an emergent phenomena as these networks get bigger and bigger, right? So the issue with YouTube and say extreme veganism, which is probably a thing, right? You search up for a vegetarian recipe and five clicks later and you're like extreme veganism. You probably couldn't test that when YouTube was 100,000 users, right? Or a million users or before the content was on there. So I'm curious about the kind of operational practices that developers need to then think about when some of the risks that you identify where these algorithms do start to hack the system, it feels to me can only emerge once the thing is already quite big. And that's also true for human systems. I mean, so get back to Los Angeles Police Department.
How do you test it beforehand to see if it's going to be racist? You can't. You have to look at it in operation, in situ, in the real world, and then start saying, hey, this police department is obviously racist. This algorithm is obviously sexist. And then you have to go back and fix it. Now, it's easier in a computer system. In the Los Angeles Police Department, it took many years and a lot of court battles.
to get that group of human beings to change its practices. It would be trivial if there was some rule that forced YouTube to change its algorithms or Facebook to change its algorithms. Yeah. When I think about AI and hacking, I think of this collision of computer speed and human speed. And so there are hacks of the tax code today. And they are patched or not at human speeds. Takes a couple of years in the United States to pass a new tax law. Don't know about the UK. I bet it's not that much quicker. It's about the same. Yeah. But Microsoft can patch a hack in Windows in a few days if it wanted to. So what happens when you feed an AI the world's tax codes? Does it find 10 exploits, 100, 1,000? Suddenly.
right? Tax revenues are dropping two thirds. Government's not functional. And we don't have a mechanism to patch in the same way as we have computer code because computer code grew up with this notion of hacking. To that point, right? The reason Microsoft can patch quickly or Apple or Tesla with its over the air updates is partly technical, but it's also about their internal processes, right? Their change management, their agile processes. So if they didn't have those agile processes, if they had a strict Six Sigma waterfall approach to patching software, you'd get an exploit announced on a Monday.
10 months later, you're still waiting for an update to occur. So of course, there's a technical stack that's in there over the air updates and being able to sort of just send partial code that updates and patches. But there's also some human systems behind that that enable that. So could you take that sort of lesson into our human decision making and start to say, well, what would it look like to have a patchable tax code, right? Rapid response, as it were. There's something even more fundamental than what you talked about is that at Microsoft, there's a singular notion of what the proper code should be. So if I find an exploit in Windows...
everyone knows you got to patch this if i find x y the tax code it gets political suddenly there's lobbying dollars in let's keep it something there's a political party that says that's not a hack that's a great idea But the system to get that even decided as something to fix is so political. You can imagine if internal revenue, the IRS had the ability to quickly pass a ruling. And you do get this sometimes. I think we need to figure out what agile policymaking looks like, not just because of hacking, because of tech change, but some way to speed up.
policy change. The back half of my book, which comes out in September, looks at that question, how you get the right balance between flexibility, that is ability to respond to exogenous changes or endogenous changes within the system, but also a sense of resilience, which is to be able to construct with the right kind of rules and interactions so that during periods of volatility, your system doesn't go out of parameter, right? Out of bounds. So we talked about AI hacks within individual systems, like within a YouTube, as it were. But one of the key things that is happening is this sort of background autonomous economy where So much goes on because algorithmic systems, we can call them bots, are scanning the open web and adjusting prices on a different website. Or there are suppliers who are interacting with each other and adjusting order volumes. And there are sensors from one party feeding into an optimization system run by another party. And these things are increasingly connected. Can you give us a flavor of the scope of AI hacks that could happen in these largely connected systems? I think the scope gets larger because the larger the rule set, the more interactions, the more hacks you're going to find.
financial systems as the most fruitful place this will be applied. One, because there's a motivation. Two, because there's the money. And three, because the rules are designed to be understood. In fact, you could argue that the gap between what is hacking of a system in this sense of getting it to do more than it otherwise should while staying within the rules was largely...
in a way, inculcated in the global financial crisis. The creation of these synthetic bundles of mortgage debt that were all junk, but you found a little mathematical loophole that allowed you to categorize bundles of junk as small packages of gold. So one of the things that seems to be, it almost feels like it's a parable or a Greek myth, that as systems get more powerful, they can do more for us. They need to get more complex. And as they get more complex, they become more hackable, more vulnerable. I think in your essay, you pointed out the Windows 3.1, which I had.
spent many a fruitless hour trying to install on my underpowered PC back in 1991, 92, had about two and a half million lines of code, but the recent version of Windows has about 50 million lines of code. And actually, in some sense, that makes the 50 million line of code thing more vulnerable to hacks of this nature. Is that a simple relationship or is there more nuance to it? We know from computer security that complex systems are less secure. The more lines of code, the more bugs there are, the more vulnerabilities, the more hacks. It's not just the raw amount of code, it's the interactions between the systems. I mean, it's not as if developers are just sort of...
handling these enormous lines of code themselves without any automated systems to help them and to identify these weaknesses. So why haven't those automated verification systems and other verified libraries and code signing and so on, why hasn't that kept up to keep these vulnerabilities at bay? So it's helped, but it hasn't kept up because the insecurity from complexity rises faster than our defenses. And we see this in many areas. So in computer security, every year we get better. There are new tools, there are new techniques, there are new security systems, but the risk increases, as you can tell by watching the news.
because the interconnections get greater, the complexity gets greater, the criticality of the systems gets greater. So we lose ground even as we improve. And it is interesting to watch. We don't know the end game of this. We now know that we live in this exponential world of highly interconnected computer systems. And we know that this relationship of complexity and vulnerability exists, the one that you've described. So the thing that's quite helpful is that it allows us to say, we know these risks now exist. So what should we do in that case? So I preach resilience. Now, resilience is an interesting concept. It's very abstract. It's not something you achieve directly as an emergent property.
And it has various components. It has redundancy. It has fail safe. It has compartmentability. It has recovery. Can we build more resilient systems so we're less vulnerable to individual hacks, to individual exploits, to a piece of ransomware? And if you had a backup to your computers, you wouldn't care about the ransomware. You just restore from backup. And that's a way to get resilience through recovery. I can also get resilience from ransomware through better, makes a better firewall, better threat detection and elimination.
So resilience is the way to think of this. So the internet was built in the 60s to answer this research question. Can I build a reliable network out of unreliable parts? Turns out the answer is yes, which is kind of surprised everybody. I have a similar question. Can I build a secure internet out of insecure parts? And the answer is not obviously yes, and it's not obviously no.
And then can I take that knowledge and move it to other systems? Can I build a resilient tax code, even though every single law is going to have a loophole? For people who've been involved in the computer industry for a long time, a hack is not a bad thing, a priori. Actually, hacks are really helpful. They just help you get things done when the system wasn't quite conceived of in this way. So we've worried a lot about the negative types of AI hacks, but there are bound to be some positive AI hacks, just as there are positive hacks in every other walk of life. What kind of positive AI hacks can you imagine?
I mean, same thing we look at hacking in general. Dunking in basketball was illegal when someone did it at first. And then the league realized, you know, fans really like this. It makes for a better game. Hacks and financial regulations. Some of them have turned out to be very positive. So hacking isn't necessarily negative. In the computer field, they tend to be negative. When you find a hack in Windows, it is rare that Microsoft will say, all right, that's a good idea. Let's make it a feature. But it sometimes happens.
So that's a way that hacks can become innovations. But Microsoft does a lot of testing in-house before they release any code to look for these vulnerabilities so they can fix them. So while an AI can do this for the offense, it could also do it for the defense. You can easily imagine Microsoft employing these AIs, everybody employing them, they become part of the compilers. And you can fast forward to a future 20 years from now.
where we say, wow, remember when software vulnerabilities were a thing before all the AI discovered them and we patched them? That was a crazy time. Now, the intermediate time is dangerous. All the existing legacy code that already exists that can't be patched. But the far future of this is that AI vulnerability finding a computer code benefits the defense more. So it's the same thing for the tax code.
You can imagine a future where any new tax legislation is torn apart by the AIs. All the hacks are discovered. They may not be patched because it's political, but they become part of the debate. Now, again, the intermediate times are dangerous, or the existing legacy tax code. But in the long run, AI vulnerability finding benefits the system designer. So in a sense, what you do is you're able to simulate the system in some digital domain where you can allow AI agents to run amok.
and try to figure out both the vulnerabilities that need to be addressed and output in a list, but also the places where there are new use cases or benefits you hadn't conceived of when you had designed or pulled the system together. And you do that before the system is released, before the law is passed, before the regulation is promoted. You do that when you design the rules for the sport. And you end up, I think, with a much better system because of that. Some of that's far future, but it is worth thinking about. The end game of this actually benefits the defense, not the offense. John Maynard Keynes had the phrase, you know, in the long run, we're all dead. And how long does this end game take to play out? I don't know. And I think you probably know as well, it is really hard to put prediction dates on anything AI. When I was a college student, we learned that Go would never be solved by an AI because it just blows up. It's just intractable.
until one year when DeepMind solves Go because they were bored one month. And it was just a matter of throwing more computing at it in a way we didn't realize. My normal thing is to say, oh, it's a couple of decades out, which means I have no idea, but I figured that's something to say. What do you say? Well, as Niels Bohr, the physicist, said, it's very hard to predict, especially the future. One thing I did predict, and I did get it right, was I would have an absolutely cracking conversation with you. Bruce Schneier, thank you so much for making the time today. Thanks for having me. This was fun.
If you enjoyed this conversation, do check out the podcast feed where you can find previous conversations around cybersecurity. Look out for my discussion with Mario Rosario Tadeo, the conversation I had with Nicole Egan, my discussion with Ian Bremmer on the geopolitics of all of this, and with General Sir Richard Barron's on AI and the future of warfare. I am sure you will love those four discussions. To become a premium subscriber of my weekly newsletter, go to exponentialview at www.exponentialview.co. To stay in touch, you can follow me on Twitter. I'm at Azeem. The podcast was produced by Maria Gavrilov and Fred Casella. Ilan Goodman is our assistant producer, Boyan Sabiachello, the sound editor. Exponential View is a production of E to the Pi I Plus One Limited.