Skip to content
Predicting the Future Is Possible. ‘Superforecasters’ Know How. episode artwork

Dec 3, 2021 52 min

Predicting the Future Is Possible. ‘Superforecasters’ Know How.

from The Ezra Klein Show Hosted by New York Times Opinion

Can we predict the future more accurately? It’s a question we humans have grappled with since the dawn of civilization — one that has massive implications for how we run our organizations, how we make policy decisions, and how we live our everyday lives. It’s also the question that Philip Tetlock, a psychologist at the University of Pennsylvania and a co-author of “Superforecasting: The Art and Science of Prediction,” has dedicated his career to answering. In 2011, he recruited and trained a team of ordinary citizens to compete in a forecasting tournament sponsored by the U.S. intelligence community. Participants were asked to place numerical probabilities from 0 to 100 percent on questions like “Will North Korea launch a new multistage missile in the next year” and “Is Greece going to leave the eurozone in the next six months?” Tetlock’s group of amateur forecasters would go head-to-head against teams of academics as well as career intelligence analysts, including those from the C.I.A., who had access to classified information that Tetlock’s team didn’t have. The results were shocking, even to Tetlock. His team won the competition by such a large margin that the government agency funding the competition decided to kick everyone else out, and just study Tetlock’s forecasters — the best of whom were dubbed “superforecasters” — to see what intelligence experts might learn from them. So this conversation is about why some people, like Tetlock’s “superforecasters,” are so much better at predicting the future than everyone else — and about the intellectual virtues, habits of mind, and ways of thinking that the rest of us can learn to become better forecasters ourselves. It also explores Tetlock’s famous finding that the average expert is roughly as accurate as “a dart-throwing chimpanzee” at predicting future events, the inverse correlation between a person’s fame and their ability to make accurate predictions, how superforecasters approach real-life questions like whether robots will replace white-collar workers, why government bureaucracies are often resistant to adopt the tools of superforecasting and more. Mentioned: Expert Political Judgment by Philip E. Tetlock “What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?” by Christopher W. Karvetski et al. Book recommendations: Thinking, Fast and Slow by Daniel Kahneman Enlightenment Now by Steven Pinker Perception and Misperception in International Politics by Robert Jervis This episode is guest-hosted by Julia Galef, a co-founder of the Center for Applied Rationality, host of the “Rationally Speaking” podcast and author of “The Scout Mindset: Why Some People See Things Clearly and Others Don’t.” You can follow her on Twitter @JuliaGalef. (Learn more about the other guest hosts during Ezra’s parental leave here.) Thoughts? Guest suggestions? Email us at [email protected]. You can find transcripts (posted midday) and more episodes of "The Ezra Klein Show" at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Book recommendations from all our guests are listed at https://www.nytimes.com/article/ezra-klein-show-book-recs. “The Ezra Klein Show” is produced by Annie Galvin, Jeff Geld and Rogé Karma; fact-checking by Michelle Harris; original music by Isaac Jones; mixing by Jeff Geld; audience strategy by Shannon Busta. Special thanks to Kristin Lin and Alison Bruzek. Subscribe today at nytimes.com/podcasts or on Apple Podcasts, Spotify and Amazon Music. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Predicting the Future Is Possible. ‘Superforecasters’ Know How.

0:00 52:50
of matches

Transcript Auto-generated, may contain errors

Hi, I'm Solana Pine. I'm the director of video at the New York Times. For years, my team has made videos that bring you closer to big news moments, videos by Times journalists that have the expertise to help you understand what's going on. Now, we're bringing those videos to you in the Watch tab in the New York Times app.

It's a dedicated video feed where you know you can trust what you're seeing. All the videos there are free for anyone to watch. You don't have to be a subscriber. Download the New York Times app to start watching.

I'm Mr. Klein, and this is the Ezra Fongio. Hey, it's Ezra. Well, I'm on Paternity Leave, but we've got an all-star team of guest hosts.

This week is Julia Galef, the host of the Great Rational Speaking Podcast, an author of the book, The Scout Mindset. Julia is unusually good at thinking about thinking, so this should be a lot of fun. My guest today is Phil Tettlock. He's a professor of psychology at the University of Pennsylvania, and a professor of management at the Wharton School.

He's also the author of several books, including the 2015 bestseller, Super Forecasting, The Art and Science of Prediction, co-authored with Dan Gardner. In my view, Tettlock is doing some of the most important research in social science today, research that has the potential to revolutionize how we run our organizations, how we make consequential policy decisions, and how we live our everyday lives. And that's because at the center of his work is a question that humans have been trying to answer for as long as the history of civilization. How can we predict the future more accurately?

And Tettlock isn't just an ivory tower academic here. In 2011, he entered a team in a competition sponsored by the US government, in which participants were asked to make forecasts about questions like, how long will the Syrian Civil War last? And is Greece going to leave the Eurozone? Putting numerical probabilities on their guesses from 0 to 100 percent.

Tettlock's group consisted of amateurs with nothing but an internet connection who agreed to make predictions in their spare time. And they were going head-to-head against not only other teams, but also the highly trained career intelligence analysts from the CIA, who had access to classified information. Yet not only did Tettlock's group win, they were so much better than all the other teams that the government agency funding the competition decided to kick everyone else out and just study Tettlock's best forecasters, whom he dubbed the Super Forecasters. So this conversation is about why some people like Tettlock's Super Forecasters are so much better at predicting the future than everyone else, and about the habits of mind and the ways of thinking that the rest of us can learn to become better forecasters ourselves.

Before we jump in, a reminder that you can always email the show with feedback or guest suggestions at EzraKlineShow at NYTimes.com. My conversation with Phil Tettlock after the break. Hi, this is Ashley. I live in San Francisco with my boyfriend.

We would love to officially share my New York Times subscription with separate logins. We both love cooking, loving in the kitchen, but I'm a 30-minute and under-efficient dinner gurly. I want a sheep and meal. He is very elaborate.

He wants to get into the storytelling. I want to be able to save my easy meals and check off the ones that I've completed, and I think him having his own profile would be great. Ashley, we heard you. It's why we created the New York Times family subscription.

You get your own login and Mr. elaborate gets his. Plus room for two others. Find out more at NYTimes.com.

Phil, welcome to the EzraKlineShow. Great to be here, Julia. I'm so excited to get to talk to you again. Before we dive into the weeds, Phil, I'd like to give our listeners a sense of just how central forecasting is to our society and to our lives.

Because I think a lot of people might be thinking of it as kind of a niche interest or pastime. And so you had a nice quote in a Foreign Affairs article you co-wrote last year where you said every policy is a prediction. Could you explain what that means? Sure.

Virtually every consequential thing we do in the world is an implicit forecast because we're making assumptions about what we'll follow later. And sometimes those assumptions are so obvious. We don't even think of them as a forecast. We think of them as virtually factual.

But the current debate over whether the budget bills in Congress right now are going to be inflationary or hyperinflationary or whatever people might want to claim. Those are obviously competing forecasts about what will happen. There's a lot of inflation, inequality, government deficits, long list of outcome variables that people are arguing about. People are arguing about US policy on Taiwan and China.

And how much age should the US be giving Taiwan? What increases or decreases the likelihood of an armed conflict over Taiwan? These are all claims about the future. So forecasting is not a disactivity.

It's central to public policy, it's central to human life. Whenever you decide to get married or not to get married or get divorced or whatever, it rests on claims about possible futures. Great. So now that we've established that we're all forecasters, whether we realize that or not, let's talk about some of your earlier work on how good we are at forecasting.

And I think the one insight that you might be most famous for is from your book, Expert Political Judgment from 2005, in which you concluded that the average expert on economics or politics was roughly as accurate as a dart throwing chimpanzee. Now, of course, that doesn't mean that every single expert is worse than a chimp, just that the average expert is worse than a chimp. But that's still a pretty disturbing result. So can you summarize the research you did that led you to that disturbing conclusion?

Well, I was a junior faculty member at University of California, Berkeley, and I was serving on a National Research Council panel, got started in 84, 85. A lot of people thought the world was heading into a nuclear apocalypse. That was one of the big debates. It was Ronald Reagan increasing or decreasing the likelihood of a nuclear war with the Soviet Union.

As you might expect, you could predict very well the positions people took from ideology. And it was a big debate also about what was going to happen to the Soviet leadership. We were coming up to that point in the mid 1980s. The aging gerontocracy in charge of the Kremlin was fading out and a young relatively junior member of the Polobeara McAlgorbit was about to become general party secretary.

And there were conflicting views, A, but whether you would get the job and B, if you got the job, whether he would be a neo-Stalinist, a disciplinarian, and confrontational with respect to the United States, or whether he was a latent liberal. And he would reform the Soviet Union. And it turned out to be the case that the people who felt it was going to be a radical reformer were quite right. He was a radical reformer.

In fact, his reforms are so radical. The ultimate led to the disintegration of the Soviet Union. Liberals generally thought that Reagan was pushing the Soviets into a corner, and the Soviet Union would become more truculents and confrontational. And the conservatives thought that the Soviet Union was unreformable and was inherently neo-Stalinist.

So neither side really predicted Gorbachev and how radical reformer he would be. But within a year or so, after it came to a parent that he was a reasonably radical reformer, everybody claimed that they saw it coming all along. And the conservatives said, hey, this is vindication for Reagan's policies. And the Liberals said, you know what?

Reagan had nothing to do with this at all. The Soviet economy was crumbling to Soviet. They were going to do this anyway. So that, like, turned on.

I thought, you know, it'd be interesting to keep score. It's kind of wild that no one was keeping score before then, honestly. I guess this is one of those innovations, like actually tracking the accuracy of people's forecasts that seems so obvious after someone proposes it but wasn't obvious to people before then. Well, it's very obvious, but it's also almost impossible to implement because a lot of political speech is not really about accuracy.

It's about laying claim to power using the most effective rhetorical devices. So your extra long study that was the cornerstone of expert political judgment, how many experts were in that study and what kinds of questions were they answering, and how did you measure their accuracy? It's about 284, so subject matter experts. They weren't all experts on the Soviet Union.

They were experts on all sorts of various regions of the world and various functional specialties as well, like arms control or economics. And we measured relatively short-term predictions, medium-term predictions, occasionally, we measured some longer-term predictions. But the main focus was on one, three, and five-year predictions. And personally, I think it's important to note that not only were the people understanding making predictions, but they were attaching a quantitative confidence level to those predictions.

Like they might say, you know, I'm 75% sure the Soviet Union will collapse in the next year or something like that. Absolutely. If I say to you, I think there's a distinct possibility that Saddam Hussein has weapons of mass destruction. How are you going to interpret that?

I say, I say that to you right before the invasion of Iraq in 2003. You hear the phrase distinct possibility. How do you translate that? You say, well, he said distinct possibility.

So I guess that means it's over maybe over 50%, or he said, he merely said it was possible. So I guess it's under 50%. Distinct possibility is almost the perfect phrase if you're a risk-averse political pundit or intelligence analyst, and you don't want hindsight tainted people coming back and criticizing you later on, you can say, either way, you're going to be on the right side, right? And you say, I'm really said it was possible.

If it doesn't happen, if it does happen, you say, I said it was distinctly possible. You didn't listen carefully enough. So distinct possibility straddles a broad range of the subjective probability continuum from about 20% to about 80%. Those are the lowest and the highest plausible readings that people attach to that phrase.

It's a really tailor-made for playing the Washington DC blame game. It's also a tailor-made for causing confusion in disagreements. I know I've gotten into disagreements with people where they said something was, you know, quite possible. And to me, I said, it was very unlikely and it turned out after much debate that we both put it at about 10%.

But, you know, that was not at all clear when we were using the big language. Right. And that was one of the stories that we used in the Super Fork asking book later, it was the story, John F. Kennedy and the Joint Chiefs of Staff.

He launched the Bay of Pigs invasion in April 1961, three months into his presidency. It was probably the biggest fiasco of his presidency. And the question was, why did he do it? It turned out that he was a little bit wary of the plan to invade Cuba.

It was the CIA plan. He was wary of it. He asked for a second opinion from the Joint Chiefs of Staff. And they gave him one.

And he said, there was a fair chance of success, which he interpreted optimistically. But they did later go back to the JCS. And they said, well, what did you guys mean by fair chance? Well, I said maybe one or three.

So I don't know whether history would have unfolded differently if they'd had a quantitative probability estimate from the JCS as opposed to the big verbiage fair chance. Right. So big innovation, in addition to just keeping score was having people actually have quantitative probabilities to their predictions so that you can actually score their accuracy. And so back to the punchline, the average expert not actually being more accurate in making these predictions than a dart throwing chimpanzee.

What were the experts doing wrong? The average expert doing wrong. The term average expert conceals more than it reveals. I mean, some experts really were substantially better than chance.

And there are other experts who actually managed to do worse than chance. And we found an individual difference variable that predicted which experts fell into which category. The more flexible Fox is tend to do better than chance. And the less flexible hedgehogs tend to do somewhat worse than chance.

And just to expand on that metaphor a little bit, what you're describing here is two types of thinkers. The hedgehogs are the kinds of people who see the world through one big idea, one ideology. And so they aren't very flexible in their worldview. They aren't as likely to update their opinions or predictions when new information arises.

And Fox is the opposite. They aren't attached to one particular ideology. They really try to incorporate a wide range of worldviews into their thinking. They tend to be more intellectually flexible.

And this distinction turns out to be really important. The finding from your big study on this was that the people who think like Fox is are much better forecasters than those who think like hedgehogs. So let's dig into that more. Can you give me some examples of hedgehogs?

The world is full of hedgehogs. And there are lots of one big idea of people. You could be a Marxist hedgehog or you could be a libertarian hedgehog. You could be a Boomsster hedgehog or you could be a Doomsster hedgehog.

Let's just take those two polarities for the moment. I mean, there's a very, very pessimistic view of capitalism, a very optimistic view of capitalism, a very pessimistic view of what the limits on growth, a very optimistic view of the unlimited growth. So you can imagine people falling toward the extremes of these continuance. And you can imagine people who are floating sort of in the middle at various points in the Fox's tendinity, centrist on the left, right dimension on the Boomsster dooms, three dimensions, and so forth.

One reason I think your work has been so eye-opening for so many people, including me, is that it really challenges our assumptions about what it means to be an expert. You know, we tend to think of someone as an expert if they have a lot of knowledge, like a PhD or in a subject's domain or have written a best-selling book on something. And your point is that no expertise actually means having a track record of actually getting things right. Which you might assume would go hand in hand with having a lot of knowledge in an area, but in practice, no, these are just two, you know, knowledge on the one hand and judgment or forecasting scale on the other are just these two different independent things.

And just because someone wrote a book about collections doesn't mean they're going to be good at predicting the results of the election and so on. When I say it, it almost sounds obvious, but it really wasn't to me. And it's really refocused my attention on, you know, not just does someone have relevant background knowledge, but also not just do they sound smart when they talk about a subject. Because I think that was also something I was kind of implicitly assuming made someone an expert that they can say smart things about something.

And I just wasn't really tracking, well, they're track record. I pay a lot more attention to that now than I did before encountering your work. Another related interesting finding from this work was that you found an inverse correlation between fame and accuracy. So the more famous an expert was the less accurate he was.

Why would that be for the reason that very few people want to talk to me, I think, experts who interweave lots of butts and however, is in all those into their speech are less charismatic and compelling than experts who are telling an emphatic causal story pointing in one clear direction. It's really hard to be charismatic. If you're building up a lot of ideological momentum in the audience, then you say, however, we have to take into account the cost of these measures against climate change, or however, you can just see the balloon deflate. So this is a good indicator of whether you are going to be a good forecast or not.

It's how frequently you use terms like about however, although. So that story about why hedgehogs with their one big idea, other one big framework would be worse forecasters and also more famous. That makes sense to me. I was just also thinking of another possible story.

I'm curious what you think about this. So it occurred to me that experts who have become famous for a particular point of view or particular argument might have kind of boxed themselves into that viewpoint. Like if you become famous for being the guy who argues that the world is getting better, or the guy who argues that big tech companies are harming society, etc. And that's what you're known for.

I think that that kind of entrenches you even more than you would already have been entrenched just from human nature. And it's especially hard to change your mind or consider exceptions to your viewpoint. What do you think about that? Well, the two psychologists who had the most formative influence on my thinking would be, I think two of the most influential psychologists of the 20th century, Dan Econam and Leon Festinger.

And you're in Leon Festinger land right now. You very clearly laid out exactly the argument you're describing that people who make public commitments to a position are going to be motivated to bolster it. They're going to become better and better at generating reasons why they're right and there would be critics are wrong. And the more you do that, of course, the more your attitudes are going to polarize.

You're going to build up ideological momentum unless somebody puts on the cognitive brakes and says, however, or although, I mean, I'm not arguing that that's not a natural cognitive response. People have the natural cognitive people response to have the dissonant or disagreeable moves is denial and bolstering. Those are the low effort cognitive responses. And they leave the polarization, of course, you're digging yourself into the trench and you have to defend your position.

And the unnatural responses are to be integratively complex to differentiate and integrate to conceit it might be legitimate alternative points of view and identify boundary conditions under which they hold. That's of course what the best forecasters tend to do, but it's not a natural cognitive act. And that's what I did the natural language processing study. I think Carvescu et al.

just reported the international forecasters thing. I think that provides some of the best evidence for that claim. This is the irony that I noticed in talking to open-minded people and less open-minded people about open-mindedness. The less open-minded people would say, oh, yeah, you got to just stick to your guns and never change your mind.

And that makes everyone respect you and it makes you successful and has no downsides. And then when you talk to the open-minded people, they're like, well, you know, I do think open-mindedness is great. But here's some potential downsides and maybe it has this bad effect and maybe it has that bad effect. And so there's no real objective vantage point from which to way open-mindedness against the opposite.

Yes. And you hear that a lot in the business school world, where a common theory is that strong leaders are decisive leaders. And there is a certain state, the course, mentality. Here you need to draw distinction between how you think in private and how you think in public.

And this actually leads to a somewhat disturbing thought that maybe effective leaders have to be to some degree liars. Here's an example of that. So Dwight Eisenhower, right for D-Day, was exhorting the troops and telling them they were going to be victorious when they landed on the beaches of Normandy on June 6, 1944. And he was projecting confidence and can do the bravado.

Now, if I was now being a completely transparent leader, he would say, well, you know, look, I think there could be a 20, 30% chance that you were going to be driven back into the English Channel. If he talked like that, you would have said he's an idiot. He doesn't know how to run a show. But on the other hand, you want someone who's aware of the possibility that the plan is going to fail, there's lots of contingency plans and so forth.

So we wrestle with this kind of moral issue, I guess, is super forecasting, Dangar, and I wrestle with it about whether leaders, in some sense, have to be liars. Good leaders have to be liars. You had another interesting argument in defense of hedgehogs, to a limited extent. It was kind of an ecological argument that hedgehogs are play a valuable role in the information ecology.

Absolutely. Where do foxes get their most interesting ideas from the extremes from the hedgehogs, which they then dilute and qualify them. But hedgehog ideas are often profoundly provocative. So the classic Marxists, they left a deep imprint on thought in the 19th and 20th centuries.

I don't know if this is how you would characterize the arc of your career film, but I think about your career as having these three chapters where the first chapter where you're studying expertise is kind of discovering this problem of the lack of actual forecasting skill in many people we consider experts. And then the second chapter is about finding solutions to that problem. This is the chapter in which you wrote super forecasting. And so I want to move on to that chapter now and talk about how you pursued this question of, well, sometimes experts are terrible at forecasting, but some people seem to be pretty decent at it.

And so what are they doing right? What can we learn from them? And so as part of that project, you led a team in a forecasting tournament that was put on by a US government agency called IARBA, the Intelligence Advanced Research Project activity, kind of like DARBA, but for the intelligence community. Could you talk about the purpose of that tournament, what the outcome was?

And for our listeners who aren't familiar with the idea of a forecasting tournament, what is it? Well, a tournament is like an athletic competition in which you have a bunch of players who are trying to achieve a goal. Here the goal is to make forecasts as possible and a tennis tournament, what the goal is and so forth. So it's just a competitive system for identifying top performers.

It was a wonderful opening salvo to bring some core enlightenment values into political discourse and into intelligence analysis, just radically simple and transparent. Here is a series of questions of interest in the US intelligence community. We're going to give some competing research teams a bunch of money. They're going to be able to recruit people to their platforms and their job is going to be to generate the most accurate probability estimates for the questions that are of interest to the intelligence community.

So there would be questions about how long the Syrian civil war lasts and what's going to happen next with Colombian narco trafficking and what's going to happen with Spanish-German bond yield spreads or whether Greece is going to leave the Eurozone. You just very long list, incredibly heterogeneous array of questions. So no individual in this forecasting tournament is really going to be an expert on more than one or two or three of these questions. I mean, they're that heterogeneous.

It's like a geopolitical IQ test. It was like a measure of geopolitical fluid intelligence. How adept are you at adjusting really rapidly to one damn thing after another? Very, very difficult.

In traditional analysis and traditional academics scholarship, we think of experts having a lot of crystallized knowledge. I mean, I just spent six years getting a PhD in China studies, then I spent 10 years getting tenure and so forth. And now some upstarts want to know whether I can out predict some guy off the street on where the Chinese economy is going. And that's annoying.

So the idea is pretty radically simple. You can see how it has a certain populist appeal because it's a way of saying, okay, you're high in my experts. We want to see whether you can actually deliver on these things. And a lot of the response to the expert community is not was not all that positive.

I mean, I think what I already just invented here is a game of kind of geopolitical trivial pursuits. So we did find some people were really good at this. We also did some randomized control experiments. We found that there's some ways or rather than others of training people to do better at this.

So we discovered some natural talent, pre-existing talent, we found ways of cultivating talent once people were inside the tournament. We found there were better and worse ways of running teams. And we found there were better and worse ways of aggregating crowd wisdom. So your team won in a landslide.

Your forecasters were by far the most accurate people in the entire tournament, including the CIA analysts themselves. How much more accurate were the good judgment teams than the runners up? Well, they were beating the other teams by margins of between 40 and 70%. They were doing better than the prediction market inside the intelligence community between about 25 and 30%.

These are analysts who have access to tons of classified information about the subject matter and the questions that the amateur forecasters that volunteered for your team did not have access to, which makes it even more impressive, I think. You're not tuning your own horn enough, so that's why I'm doing it for you. Well, bearing my two actually did the work. It was these so-called superforecasters.

And because the government didn't allow us to pay people a lot of money for performance, because they don't believe in pay for performance inside the intelligence community currently, we weren't allowed to do that. So we had to reward them with status. And what's the status award? Well, it's to be called a super forecaster.

Oh, that's where that term came from. Yeah, it was a motivational tool. So we've kind of touched on a few things that made the super forecasters super. But if you had to kind of pick one or two things that really made the super forecasters what they were, what would they be?

You know, we've already talked about one of them, which is their skill at balancing conflicting arguments or skill perspective taking, however, although, but they put the cognitive breaks on arguments before arguments develop too much momentum. So they're naturally inclined to think that the truth is going to be some blurry, integrative mix of the major arguments that are in the current intellectual environment, as opposed to the truth is going to be way, way out there. Now, of course, if the truth happens to be way, way out there, and we're on the verge of existential catastrophe, I'm not going to count on them to pick it up. In addition to these dispositions and sort of general thinking patterns that the super forecasters had, are there any kind of concrete habits that they would always or often make use of when they were trying to make a forecast that other people could adopt too?

One of them is this tendency to be integratively complex and qualify your arguments, however, and buts and all those. They sign that you recognize the legitimacy of competing perspectives as an intellectual reflex. You're inclined to do that. And that's actually a challenge to fastenger and cognitive dissonance.

They're basically saying, look, these people have more tolerance for cognitive dissonance than the unfestinger realized as possible. The other thing super forecasters do is they use comparison classes about the outside view. And this is where the super forecasters do what Danie Kahneman said people don't do. So I want to talk about that distinction between taking the inside view versus the outside view on a problem.

So when most of us try to make predictions, we'll start instinctively with the inside view, which means we'll start by looking at the specifics of a case. And then we'll base our prediction primarily on those details. But starting from the outside view means looking for comparison classes, which are basically benchmarks that help us see how other similar situations have played out. And then of course, you still look at the specifics, but only as a way of almost fine-tuning your prediction.

So can you walk us through an example of what it would look like to make a forecast by starting from an inside view versus starting from an outside view? Well, if you're at a wedding and someone has a bad taste to come up to you and ask how likely is a couple to stay married, you're going to look at them and say, well, it's a really rude question. Look how happy they are. It's also not only rude questions, it's a dumb question.

Look how happy they are. They're taking the inside view. They see the happy couple. They're basing their prediction on the particular situation inside that situation and making a prediction.

As opposed to you come up to somebody and say, how likely is they married and say, well, let's see, what's the socio-demographic category into which this couple falls? What's the divorce rate for this socio-demographic category? Looks like they've got about a 65% chance of staying married in the next five years and a 53% chance in the next 10 years. That's the outside view.

You're identifying a comparison class and that comparison class is the start of your forecasting process. You're saying, okay, my initial forecast about how long this couple is going to stay together, say, we'll be next five years, it's around 60%. Now I discovered that the husband is a psychopathic philanderer. Now I'm going into the inside view.

So 60% is going down. How fast is it going down? Maybe to 40, 30, 20. I'm updating, but I didn't start by saying it's a 99% chance of going to stay married because it's a little happy because that's a dumb forecast.

Almost everybody can manage a little copy on their wedding day. My favorite example of an outside view, inside view, tension actually comes from Danny Kahneman. I think it was thinking fast and slow. He described being on a committee trying to write a textbook and they were trying to predict how long will it take us to write this textbook.

They thought about it using the inside view. They were like, how many chapters do we have to write? How long does it feel like it should take us? Their answer was something like one year.

I think the outside view was closer to five or six years because if you looked at other committees that were trying to write textbooks, I don't think any of them finished in under four years. That was the outside view they should have been adjusting from and indeed I think it took them well over six years. So taking the outside view and especially starting with the outside view as you're kind of baseline and then maybe adjusting it based on unique or special features of the situation, that's a very helpful tip that aspiring super forecasters can learn to use. I was hoping we could go through a concrete example of a forecasting question and just, I mean, you're not a super forecaster yourself, I'm not a super forecaster, but we could maybe talk about how a super forecaster would approach it.

And my nomination is, are we going to get to full self-driving cars by the year 2025? The kind of car where you can really just take a nap and let the car drive you to work through busy city streets? Will something like that be available by 2025? So do you have any thoughts on how a super forecaster might start thinking about that question?

The first thing they would do is they would pick holes in it. In the question? Yeah, they pick holes in a question. They would be like you're talking to a contract lawyer and they'd say, oh, Julia, my God, this is so underspecified.

Okay, you mean anywhere in the world? Yes, yes, I do. Yes. Okay, so you have more of a shot at happening in Singapore because they're pretty far advanced with the infrastructure and they have a different legal system.

Even if we had close to perfect technology, technology is never going to be perfect, given the US tort law set up and given the dynamics of US politics, it would be very difficult to imagine that happening by 2025 in the US. Could it happen in certain authoritarian or semi-authoritarian systems that are technocratic, maybe in Dubai or in Singapore, if that's the question, now I'm saying, okay, maybe 25%. Now, take a nap, I'm not so sure. I think even there, the laws would be you have to be conscious and you probably shouldn't be drunk.

There would still be laws and essentially observing, at least in principle in charge. But yeah, that's how they think. They think in very detailed terms and they think about boundary conditions and it's not just the technology, it's the politics and the law, right? I guess this is how a lot of forecasters go awry.

Is they're thinking of one specific kind of scenario? Like I was thinking about the US and I was only focused on the technology itself and not on the legal framework surrounding that deployment of the technology, etc. And so I can a little bit empathize with the experts in your tournament who, you know, when the questions were solved and they were wrong, they kind of put us, oh, but that's not what I was thinking of when I said, you know, 75% chance. Yes.

The thing I was thinking of was blah, blah, blah. But that is part of the game is you have to be thinking about all the factors and I guess that's what the super forecasters are good at. You know, it's a wonderful experience working with them. The vast majority of psychological work in the last few decades, several decades, has been on documenting how people aren't all that smart, the various areas and biases that people fall prey to.

So working with human subjects who are, you know, probably smarter than I am was a real privilege and it remains so because I've found among them they're not subjects anymore. They're the collaborators. I did just have a thought about how I might approach the forecasting self-driving car question. Tell me if you think this is a good use of the outside view.

So I was thinking it might be useful to look at other technologies in the past that also got a lot of hype like self-driving cars and also seemed to many people to be only a few years away like self-driving cars. And then I would look at how long each of those technologies actually took to come to fruition or whether they did it all and treat that as kind of the outside view for self-driving cars. What do you think about that? I think it's a good idea that the devil lurks in the details of defining the relevant comparison class.

It's pretty easy for the divorce case, right? We say what's the most sociodemographic category? So, you know, Harry and Megan, you know, what's the divorce rate for British royals? And now that they leave the royal family, you know, what's the divorce rate for Hollywood celebrities or, you know, changing based rates, changing comparison classes.

Because I don't have the requisite technical expertise, I'm simply going to punt on what the relevant comparison class would be, but that in principle would be the right move, yes. I also just wanted to note that one reason I liked this question as a test case is that I think it's kind of hedgehog bait, really some kinds of hedgehogs that I know. The techno-optimist hedgehogs, for example, tend to get very excited about progress and they really want to be bullish about everything. Whereas the techno-pessimists just find this kind of hype so annoying and they like to focus on the ways and all the instances in which progress is slower and worse than we expected it to be.

And both of those world-users sometimes, right, but if you're only sticking to one reflexively, I think you're going to miss a lot of calls. Yep. So many subspecies hedgehogs. So, Bill, what have you learned about how trainable these super-forecasters skills are?

Like you can identify specific things they're doing, like taking out that view, but have you studied can non-supercasters develop those habits and learn to deploy those tools as effectively as super-forecasters can? Could we teach these in a college class? Yeah, we certainly have. And in classes, yes.

I wish it had been formally embraced by the intelligence community, which of course meant millions of dollars from developing all this stuff. Do you happen to remember any results about how much improvement people could see if they, like, through practice? Yeah, about 10% Oh, great. And how long was the training that they had to go through with this whole semester that they had to practice?

The training only took about an hour. An hour? An hour? An effect for a severe tournament year, which is about nine months, and about 100 plus forecasting questions.

So, count me as modernly astonished by that. Yeah, maybe some people in the audience aren't as impressed as we are by this. If so, that's just because they haven't been beaten into deep cynicism by all the failures of other educational interventions to do anything at all, so they can't. You have to take it in context.

We appreciate what a big deal, even 10% improvement is. One other kind of cool result from your experience with the super forecasters I wanted to highlight is that not only were they more accurate than everyone else, they also improved much faster than everyone else. Am I saying that right? Yes, they are fast learners.

Yeah, they are so interested in why they get things wrong when they get things wrong. And so if their forecast is way under confident or way over confident, instead of trying to make an excuse for themselves about, you know, well, you know, I meant such and such a while, I was really thinking of this other thing, they get really curious about Oh, well, you know, what process was I using that made me get the wrong answer and like, how can I fix the process and so on? It's really cool to watch, actually. It is.

It's a wonderful thing to watch them do post-mortem and they not only do post-mortem on their failures, the best ones do post-mortem on their success as well, because they know that you can be right but lucky. We might have gotten the right answer because we made a set of offsetting errors, not because we had the fundamentally correct diagnosis. You've talked about Danny Kahneman a few times in this conversation and I expect many of our listeners have read or at least heard about his book Thinking Fast and Slow and Phil as you're probably aware, Kahneman is more pessimistic than you are about the possibility of training people in better thinking skills. Like just in general, he's pessimistic about our ability to overcome the cognitive biases that he writes about and improve our judgment in the lasting way.

So I'm just curious, why do you think you and Kahneman disagree about our ability to improve? I think it runs deep. I mean, he had a profound impact on my thinking and the cognitive illusion metaphor I always thought of as quite compelling. If you believe that cognitive biases are as tenacious as perceptual illusions, you're going to be really pessimistic.

Perceptual illusions continue to trick you even after they've been explained to you. You still see them the wrong way. Famous example, the Moer liar illusion that Danny uses in his book Thinking Fast and Slow, there are two lines that are of identical lengths. One of them looks longer than the other because of the way the arrows are pointed.

Beautiful illusion. The reader should look it up if we can't, we don't have video videos. We can't show it to them, unfortunately. But it's a beautiful illusion.

And you know what? You're going to see it the wrong way. You're going to pull out a ruler and you're going to measure it and say, oh my god, do the same lengths. Then you take the ruler away and you're going to look at it again.

And you're going to see it wrong again. That's why Danny tends to be more pessimistic because he thinks of cognitive biases as being as tenacious as perceptual illusions. I think some cognitive biases do have that property and are that tenacious. I think other cognitive biases can be checked to some degree if you have the right cognitive style or the right training mindset.

I think you can be trained to use comparison classes to some degree, even though the natural first thing to do is to take the inside view. I think you can be trained to be a little more tolerant of cognitive dissonance. And I say, I'm asked however, although but sorts of questions, even though the natural thing to do is to dislike cognitive dissonance and prefer everything to come together in a nice need, a value to get stalled. So it's a matter of degree.

Your point about what does it really mean to overcome a cognitive bias? And does it still count if you you still experience it, but you've put in safeguards to correct for it, even though you still feel the bias on your inside view is an excellent point and makes sense to me. I have this other theory about why Danny Kahneman is more pessimistic than you and me, which is that I think his work in and a lot of psychologists work on cognitive biases and trying to train critical thinking skills focuses on college students who are participating in these experiments that are being run by the professors on college campuses. And they're not necessarily people who have come to this experiment because they really want to improve their critical thinking skills and they're excited to put in the work to improve their participating in the experiment for a credit in their psych class or something like that.

Whereas the people that you're working with, the super forecasters or the aspiring super forecasters, actually do want to improve. They have the motivation to be more accurate and they're willing to put in the effort. And I'm much more optimistic about that group being able to improve than about the group that's just there to check the box. Does that make sense?

I think that makes a lot of sense. College students who participate in most psychological experiments are not all that motivated to get the right answer. They're motivated to get in and out pretty fast. And the researchers' hypotheses are often assuming something like that.

So it actually works to the benefit of the hypothesis testers that the subjects are processing things in fairly superficial ways. I think the Kahneman-Tiversky heuristic devices work cannot be dismissed as an artifact of under-motivated undergraduates. But I think a substantial amount of psychological literature can. But the Kahneman-Tiversky effects are remarkably robust.

And you see versions of them holding up in complicated real-world situations as well. But yes, that is certainly one of the differences. Well, unfortunately, I think that if you and I are right about the importance of motivation here, that doesn't bode well for being able to teach kids super forecasting skills in their mandatory high school classes or even college classes where they're there to get a credit. It does bode well for our ability to cultivate a class of super forecasters who can help our society make better decisions.

But that's different from training kids and critical thinking, which is something I know many people do really want to do. But I think that's not a promising sign about that. I understand that they desire to make it mandatory. But it does have some perverse side effects.

So switching gears a bit. Now, I feel an elephant in the room in this discussion of how affected the super forecasters were and how they were 30% more accurate than the CIA themselves. I think a lot of people are probably wondering, well, has the government adopted super forecasting? Like, do they have a huge team of super forecasters that they consult about policy questions?

Like, when will the supply chain issue ease up? Or what's trying to do if you type one, things like that. And it sounds from some things you've said so far, like the answer is no. The government has not bought into super forecasting wholesale.

Is that mainly because of the social frictions, like the incentives issue you talked about, where the people who already have positions of prestige don't want it to be discovered that they're not actually any more accurate than some random 24-year-old on the internet? That's right. So the senior people my age don't welcome some upstarts from an R&D agency like IAR about coming along and saying, hey, let's put the 60-something up against the 20-something and see who can do a better job predicting. Now, what Xi Jinping is going to do on Taiwan in the next two years?

That's a hard sell. Have you found any kind of ways to work around this incentives problem in your efforts to get forecasting more widely accepted? Like, is there a way to persuade people to do it, even though it might risk exposing them as accurate as they thought they were? Yeah, it definitely threatens people.

So my first instinct is to say, well, maybe you can reduce the resistance to forecasting tournaments by changing your organizational culture and say, look, we're going to treat this sort of as a game. We want everybody to play. We want everybody to try to get better and we're going to reward people for playing and getting better. But we're not going to punish people for being wrong.

You know, it's still going to be somewhat threatening, but I think you can certainly create more psychological safety running around the activity of people think their jobs are on. They think someone's going to come along at the end of the day and say, well, let's take the top 10% of the best forecasters and a few other guys. I guess we really don't need you. That's going to be a threat.

It's interesting what is so much an easier sell to work on super forecasting to sophisticated private sector firms in technology and in finance that is in the government. I mean, there are parts of the government that are open to this. It really depends on who's in charge at that particular moment. The right person's in charge of the National Intelligence Council, right, versus the DNI, not Director of National Intelligence, the right person's in the Federal Reserve, right person's here and there.

It'll go forward. Oh, change jobs backward. Well, hopefully it's two steps forward, one step back and not the reverse. There was another big and important reason that I think there's a bit of a slow uptake of forecasting by institutions like the government.

And I know you've thought about this and are working on it currently. It's something you call the rigor relevance gap. Could you explain what that means? Well, the forecasting tournaments are really good at answering very specific, resolvable questions like how many people are going to die on the Israeli Lebanese border in the next six months in military clashes or how many are this or that?

I mean, very specific. What is Spanish-German bond yield spread is going to be in the next six months? So very specific questions. Policymakers, however, care about more general questions.

They care about, well, how compatible or incompatible are the geosynthetic interests of the US and China? Is a war between Israel and Iran just a matter of time, a full-blown war between the two between the two entities? And if it is a matter of time, is it a potential becoming a nuclear war? Is the Euro zone viable over the next 10 years?

I mean, these are long-term questions that you really are difficult to pose in a forecasting tournament. But what you can do in forecasting tournaments, though, is try to bridge the rigor relevance gap. These big apps are not questions we want answers to, but they're not rigorously resolvable in the short term. What can we do?

We say, the idea here is you take a big scenario you're interested in. Say you're interested in whether advanced forms of artificial intelligence are going to cause major dislocations in white collar labor markets in the next 20 years. No more radiologists, no more airline pilots, no more they're all replaced by machines. If you believe that in that scenario and you want to figure out more rapidly than you could otherwise could, whether we are on that historical trajectory to wear that outcome, you get together some creative experts in groups and you get stuck in the work together in certain ways to generate insightful early warning indicator questions like, you know, would artificial intelligence when the world beat the best player in goal by the end of, I think, 2016 or 17, that did happen.

Another question is whether an AI system would defeat top US air force pilot in a simulated dogfight. And I think that did happen before 2021. Another question is whether Watson MD would defeat the world's best medical diagnostics in a medical diagnosis tournament by the end of the 2019 or 2020. That did not happen.

So what does that tell you? It tells you that there is some truth to this idea that AI is going to cause major dislocations in white collar labor markets. Is it going to be as massive as implied by that scenario? Will it be as early as 2040 or 2035 or 2030 or will it be 2060 or 2070?

It's probably going to happen, but the time framing might be a little bit slower than the techno optimist would hope or the techno pessimist would fear. So one aspect of the rigor relevance gap is that we care about things that are going to happen far in the future or not going to happen far in the future. A separate issue is that the questions are just kind of big and nebulous and not well defined enough to make predictions about the big things that people are debating are not typically well defined enough to put into a forecasting tournament. Are we moving into an era of hyperinflation?

You know, that's a that's a vague question. You need to specify it a lot more. And similarly, with respect to strong AI causing major white collar market dislocations. Oh, that's kind of a pretty vague too.

Isn't that when you think about it? It sounds smart. It sounds intelligent. It sounds meaningful.

But on close inspection, it needs to be decomposed into micro indicators. And those micro indicators are things you can run in forecasting tournaments and can help you make the conversation smarter, faster than otherwise would advance. There's another technique that I know you've been working on recently that I think is designed to help close this rigor relevance gap. It's called reciprocal scoring.

Could you explain what that is and what purpose it's supposed to serve? It's a brilliant idea that is still being tested in the lab. So we're pretty mature to announcing your rika. But the core idea is this, you put together two panels of people.

On those panels of people, you put individuals who have very good track records of being accurate and value only one thing. They only value their reputation for continuing to be accurate. So let's say you populate the two panels with superforecasters. And the ideal superforecaster has no ideological skin in the game.

The ideal superforecaster just wants to continue to have any elite track record. And you know, that means I'm going to be expelled from a political party. I don't care. Okay.

So you have two panels of people each of whom have this desire. And you give them a difficult question that cannot be objectively resolved in a regular forecasting tournament. You give them a question about, well, you know, if the Federal Reserve were suddenly to do X, Y, and Z and tightening the money supply, what would U.S. economic growth inequality and so forth look like in 2023 or 2024?

You ask the question, if they did this immediately right now, now as soon as you ask that question, it becomes kind of factual, right? Because it's not going to happen immediately. It's already a kind of factual question as I ask it because it hasn't happened. It's in the past.

But I said, if they did this right now, this second, what would the effects be on U.S. economic growth inflation, etc. And you get each panel to try to predict the predictions that the other panel would make. So now we have an accuracy standard.

How accurate are you predicting the predictions of the other? And you know that the other side highly values its accuracy. That's the one thing they really want to be. So it's really I'm challenging you to be your best cognitive self.

I'm saying to you, look, you might as well put aside your ideological priors here and any other stuff that you know, deep inside your soul is BS. You know, put all that aside and just try to do your very, very, be all that you can be cognitively by trying to predict the prediction to the super smart people on the other side. And by the way, they're doing the same thing. So that's a very interesting exercise.

And you think about it, you know, imagine you're the chairman of the Federal Reserve, what would you value more knowing the answer to the question I just asked, what would be the impact of these policy options? Or would you value more knowing what the inflation rate and economic growth is going to be a year from now? You probably want to know what the policy impacts would be. And you don't want to have to wait a year to find out what the inflation rate is, because that's what you're going to have to do.

Now, for the reciprocal scoring, you can find out tomorrow what the right answer is, because you just have to look at how well each team forecasts the predictions of the other teams. So you get an answer virtually immediately. Now, it's a different conception of truth. It's not an objective ground truth.

What you have is inter subjective ground truth. That's what you have. And now you say, well, that's not as good as objective. And I say, you're right, Julia.

It's not as good. But it's a heck of a lot better than purely subjective. Right. The big advantage of doing reciprocal scoring is that you can extend it into domains where there is no objective answer.

And it turns out there are many things in life like that. Great. Well, we're just about out of time. But since this is the Ezra Klein show, we're going to end the podcast in the traditional Ezra Klein show manner by asking you for book recommendations.

So what are three books that thoughtful citizens should read? Okay. Well, no surprise. I'm going to recommend Daniel Kahneman's Thinking Fast and Slow.

For all the reasons we've been discussing over the last many minutes, I would also recommend Steve Pinker's Enlightenment now, which is of course, we're much healthier and wealthier than we otherwise would be. So here's a kind of factual, we're much healthier and wealthier than we otherwise would be if it hadn't been for the Enlightenment. It hadn't been a major pivot in how human beings think about knowledge in the 17th, 18th centuries. So those would be two.

And the third book is, and okay, they'll both those people are famous. They're famous writers and they reach popular ideas. Here's a third book written by a Columbia professor of political science, who I've known for many years, admired for many years, book never gone on the bestseller list. But it's an influential book in political science.

I think it was brilliant. It had a big impact on my early career. It came out in 1976. It's by Robert Jurvus, perception of misperception in international politics.

If you look at the description in that book of how to balance deterrence and reassurance in relations between the US and the Soviet Union or how it failed with Germany in the 1930s, if you look at the descriptions of those sorts of things and you think it's totally relevant to US China today, it's totally relevant to everything going on in the world, it's a rare scholar who has a deep understanding about international politics and human psychology. And I continue to think of it as a masterpiece, but much less well known than the other two books. Well, maybe the Ezra Clancheau can bump it onto its rightful place on the bestseller list. We'll see.

It's a somewhat difficult read, but okay. The honesty of a fox, I appreciate it. Well, thank you so much for coming on the Ezra Clancheau and letting me pick your brain about the difference between foxes and hedgehogs and super forecasters and unsuper forecasters. And I'm excited to see where your new experiments and ways to close the bigger relevance gap lead us.

Well, thank you very much. I'm going to talk with you, Julie. The Ezra Clancheau is a production of Your Times Opinion. It's produced by Jeff Gelt, Rache Karma, and Annie Geltin, Spack Check by Michelle Harris, and original music by Isaac Jones, mixing by Jeff Gelt.

Frequently asked questions

How long is this episode of The Ezra Klein Show?

This episode is 52 minutes long.

When was this The Ezra Klein Show episode published?

This episode was published on December 3, 2021.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Ezra Klein Show episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.