This is Hidden Brain, I'm Shankar Vedanta. We're surrounded by people who tell us they know what's going to happen in the future. A lot of people have no idea that Trump has headed for a historic defeat. Bear Stearns is fine!
Don't move your money from there! That's just being silly! These predictions have a few things in common. The commentators have complete confidence in themselves.
We as the audience love to hear them make a complicated world seem simple. And finally, no one ever pays a serious price for being wrong. Donald Trump wins the presidency! Bear Stearns in the bargain bin sold to rival JP Morgan Chase for just $2 a share.
On today's show, our fascination with making predictions and why we may need a revolution in the way we make them. If you play the game the way it really should be played, the forecasting game, you're much more subject to embarrassment. We look at how some people actually are better than others at predicting what's going to happen in the future. Ironically, these aren't the people you usually find on television, blow-veating about what's going to happen next week.
They're ordinary people who happen to know a very important secret. Predicting the future isn't about being unusually smart or especially knowledgeable. It's about understanding the pitfalls and the way we think and practicing better habits of mind. Starting a business is tough.
Shopify makes it easier. Build a beautiful store, write with AI and manage marketing, inventory and shipping all in one place. Join millions who trust Shopify to launch and grow their business effortlessly. If you're ready to sell, you're ready for Shopify.
Turn your big business idea into sign up for your one euro per month trial at Shopify.ie. Go to Shopify.ie. It's time to see what you can accomplish with Shopify by your side. At Vodafone, you get more.
Like more than more. Get a brand new Samsung Galaxy S26 Ultra for zero euro. Plus, get more peace of mind with an extra 12 month warranty when you switch to our unlimited bill pay mobile plan. Now that's a whole lot more.
Vodafone. Together we can. Subject to availability, eligibility and switching to our 24 month 65 euro per month on limited bill pay plan by 30th of June, 26th, each April the price of your plan will increase by three or fifty per month for full-term C Vodafone.ie for slash terms. The bank holiday is finally here.
Enjoy summer savers at Centro with great offers. Like Centro, Freshyard, Stripperone, Stay 2 Pack only 8 euro offer in Sunday. Selector Pringles 165 grand, 175 each. Plus Diet Coke and Coke Zero, 12 grand packs only 6 euro each plus deposit.
Bank holiday value sorted. Centro. Live every day. Phil Tack is a psychologist at the University of Pennsylvania.
Over several decades he's shown that the predictions of so called experts are often no better than what he calls dart throwing chimpanzees. After spending years criticizing forecasts and forecasters, Phil decided to look at people who were very good at predicting the future. In his book Super Forecasting, The Art and Science of Prediction, Phil explores how we can learn from these people to become better forecasters ourselves. Phil, welcome to Hidden Brain.
So, Phil, lots of people watch television at night and millions of people feel like throwing things at the television set each evening as they listen to pundits and prognosticators explain the day's news and predict what's going to happen next. Of all the people in the country, you probably have more cause than most to hurl your coffee cup at the television set, because starting in 1984, you conducted a study that analyzed the predictions of experts in various fields. What did you find? Well, we found that pundits didn't know as much about the future as they thought they did.
But it might be useful before we start throwing things at the poor pundits on the TV screen to consider their predicament. They're under pressure to say something interesting. So they resort to interesting linguistic gambit. They say things like, well, I think through the distinct possibility that Putin's next move will be on a stony.
Now, that's a wonderful phrase distinct possibility. It's wonderfully elastic, because if Putin does move into Estonia, they can say, hey, I told you there was a distinct possibility that he was going to do that. And if he doesn't, they can say, hey, I just said it was possible. So they're very well positioned.
Now, if you play the game the way it really should be played, the forecasting game, and use actual probability. So you play it the way Nate Silver plays it, and you wind up with a 70% probability that Hillary will win the election a few days before the election in November 16. You're much more subject to embarrassment. If he said there's a distinct possibility that Hillary would win, he could he would have been very safely covered.
Because when you ask people to translate distinct possibility into numbers, it means anything from about 20% to about 80%. I want to talk about how and why people often arrive at the wrong forecasts. In your book, you tell the story of a doctor in 1956, so we're going to see a dermatologist because he saw spots on the back of his hand. Tell me the story of Archie Cochran.
Well, Archie Cochran was one of the early pioneers of evidence-based medicine. He was a great believer in running experiments, randomized controlled trials, and he was very skeptical of his fellow physicians. He thought that they were systematically overconfident in much the same way that my data suggests a political pundit started systematically overconfident. So he was a skeptic by nature.
Yet when he did get this cancer diagnosis from an eminent specialist, he didn't wait for the biopsy results to come in. He just allowed them to go right into surgery immediately before the biopsy, which was a shame because it turned out that he did not have an emergency. We use that as an example of the bait and switch heuristic. So Archie Cochran didn't know whether he had cancer, but he was looking in the eye of a very renowned specialist who looked exactly like the sort of person who would know.
So he took a really hard question, do I have cancer? And he substituted a somewhat easier question. And that is this guy I'm talking to right here, it looked like the sort of person who would know if I have cancer. And the answer to that was a resounding yes.
So he took the answer to the easier question. He assumed it applied to the harder question, and he forged ahead and got massive surgery that proved to be unnecessary when the biopsy results came in. Archie Cochran didn't make that mistake again. He went on to have a very distinguished career where he developed a series of techniques to help doctors guard against the pitfalls of following their intuitions.
He said, look, you might be very smart, you might know a lot, but there's something you have to do before you act on your intuitions. This did not make him popular with his fellow doctors. Right. And Cochran now is honored, but I think during his lifetime, a lot of his colleagues, Tom is a pain in the butt.
He would run experiments that would repeatedly show that the intuitions of his colleagues were off base. There was a debate, for example, among cardiologists about how long heart attack patients should stay in the hospital after the heart attack, whether the mortality would be higher if they went home or if they stayed in the hospital. And the conventional wisdom among cardiologists was that it was better for them to stay in the hospital. But Cochran said, look, we have to test our intuitions.
People would be happier, it would be less expensive, and so forth if people were resting at home. And they thought that would be an ethical experiment to run because it could kill people. If it showed that the people getting the home rest were actually dying at a higher rate. But Cochran persisted.
He wasn't a guy to take no for an answer. And they ran the experiment. And he also had a mischief history. So he presented the results to his colleagues and he flipped them around.
He showed them a chart that indicated that the heart attack patients who went home were dying at a higher rate just as a conventional wisdom predicted. And the doctors were outraged. They said, Archie, we told you, you're a fool. Look what you've done.
You've caused a great harm. You've got to stop this experiment immediately. And then he paused and he mischievously smiled and he said, oh, I got the data turned around. It turns out that the people were going home or living longer.
The truth is making predictions is difficult. But many biases also get in the way of making accurate forecasts. When we make a prediction and it turns out wrong, most of us don't remember that we predicted something different. In fact, the hindsight bias prompts us to believe we've gotten it right all along.
We also hear people who make predictions that turn out right, whether in the stock market or politics or sports. But that keeps us from seeing the role of luck. Many people who get the right answer are just lucky. And some people who get wrong are just unlucky.
Over time, the laws of probability mean luck can only take you so far. One reliable way to check if someone's success at predictions is driven by skill or by luck is to have them make lots of predictions and see how they turn out over time. A few years ago, the federal government launched such an experiment. They conducted a forecasting tournament where thousands of people logged into computers to make thousands of predictions.
As time passed, the forecasts could be checked for accuracy. It was a remarkable thing for the professional career government officials to do, to sponsor a project that had the potential to be embarrassing. We were one of five academic research teams that were competing to pull together the best methods of making probability estimates of events that national security professionals cared about. What kind of questions were they asking?
All over the map, quite literally. So there would be questions about violent clashes in the East or South China Sea. There would be questions about the Syrian Civil War, about Russian-Ukrainian relations, both the Iranian nuclear program, Colombian narco-traffickers, literally all over the map. So you found, after some time, that people who were not necessarily trained experts in a specific domain were actually able to perform as well, or maybe even better, than people who were the experts.
Yeah, that's essentially right. David Ignatius broke the story, I think, in 2013 in the Washington Post about how the best forecasters in the Good Judgment Project are outperforming professional intelligence analysts who had access to classified information, and were working on an internal prediction tournament. How could this be? Are these people smarter than the rest of us?
More knowledgeable? When we come back, I'll talk to Phil about what makes a Super Forecaster a Super Forecaster. Stay with us. Get more peace of mind with an extra 12 month warranty when you switch to our unlimited bill pay mobile plan.
Now that's a whole lot more. Vote a phone. Together we can. Subject to availability eligibility and switching to our 24 month 65 year-per-month unlimited bill pay plan by 30th of June, 26th.
Each April the price of your plan will increase by 30 or 50 per month for full-term C-vodafone.ie forward slash terms. The bank holiday is finally here. Enjoyed so-and-a-saver's act centroid with great offers. Like Centra Fresh-R-Strux and Stryp-Rone's Day 2-Pack Only 8-Year-O offer ends Sunday.
Select your Pringles 165-gram-175 each. Plus Dye-Coke and Coke-Zero-12-Compacts. Only 6-Year-O-Reach plus deposit. Bank Holiday value sorted.
Centra. Live every day. If you were asked to pick someone to answer a difficult question about the economy or foreign affairs, you might turn to an Oxford educated public intellectual who writes a column for a very important newspaper. You probably wouldn't turn to a retiree in Nebraska who spends his time bird watching.
But Phil Tatlock says, maybe you should. He was the opposite of Tom Friedman in the Super Forkasting book. Tom Friedman, of course, being an eminent New York Times columnist, very, very elegant writer, best-selling author. Well-known for his explanations, but nobody has any idea how good a forecaster he is.
And Bill Flack is an anonymous, retired irrigation specialist working in Nebraska, working out of the public library, right at his home, and doing a fabulous job making probability estimates in the Good Judgment Project in the intelligence community forecasting tournament. So you helped discover Bill Flack. Was he part of your team that you entered in this forecasting tournament? He was indeed.
He was one of the very best. The very best forecasters in the tournament were called out each year, the top 2% and we put them into elite teams called Super Forecasting Teams. We let them work together and they did just a fabulous job. Super Forecasters like Bill Flack, turn out to have some things in common.
Tell me about the kinds of philosophies they have and the kinds of thinking styles that you seem to find in common among many of these Super Forecasters. I would say the most distinctive attribute of the Super Forecasters is their curiosity and their willingness to give the idea a try. And when I say the idea, I mean the idea that forecasting is a skill that can be cultivated and is worth cultivating. Because it doesn't matter how intelligent you are or how knowledgeable you are.
If you believe that it's essentially impossible to get better at these kinds of tasks, you're never going to try and it's never going to happen. It's as simple as that. So a big necessary condition for moving forward is having the attitude that this is a skill that can be cultivated and is worth cultivating. Super Forecasters tend to gather information and update their beliefs in a very particular way.
Flack Lock points to Aaron Brown, the chief risk officer of the hedge fund AQR. Before he was a big shot in finance, he was a big shot in the world of poker. He was a world-class poker player. And we quote him as saying that you can tell the difference between a world-class poker player and a talented amateur.
Because the world-class player knows the difference between a 60-40 bet and a 40-60 bet. And he pauses and says, oh, maybe about 55-45-55. Distinguishing more degrees of maybe is an important skill. Why is that?
Well, the very best forecasters are well calibrated. So when they say events are 80% likely, those events happen about 80% of the time. When they say things are 90% likely, they happen about 90% of the time. So it makes a difference how frequently you update your forecasts.
If you don't update your forecasts reasonably frequently, you're going to fall out of phase with events. And that means often making adjustments that are relatively small. You suggest that forecasters should do something that doesn't seem to be very intuitive. Instead of looking at the particulars of an individual case, you say forecasters should zoom out and figure out how often something has happened historically.
So Daniel Kahneman is probably one of the greatest psychologists of the last 100 years. And he calls out the outside view. And he says, people rarely take the outside view when they do forecasting. They normally start from the inside and they work out.
But there's a big advantage to you as a forecaster from starting with the outside view and working in. Take another example. Let's say you're at a wedding and you're sitting next to somebody who has the bad taste to ask you how likely do you think it is this couple is going to stay married. And you look at the person that has bad taste and all that.
You see how happy the couple is and you can see it's a joyous occasion. You say, I can't imagine these people who are so happy together getting divorced. I think maybe a 5% chance they're going to get divorced. Now if you ask that question of a super forecaster, they'd say, well, let's see.
Let's look at the socio-demographics of the couple. And let's see. What's the base rate of divorce within this socio-demographic group? Let's say it's 35 or 40% over the next 10 years.
Okay. I think it's about a 40% chance they'll get divorced in the next 10 years. Now, that's not the end of the forecasting process. That's just the beginning.
The real value though of doing it this way of starting from the outside and working in is it puts you in the ballpark of plausibility right away. 40% of much more plausible number than 5%. Now, then you can start adjusting the 40%. So if you discover things about the couple that suggest they really are deeply bonded to each other and they've known each other a long time and they really understand each other and they've done great things for each other, you're going to lower your probability.
If you discover that the husband is a sociopathic philander, you're going to raise the probability. Those are inside you start some pieces of data that would cause you to adjust. Or you might just see them having a small fight and say, well, okay, I'm going to move from 40 to 41%. And that's one of the interesting things about super forecasters.
They do a lot of updating and response to relatively small events. And most of the news most of the time is what statisticians would call low-diagnosticity news. It doesn't change things grammatically, but it does change things a little bit. And appreciating how the news gradually builds up toward one conclusion or another is a very valuable skill.
I'm wondering if one reason many of us start with the inside view rather than the outside view is that just at an emotional level, that's how our minds think. You know, you see a couple and you put yourself in the shoes of that couple and you try and imagine what's happening in their lives. And we think in stories and we imagine what life must be like for that couple. And we're trying to see how that story will turn out.
And we're trying to follow the narrative where it leads rather than do this much more, you know, abstract remote process of saying, let me start with the rough estimate of how often something happens. There's something about that in some ways that requires us to step outside this narrative frame that we often use to understand the world. I couldn't put it better. I think that's right.
We're quite readily seduced by stories. Another example might be, it comes from research. This is again related to Daniel Kahneman and his work on the conjunction fallacy. It's another source of error in forecasting.
Let's say I ask you how likely is it that in the next 10 years there'll be a flood in North America that's more than a thousand people and that's going to make an estimate on that. Let's ask another person to make the estimate. How likely is it that there'll be a flood in California that will be caused by an earthquake cracking a dam leading to a massive outflow of water. Now, when I put the two questions together like that, it's pretty obvious that a flood anywhere in North America due to any cause has got to be more likely than a flood in California caused by an earthquake cracking a dam, right?
The California is obviously a subset of the more general North American flood thing. But people don't see it that way. The California earthquake dam story is more like a story. It's like a story.
You can actually put it together in a more meaningful way whereas a flood anywhere in North America is kind of abstract and vague. So people can transport themselves into the world and you can imagine it's like a movie, like a Hollywood movie playing out and they can see it happening. Yes, I can see that happening. In fact, I can see it as your speaking right now, Phil.
And that pumps up the probability and that screws up your forecasting track record. So again, think of forecasting as a skill that can be improved with practice. When you're making a prediction, start with the base rate, the outside in view. Beware of the risks of storytelling.
Finally, amateurs make three kinds of predictions. Yes, no, and maybe. Professionals have many gradations of maybe. And they attach specific probability estimates to their predictions, allowing them to go back and learn where they went wrong.
But even if you do all these things, I asked Phil how you can be really sure that predictions that turn out correct are because of the technique. You know, there's an old trick that they play in business schools to talk about the role of luck, where they divide the class into pairs and they say, you know, have a coin toss between each person and then the winner of each of those coin tosses competes against another winner. And after 12 rounds, there's one person who has declared the winner. And of course, that person in a business sense might seem to be extraordinarily good.
But really, all that's happened is they've happened to win 12 coin tosses in a row. They've just been very, very lucky. How do you distinguish between people who are lucky and people who are actually very good? Well, that is indeed the $64,000 question.
And it comes up in finance too. I mean, there are some finance professors out there who would argue that the really famous super investors, Warren Buffett or Ray Dalio and people like that, are in some sense like coins that come up heads 20 or 30 or 40 times in a row. We have a lot of people competing in financial markets and they're making a lot of predictions over long stretches of time. So you're going to expect some streaks.
And when we get really streaky performance, we declare we found a genius. So the skeptics would say, well, Phil Tuttlock is doing essentially the same thing here. He's anointing people who are essentially lucky. So we built in a lot of statistical checks, but you can never be 100% sure.
One of the other critiques of Phil's work is the kinds of questions that super forecasters are answering are not really the questions that people want answered. Most of us are interested in the big questions. Who's going to win the current standoff between the United States and Russia? Super forecasters tend to answer much more narrow questions.
Is Russia going to invade Ukraine in the next 18 months? I think that's a very fair criticism of the first generation of forecasting tournaments that we put all of our effort into improving forecasting accuracy. Now, I'm not going to say the questions that people were trying to answer were trivial because they were by no means trivial. But no means trivial whether Russia was going to invade Ukraine or the Syrian civil war was going to drag out as horribly long as it has and so forth.
I mean, the questions were really important things. But were they about the things that matter the most from a public policy perspective? Could we have made the questions more relevant to deep policy questions? I think the answer is yes.
And science proceeds incrementally. You do one thing at a time. So the first generation of forecasting tournaments generated is that there is a skill that is worth cultivating and can be cultivated. That's step one.
Now, I think the next generation of tournaments should be focusing on exactly what you're saying here. I think we should be focusing on as much on the encycleness of the questions as the accuracy of the answers. I'm wondering if some questions just are too hard to forecast. If we were having this conversation in 1917 rather than 2017 and someone would tell you, you know, 40% of the country right now is in farming.
And in 100 years from now, you know, 2% of the country would be farming. It would be reasonable to sort of conclude that, you know, a good chunk of the country would be out of work, that our unemployment rate would be, you know, 35%, 40%. And that would be a perfectly reasonable conclusion to draw. And of course, there just would be no way to fully understand how that would not have come to pass.
And I'm wondering in some ways, you know, is the point of super forecasting to break questions down so that they're so simple, so that they're answering questions with accuracy, but not necessarily questions that are of importance. And there's the old joke of the guy who loses a coin on the street and he searches for the coin under the street lamp because that's where it's brightest. Not where he actually dropped the coin. And is that what we're doing with super forecasting?
Is it possible that we are asking the kinds of questions that are the easiest to answer, the easiest to which we can attach these probability estimates? But the moment you go a little bit further out, we're essentially just guessing. Well, that you've raised a host of interesting questions and I would be the first to agree with you that forecasting tournaments can degenerate into games of trivial pursuit. And I'd also agree very much with your premise that forecasting accuracy falls virtually to chance to the dart throwing chimpanzee when you try to forecast more than about 10 years out on geopolitical and geoeconomic questions of consequence.
So both of those things are true. There's a limit on the forecasting horizon. And it's true that forecasting tournaments, if you don't pick the questions carefully so that they add up to something important, they can become trivial. So both are true.
I don't think that they mean we shouldn't be doing forecasting tournaments. I think that we should be doing forecasting tournaments as well. I'm going to ask you one final question. And this is also, I think, a potential critique of super forecasting, but it comes to the form of a forecast that I'm going to make.
The reason I think many of us make forecasts or look to prognosticators and pundits make forecasts is that it gives us a feeling like we have a handle on the future. It gives us a sense of reassurance. And this is why liberals like to watch the pundits on MSNBC and conservatives like to watch the pundits on Fox. A more cautious style that sort of says the chance that Donald Trump is going to be impeached is 11.3% or the chance that you're going to die from cancer is 65.3%.
These estimates run up against a very powerful psychological impulse we have for certainty that we actually want someone to hold our hand and tell us you're not going to die. We don't want a probability estimate. We want actually an assurance that things are going to turn out the way we hope. So here's my last question for you.
If someone advises people to do something that runs against their emotional need for well-being and reassurance, I'm going to forecast that that advice, however well-intentioned, however accurate, is likely not going to be followed by most people. What do you make of my forecast for? Well, I think there's a lot of truth to what you say. I think people think about the future.
They have a lot of goals and they want to affirm their loyalty to their ideological tribe. They want to feel good about their past commitments. They want to reinforce their preconceptions. So those are all social and psychological goals that people have when they do forecasting.
And forecasting tournaments are very unusual worlds. We create a world in which only one thing matters. It's pure accuracy. So if someone analogous to sort of world is created in financial markets or London bookies or Las Vegas bookies, all that matters is the accuracy of the odds.
I would say this. I would say people would be better off if they were more honest with themselves about the functions that they believe serve. Do I believe this because it helps me get along with my friends or my boss? It helps me fit in, helps me feel good about myself, or do I believe this because it really is the best synthesis of the best available evidence.
If you're playing in a forecasting tournament, it's only the latter thing that matters. But you're right. When people sit down in their living room and they're watching their favorite pundits, they're cheering for their team. It's a different kind of psychology.
They're playing a different kind of game. So all I'm saying is you're better off if you're honest with yourself about what game you're playing. Psychologist Phil Tackloch is the author of Super Forecasting, the art and science of prediction. Phil, thank you for joining me today on Hidden Right.
My pleasure. This week's episode was produced by Rayna Cohen and edited by Tara Boyle. Our team includes Jenny Schmidt, Maggie Penman, Rayna Klar and Bart Shaw. Our unsung hero this week is Portia Robinson-Migas from our marketing team.
Portia helped us brainstorm many ideas in the early days of Hidden Brain and has remained a steadfast friend. Thanks Portia. You can find us on Facebook, Twitter and Instagram and listen for my stories each week on your local public radio station. I'm Shankar Vedantam and this is NPR.
Starting a business is tough. Shopify makes it easier. Build a beautiful store, right with AI and manage marketing, inventory and shipping all in one place. Join millions who trust Shopify to launch and grow their business effortlessly.
If you're ready to sell, you're ready for Shopify. Turn your big business idea into sign up for your one Euro per month trial at Shopify.ie. Go to Shopify.ie. It's time to see what you can accomplish with Shopify by your side.
At Vodafone, you get more. Like more than more. Get the brand new Samsung Galaxy S26 Ultra for 0 Euro. Plus, get more peace of mind with an extra 12 month warranty when you switch to our unlimited bill pay mobile plan.
Now that's a whole lot more. Vodafone. Together we can. Subject to availability, eligibility and switching 24 month 65 Euro per month on a limited bill pay plan by 30th June, 2026.
Each April the price of your family increased by three or 50 per month for full-term C Vodafone.ie for slash terms. Here in or there in wishes to advise customers that this Saturday, Sunday and Bank Holiday Monday, the Northern Rail Line between Dublin, Connolly and Drade will be closed for essential works. There will be no dark service between Connolly and Malahide or host. A limited bus service will operate between Drade and Dublin.
Thank you for bearing with us as we deliver improvements and ensure a safe and sustainable rail network. For full details, see Irish Rail.ie.ie. Here in or there in this part of the Transport for Ireland network. Sometimes your search for a new home can feel like it's going nowhere.
But every month at AIB we help hundreds of house hunters become homeowners. Our specialist mortgage team are on hand to help you every step of the way. Moving mortgage applications to mortgage approvals, changing can we into we can and turning hoping into owning. Because we don't just help, we help make it happen.
Apply for your mortgage today, online, in branch or on the phone. AIB for the life you're after. Aladarsh Banks PLC and AIB Mortgage Bank UC are regulated by the Central Bank of Ireland.