This episode of Radio Lab is supported by Stitcher Smart Radio, offering the Stitcher mobile app. With Stitcher's free app for mobile phones, listeners can get the latest episodes of Radio Lab, NPR, and thousands of other podcasts on demand without downloading or syncing. The free Stitcher Smart Radio mobile app can be found in the iPhone or Android App Stores or at Stitcher.com slash Radio Lab. Wait, you're listening to Radio Lab.
Radio Lab. Short from WNYC. And NPR. Hey, I'm Jad Abunrod.
I'm Robert Kowich. This is Radio Lab Podcast. And this week on the podcast we did a collaborative thing with our friends. We have our friends at On The Media, a wonderful show produced here at WNYC.
We decided to give them a headache. It was a very provocative idea. We got to us by Jonah Lehrer, one of our regular contributors, and we just couldn't get it out of our heads. It was such a...
Boogie! Yes. It begins with the work of a psychology professor, Jonathan Schooler, who many years ago, too great a claim, got a bunch of people in a room together, and he had them watch a video. They basically watch this bank robber walk into a bank and he hands a note to the clerk and he says, don't press the alarm and you won't get hurt.
Clerk then hands him some money and he exits. And these people watching the video, do they get to go look at the guy? You got to straight on look at the bank robber, absolutely. And here was the test.
After everybody watched this thing, he had half the subjects, only half. Right down in his much of tail for five minutes, everything they could remember about the appearance of the bank robber. So they'd write, you know... Clearly brown here, mustache, glasses, whatever it was.
Yeah, they just described the guy that they just seen. Now, only half of them did this. The other half did nothing. And then later, he had all the subjects look at a police lineup and try and identify the robber.
Pick the bank robber, so you can pick them. Yeah. Now you would think that the people who had to describe the guy right after seeing him the first time, they would do really well at this, you know, because they had kind of set the memory. Yeah.
That's not what they found. They would describe the face in great detail. They were actually less good at recognizing the face than if they didn't engage in any description at all. And not just a little less good.
They were like 30 or 40 percent less good. So it was pretty whopping. Wapping and just odd. Yeah.
Even more odd is that as he did more studies, he found it wasn't just a face thing. It happens when you try to remember all kinds of stuff. We found the effect with colors. We found it later on with tastes.
Choices? The effect was so strong and in so many different places that he gave it a name. Verbal over shadowing. Verbal the words over shadow.
The truth. Yeah. He was so much more kind of a schooler and kind of a rock star. Yeah, I did get it got some press at the time and...
He appeared everywhere really. Including on a little show called Radio Lab. He was even here. And this is where our story really begins because just as people like us were getting very excited about his work, the data began to go a little funny on him.
That's right. And it all began when he tried to replicate that original experiment. Yes. Well, as you kept doing it, what happened?
Well, over the years, over the next five or six years when I attempted to do it, again, I would get the effect but not to the same degree that I did initially. And this is a little troubling for him. That's Jonah Larris, science writer, one of our contributing editors. He turned us on to the story and it went like this.
The first time schooler tries to replicate that study, that effect? Falls by 30%. And so he tried it again and again. And as we kept trying to replicate this study, the effect size got smaller and smaller and smaller.
Meaning that big difference between the people who wrote about the bank robber and that are wrong, and people who didn't write about the bank robber and they got it right, that big difference began to decline. This slow downward trajectory. It did sort of gradually get smaller. It wasn't as if all of a sudden it disappeared.
Now it's still significant. It's still publishable, but it's not nearly as exciting as it was that first time. So, as you can imagine Jonathan Schuller was sitting in his office and he was like, what? It's happening here.
I mean it was so good the first time and then it started to fade. He's a very good experimental psychologist. He's not sloppy or anything. No, so he's thinking what's happening here?
And the first theory he has to really wrestle with? It's something known as regression to the mean. I mean those are three of the most uninspired words. But together- They're four words though.
Two. But Jonah can you make a concrete for us? Sure, so you flip a coin, right? Let's say you flip a coin ten times.
You may get eight heads in two tails. And you may say, oh my gosh, I discovered a new law of coin flipping. When I flip coins in this room, they're almost always heads. But if you kept on flipping that coin for say a thousand times, your data would show almost certainly unless you really had discovered something very peculiar about that room.
The results would get closer to the true result, which is about 50%. The results would regress to the mean. Sorry, in case this first thought was maybe that's what's going on here. When we first did the study for whatever reason, we got lucky, or unlucky as the case may be.
You saw an outlier. Exactly. That reality is full of quirky surprises we can't explain. But over time, and this is the miracle of the scientific process, you regress to the true effect size.
But one thing about the regression to the mean account is it doesn't really explain why the effects gradually get smaller. Regression to the mean, you predict one big effect, and then it should basically tauter around the actual value. This gradual decline doesn't naturally fall out of the regression to the mean account. Yeah, I mean the effect could just go away in which case you knew you were wrong.
But why would it slowly get worse? Well, one possible explanation is that there was some aspect to the procedure that was important, that we never really realized was important. And somehow we were gradually not including whatever that secret ingredient was. Meaning, you know, as a scientist when you try and do an experiment, you try and do it the same way every time, down to the floss that you've lost your teeth with before you did the experiment.
There are too many things that you may not pay attention to. Yeah, I mean, there's a little thing off in the side that you're not even thinking about. It could have been the color of the room in which he was conducting the experiment. It could have been how charming his grad student was, who was actually asking the students to describe the bank robber.
Totally making up a story here. That's why the grad student was so charming, so good-looking, so charismatic, that he distracted the students. Then the grad student goes off, leaves the lab, and has got a much less exciting grad student, so I'm here he has distracting. And now the effect size of verbal overshadowing has gone down.
The only problem with that is that little sound that Jan and I made, beep, beep, beep, beep, beep. Means that you have to have your charming grad student at the beginning, and your less charming grad student in the middle, and your even less charming grad student. Yeah, they have to get slowly less charming. That's right.
And so if you're thinking something is changing here, what is it? Did you go on some kind of mad search to figure out what you might be doing differently? We tried lots, a lot of different things, and in the end I just moved to another area of research. You got out of town.
And apparently one of schoolers' colleagues told him, Don't worry about it, the only mistake you made was trying to replicate in the first place. Really? Yeah. But here's the problem.
Not just me who has experienced this peculiar decline effect. As you start to look around, he realized what was happening to him was happening all over the place. Other scientists, in all kinds of other sciences, were having the exact same problem. In biology there was a meta-analysis of many different biological findings showing...
There are a ton of examples he says, and here's one. In the 90s there were a bunch of studies about animals using symmetry to find mates, like birds, females, females choosing their sexual partners based on how even the male's tail feathers were. It was a very exciting idea. And the first year there were eight tests of it, and all eight found the fluctuating asymmetry.
That's what the phenomenon is called. Is real, we also got an effect size. So it seemed true. Yeah.
All over the world and all these different species, females had evolved this unconscious tendency to prefer symmetrical males. The next year it's tested 12 times, and 9 of the 12 confirm it. And then things start to fall apart. You can make the sound effect exactly.
Until by the end of the 90s, you're going 1 for 13. 1 for 13? 1 for 13. Now of course these studies are not black and white, yes no studies, there's some gradation, but this was the basic trend that Jonas saw.
In just case birds seem a little distant, here's another example. And I think this is for me the most troubling area of the planet effect. Because you see like second generation antipsychotic. Second generation antipsychotic.
These are drugs used to treat people with schizophrenia, bipolar. When they first came out in the late 80s, early 90s, some studies found that they were about twice as effective than first generation antipsychotics. And then what happened is the standard story of the decline effect, Q, the sound effect. Which is clinical trial after clinical trial, the effect size just slowly started to fall apart.
And that's not all. You see a similar decline with things like Prozac and antidepressants. The effect of the drugs have gotten weaker, but the placebo effect has also gotten stronger. I was talking to one guy at a drug company who was kind of interesting.
He blamed that on drug advertising. He said that they started to see their placebo effect go up in the late 90s when these drug companies started advertising. But then wouldn't that actually offer an explanation for this decline thing? Because if you know about what this drug is supposed to do, maybe it works differently somehow?
Certainly there are areas of psychology where that can change the outcome in one way or another. But it's very unlikely that in say these female preferences for symmetrical feathers, that the birds got wind of the symmetry finding. Now all of a sudden they're not into it anymore. I don't know.
You haven't been around chickadee conversation lately. Fastest quickly, but it's a chickadee. Does that mean that you can you explain why what you found at the beginning? Isn't that what you find now?
And why it gradually went away? The gradually still puzzling. I tell you, I find it very puzzling too. I'm personally baffled.
It's tough to come up with an all-purpose explanation or some easy fix. It could be a lot of different things bundled together into one phenomenon. He says maybe in some cases it really is statistics. The Russian to the mean is almost sure to be a part of it.
And maybe in some cases it's you know, this gradual change in the procedure and something that we just don't know what it was that happened. Can't rule it out. But I would probably be less shocked than most people if something unconventional is actually involved in this as well. Unconventional like.
Like I say this with some trepidation, but I think we can't rule out the possibility that there could be some way in which the active observation is actually changing the nature of reality. But somehow in the process of observing effects we change the nature of those effects. Oh, you're in real trouble. Essentially what he's saying we think is that when he discovered that thing with the bank robber experiment that maybe the discovery itself somehow set in motion a series of events that made the thing he discovered start to sort of run away.
Well, I'm not going to say that. I'm not going to say that there's some sort of intentionality to these effects disappearing. More that it's almost, again, this is just speculation. Some sort of habituation.
So just as when you put your hand on your leg you feel it and then as you leave it there it becomes less and less noticeable. Somehow there may be some kind of habituation that happens in with respect to these findings. What is the hand and what is the leg in this? Well, in this most radical conjecture there could be some sort of collective consciousness that's habituating.
Again, radical speculation. Keep in mind the notion that the laws of reality are unchangeable is an assumption. It's a reasonable assumption. But we don't know what for a fact and there have been physicists who have been speculated that perhaps the rules change as time goes on.
The problem with this idea is if you really believe it then you can never really know anything. We're sliding into that kind of door. By this logic you could never really know for sure because reality could change based upon the observer's position, habit, spices, information, whatever. So far we have not really seen these types of things in the domain of physics but an aspirin might not do what it used to.
There's a question that you haven't asked which is let's say that we were to do a study and demonstrate this decline effect when you keep running experiments that they get smaller. Well, what happens when you try to replicate that effect? Does the decline effect decline? Yeah, that's a good question.
Maybe we could just get rid of the decline effect by studying it. But then if you were to study the decline of the decline effect then it would...on decline. It would come back. Do you know what I'm saying?
I see what you're saying. You just stuck forever in the great seesaw of the universe. We should thank our friends at On The Media. On The Media is a show that analyzes the media as sure you know.
And it's an amazing show. It's an amazing show. A broken Bob who hosts the show are funny, hysterically funny and brilliantly smart. And it's the kind of show that it's just kind of essential.
It's one of those shows. We do not recommend it more OnTheMedia.org. All in word OnTheMedia.org. Check it out.
They're going to do an entire hour on the subject of data. And we have sort of snuck this issue that we've just talked about in the middle of that show. So it's the same thing but in a very different context. You can go to their podcast on Friday the 13th of May and there it and we shall be.
Until it and we decline. Yep. Into a living end. I'm Chad Oumran.
I was and we'll continue to be I hope for the moment. Back. This is Colin Vongheering. I'm a Radio Lab listener from Portland, Oregon.
Radio Lab is supported in part by the Alfred Peace Loan Foundation. And I'm in public understanding of science and technology in the modern world. More information about Sloan at www.slown.org. End of message.