Hi, you're listening to Science Versus the Show where we pick fact against fad, I'm Wendy Zukman, and on today's show it's the mother of all science versus episodes. We are pitting science against science, and we're asking, can you trust scientific findings? Researchers have been celebrating a breakthrough in covering the synthesis of scientists across the world. The idea of science is that it lets us understand how the world and everything in it works.
Ultimately, science tells us that by using the scientific method, we'll find facts. So, what is the scientific method? According to Richard Feynman, who is this ridiculously clever physicist who have these amazing one-liners, the scientific method is about taking a guess about the world, perhaps based on observations, and then testing it in an experiment. Now, if your guess...
It's agreed with experiment, it's wrong. In that simple statement, there's a key to science. It doesn't make any difference how beautiful your guess is, it doesn't make any difference. Smart you are who may be get what his name is, if it disagrees with experiment, it's wrong.
Science, to me, is an extremely good way of figuring out what's actually happening in the world. This is Ivan Oransky, a professor of medicine at New York University. Now, in Ivan's mind, science is so effective because it fights a phenomenon called confirmation bias. The notion that we see things sort of the way we either hope to see them or saw them before and are just looking for confirmation of it.
Science fights this, not because scientists don't have confirmation bias. Of course, they do their people as well. But because scientists will do experiments to test their ideas that they have about the world. And in many cases, scientists will be blinded so that their bias can't infiltrate the results.
For example, with clinical trials, if you're testing a new drug, you'll compare it to a placebo or the best medicine at the time. But we don't tell people what they're getting. And in fact, the people who are administering the drugs, they don't know what the person's getting either. And so that is a way to actually look at what's happening.
And then once the studies are completed, the findings are analyzed and the papers get submitted to journals where they are reviewed by other scientists in this process called peer review. And if they're deemed to be good enough, then they'll get published. But that's not all. Embedded into the scientific process is this idea that studies will be repeated by different groups of scientists who perhaps never had that original bias or idea about something in the first place.
So you end up with an idea with a finding that is really, really scrutinized. So this is the ideal, the dream of what science should be, an unbiased way to uncover nature's secrets. But then, there's science. Oh geez, this is like inception, except we've got the exposition.
Thank God. Okay, the first phase of evidence that we have that the dream aimed at what it should be are retractions. Well, there are about 500 or 600 retractions right now per year, best we can tell. Ivan Rensky isn't just a professor at NYU.
He's also the co-founder of Retraction Watch, a blog that tracks the published scientific papers that get retracted every day. Not quite too a day, but on average it's more than one. So what are these papers getting retracted for? About two-thirds of the time retractions are for something that we'd be considered misconduct.
That would be maybe faking results or plagiarism. About a third of the time these retractions are due to what I think we can sort of loosely call an honest error of some kind. According to Ivan, these retractions have gone out more than tenfold since 2001 and that's outpacing the rising publications. So there's something happening.
But he points out that around two million papers are published every year. So if there are 500 or 600 bad eggs, we're still dealing with a vanishingly small percentage. Plus, according to astrophysis Dr. Neil deGrasse Tyson, we should be celebrating these retractions in a way.
Instead of complaining that someone cheated, we should celebrate the fact that it was other scientists who found that transformation and that science added scientists, if you will. According to Neil, this isn't just a fluke that we spot these papers. Scientists are human just like everybody else. And so they might want to fake a result or whatever.
Go ahead, the process will find it and when it does, that is the end of your career. The punishments are huge. And when Neil says the process will find it, he means the process of replicating results will ultimately ensnare these dodgy scientists. Conclusion.
While there are instances of scientific fraud, they are very, very rare. And this leads us to the idea of self-correction. So as we mentioned before, a really important part of the scientific process is that results are repeated by different teams of scientists. Now, according to Professor Barton Zweebuck, a string theorist at MIT, this process of self-correction is- This is the most important thing.
And here's how it works in the world of particle physics. If you publish something that is somewhat surprising, then the first reaction of your colleagues is to show you that you're wrong. They kind of say, if it would be right, I would have had that idea myself. So it must be wrong.
The next step is to say, well, it's not wrong, but it's not that interesting. And then the next step is when they find that it's actually not wrong, but it is interesting. The next step is to say, well, somehow that idea was in basic form in my own papers, but I didn't develop it. So it just goes like that.
So any interesting or novel proposal will experience a lot of fire and attempts to prove it wrong. And therefore, if it survives, it generally is right, or at least nobody would be able to figure out why it could be wrong. So, take for example, the study published in September 2011, which found that neutrinos, which are subatomic particles, which are traveling faster than the speed of light. I know right, but seriously, according to Einstein, that shouldn't happen.
Wendy, I must say, I was having nightmares, those things. Everything we do in physics is based on Einstein's special theory of relativity. It would be like suddenly somebody would tell you that one plus one is not two. So as soon as these results were announced, people tried to reproduce them.
And by June of the following year, five different teams of physicists had independently verified that neutrinos do not travel faster than the speed of light. Experimentalists found that they had made a mistake, a very subtle mistake in synchronizing their clocks. So this was a case where the laws of physics were in question and self-correction worked like a dream. But does everything get tested through this rigorous process?
Martin says no. If you publish something that is kind of dull and not very interesting, and it kind of on the sidelines, it could remain there, untested, unverified, unconfirmed, and possibly wrong for a very long time. But in other areas of science, like psychology and chemistry, there are concerns that even interesting findings are not being widely reproduced, particularly if the original study wasn't all that controversial. Dr Alice Williamson at Chemistat University of Sydney says that when it comes to reproducing studies, it's not being done widely, but there is an increased push for people to do this with their work.
Conclusion. Important and controversial findings often get replicated, but in many cases, studies can stick around unchecked for some time. And this takes us to another problem. When studies are being replicated, so the process is working.
Scientists are finding that they can't reproduce the original results of papers. So it's believed that many findings out there are in fact wrong. So some people have called our current situation a reproducibility crisis. This is Alex Holcomb, an associate professor of psychology at the University of Sydney.
It's hard to know how to define exactly the word crisis. Crisis. A time of intense difficulty or danger. Oxfusionary doesn't have any trouble.
But what we do know is that of the efforts to try to systematically reproduce findings, whether they be in cancer biology, whether they be in psychology, the success rate has not been impressive. So for example, a 2011 paper from Bay of Pharmaceuticals reported that the company could only reproduce the findings from a quarter of the studies that had been published looking into particular drug candidates. While another company, Angin Corporation, reported that they could only reproduce 11% of the landmark cancer and blood studies that they tried to replicate. The authors of the Angin paper wrote, quote, this was a shocking result.
End quote. Here's Alex again. In fact, all the efforts that I've seen, the success rate has been less than 50%, which is really sad commentary. If you think about all the science and health stories, let's say the ABC reports in a month, the idea that maybe more than 50% of those studies, if they were redone, you'd get a different result.
That's quite disappointing. And your drug is being unveiled that scientists say has the potential to cure Alzheimer's. Right, with broken bones yield much quicker when given the non-psychotic marijuana component, kenobedial. It isn't often that HIV researchers declare a brief...
The report in the journal Gemma Oncology finds that chemotherapy often harm these patients at the end. Well, researchers found that the unique shape of the seahorse head, particularly their snout, allows them to move through the water with extremely little disturbance. Ha! But really, half of those findings might not be reproduced.
And if you want to call it a crisis. What about the seahorse one? But we really do have a problem that has to be addressed. Next question.
Why are so many of the findings from studies not getting replicated, and does it mean that they're wrong? Well, in some cases, yes. And the reason behind that can be bad statistics. And I know statistics are boring, they're really, really boring, and the more they say statistics, the more that you want to stop listening.
But just stay with me, I promise things will get very, very interesting. Meanwhile, here's a fart noise. Okay, so why do we use statistics? You have to distinguish your effects.
Here's the drink theorist, Button Zuibach again. That you claim have happened from random occurrences. Hot potato, hot potato. Today, a study finds that people feel older after listening to hot potato by the wiggles, compared with say when they listened to Kulema by Mr Scruff.
Statistics are how we work out. How confident can we be that people felt older because they were listening to the wiggles, or that that finding was as due to something else, say random chance. Now statistics and science are calculated at particular levels of confidence. Which change depending on the discipline.
So for example, 99% is not sure enough in particle physics. In particle physics, the standard is 99.999. And in psychology and neuroscience, clinical trials, in fact, most sciences, the standard to use is a 95% level of confidence. So that means that there will be a 5% chance, or a 1 out of 20 chance, that the effect you find in a study is due to something other than what your claiming it is.
I mean, everybody's gone in for 5%. That's associate professor Alex Holcomb again. And there's a couple of reasons why the level of confidence is much higher with the Higgs boson than say other areas of science. And one is because you have so much more data when you're talking about particle physics.
That collision experiment is happening at millions of times a second, for month after month. So if you can't find what you're looking for in all of those trillions of protons getting smashed, it's probably not there. On the other hand, a pharmaceutical company can't test their new depression drug on every single person with depression. So even if their results aren't statistically all that significant, it still might mean that your drug could have helped some people who weren't involved in the trial.
But still, when you set your confidence level at 1 in 20, it means that you're accepting as a community that 1 in 20 individual findings will be wrong. On top of this. Yeah, so in some areas, unfortunately, it's very easy to tweak statistics. Right, and what Alex means by that is you can look at a particular data set that you've got, and you can analyze it one way and do the stats if you don't get statistical significance.
You can analyze it another way. And that's a bad thing because then I can publish my paper, not reporting that actually I tried all these other ways, and they didn't work. Alex says that statistics tweaking is particularly problematic in small studies, whether the effect you're looking for is also small. Like with that Wiggle study, listening to Hot Potato is not going to have a huge effect on how old you feel.
Oh, and that example, I didn't just pull that out of nowhere. A paper published in 2011 by Joseph Simmons at the University of Pennsylvania and his team. Went about demonstrating how quote, unacceptably easy it is to accumulate and report statistically significant evidence for a false hypothesis end quote. And indeed, they demonstrated this quite well.
They found that they could statistically show that people in their study did feel older after listening to Hot Potato. And in fact, using some very, very nasty statistics, they even showed that people were younger, were younger after listening to the Beatles when I'm 64 compared with when they listened to Colombo by Mr Scruff. Conclusion statistics are so boring, after all. Now I know they're still boring.
The real conclusion is that we have to be very careful when interpreting studies that find statistical significance, particularly when they have small sample sizes and small effects. Next up, Shoddy Study Design. Sometimes studies aren't as accurate as we'd like them to be because their initial design is set up in a, well, a little bit dodgy kind of way. So it makes it look like the scientists have found something very exciting.
These searches have been celebrated, but they haven't really. So for example, scientists might have selected people from the start of a clinical trial that they thought were more likely to respond to a particular drug, or perhaps they picked a kind of dodgy endpoint to their study. So if they were testing a fertility treatment, did they record pregnancies or live births? Or perhaps they even decided to stop a trial early, say as soon as they reached that magical marker of statistical significance.
Indeed, a 2011 survey found that 70% of behavioral scientists had admitted to stopping data collection at the halfway marker of a study. But I don't know if that result has been reproduced. It's not technically called misconduct. This is Professor Ivan Oransky at Retraction Watch again.
Because the protocol, the experiment, actually sort of passed muster in a number of ways. But in many ways it's more pernicious than that because you can't point to it and say it's raw, but it is absolutely warping the results that you end up seeing. To counteract these effects, there is now a push in many scientific disciplines who register studies in advance. So announcing what your endpoint will be, for example, before you begin a study so you can't change it midway.
Plus in some cases, this sort of shoddy study design can actually be spotted because scientists still have to write down their methodology. So once the paper is published, say it's snuck through peer review somehow. Once it's online, other groups can pounce on those crappy papers and criticize them. And Alex says this is actually becoming more common in the new online world of academia.
The good thing of science moving online was that it's been easier for different scientific camps to interact and sort of fight each other. And Professor Ivan Oransky agrees, saying that all this online criticism will... Really change the rules of the game a little bit, and also change what we think of a paper just because it's published. Doesn't mean that it's gospel in the final work.
So when it comes to science versus science, does it stack up? Well, the big and controversial findings. Climate change being largely human-induced, neutrinos not traveling faster than the speed of light. That stuff is really getting scrutinized, and the scientific process is working very quickly.
But for those less controversial studies, well, they may find themselves under much less scrutiny. Now where we see the results of papers not getting reproduced, whether it's because of the dodgy study design or bad statistics or whatever it is. That doesn't necessarily mean that science is broken. In fact, the fact that these papers are getting picked up and thrown out means that the process is working.
The scientific process is in action. But it also means that on a day-by-day basis, when we read reports in the news about a great new drug or a great new diet or a part of the brain discovered for porn addiction, even if it's a study, even if it's been published in a peer review journal, maybe just follow it up in a few years or a decade and see if the scientific process hasn't kicked it in the arse. Now I'm going to leave the final word here to Dr. Neil deGrasse Tyson.
I wonder whether without anybody going to the moon, you don't have the daily headline reminder of the great achievements that science makes. We all have smartphones now, but the smartphone is talking to GPS satellites orbiting the Earth. We can excel in on it and have an image that's test you're taking. And a built-in camera to take images that you can then check.
And this kind of ditched up on us on a level where most people who use their cell phones are not super-fried by how powerful it is. Yet it's all science. So I fear that people in modern times are taking science for granted. So that's science one and science one.
And didn't you just know it. So this brings us to the end of our first season of science versus next week. We're going to bring you some highlights of my extended interview with Dr. Neil deGrasse Tyson, because I didn't just talk to him about science.
I asked him about a whole lot of other stuff, and he's just so brilliant that I wanted to share it with you. And that interview was originally recorded for a Saturday paper. Plus, this is a very short goodbye because we're going to be back in October with a brand new season of science versus. And to get updates on the show, you can find me on Twitter.
I'm at Wendy's hook. Also, if you're there, you can flick me some ideas for next season's show. What would you like to be? Science vist.
This is science vist. What noise, fishman. So they can't hear it. And the next minute I'm looking at Never Seen Before Footage of a Hopping Mouse from an island in North Australia.
It helps around like a kangaroo. It keeps track of the moon cycles, and it digs its own boroughs. Oh, and while you're on iTunes grabbing off track, leave us a review. We'd love to get your feedback.
Thank you so much for listening to this first season. It has been wonderful. Thank you very, very much. I'm Wendy's hooker moon.
Back to you next time.