Naturally speaking Well, here to talk about this more here. Also with us is Paul Johnson who's a research fellow here in our department Paul your background is in genetics correct yes And genetics published genetics and statistics so I can produce the values of the laws your sort of the turnaround statistician Yes, right people coming to you asking about there's a baby crying in the background during a stock It's statistics Becoming scared, but yeah, so a lot of people come to you and ask you about statistical issues Yes, and then finally with this year from University of Roehampton It was how see tell us a little bit about your work that you normally do studying Yeah, yeah, so I'm only so yeah So normally vertebrates always and actually sometimes sea birds are not really a seabirds suspicious I know a little bit about them not as much as James no doubt But I will be going off to the Caribbean in June to be studying them some more just like that That's nice. Yeah, I do So normally that's what I'm a whole lot of biologists taking interest in genetics, right? You're not a statistician and yet you've recently had a paper in nature methods where you open up a can of words of controversy surrounding Sort of a pillar in science or it has become sort of a pillar in how science is done and how conclusions reform in science And that's this use of p values to infer conclusions or meanings or meaningfulness and data Do you want to I guess we should start off by talking a little bit about what p values actually are and how they started to become used in science the way They're used today.
Okay, so I'll try to talk to that So I guess I kind of lifted the can on some worms Which was very slightly open and some special statisticians I make a few other like people seem to be aware of the can but not many people at all very well Hopefully the paper which actually says nothing that's entirely new. Hopefully it says a few things to say clearly That's the point pulls a few ideas together which some people may not be aware of So yeah, so it's based on this issue of p values Which I guess is one is everyone who's done these steps will have heard of p values And at least heard of them for you to them and they seem to be the basis for most Statistical analysis in most areas of science is Lisa in biology They're based around the idea of using probabilities to infer how likely it is that there's really interesting pattern interesting signal in your data Because when a scientist collects data or they can then collect a limited amounts of data They can't measure everything that's out there that's interesting to measure a sample of those days So they'd like to know about a pattern in general the patterns in general in the world They can't actually know that they're written because they can sample some of their data Hmm, so say okay Well, how well can I infer how confident I can be that my sample is actually representative off of the patent or generating the world And they use for quintess statistics based on probability that comes up to conclusion about how confident they can be any pattern They find in their sample so they get difference between two groups for example How likely that is to actually also exist in a broader sense So when we kind of you start doing maybe undergraduate maybe when you're doing a little math You'll do some kind of simple experiment where you go out and sample a couple of groups say heights of people And you can compare those two groups say men and women and you would use a p-value in some kind of test to decide whether or not those two groups Are the heights of different? Yes? Yeah, it's what's all perfect as well.
Right, so and that's sort of how they're used today As I mentioned in science a lot of times it comes down to just looking at this p-value to tell you whether or not There is actually a difference between yeah, the heights of men and women or the food preferences of one type of seabird and another and It often comes down to using sort of threshold p-value for telling you whether or not to consider whether or not there's a difference actually there There's quite deep history to this But maybe from around the 1930s or so there was some of these popularizing off of the p-value and famously one official was was the lead on that without the only person and there are a couple of systems set up by different people for kind of interpreting a p-value and over the years They kind of become blended together and so what people tend to do who want to use p-values to interpret their data and most people use p-values Including myself and optically mathematically strong like a nice simple system saying, you know, if the p-values high I don't have any any evidence for up for a difference So it's the p values low I've got good evidence for difference and that's kind of been simplified down further to a significant result It's good evidence for there's a difference based on my sample. I'm non significant result There's not going to be this difference based on my son. There's very strict cutoffs like like I'm saying like why if your p-value is .05 or it's below .05 and that's when you say oh We found something interesting Yeah, that's the threshold So there's a low than zero five It wasn't say if I found something that is high up people tend to say I hadn't found something That would be that the crudeest and probably the most common way of interpreting p-value And the crazy thing is people use that even if p is .04 or .06 yes So this is where we're starting getting into why these P values can be so controversial. And so I guess this is where your paper Lewis comes along.
Yeah, so one issue as James points out straight away is, okay, well, what's the right cut off to how, what's that right point to say, what's the correct sort of P value to say, okay, to lower than that we've got a sniffer result, it's higher than that, we've got a nonsense unit, it's an arbitrary choice of 0.05. And then beyond that, whatever your choice is, then it seems a bit iffy to say, well, okay, it's just a teeny bit below that. Yeah, we've got a result, a teeny bit above that, we haven't got a sniffer result, that's a problem. That issue becomes exacerbated when, and this is kind of what the papers show in.
When we then realize that the P value is what I call very fickle, so it's, it'll spit around below the place around the value here. So if you run an experiment, okay, you measure the height of the minimum and you get a P value of 0.03. So you get a, there's no difference in the height between minimum. If you run that analysis again, so you take a new sample of minimum and same sample size from the same populations, you run that as again, your P value could be more or less anything.
Whereas I think people typically, at least in place it is, you know, if they've got a P value of 0.03, first time around, they're going to get something somewhat similar to that next time. If they've got a P value of 0.8, they might get something like 0.8. If they've got a P value of 0.01, they get something similar to 0. The reality is they could get almost anything, you know, from the range of 0 through to 1.
And so simply put, that makes the measure almost untenable because from one measurement to the mixture you get different values. And so how do you go about showing this in your nature method? Right. Well, I just ran some basic Excel models to start with, where I just kept creating random data based on some populations, which I knew the parameters.
And it's someone who's on the paper, Doug Corinnevere, who was actually a statistician, I did this in property, they're very much better than me, producing beautiful grass, which are in the article. So that's great. But basically, it's based on model data. So it's just two populations.
We want to see if there's difference between them. We know the background beyond populations, which is keep running iterations of the experiment where we sample from those two populations and then run a t test to see if there's a difference. And the P value over and over again is really different. And it's not because these populations are very similar.
You know, the standard deviation between them, the absolute effect size of the absolute difference between these two populations is common to what you find in standard experiments. And yet the P value is very, very fitful. And it's like going in your car, you get out to 100 miles an hour the next time, you're going to get it to 50 miles an hour, the next time you get it to 150 miles an hour. You don't really know how fast you're going to go next time you get in the car.
So you guys were really only able to detect this because you basically simulated two populations. And then you're able to repeatedly sample those populations over and over and over again. So the only thing is that most of the time, most of us will run an experiment once. Yeah, exactly.
And then we move on. It's not always the case, but it's too big in the case. Here, what we're able to do is take almost that overview. We're looking down, if you're like, on this situation where we can run the experiment many, many times, thousands of times and see what happened if that was a reality.
And as I say, you just kept being back in different ways. So then if you're, you know, you're doing your 30 dissertation or PhD thesis or whatever, it is a report for the council. You do that study once. It's almost all bets are off.
What people are going to get to you're going to report, you're going to be able to report some exciting result, or you're going to have to say there's nothing there. It's really just down to, it's not far of down to chance in most cases, it's about a while. Right. Which is obviously quite controversial.
It's not easy to touch it. It's backed up by the solution to the argument. And you can't do the argument with what you're using. You're sitting studying studies as a real estate, real estate sample science.
There's three differences between the populations. These are the kind of studies where you're not so large that you're going to get really, really strong against from your quite likely to get, if you do get a significant results, or if you don't have less than 0.5, then it's quite likely to be 0.4, 0.3. So for studies of this sort of science, this is what it's about to be. If you do this again, then you're, then you're, you can be anywhere from 0 to 1.
That's a scary thought. And it also kind of reminds you of how there's so many logistical constraints when you're doing experiments and science, you can often only do an experiment once when another one of the tenets of science is meant to be that, you know, something should be reproducible and there should be a consensus that's built around reproducibility in different labs doing the same experiment. But oftentimes that doesn't happen. Well, no, it's going to be driven, aren't we, by the systems, if there are systems, you know, like REF or what these fancy journals in particular want, or maybe what, you know, in some universities, you know, the politics there, because they've been driven to gain money and, and jumping up.
We're scientists that push towards not repeating stuff, but actually taking, going from novelty, yeah, no, it's where that big steps forward, right? You're taking big steps based on something with a Riccati P value. You can take a few more steps forward and then look back on the whole thing as fallen below your house cards. You're not necessarily the joints upon which shoulders your standing may have very dodgy knees.
Right, right. That's a good way to put it. Because there's always going to be a difference, right, when you take any two groups of any of minimum heights. So it seems to the P value that the only way I should try to look at it again, I'm not about to say I get that limited here, but there seems to be a high gearing between variation and sample that you get and what happens to the P value.
In other words, a bit of a change in the sample that you pull out from the population each time you run the experiment, a little bit of change there can have a big change on the P value. Right. So because when you come back to experiment, sorry, you're re-solving getting a new sample, it's very likely that the P value at the end of the process is going to change a lot. Yeah, right.
Right. But one thing that you may be going to start talking about is the role of the sample size in plainness. So one of the things that you did in this study was you just like ranking up the sample size. And also you can never wholly, very unlikely that any study will hold you sample in the type of population.
So, but there's a chance that you can increase the numbers of samples that you take from that population and you would hope that when you start to get to much larger sample sizes that you'll have fewer or much better chance of getting at a significant P value, if there's an effect for it. Yes, and that's absolutely right. We do show that. So it's intuitive and it comes out in the mass that as you increase sample size, you start to hone in a bit better on any difference that there is between the population because your sample is bigger and bigger because you need a better and better description of the population.
And so, you're doing that, you're increasing the power of your study. So the power of your study by definition is the probability that if there is an effect, if there is a difference between the two groups, for example, that you will pick up in your analysis as a significant result, you will get a significant result when you compare your sample. And so, you increase the sample size, you'll get that. And also, in fact, if you increase the sample size, the fickle is a P decreases.
So, if you were to run that study lots of times, P value becomes less fickle. Most people don't run the study lots of times, but if you can start to be a bit more confident about the P value that you've got that if someone came along and reproduced your experiment with the same sample size, it gets a result. However, the kick up is that by the time your power is high enough, there's your sample size is big enough that your power is high enough that your P value starts to stay by. You don't need to run and need statistical analysis anyway because if you just plot the data, all the data is like, no, it's one bar, so we can get this, we can disagree.
One bar is up here in the data points, all the other bars down here, you look at it. And as Jeff comes in, it smacks you between the eyes, you really don't need to have P value of 0.00, one to 10, there's a difference. But it becomes any fun in that situation. But it's also an issue in that this is slightly different from power, but it's power involves a fact size.
But also the more and more samples you take, eventually you're going to detect a difference there statistically, with respect to the P values and significance. So I might not mean anything biologically or in terms of whatever it is you're actually interested in. We were talking about this earlier, Lewis, and you give the examples, you can stand out by the motorway and count blue and green cars. Do you want to give that analogy?
Yeah, so if you want to know if blue cars go fast in green cars, I can't teach you there is a difference in the speed that they go even though there's no reason why they should be a pro because of random chance, even if they've got the same engines in them and have a blue and green cars, they probably basically do. There's going to be some degree of difference, there's going to be some difference in the meat between the average speed that the blue and green cars are going. They're just what we use, even if it's to a tiny, tiny decimal place of miles per hour, there's going to be a difference. So if you measure enough cars, if you measure enough cars, you will come out with a statistically significant difference to it.
If your sample size is big enough, you will do that. You could argue therefore that when you get a sniff from a result, what you're really showing is your sample size is big enough to show the difference between the two groups, which is inevitably there. Right. That's big.
So there is an issue there may be, I mean, it's not quite what stays in the paper, but you could argue that typically if you pump up the sample size in us so that your P values stabilize, you're all way away. You're also going to find a different result regardless of whether it's interesting. Then you really want to go to the effect size, the effect size of interest because it's the difference in the speed of the quality of the miles per hour who cares. You're sort of throwing your brow up, I'll do something for this disagreement here.
Well, I was talking about the what kind of attacks on P values that are so far. So we have the attack on the arbitrary threshold of, you've got stock elements of your P numbers, you've got 0.5 and 0.05, which is silly. They have the unreliability of the the problem is that people invest too much certainty in a P value that's of say 0.03, they see that as strong. So really it's quite weak evidence.
And the proof that we have in this can be seen by by by by by by uses. It's not highly high the people. So it's really scientific to recalibrate their evidence on which it is to not stick a P value that's slightly less than 0.05 as strong evidence. Right.
But it's surprising. So this is a social dispute to go through the entire site, to do a fine-grant place of significance for my surprise, maybe the paper, or you know, you're looking for the strength of the evidence. Yes, excellent point. Yeah, that would be an interesting exercise for sure.
There was another recent paper that came out sort of getting back to this idea about threshold P values and that determining sort of the beyond all the conclusions that are made in a given study. There's another paper that came out in PLOS biology recently where they discussed this idea of P hacking where researchers sort of unconsciously or consciously will sort of manipulate their studies or go back and collect more data or do various things so that they can try and get the P value below this threshold of 0.05. Yeah, I mean, is this tied in directly with this issue of using this threshold cutoff? Yeah, I think it's not really an issue with P.05.
It's not an issue with something that's special when very often you don't need a threshold. I mean, this really sort of was saying that this, when you use a P value as some kind of measure of the strength of evidence of the pattern there, what you're doing is you're assuming that you're proposing a null hypothesis. So there's no difference at all. And then you're trying to engage the evidence against the hypothesis that there's no difference.
And as we say, it's very difficult to ask. It's extremely unlikely that this null hypothesis is true. So there really isn't any point in testing the policies that have no chance of being true. And that situation, what you want to do is as loosely, measure the effects on some measure of the size of the difference with some idea of uncertainty of position about that, which would be a confidence interval.
Right. We'll come back around to that a bit later. Put a pin in that. We'll talk about alternatives to P values a bit later.
But I guess is there anything to say in defense of the P value? I mean, why are they? If it was used as, sorry, I don't know if it's used as Fisher who was mentioned earlier, if it was used as, I haven't read his book yet, I never will. But he's used in the way he advocates and sort of almost the origin, if you like the P value, I think it's quite good actually that comes back to what Paul said, which is, you know, if there's a different word, you know, if there's a value of around 0.05 fish, then the data's getting interesting.
Maybe there's something going on. It's a bit surprising, you know, take another look, collect more data, maybe explore that avenue. And if it's much less than that, then you've got a bit more evidence. But again, play it cautiously.
And this is what, this is what Fisher said. I don't think what I understand that Fisher was probably a genius. But of course, he didn't have the computer, he didn't have a computer at all in his day. And I don't think he'd probably aware of how fickle P can be.
But, you know, the way he was very cautious about it, instead, it could be used as a pointer. And if that was done today, I think it would be okay. So a pointer for replicating that study? So rather than the way that we're more or less indoctrinated to say, right, if you're peeping just 0.05 bang, you're in, you've got that.
That's yours. So you can't lose that. You know, you go home with that, whatever happens next, and you build on that to the next step, whereas Fisher's advocating, there's a suggestion there could be something there. Well, let's go and look, see if we can develop more, or it may well disappear in a puff of smoke because the new reality is there's nothing really.
So this attitude, as you mentioned, going back over the science, looking again, building a fetish, looking at different angles and developing it all, which unfortunately, you're probably not going to get no, it'll be this I'll see something to do. But could you simply, I mean, is it something as simple as rephrasing how you describe your results in a paper? So rather than saying, we tested this hypothesis, you say we examined the chance of this hypothesis, and this is a p-value that suggests there may be something going on, rather than saying it's black and white, you do say there are however many shades of grey you want. Yeah, I think there's too much of the people often writing black and white one because they want to give a stronger, clearer argument because that makes the paper seem better.
And also there's a way of writing concisely, you know, you're on a waffle and you've got limited space, but absolutely, we should be using more caveats, more qualifiers, although, you know, there may be some English stylists who don't like that, but we should be writing them all cautious way. And I think that's probably one of the reasons why p-values become so popular is because they are a way of sort of distilling down very complex experiments and datasets into a simple sort of take home message that you can put in the title of the paper or the abstract. In fact, that takes quite a lot of skill to do well, even basic stats takes a lot of skill. The reality is I think that there's a whole layer of skill and experience needed on top of what we already think is needed to choose the right test and get it into SBS or I'll correct it, which is based on really carefully assessing our data by viewing it, by running statistical tests, by considering it within the context of the science we're doing, and then writing about that accurately, concisely and eloquently, so that we convey a realistic level of meaning about the data to which we can slide, you know.
And at the moment, people don't tend to do that. They may get the right test. They may get these experiments out and they might even get the right test, but I don't think people necessarily are very good at interpreting astral data, is it weaning that into a discussion of the council where it's still on or on matters of the week. So we're sort of touching on a little bit earlier when you mentioned funders and sort of approaches to these things.
There's sort of an ingrained culture in science and using P-values and hypothesis testing in this way. I mean, how do we go about changing this culture then? If there are these issues with P-values, what role do journals play or what will the funders play? It's quite obviously a discussion that's going to start off by the journal basic and depletion of the social sciences.
And they're going to ban P-values from there, they're journal. Right. That is one quite extreme approach. Most people would say that that's an early extreme approach.
I don't think the funders have given up on this with the representation of the audience. I mean, they're used too much and often they're used badly. Fewer D-values better turn into the answer more nuanced instruction that journal could give to authors and reviewers. Is there a problem there with throwing out one dogma and replacing it with another?
I mean, they said they're going to ban P-values. Is it just the case now that every paper is going to use AIC or something instead? It's all right. I think they go to confidence and falls.
Yes. The advantage of the importance of well-in-fact. Does it happen? Does it happen?
I think it also may be another story. But there is a difference. I think, yes, I mean, this paper briefly touches on other possibilities, not because the paper is about other possibilities because it seems a bit rude to say, this is very good. Don't use this.
And then what else might you use? And there are lots of other things that AIC and Bayesian statistics. The option that this paper does talk a little bit about, which is perhaps the easiest to use. That doesn't necessarily make it the best.
It's simply effect size combined with... Pals mentioning earlier. Yeah. And that's very important.
So it doesn't address the issue of whether there's one dot ball will be replaced with another. But one thing I would advocate is actually moving away from hypothesis testing per se, regardless of whether that involves P-values. Because I think it limits. I don't think it actually reflects what most people want to study most of the time.
There are examples of where they do, perhaps in Paul's area more. But for example, if you want to know the quickest way to get to work, right, and you could take the car, you could take the train. Okay. I don't know which way is quicker.
It depends on the traffic, it depends on the time of year and so. So what I'm doing every day, I'm going to alternate between taking a car and train to whether a time how long it takes you to get might be worth. What isn't that interesting? What isn't that valuable is to know if there's a significant difference.
That's not that useful. Because there could be a really tiny difference. It could be, for example, after 10 years of measuring how it's taking a different time, is, you know, it's actually 37 minutes, exactly by car on average, and 37.1 minutes by train and temperature. And you probably get sufficient results with that.
But that's not going to be too useful here because that's kind of 10 to the minute 6 seconds difference. If you ask yourself, what is the effect size, which sounds fancy but it is just what is the difference between the mean times of these two transport, even comes out at 10 to the 10th of a minute, then you probably think, actually, doesn't matter. Time lies, neither one, better than me. So I'm going to think about other things that might help me decide whether I'm going to take a car on a train, like, do I have a work on the way to work?
How expensive is it? You know, do I have a car on a bunch of times to get into town? Stuff like this? So it gives you a context within which to assess that effect size.
Again, you could have a very, very small difference between the two, but you'll find a significant difference if you run enough information, if you run enough data on the times. So I would say that that generalizes to the majority, probably not all but to the majority of scientific situations, but really the question we want to ask is, how big is the difference? And that might sound presumptuous. I'm not sure if it's seen as different.
There's always difference. There's always a measure enough to act, enough with your laser or whatever it is. There's always a difference. The question is, you're going to suddenly have to get that difference between people.
You're also going to say, the car train examples are an easy one, right? Often when biology is not so easy to know what the context is to make a decision on this effect size being enough. But that's, we need to be asking a really certain question. Is there, how big is the difference in the time it takes to get to work between car and train?
Not high policies question. Is there a difference in the time it takes to get to work versus car versus train? So you're completely reframing the question basically. Yeah, and we're all, most people are indoctrinated into the P value.
And it's quite hard to come out of this. But if we start asking the research question, how big is the difference, not is there a difference? So do you envisage that students, like being taught differently now as more of these papers start to come out sort of pointing out the issues with P values? Because it is so ingrained that the people who are going to be teaching them will have run off in a P value culture.
Or do you have, any of you noticed that maybe you're shifting towards different teaching strategy or you see that happening in your department? I don't know what's right, what was on in not to make teaching, but I guess that the movement against the value is quite recent. People who teach statistics are generally quite slow to change. So I would be surprised if it would be any major change.
But I mean, there are fields like in ecology where there has a movement that we have to use towards other methods of making inferences such as the medical AIC. I was wondering if I could talk about what we could call the COT of the AIC, which is basically the point that we've all moved into in different fields of ecology. People are using this other measure of actually those using the same way as you know. You can see the same as the COT, the same as the other, they haven't changed anything.
They just sort of swap one, one flawed method of inference that you can use badly for another. Right. So there's one thing that you use for with something like AIC is that you can rank different models. So you've found you're looking at some kind of response.
You could compare the roles of different variables and try to explain response. And you could try and rank a series of different models with different numbers of variables. And that's not kind of something you could do in the same way with p-values, but is it a more useful approach than having all of them in and taking them all out and testing? Can you get any sort of effect size that way though?
If you're just ranking models? Not for me. Yeah. You don't really get an effect size measure.
You don't get an effect size measure. I think back to the stock. You know, you did a short slide showing that you had to rank variables using p-values. You can add up to the same thing with AIC.
But I think that there are things you can do with AIC. I think it's fundamentally less arbitrary than the other things you can do with AIC. So you can, you know, with the terms options, you can see this model is about 8% of the best one is 50% of the best one. And that's 5%.
And you don't have to reject any of them. You can just keep them there and say, okay, so that's that like, like, you have to say, that's not the ordering of the idea. Right. But then to build them, that's quite nice of you.
You have several models describing data and you can put in the order of how likely they are to be the best model given the different models you've got, the data you've collected. And that's good. But then to me, that cries out for needing, well, okay, let's take this. Maybe you've got one more, for example, with his hands and headshots.
Sorry about the rest. That maybe, okay. But how good is that model? So it may have finished first in the race.
But was it fast and run or what are they still really, really slow running? Because if they're explaining a very small amount of variance, the data's at the effect size is very small. Then you make the best, but it may still be pretty rubbish. And so it's nice.
I think AIC is a nice approach. But for me, at least a slight extent, the argument would still come back to, okay, well, what's the effect size? The effects size could be the difference between two groups. The effects size could be how strong the correlation is basically, maybe for these models, it tends to be more how strong the correlation is.
But it seems like strong, isn't it? Sorry. Because maybe it seems like relying too heavily on anyone's technique is probably not a wise thing to do. But having all these things in the toolbox and being used appropriately in different combinations, in different contexts.
I think that's kind of true. I'd say I think it's slightly pushy for interest, I'd say that again, I think we need to be voting when we can on how big the effect size is. Otherwise you get, you can use AIC and they've got their value and use both of them. And you can get focused on a winner and a loser.
You don't know how strong the winner is. And if it's a very weak winner, you know, if someone wins the running race at a local sports day, you're not necessarily picking them for the Olympic team, right? I mean, because they're going to be a long way behind doing the same goal, whatever it is. So you want to see what that rate is.
I think people should again be asking, how strong is the effect? And actually, some people doing complicated models, it's not easy to pull those sorts of data out. Okay. So, you know, I don't really have a stat sort of police about this.
But maybe people should ask for something people should they be doing models that are complicating clever if they can't actually get a reasonable answer now. I'm not sure. I mean, that's a rhetorical question. I think so.
It seems like AICs can be sort of valuable for honing in on what model you should be considering in any of the individual. Maybe even if none of the models, if they're all very weak, you say, okay. But I'm not saying I'm just saying fairly sensitively, I'm not. But it's really, because we have these three models, as we said, some inconceivable alternative model that's much better, they have a test of it.
But then you need some kind of effect size measure. There are things like R-squayers, which, which is a number of those from zero to one, and one means the model's perfect. So, you can also use these kinds of things to get some sort of objective assessment of how good your model is. And you can combine that with B-value or EIC or whichever approach is appropriate to your analysis.
I think we should probably consider wrapping up here. There's a few less punches thrown there than I hope for. So, it's jeez. Anyway, thanks a lot, guys.
What we'll do is we'll post a link to Lewis's paper and other papers that we've touched on and a few other resources on our blog site. We're in the podcast as post it. And thanks a lot, guys. Thanks again for joining us here today.
Thanks to you listeners. Bye. Bye.