Skip to content
#UnblindingResearch Eighth Session: Outcome Measures, Confidence Intervals and p Values episode artwork

Oct 26, 2018 31 min

#UnblindingResearch Eighth Session: Outcome Measures, Confidence Intervals and p Values

from Take Aurally Hosted by Take Aurally

In this live recording of the eighth #UnblindingResearch session on 'Outcome Measures, Confidence Intervals and p values' from DREEAM through a made up beta blocker 'Superlol' we go through: - p values and significance - how mean values move and standard error of the mean - how outcome measures are made and revisiting QALYs Don't be mean and bear the pressure! The blog entry containing the slides, #TakeVisually and more info can be found at https://www.takeaurally.com/new-blog-1/2018/9/30/outcome-measures Subscribe to Take Aurally on SoundCloud and iTunes Remember you can find Take Aurally and NUH DREEAM on both Facebook and Twitter The amazing DREEAM Research Team can be found on both Facebook and Twitter Our next #UnblindingResearch session will be on Randomisation on 21st November 1000-1100 in DREEAM. Next week's episode will be on Cardiac History Taking

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

#UnblindingResearch Eighth Session: Outcome Measures, Confidence Intervals and p Values

0:00 31:32
of matches

Transcript Auto-generated, may contain errors

Thanks for downloading and welcome to Take Aurelie, the podcast from Dream Cruise Medical Centre at Nottingham. This episode is a live recording of the Unblinding Research Session on P-Values, Confidence Intervals and Outcome Measurements. As ever, any information is correct at the time of recording. Any guidelines mentioned are correct for Nottingham University Hospitals NHS Trust. Other trust guidelines may vary or views and opinions of the speaker's own.

Cool, so today we're going to talk about p-values, confidence intervals, and outcome measures. We've got some activities to hopefully explain some of these principles. Really what we're talking about is these numbers that you'll see. So if you ever read a paper, you'll see these random numbers. So you'll see this p followed by a number. So hopefully by the end of today you'll understand what that is and why we put it there. And then you'll also see sometimes a value is given, followed by, in brackets, a percentage, ci, or confidence interval, and then a range of values, which is what I've highlighted there for you.

So again, through another activity, I'll hopefully show you why they do that and what it means. Just so that when you ever encounter it again, you can go, okay, I understand what that means. So, and then we'll have a bit of a chat about outcome measures as well. So, we'll explain outcome measures, we'll explain what p-values are and why they are used, and then explain what confidence intervals are and why they are used as well. This is a nice friendly environment. Two stormtroopers looking into the sunset. This is nice and friendly. There is no such thing as a stupid question. If you want me to slow down, if you want me to go through something again, please just let me know and we will do that. So, first up, I want us to imagine that we are a drug company. We want to make lots of money, which is what drug companies are there for. And we have invented a brand new beta blocker that we are going to call SuperLol.

And this is going to be the best beta blocker that money can buy and hopefully make us a lot of money and make us rich. We want to know that Superlol works, don't we? We're going to market it as being better than anybody else's. We need to know that it works. Now there's a problem when it... What have I done with my coin? What did I do with it? There it is. So we want to know if A, will it work? B, does it not work? So that's what we want to show. Now the trouble is whenever we give a drug...

Things can happen to muddy the waters and muddy our results. The placebo-affected one, so we all know about that. There was recently a study that was released where people with chronic back pain were recruited into this study. They were told that they would either be given a placebo or a brand new wonder drug. They were all given a placebo, and they reported amazing results. So people who couldn't walk were walking. People who couldn't go to work were going to work.

Amazing. And even when they were told this is a placebo, people were like, can we carry on taking it? Sorry? I don't know if it's on TV. I was listening to it on the radio the other month and it was incredible. So it's incredible. It opened up an idea to them that they could do it. And just that alone had got them over this major hurdle. So it's quite incredible.

Other things as well can affect. So the way that we recruit patients, the way that we design our project can affect different things. Okay. And chance is a major thing as well. And to prove it, I'm going to test the psychic ability of this group. So I require a volunteer to come up here and toss a coin 10 times for me. Who would like to toss a coin 10 times? Okay. So this shows that with a simple binary choice of one of two things, two out of 10 times purely by chance, you get the right result. Okay. Chance is a major thing. Okay. So if I give...

drug to a patient, there is a chance that that drug will work, even though it will show positive results, even if it doesn't actually work. What it also shows is there's a number of different types of error. So you could have a positive that turns out to be negative, so you can have false positives. You can also have negatives that turn out to be positive, so you can have false negatives as well. So your result can be masked by the fact that you have these false positives and these false negatives. I.e. a false positive, your research shows that your drug works, when in actual fact it doesn't. Too many false positives, you would broadcast superlol is the greatest drug ever.

give me lots of money and you'd end up being sued because it doesn't actually work. Too many false negatives, you spend millions of pounds on research and you find that it doesn't work when actually it does and you kill off that research and you kill off a potential cure for high blood pressure. Does that make sense? So really there's three things going on. Through chance alone, so two out of ten times we got it right. I'm not going to call any of the psychic in this room, but two out of ten times we got it right. And also there are false positives and there are false negatives. The p-value that you see there in the research, p-value is all about false positives.

The gold standard of research is to document a p-value of less than 0.05. If your research has a p-value of that, that means that there is a less than 5% chance that the effect in that research was due to chance and chance alone. If you have a p-value of less than 0.05, that is what we call significant result. So when you hear scientists talking about significance, that is what they mean. It means that, yes, your drug works, your test works. There is a positive result there. Does that make sense? Now, as we did in last week's month's session, like Celia will hopefully remember,

We can set our p-value right at the beginning of our research, and that affects how many people we bring to our research, because we say, I want a p-value of this when we're setting up the research, and that decides how many participants we get to our research. We can also calculate a p-value based off a group of data as well when we're given it. So you can do it at the beginning of research, you can do it at the end of research. It's all complicated, I'm not going to put us through that, don't worry. But hopefully what that shows is if you see this, that means that that research is significant. You'll notice here, they've called a p-value of less than 0.01, so that's even more significant.

So, and you'll find that for the really big research, they'll want that, a p-value that's even less than 0.05. What basically means is there's a less than 5% chance that that was just due to random chance alone, i.e. that drug works. Does that make sense? So if you see research that's got a p-value above that, it is not significant. And this is one of those effects, this is the thing that comes to trouble when scientists are reporting data, because in everyday language, significance sounds like this massive thing, whereas actually, significance when it comes to research is a very tiny little thing that can make all the difference.

We also talked about, we saw some false negatives there. And this is where we come into what's called power. So you might hear about research being powered for something. The gold standard for power is something called 0.8. So we accept that false negatives will happen. The gold standard in research is that we accept 20% of negatives to be false, which seems like quite a lot. What a power of 0.8 means is that 4 out of every 5 negatives that that research finds is a true negative.

1 out of 5, a fifth, 20% will be a false negative, they accept that. A power less than 0.8 means that that study is not powered enough. So a power of 0.9 means it's even more powered, because there's even fewer false negatives. Do I need to go through that again? So yeah, so again this goes into when you're designing your research, but you design a power and it has a decimal like this, 0.8 means that 80% of any negatives that this study finds will be true negatives, 20% of any negatives this study finds could be false negatives, but we accept that because we know false negatives will always happen. There is no such thing ever, you are never ever going to have a perfect thing that will find only true positives and only true negatives. We accept that, okay, and this is the...

Gold standard, this is the consensus of research. So the power is the agreement, the positive note, the true note? Basically, yeah. So positives, you're into your significance and your p-value, your negatives, you're into power. And again, you do this right at the very beginning, and this helps decide how you do your study and the number of people you have to bring forward. And again, it can be very complex, and you need to get in high-level people in order to do it. Again, you could get a power that's more than this, so 0.9, and some studies, especially if there's a lot of money riding on it, will want a power of 0.9, i.e. we only accepted 10% false negatives. So if you're bringing forward the next cure for dementia, for example, there's a lot riding on it, they'll want that, they'll have a power of 0.9.

Only 10% of our negatives are false negatives. Yes, here. How are these numbers chosen as gold standards? I knew you were going to ask, and I don't know. I honestly don't know. But they are. And on his way out of here, Chris, having seen this, was not moaning. He was saying to me that actually p-values he thinks are worthless. And there is debate about it, like there's debate about everything. But this is the gold standard. This is the consensus. So this is what a p-value is all about. And if you hear about a study being powered, this is what it means.

Does that make sense? Good. And it's all about the type of research that you want to do. So if we wanted to show that superlol is better than, say, bisoprolol, that would affect our sample size compared to if we just wanted to find that superlol is not worse than bisoprolol. So whether you want to show something is better or just not worse, a non-inferior one. So this all goes into that.

So if you see that p-value, that's what that means. If you ever see that in research, that's what it means. It's what it's about. It's about significance. Does that make sense? That's written as a power to not behave. Yeah, it'll just say that. It'll just say that. So you'll see a p-value, that's about the significance. And you'll see this, this is about power. I don't think there's anything about power on this particular one. But they may talk about that in their design. This was powered for this. Cool. Right then. So...

Blood pressure. We're interested in blood pressure, aren't we, obviously? We've created Superlol. We want to reduce people's systolic blood pressure. How variable is systolic blood pressure? Do you have the same systolic blood pressure all the time? No. And we're hopefully going to demonstrate that now. So, two groups? Is that okay? We've got a stigmamometer. I've had to steal from the department. I will return it. We've got some stethoscopes.

Would you be able to chest, I just want you to start blood pressure, I don't need your diastolic blood pressures, but if you are able to take each other's systolic blood pressure, shout them out to me and we're going to write them up and we're going to move on to confidence intervals now. Okay, so, in checking our blood pressures, as you were giving them out to me, I was then calculating the mean for our group. Okay, are you all happy with how the mean is calculated? Just a simple average, you add it up and you divide by the number of people. Okay, so when one person shouted out the result, it was 142, that was 142 over 1, 142. So as you see, as then two people came, our mean was this, and then by the end, we've got this mean here. Okay, so what I'm hoping that this demonstrates is that as you take in more information, as you take in more means, your overall mean jumps around. Okay.

The more people, participants that you have coming in, that's going to jump around your mean blood pressure. And you're not going to just give your drug to the people and ask them to take one blood pressure and go off that, are you? You're going to be asking them to take Superlol and check their blood pressure every day. And as you know, that changes things. You can have a bit of an agitated day that day. They might have just gone for a run. They might have had their yoga and be super chilled that day. All these different things that can affect. So as a result, when we're writing up our data, we're never actually entirely sure what our mean is because it jumps around.

So that amount of movement is something we call the standard error of the mean, or SEM. Okay, don't worry about that. There is a way of calculating it, but don't worry about that. But it's just an idea that it jumps around. So that if we were writing up our results about superlol and saying this is the mean to stoic blood pressure of a patient who takes superlol, we are never actually certain that's that accurate number because of how much it jumps around. So what we have to do is give a range. We know it's within this amount because of how much it jumps around. That is our confidence interval.

So let's go a bit more into that. So say that these are some blood pressures of our patients that we've got, and we calculate their mean. So the mean of these patients, the mean systolic blood pressure is 108 over 4 for these patients. We want to then find out the confidence interval. Now, again, the gold standard for a confidence interval is 95%. You can have a lower one. You can have a higher one. But convention is usually a 95% confidence interval. And that means that you are 95% certain that the true value is within that range.

So the mean systolic blood pressure on superlol, we are 95% certain that it is within this range of data. And that is because, as we've demonstrated, the mean jumps around so much. And it jumps around even more if you're asking for multiple sets of data from one person. That's the same for heart rate, it would be the same for white cell count, it would be the same for D-dime, it would be the same for, you know, we're dealing with human beings, not machines, it's never going to be the same thing. There is a formula for it, but the mean, the confidence interval is the mean plus minus a value which is called the Z number times our standard error of the mean, which is how much it moves around.

The z-value, don't worry about that, that is a number that comes off a table. You look at this table and go, I want this confidence interval, it gives you a z-value. The z-value for a 95% confidence interval is 1.96. Again, don't ask me where that comes from. We would work out the standard error of the mean for this data. We would times that by 1.96. Our confidence interval is the mean plus and minus that value. If we say that 1.96 times the standard error of the mean for this data,

comes to 6.23 we know that our mean is about 108.4 our confidence interval is 108.4 our mean plus minus 6.23 so it's from that minus 6.283 to that plus 6.23 gives us that range so we are 95% certain that the true mean for these patients lies within this range that is our 95% confidence interval CI confidence interval everyone with me say if not Don't so much worry about the calculation, because not everybody's going to be doing that, but you will be reading papers and you'll see this, and it's just a point of knowing what that means. So if we said, so we would report this as going, our mean systolic blood pressure was 108.4, and then we'd have in brackets, 102 to 17 to 114.63, 95% confidence interval. That was how we would demonstrate that. That's how we would write it up in our paper about super long. Sorry, how do you calculate the SEM? Don't worry about that.

It's the, you need the standard deviation of your data, so don't worry so much about that. Does that make sense? Yeah, I don't want to bombard with two, yeah, standard deviation, everyone's favorite bit of maths. Okay, so we've talked about false negatives, false positives, we've talked about p-values and how that comes into it. We've shown how mean jumps around and why we therefore have to use a confidence interval. So it then finally comes into thinking about health outcomes and outcome measurements. Okay, there's a number of different considerations that come into us when we're thinking about an outcome measure.

It's the impact of the disease and the impact of the treatment, what the individual wants, and also what society wants as well. So if we think about those three things, what sort of things do you think we would want to measure then? If we're putting patients onto a treatment, say Superlol, we want to look at the health outcomes for Superlol. We're thinking about these things. What sort of things might we therefore look for as an outcome measurement?

yeah we could just look at sheer value yeah absolutely a sheer value here's a number we know it decreases by this we do know it decreases blood pressure by 50% versus something else excellent good yeah brilliant sheer money absolutely so society there's a budget in front of us this is our health budget what does it actually cost good yeah yeah impact on the person so what are the side effects it's great oh yeah it reduces my blood pressure but I can't get out of bed in the morning for about half an hour because my blood pressure is so low I get postural hypotension I've had to stop doing you know going running which I love because of my low blood pressure. Brilliant, yeah. Anything else? It's like how many times do you have to take it? Yeah, exactly. Is the patient going to take it? Yeah, absolutely. You know, how often do you have to take it? How do you take it? So, you know, allendronic acid that people take for osteoporosis, you have to stay standing up for 30 minutes because it's so caustic, there's a risk of heartburn. It's not nice to take. You talk to people about it and go, yeah, I don't always take it because of that. It's not nice.

What about warfarin and INR? If you're starting a patient on warfarin, you're committing them to weekly INR checks. So there's an impact there on the person, isn't it? Yeah, good. What's another measurement? So why do we worry so much about blood pressure? What does high blood pressure do to you? Risk of stroke. Yeah, exactly. There's another one, isn't it? So actually you could then look at stroke risk. So you could use a stroke risk score and see how much you've reduced people on the stroke risk score. So that's another outcome measurement, isn't it? Anything else?

Yeah. I'm thinking more like kidneys. Yeah, absolutely. So like fruizomate, you know, I've started you on a drug, that's great, but our number of AKIs has shot up because we've started everybody on this new one. Yeah, brilliant, good. So you see how it all sort of comes together and these sort of things. And, you know, you've got to take the individual in front of you as well. They're going, do you know what? I don't want to take anything. I'll take the risk. I don't care. I'll eat more fruit. Don't want it, you know. And that also comes into, again, we're not dealing with machines, we're dealing with human beings. So, you know, that comes into it as well. Good. One of the outcome measurements we talk a lot about is a quali. So I talked about this at the ACP study day. Have you heard this expression, a quali?

Guys who are nodding, what's a quali? Yeah, brilliant. So it's quality adjusted life years. So this is a concept that's used in health economics a lot. Basically, one year spent in perfect health is one quali. Your quali is essentially measured from zero, dead, to one, perfect health. And you can be somewhere along that on a spectrum.

So one year spent in perfect health is a QALY. One year spent in half perfect health is half a QALY. Quarter QALY, quarter perfect health. Does that make sense? You may ask, how do we measure health? Because that sounds very subjective, doesn't it?

So one of the models that's used and is used a lot is the EQ5D model. I've heard of that. Celia, we use this in our study, don't some of our studies? Yeah, so basically... Yeah, so the EQ5D model, this looks at a patient across five domains. So there is their mobility, their self-care, so washing, dressing, etc. Usual activities, so going out, shopping, walking, etc. Freedom from pain and freedom from depression, anxiety. And it's a subjective thing, so you ask the patient.

but it's also objective so you can fill in yourself as a clinician what you think and you can ask carers relatives etc tick full boxes in all of those for a year you've got one quality basically um it's not perfect is it and i think whenever you start talking about health economics it can become a bit uncomfortable but this is one of the main health outcome measurements that we use and this is what we use to decide a lot of what treatment gets funded and what treatment doesn't get funded basically the idea is here you see one patient living for 40 years in full health all the way they may have had an accident which happens suddenly they're dead Or a second patient who spent 80 years in half perfect health all the way before dying. Both of these patients have 40 qualities. So it's not about your length of time of life, which is actually what we often think about. Well, my relative lived to be this age, that's a good age. It's actually about the quality of that time. Again, it's a philosophical thing. It's a very objective thing. And when I talked about this in the ACP study day, there was a lot of debate about, well, I don't agree with that. I don't agree with that. It's a very personal thing. But again, this is what happens when you have a health care budget in front of you and you've got to fund certain things.

So basically, how does it work out? So say that we've got our Superlol and I'm presenting some data in front of you guys and I want your money. I want you to fund Superlol. Stop funding by Superlol. Fund me. Okay. So say with my Superlol, I can give your patients 79 years of perfect life. Without it, they get 66 years of perfect life. There's a benefit of 13 qualies if you give Superlol. So I can say that for so many millions of pounds, you get 13 qualies.

Say my competitor says, for the same amount of money, I can give you 20 qualies. My competitor gets the bid. Do you see what I mean? This is how it works. So this is why some things get funded, some don't. So, you know, with my treatment for X conditions, to say patients had a really nasty stroke, with my treatment, I can keep them alive for 10 years. But actually, when you look at it that whole time, they're highly dependent on care. Even though it's 10 years, you'll be looking at a very small quali. It doesn't get funded. So this is why.

And this is why your things that are for prevention, your statins, aspirin, all of that stuff, they love it because you get the qualies from it. You're bang for your buck. Does that all make sense? So more money spent in prevention than cure? Precisely. Far better to spend a little bit of money on your beta blocker, prevent the stroke, get those qualies, versus thrombolysis when you've had a stroke, and your qualies afterwards is very, very difficult, and there's, you know, care afterwards, gone from being independent to not independent. This is why things get approved and they don't get approved. Does that all make sense?

It's a bit uncomfortable. It is uncomfortable. I'm glad I don't have it. I'm glad I don't have it. And you can imagine what it's like in America where I go, well, you need a CT head and you can't afford it. I can do that. I'm very, very proud of our NHS. I can do that.

And I remember when I was an F2 in healthcare of the elderly, we had a bit of like an M&M, we had a grand round. It was kind of a philosophical question. One of the consultants says, I've currently got a patient who is not from the UK, who is relying on health insurance. They come in with this condition. These are the prices of all of the tests that you want to deliver. This is how much they can afford. What tests do you do? And nobody could agree. None of the consultants, none of us could agree. Under us, they'd have got everything.

Yes, they need an endoscopy. Yes, they need that. Yes, we should transfuse them. Yes, yes, yes. Probably got a tumour. I'd scan them. Yes, yes, yes. When you actually see the money, what it's against, like, nobody could agree. I'm so glad we don't have that. I'm very glad it's not my job to do this. Cool. Thank you very much, guys. Next month's session, we're talking about randomisation. Hopefully there'll be more sweets involved in that one.

I'll put this on the Take Already website. There's more information about how we calculate standard errors and a bit more information as well on there. And I've put up a BMJ link. There's an article, a really good BMJ article about health outcomes as well, which is quite interesting, which is a good read. I'll put a link up to that as well. I always try and end it on a bit of a pun. Don't be mean. Help relieve the pressure. Thank you very much, guys. Hopefully that demystified some of those things for you. Thank you, guys. That was the Unblinding Research episode on p-values, confidence intervals, and outcome measurements.

You can find the blog entry, including the tape visually for this episode, at TakeOrally.com. Remember to subscribe to Take Orally on both iTunes and SoundCloud. You can find Take Orally on both Facebook and Twitter. And for more information about research and education opportunities within emergency medicine and acute medicine and major trauma, you can check out a new mainstream on both Facebook and Twitter.

Frequently asked questions

How long is this episode of Take Aurally?

This episode is 31 minutes long.

When was this Take Aurally episode published?

This episode was published on October 26, 2018.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Take Aurally episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.