Skip to content
#UnblindingResearch Session Seven: Sampling episode artwork

Sep 21, 2018 24 min

#UnblindingResearch Session Seven: Sampling

from Take Aurally Hosted by Take Aurally

In this live recording of the seventh #UnblindingResearch session on 'Sampling' from DREEAM through the medium of Gummy Bears we go through: - the principles of sampling and why we do it - significance and power and how we quantify them - how sample size is calculated Hopefully this session will help you to bear sampling! The blog entry containing the slides, #TakeVisually and more info can be found at https://www.takeaurally.com/take-visually/2018/9/5/unblindingresearch-session-seven-sampling Subscribe to Take Aurally on SoundCloud and iTunes Remember you can find Take Aurally and NUH DREEAM on both Facebook and Twitter The amazing DREEAM Research Team can be found on both Facebook and Twitter Our next #UnblindingResearch session will be on Outcome Measures, Confidence Intervals and p values on 17th October 1000-1100 in DREEAM. Next week's episode will be our second Redthread special!

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

#UnblindingResearch Session Seven: Sampling

0:00 24:56
of matches

Transcript Auto-generated, may contain errors

Thanks for downloading and welcome to Take Orally, the podcast from Dream Queen's Medical Centre, Nottingham. This episode is a live recording of the Unblinding Research Session on Sampling. As ever, any and all information is correct at the time of recording. Any guidelines mentioned are correct from Nottingham University Hospitals NHS Trust. Other trust guidelines may vary. All views and opinions are the speaker's own. Right, hello guys. Hello. Hello. Everybody got some gummy bears? Remember to keep passing them along. Yes. Welcome to this session on sampling. We're going to have a look really at sample size and what determines a sample size.

and we'll have a little bit of a chat about significance and power as well and what the numbers mean and why maybe they aren't always as correct as they should be. Megan, looking forward to that bit. Cool. So not very many objectives again for this session. So be able to perform sample size and power calculations. I'm not going to ask you to do that, but I'm going to ask you to be aware of them and have some ideas about where you can go to do it. Explain the difference between incidence and prevalence and describe generalizability. Does that sound okay? Awesome.

It's a nice friendly environment as ever with our two stormtroopers. You get very used to this slide. So yeah, any questions, if you want to make a point, please put up your hand. Cool. Okay. So why do we sample? Okay. Why do we need to make sure we have the right sample size? So it's not a fluke. So what's the trouble if our sample size is too small?

You're all talking, yeah. What was that Mark? Might happen by Charles. Might happen by Charles, what are you saying, Celie? Maybe criminalise it. Brilliant, yeah. We'll have to represent the whole population. Yeah, brilliant, absolutely. And if you think about these gummy bears here, those gummy bears that you've been eating, and you think about the human species, why do you think I've chosen gummy bears as a metaphor for human beings? What do they show that human beings also show? Variation. Well done, Louise. Variation.

yes variation okay so variation yeah so in what ways so ethnicity being one of them in what ways other than ethnicity do human beings vary Sex, age. Gender, age. Sugar content. Sorry? Sugar content, yes. Are some of them DKA or not? Yeah, absolutely sugar content, yeah. Size, brilliant, yeah. Their geographic area. Yeah, where do they live, what country they're in. Demographics. Demographics, brilliant. Comorbidities. Comorbidities, brilliant, yeah. And then within comorbidities, there's things that you were born with and then things that you've done to yourself and maybe things that happened to you that weren't completely out of your hands yet. Anything else? How else do human beings vary? Thinking about health economics.

Things that aren't really biological, but are cultural. Lifestyle, brilliant, yeah. Poverty, brilliant, yeah, absolutely. So the north-south divide, the divides that we have within cities, so you've got, you know, areas that are more or less deprived within a city, okay, brilliant, yeah. And poverty and money, access to healthcare, et cetera, et cetera. Lots of variation, so we've got to make sure that our sample is going to be representative of our population. What could be the trouble if our sample size is too big?

Generalizable because it could be like everything is a contributing factor. Yeah, so you might have too many confounding variables if your size is too big. Yeah, brilliant. Yeah. Anything else? Processing would be a nightmare. Processing would be a nightmare. Imagine if you sent out a million questionnaires and you want a million questionnaires back. Shit, time alone, resources. You guys, you're not doing this for free. You all paid money as well. So somebody coming in to do that, that's time, it's money. Good. You get a high attrition rate. Very high attrition rate, yeah. So you might get lost to follow up if you run the risk. Brilliant, yeah. And what else from the patient's perspective?

Say I wanted to do a study where I'm going to do full-body CTs on a million people. Time. Time. Money. Money. And? Risk of cancer. Yeah, is there a risk of harm? So sometimes if you're doing something that might be harmful, might not be nice, you know, say it's an invasive procedure you want to do, if you're doing that to too many people, it's not nice, is it? Unnecessarily. So the risk of your sample size being too big and too small. So let's have a chat now about incidence and prevalence. What is the difference between the incidence of a disease and the prevalence of a disease?

Because this is one of those examples of where the media and politicians can manipulate data quite easily by choosing which number you want to choose. So it's incidence saying like one in a hundred people develop diabetes. The prevalence saying in Nottingham a certain area is more at risk. There's more diabetes in that area. Yes, good. So say you've got a condition, we'll call it red gummy bear fever. And this is a condition that turns these clear gummy bears red. Incidence is the number of new cases that you get in a certain time. It's usually a year because we're human beings, we live in years.

So say that for a thousand clear gummy bears, 30 each year turn red. We would say that there is an incidence of 30 per a thousand clear gummy bears. It's usually expressed as a number of the population, hundreds, a thousand, maybe even more. Prevalence is the total number at any one point in the whole population that has that disease. That includes people who've had it within the last year, but also people who had it going into the year as well. So you could have a population of a thousand clear gummy bears, 300 of them.

I've got red gummy bear fever. 300 of them are red. 300 over 1,000, you've got 30% prevalence. Statistics can be manipulated. Ah, this disease has a 30% prevalence, or whatever it is, versus, ah, but it's only 30 per 1,000 every year. So this is how statistics can get manipulated. Does that make sense? What does generalizability mean if we say our study is generalizable?

Yeah, brilliant, yeah. So generalizability is external validity. Okay, so all very well and good me going to London to present a study on Nottingham, but does my findings from Nottingham mean that somebody who's from London can take it to their population? Okay. There's also internal validity. Do you know what that is? The internal validity of a study. Is that how it was? Kind of. Everyone in the study did it in the same way. Kind of, yeah. So essentially it's about saying how much does your study actually represent what you were studying. Okay.

So there can be a difference between the two, though, that can be quite difficult. So if we do a study here on traumatic injuries, we could do a really good high internal validity study. But we are a major trauma centre that has a 24-7 consultant-led A&E. Does that mean that we're necessarily as generalisable? So actually what you find, the more high internal validity a study is, the less generalisable it is. And vice versa.

The best internal validity comes with randomised controls. The best external validity come with very broad inclusion criteria. So those studies that have lots of exclusions tend to be the least generalisable. Does that make sense? Marvellous. Right then. So, say you're wanting to work out our sample size, just thinking broad concepts. What sort of things are we going to want to take into account? What do you think when we're calculating a sample size? Of that population that exists in total. Brilliant, yeah, our total population, very much so.

You could argue if your population was only 100 people, is it even worse sampling? That could be quite relevant. But yeah, we talked about it earlier. What type of study we're doing? Brilliant, yeah. Significance is one, yeah. What significance do you want? Yeah. And what goes with significance as well? So significance and power are very, very big considerations when we're working out our sample size. And we have to determine what significance we want and what power we want right at the very beginning of our project. This isn't something you do later on. It's right at the very beginning. Okay.

Significance is to do with type 1 error. Power is to do with type 2 error. So you may have heard of these. No, that's okay, don't worry. So, Megan, you nodded. What is type 1 error? You've heard of them, that's fine. Simon? Don't remember, that's fine. This is why we have these sessions. That's why we have these sessions. Significance is to do with type 1 error. Type 1 error is to do with your false positives. Okay.

So type 1 error, say you've developed a treatment for a disease, and type 1 error would be you falsely finding that your drug treats that disease when it doesn't, i.e. it's a false positive. So what do you think type 2 error is? Brilliant, yeah, absolutely. So power is to do with type 2, and type 2 is about false negatives. So that would you be wrongly finding that your drug doesn't treat the disease when in fact it does. Does that make sense?

And whatever we do, there's always going to be false positives, false negatives. No trial is perfect. So we accept that there will be a proportion of false positives and false negatives, whatever we do. But it's how much of those that we accept. And we express those. You may see things like this. Does this ring a bell? Yes, we've got good nods. This is good. OK. So type one error is also sometimes labelled as alpha, the Greek letter alpha. So you may sometimes see people writing it as alpha rather than type one error, but it means the same thing. OK. So significance, the gold standard is expressed as a key value. The gold standard.

Some want lower, but the gold standard is P is less than 0.05. Do you know what that means? Yeah, absolutely. So it's 5 in 100, isn't it? Yeah, 5 in 100, isn't it? Yeah, 5 in 100, isn't it? Yeah, 5 in 100, isn't it? Yeah, 5 in 100, isn't it? Yeah, 5 in 100. Basically, it's a 5 in 100, or 5% chance that whatever this trial finds is due to chance and chance alone. That's basically what it's saying.

So we set that right at the very beginning. We make sure we have a sample size that will give us this p-value. And that means that whatever we find in that trial, there's a less than 5% chance it is due to chance and chance alone. Some trials want an even lower p-value and want 0.01. But these are those big ones with lots of money, lots riding on it. You know, we want to be the next wonder drug, etc. Power, like we talked about, is due type 2 error due to false negatives. That's expressed as a decimal. And 0.8 is the gold standard. What does that mean? Does it 0.8 power mean?

basically yeah basically what that means is 80% of any negatives we find are correct negatives so four out of five things that we find negative will be correct negatives one in five will be a false negative okay it allows a 20% false negative rate these are the gold standards does that all make sense some have a power some want a power of 0.9 and we'll go for it that way i.e. there's only 10% false negatives again these are your really big studies lots of money riding on it we've got shareholders etc etc you know You can guarantee any future treatment for dementia will have a p-value of less than 0.01, power of 0.9, because there'll be so much riding on it, they'll want it to be proven that this new wonder drug works, because if you can imagine, they'll take over the world if it can treat that. Does that all make sense, guys? Please say if it doesn't. Good. Power of gummy bears. Excellent. Yes. One of the questions I've always been asked is, what sample size are you looking at?

So I'm going to show a demonstration of how we put it together. So this will be right at the beginning of the methodology. We wanted this p-value, we wanted this power, therefore we have made this sample size. Excellent. So then, just another three things then to think about. Our estimated effect size, our event rate, and our standard deviation. So our estimated effect size, this basically comes from our literature review.

What effect do we think our drug will have? How much will it reduce mortality? How much will it reduce mean systolic blood pressure? So you have to look back at future, look back at past literature, look back at your preliminary test. The smaller your estimated effect size, the bigger your sample needs to be. Event rate, that's basically our prevalence. So how much of this condition is there in the population we've got?

And then standard deviation, everyone's favourite part of maths, do you remember that? That's basically looking at how varied our population is. Have you got a very broad range of blood pressures in your population, or are they actually quite narrow and quite similar to each other? The smaller your standard deviation, i.e. the more homogenous your population, the smaller your sample can be. If you've got big variation, you need to have a bigger sample. Does that make sense? In what way?

Why do you need a bigger sample if you want a smaller estimated effect size? So basically what you're doing is you're looking back at the literature. You're finding out that say this drug was used in China and it reduced mean systolic blood pressure by 10% there. So that's our estimated effect size. But if you've got such a small effect, there's a high chance that you could miss it very easily due to error.

So you're going to want a bigger sample in order to catch it. Does that make sense? Basically, we are never completely going to eliminate type 1, type 2 error, we can't. No trial is perfect and anybody can tear a paper apart. Well, maybe not apart, but you can have a go at it. Okay, cool. So hopefully... Yes! Okay, so I did some googling and basically looking at different online sample size calculators. This is a very simple one.

But I think it sort of shows the principles there. Then you can go off and purchase really big, complicated software as well, speak to the guys in the statistics department. So this one here is from ClinCalc, which is good. Basically, it's looking at are you comparing two independent groups or have you got a study group versus the population? So let's say, for example, we are looking to decrease mortality for a certain condition by 10%. Shall we say that? So we've got our study group versus the population there.

We're looking to decrease mortality, so that's a dichotomous, are they alive, are they dead? So we can keep it there. If we say that... 50% of the population die from this condition. Triple A. There we go. That kills 50% of people, doesn't it? Well, before we even arrive at hospital. Should we say that one? Pre-hospital Reboa. There we go. It's pre-hospital Reboa. There we go. You can make us think for anything. Okay, so we've got our percentage decrease of 10% that we're looking for. We've got our alpha, i.e. type 1 error. So we've got our p-value there. We've got our power there. And now we can calculate it. And hopefully that'll work it out for us. So this means for our...

For all the things we've put in, we would need to recruit 783 people to have a meaningful result that we could publish. Do you want to see the calculation that goes into that? Just for nothing. Forever. Well, there again, it'll be like, now you need to be a multicenter trial then, don't you? You know, therefore, straight away, I need to recruit other people. That is the amount of math that goes into it. That makes my head hurt, so we're not going to go into that. But this is why, this is why, get the software. Okay, and I think that... Is that free? This is free, this is free. There's lots of different places. So, University of Hong Kong has got loads you can just use quite freely. Yeah, lots of different places I've found, but this is the one that looks the nicest up on a projection. And it's not in Chinese or Mandarin or Cantonese, is it in Hong Kong?

So does that make sense as a kind of way of going around it? Excellent. No, say it wasn't. Yeah. Will it affect your, or is that still the gold standard for your numbers? That's still the gold standard for your numbers. Okay. But then again, that then comes into, the question will be, well, this is what we were hoping to find, and this is how many people wanted to find it, but then people will say, well, do your methods actually go with what you wanted it to do? So that's where it always goes into it. At the end of the day, you're always asking yourself, did these people find what they were looking for? To sort of quote Bono. Excellent. So obviously I'll put this out on the podcast, and there's a blog that goes through all of this as well, so you can have a read of that at your leisure.

Next month, which will be between 10 and 11, are we saying now? We're going to do that on the Wednesday, between 10 and 11. We're going to be looking at outcome measures, confidence intervals. Sorry? So you guys, who've already heard me talk about p-values, are going to be bossing it. You're like, oh, Jamie, this is about significance, isn't it? Less than 0.0. So we're going to have a look at outcome measures, confidence intervals, and p-values.

Anything about today's session you want me to go through again? Does it make sense? No, that's really clarifying a lot. Cool, cool. Good. Coming, Bears. So, as you know, I like to stick a pun at the end of these things. What do you think the pun is? Bears. Bears. Bears. Bears. No, well, that's a good one. I like that one. I wish I'd like that one. It's a Bears. Thank you very much, guys. Thank you. Thank you.

That was the Take Orally Unblinding Research episode on sampling. You can find the blog entry and take visually for this episode at takeorally.com. Don't forget to subscribe to Take Orally on both SoundCloud and iTunes. For more information about research and educational opportunities with emergency medicine and acute medicine and major trauma, you can find a new age stream on both Facebook and Twitter.

Frequently asked questions

How long is this episode of Take Aurally?

This episode is 24 minutes long.

When was this Take Aurally episode published?

This episode was published on September 21, 2018.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Take Aurally episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.