Hello everyone. Hi Matt. So welcome to our session on blinding research. We've got quite a bit to go through. Validity, reliability, generalizability, bias, and practice. Essentially, I've created some exercises, that needs to be there of course, to help us demonstrate the principles. Once again, this is a nice open environment, everybody can ask a question, everybody can say anything that they've got or would like to make.
So we're going to use Frank, these pictures of Frank up here on the board. We're also going to use one of our former colleagues, James Pratt. This is him having run the Great North Run. He did it in 1 hour 39 minutes, which he tells me is a very, very good time. So between James and between Frank, we're going to have a discussion about some definitions and some principles. Right then, first up, we are going to play a game of dot Frank's nose. So what I'd like is six volunteers to come up and be blindfolded.
and try and put two dots, one at a time, on Frank's nose. So here's the point of what accuracy is. Accuracy means the true value. So just like when James Pratt was running and he got his value of 1 hour 39 minutes, an accurate result is a result which closely reflects the true value. So in which case an accurate dot is close to Frank's nose. Like an accurate arrow hits the bullseye. Precise data doesn't necessarily have to be accurate. It's just near each other. You haven't got a large range. So actually...
Up here, they're quite close together. Lucy, you've managed to be quite accurate and also quite precise, because you're quite near each other. Whereas over here, we're not really very accurate, and we're not particularly precise, aren't there? Because you've got the wide range. So if James Pratt was to run the race again and again and again, precise data would be close together. You get 1 hour 39 again, 1 hour 38, 1 hour 40, close together. So accuracy and precision describes it. Does that make sense? So you don't have to be accurate to be precise. You don't have to be precise to be accurate.
Good, marvellous. So that's, well done. Well done Lucy, I think you win that. What I'd like you to do is, you've got a couple of minutes. I'd like you to draw Frank. So underneath here, I'd like you to draw Frank. We need to get photos of this. Try and be as accurate as possible. We don't want cartoons. I want a nice... You don't want cartoons? I don't want a cartoon version of Frank. I want you to replicate that till I know. What do you think, guys? I mean, there we go. I forgot the name. I don't know the last time.
So let's say we are judging these pictures of Frank, and we are going to award a prize for the best picture of Frank. We want to be reliable judges. Reliability describes a test. So while accuracy and precision describe data, reliability describes our test. And essentially, we wanted that if a test is reliable, we would give consistent results in the same thing.
So, if we were judging these two beautiful pictures of Frank, how are we going to be reliable judges and make sure that we are judging them to the same standard? This was being judged by one judge who likes clowns, and this one was being judged by somebody who hates the colour red. Is that fair? No. So that's what we call inter-rater reliability. Reliability between two races. So say you've got somebody in one site who thinks that there's a certain criteria versus another one who's got a different criteria, that's what we call inter-rater reliability. Cool. If Alex was to use this exact same picture again, should she get the same mark again?
Yes, and that's what we call test, retest, reliability. So if you do the same thing again, if James Pratt ran the exact same race again, he should get the exact same time again. If you do the same work again, you should get the same result again. Has anyone here ever done a day's worth of marking for an OSCE or anything like that, had to go off and do a day worth of marking or anything? What's it like, guys? They get largely better than shit.
I would say your market is not as good as on the first one. So say you've been judged by a panel in the morning who are nice and happy, they've just had their coffee, they're raring to go, it's really exciting, and they give this a 5 out of 5, then by the end of the day their coffee's worn off, they hate life, and they give this a 1 out of 5 just to get a hold of me over. So that's an intra-rater reliability, you've only got one rater, but it's how I bear each other time. And this is one of the big reasons why, medical finals have changed because it used to be across many different sites with different examiners and there's too much unreliability because examiners are different sites some are more old school some want to do it a different way some have certain patients some other patients and so there's too much unreliability which is why now all the exams are held in one place they're trying to end the same day as much as possible so accuracy and precision describe the data reliability describes the test internal validity do you know what that means has anyone heard that expression before it was infinite can we talk about the internal validity of a study
It just seems to be a kind of how it would be comparable compared to the states and would also the outcomes it gives. Not quite, no. I'll speak with Ken that, as in how strictly high states are rules and I actually chose the most accurate patients and all that. Yeah, so internal validity essentially describes how much does this test result accurately reflect what's actually been found. So that if James Pratt comes fifth out of the runners, and internally validated, if there's high internal validity, it means he was the fifth highest, that is the accurate representation.
Does that mean he's the fifth faster to run in the world? No, just that particular way. And that's where external validity comes into it, and this is what we also call generalizability. So this is when you guys, when you're off in Prague or wherever, and you see a study that's been performed in, say, Azerbaijan. The generalizability of that is, how much can I actually take from that and put it into my own setting? So a high validity would be one that accurately represents what's actually going on in that site. External validity represents how much can that be extrapolated and used in others. Do you think they can be both high internal validity and high external validity?
So this is something that's difficult, because when we do study here, we are a major trauma centre. We're the 24-hour radiologist, 24-hour major trauma unit. So when we do trials, we can show high internal validity, but your DGH that's listening to us might not be able to go up. That doesn't work at our site, so it's not so much height. So there is a balancing act to be done. And the way you can go about doing that is to have a randomised trial that's being well performed, but also to have as loose criteria as possible.
And that's how you can try and create as high internability as possible, as high internability. But it is balancing. Does that make sense? Cool. So bias. What is bias? What does bias mean? To favour something. Or conversely not to favour something.
So bias is anything that skews your results, it can be one way or the other. You can be biased against a certain type of person, you can be biased against a certain football team, you can be biased against a certain politician, etc. And you can have bias in your research, and that is anything that skews your data, one way or the other. And you can have bias at all the stages during design, during performance, and during the mindset. So what we're going to do is now, that's another exercise, I've got to get you into some groups.
What we've got is, these are some scenarios, and these are some definitions of different types of bias. What I'd like you to do is over the next five, ten minutes, is just work through... One per pair. Thank you.
So, you might find that there's more than one bias for some of them, but there is one that fits best for each particular one, so all the biases should be used after. Yeah, yeah.
An alarmist? Somebody trying to cause alarm. OK, make sense.
Thank you very much. Yeah. Yeah.
That won't happen actually. How we doing guys?
How are we doing guys? Alex and Matt how are you doing?
Thank you. Thank you.
some of them might go for more than Yes! So by the process of the nation it's well left, you can go back.
You're interviewing patients about sweet consumption and developing diabetes. You want to know who has diabetes and who doesn't before the interview so you can probe more about risk factors if they've got diabetes. So I wrote this with interviewer bias in mind. The interviewer may know the outcome to change the question based on this. So say you were doing this study, if you know that somebody coming in doesn't have diabetes, you can treat them differently to somebody who does have it coming in. So there's a clear sort of conflict there. And that can be the bias in the interviewer. Cool. You recruit only young, healthy patients to the intervention arm, and you put older, frailer patients onto the placebo.
What have we got? So it's slightly more channelling, but there is selection in there as well. Is it 1, 2, 3, 4, 5, or we're down to 11? Sorry? Did they match? Don't know. I hope not. Because it's 3 and 4 on 3 and 4. Oh god, baby. Oh no. Oh no. This is going to become very easy, isn't it? Okay, cool.
So you are using slightly different criteria to go on different arms, but the reason I put this as channelling is, imagine if you were trying to show that your intervention works, you're pushing those who are going to have better outcomes anyway onto your intervention arm, whereas those who are going to have worse outcomes because they're older, frailer, polio, they're going to go on to the other one. So that's an important thing to look at in any study tool. What were the criteria? How did you get into the different cohorts? This is why we randomise, to completely rule this particular bias out. If you want to start a study looking at surgical outcomes, you will ask surgeons if they are pleased with the outcomes of their procedures.
Flawed study design. That's the first one, that's the first question. I've flawed this one. Brilliant, yes, flawed study design. What's wrong with, why is this a flawed study design then? Why is it wrong?
Yeah, absolutely. You've done this piece of work. You did this operation. Are you happy with how it's gone? Are you happy with how this has gone? Are you happy with your work? Everybody's going to say yes. It's completely flawed study design. People will tear your study apart. And you go, we found that all of our surgeons are really happy with their work. We did good surgery. You go, no, that's a flawed study design. That's how we are. Cool. You find that coffee drinkers have an increased risk of heart disease?
How is this a confounding factor then? How is it confounding? What do coffee drinks also do? Smoke. So this is why studies will find that back in the day people who went to the pub a lot got lung cancer.
Even if you're not a smoke, you go to the pub for a drink, you've got lung cancer. Therefore pubs cause lung cancer. Or drinking pints cause lung cancer. So, you know, that's a confounding factor. Something that's a third association that you don't think of, that actually gives you a false association. So in this case, it would be that the coffee drinkers you ask like to have a cigarette with their coffee, therefore there are increased risk of heart disease. Having looked at the coffee I drink, I hope that's not a true association. Cool, so this one's going to be easy. Next one, you're investigating lung cancer to see if there's a link between drinking coffee, looking back at how much coffee they drank before they're diagnosed.
It's electric. Well done. Why is that selection bias? Why is it selection bias? So selection bias is very, so retrospective studies where you know an outcome are very open to this. Because you can go, I know you have lung cancer, I'm therefore going to look back at anything you did beforehand and try and root out an association. So this is written for that as a selection bias. And so it's very, very dangerous in that we can go, oh, you know, you've had X exposure, I'm going to look back, therefore you've had X outcome, I'm going to look back at any exposure to my binder. Cool.
You compare the success rates of different surgeons, you notice the more experienced surgeons have the best results. How is this performance biased? How is the performance bias, guys? The more experienced you are, the better you're able to be. Yeah, absolutely. So that can be, we've got more experienced surgeons, that can be, this is a particular piece of equipment that we always use, and we find that we're better at using this piece of equipment. There's another place that's only just Scott has taken on. So this is something that surgical studies are particularly open to, is this performance bias aspect of it. And the other thing to find out with surgeons is, each one will have a particular way of doing it, and they'll all be really good at that one particular way of doing it, whether it's laparoscopic, whether it's open, or however they do it. So performance bias comes into that.
This has been very easy. We have 25% placebo patients, lost a follow-up. Well done. So it should alarm you if you're looking at a study and you find there's a huge difference between which cohorts and the amount of people who are lost to follow up because it shows that how they were doing it was not fair across the board and that there's something going on so they meant that people were lost to follow up. So it should alarm you if there is this big difference between which arm of the study. They should always be very similar. They should show how many people were lost to follow up and they should be similar. Is that potentially an incentive to say if you've got the potential? Yeah. So we didn't care about these placebo patients.
We just logged onto, noticed to check if they were still alive, that was it, fine, moving on. Whereas this one set, we rang them up every other week to make sure they were still OK, we asked them X, Y and Z, all of this. So if you see this sort of thing, you'd be like, well, your study was bad. Yeah, this is it. Again, why are we blind? Because if I know that you've just had some sugar water rather than my brand new drug, I don't care about you. It doesn't matter. I only care about the people I know, I've got that, which is why we're blind. Who is this about? Well done. So you're a famed hungry alarmist, you ask patients of children with autism to link it to the time you had their vaccination.
Absolutely. What type of bias is this? Recall bias. Yeah, brilliant. Essentially, Andrew Wakefield went up to some parents, 12 of them, 12 parents. Big cohort. Very big cohort, and asked them, your kid has digestive problems and or autism. What time did they start to have it? Was it around the same time as their MMR? And the parents went, yeah, I think so. And that was enough for him to write a paper causing the trouble that we have now. Scary, that. And he's now in prison. No, sorry, he's in America, dating Elmer Pearson, touring America with the Make America Great Brigade.
What follows him as he goes across America? Donald Trump. Measles. Wherever he goes, measles follows him. New York is currently in the midst of an epidemic, thanks to him. If you're under 18 and you're not vaccinated, you're not going to a public area in America. In New York. You're investigating a new intervention, and as your comparison, you go back and use 20-year-old guidelines.
Brilliant. So if you find out that your new intervention is absolutely brilliant but you're actually comparing it to something that was used in the 1980s, 1990s, that is not a fair study. You should be comparing it against the most up-to-date guidelines. That's fair. Cool. Your trial finds the intervention doesn't have a significant effect and your sponsored air forces don't publish.
Is it worth pointing out that a lot of these have got many names? Just if you read the journal, you'll see, instead of the citation bias, you'll see publication bias. So just be aware, there's 70 names of bias, and then about five or six real types of bias. And each time you Google, there's a new name in something, absolutely. So citation, publish, attrition bias, yeah. People publish positive results much better than they publish, negative results, if they want to name that. So this would be a citation or publication bias in that result.
If you want to investigate the rates of PDBT, you will record all patients who report chest or calf pain. A misclassification. Why is this a misclassification? Because not everybody report chest or car pain or DBT. Absolutely. So if you were looking at this study being presented and you found out that we were checking our EE rates following surgery, and how we did that was asking everybody, do you have chest pain or car pain following the surgery? Does that mean that each one of them had a PE or a DBT? No. What would have been better? You're actually ultrasounding, scanning, doing diagnostic testing, actually finding out they've got confirmed PE or DBT. So this is misclassification. So if you absolutely look at this study, you know, no, that's wrong. Cool.
Thank you guys. Well done, that got better as we went on, didn't it? Never mind. Next one, we're going to be looking at writing a paper. So that'll be interesting. Cool. Thank you very much. Thank you for our artists. And a take-home message. Be accurate and precise. A free-shape bias. Don't be a praff. Thank you very much. Boom. Ten minutes is fair. There we go. Go on, don't steal my head.