Now hopefully you guys can see. So today we're going to talk about AI, and I'm going to go first with AI in medical education. I'm going to start with a full disclaimer. One is that I'm not an expert. Despite standing in front of you all today speaking about this, I'm not an expert. I'm learning with all of you, and I think you can appreciate this. This field is moving fast and furiously, and it's exciting. But while I'm awestruck, I'm also careful. I think that it changes the way we think about things. It changes the way things are brought to us, how we process things, and really kind of how we approach almost anything that we do is going to be influenced by AI.
Our roles as educators have continued to change over the years, and I think that that's what it's supposed to do. But with technology comes a bit of hesitancy, especially with something as powerful as AI. So in my opinion, we have to provide opportunities for our trainees from undergraduate medical education all the way through graduate medical education to learn about the changing technology landscape and the impact that it's going to have on the future. So we in the medical school are shifting our approach to AI, moving from apprehension and how do we prevent the use of AI or how do we regulate the use of AI, which I think is still important, and we are focused on that, but how do we actively integrate AI into our curriculum and prepare future physicians and researchers? Really, if you look at what's happening in the literature, you'll see that across many institutions across the country and the world, students are now learning the responsible use of AI for tasks like diagnosing fictional patients, analyzing medical data, and conducting research. And so this is kind of the landscape that is now becoming routine. It's a routine tool that's used in patient care. Just for a show of hands, I have some people in the audience here and someone Zoom, but raise your hand if you use AI in your regular life. Regular life. So now keep your hands up. How many of you use it in your personal life?
How many use it just in your professional life? And in both, right? So there's a mix of this. But I think that we're seeing it more and more used across the spectrum, from our personal lives all the way to our professional lives. So what is AI? And before we can explore how we use AI in medical education, I want to make sure we're all speaking the same language. So AI is the abbreviation that I've just used multiple times at the beginning of this talk. But really, there are two different kind of terms that AI can stand for. One is artificial intelligence, the ability of computers to perform tasks usually associated with a rational human being. And then there's augmented intelligence, a set of computational methods and systems that enhance human capabilities and decision making. And the AMA favors the term augmented intelligence. And I think that I like that too. I like the idea of...
using these methods and systems to enhance our ability and our decision-making. And so I think augmented intelligence, while all of us, I think, are used to saying artificial intelligence, when I say AI, I'm thinking about augmented intelligence. So let's talk about some of the benefits and challenges. So in terms of benefits, a dramatic reduction in the time to generate educational materials. Materials can be generated quickly and can be refined to meet expectations. In fact, the better you instruct whatever tool you're using, the more parameters you give it, the more it'll be refined, the output will be refined to meet your expectations.
In terms of challenges, difficult to decipher AI-generated versus user-generated materials. And there's this question of, does the best AI win, and does privacy lose? One benefit that I didn't have here that I want to highlight is that it really allows us to, when we think about augmented intelligence, it allows us to meet the learners where they are, tailor the medical education being delivered to that particular learner to provide the best precision medicine for our patients in an equitable way. And I think that that's really important as we think about this, and we'll touch on that a little bit later. So in terms of further glossary of terms, you may have heard the word machine learning.
That's self-learning algorithms that derive knowledge from data to predict outcomes. Deep learning is a subset of machine learning, which extracts complex patterns from input data and eliminates some human intervention. And then neural networks are a framework for deep learning, or node layers where you have input, hidden layers, and then outputs with neurons, quote unquote, that connect them if a certain threshold is reached. We'll talk about that a little bit more. pizza, burger, and taco, a human identifies these based on features. So it might be, OK, there's bread involved here, and it's in a crust form, so that's going to be a pizza. If it's just bread that you said, you wouldn't know if it's pizza or a burger, per se, or even a taco if it's a soft shell. So machine learning is going to determine features that distinguish something specific from unstructured data. And then deep learning would find the similarities or differences and put these three things together. So that's kind of an analogy that's there.
When you think about the relationship between AI, machine learning, and deep learning, augmented intelligence is going to be the big circle. Machine learning is going to be a subset of that, where the data can be utilized, and you can train it on and work effectively with smaller data sets. The training time is faster or shorter training due to fewer computational resource requirements. And then deep learning is a subset of that, where you require a large volume of data to train and operate efficiently. So this takes a lot longer and can take a lot more computational power to establish. When we think about the deep neural network, I think that this is kind of
one of the things that took me a while to wrap my head around, that you have all these input layers that then feed into these multiple hidden layers that are then analyzed and cross-analyzed, and then you have, at the end of the day, this output layer that comes, and that's your product, right? That's what you're getting. And what's really important is that there's a lot of stuff that can happen in this multiple hidden layer period that can result in output that's not actually accurate. And you can see that the input really is important as you think about how you're going to get the output that you desire, and that's the more specific you are, the better it is. We all use this almost on a routine basis. We come up with lists. We come up with things that we're going to think about that we're going to do. The reality is that it's already integrated into your daily life. Google Maps, Uber, Siri, Alexa, all of these things that are used. This past weekend, my daughter used
chat GPT in the car ride home from North Carolina to come up with a list of places that she's going to apply to for college. And she didn't like the output. And so I asked her what the input was. And she mentioned that she downplayed her GPA, anticipating that she's going to get a B in physics and a B in calc this year. And we had a whole conversation about that because that's just not OK. But it's insane, the amount of use that happens with AI and using Google Maps or Waze or whatever to get to where you're going, smart thermostats, YouTube recommendations, et cetera. The list goes on and on and on. So in terms of machine learning terms, there's natural language processing.
So a lot of us have worked with NLP models where you basically use computational techniques to understand, interpret, and generate and manipulate human language in both spoken and written forms. And one of the things that we were working on here with NLP processing was looking at all of the notes that are written by a medical student and analyzing what the language that they're using is, thinking about desigmatizing language and how we say stuff patient complains of versus this is their concern, et cetera, things like that that can change the way we think about a patient as well. So NLP, you have large language models. a significant number of parameters, often the millions or billions, which enable them to learn complex patterns and relationships. And then VR, which is, I think all of you know, computer generated simulations to create immersive interactive experiences in a virtual environment.
So the presence in medical education is already out there. It's already in a lot of different places. Hopkins has an AI and medicine focus area for trainees in the graduate school. NYU students use AI to generate differential diagnoses and plans for problem-based learning scenarios. Stanford uses AI-powered clinical decision support tools. You're going to hear a little bit about AI's use in critical care medicine from Dr. Seem. So I wanted to highlight a few case examples of how we use AI and med ed at the School of Medicine here in Maryland. So in the pre-clerkship phase, we've demonstrated the ability to extract goals and objectives and mesh terms from panoptic captions from recordings of pre-clerkship didactic. So for example, if this was being recorded right now, you can then...
put this into, at the med school, we use Panopto to record things. You put it into Panopto. It generates captions. It transcribes the entire presentation. And then you can say to AI, basically, identify the top three goals and objectives that were discussed in this. And you can then go back and see, did I actually discuss those three goals and objectives? And are those the ones that I think are most important? You can also say, these were my goals and objectives at the beginning. Did I achieve my goals and objectives by looking at the transcript of the didactic?
Also, use it to generate multiple choice questions to be able to aid students to test whether they're, A, coming in prepared to a lecture or a small group session. And then also, they gain from the content what we expect them to gain by looking at multiple choice questions that are developed. And then we've created formative assessment dashboards that help students focus on their areas of need. So this is an example of the Panopto software where you can see this was an lecture on inflammatory bowel disease given in May. And it'll basically transcribe the entire thing, put the headings there. It'll give you the slides. You can put in a word. You can say, you know, an inflammatory bowel disease. I want to see everywhere that.
Crohn disease showed up. And we can put Crohn in the search, and it'll show you each slide and timestamp. You can hear specifically what they said. We can even use this to fact check when we think when a student comes to us, how many people when they were students said, that was not taught to us. It was on the assessment. We can go and put that into Phenopto and say, show me where this was in this lecture and listen to it and say, OK, maybe you were right. They said, don't worry about this. And they still put it on the test anyway. This is an example of the formative assessment dashboard. This is a blank one. But you can see here that exam grades by course, student grades by course. And then as you go from the top left to the bottom right, this is strengths and opportunities. So in the top left are going to be the higher
percentage categories. So this person here does really well with the renal and urinary system, with intro to clinical medicine. But physiology, gross anatomy, histology, these are areas of weaknesses or cardiovascular compared to the others. In addition, it'll give them their attendance. That's always a big question. Am I going to give my points for attendance? And then peer evaluations. How are they doing in relation to their peers?
In the courtship phase, we've used it to create case-based scenarios for learners. We've done a scenario with a fractured arm that included an intimate partner violence scenario, aortic dissection, appendicitis, things that potentially you're not going to necessarily see right away all the time, but you're going to need to know how to approach something in a meaningful and thoughtful way when you do see it. So, for example, intimate partner violence, the first time that you trial what you're going to say or how you're going to act in that scenario is not. in the actual scenario when someone is emotionally and physically hurt, right? So you want to be able to do these things in a simulated way. One of the things that you're just maybe not going to see as frequently, aorta by section, once you see it, you won't miss it, but you may not see it in your rotation in four weeks on that. We also used AI to generate a transition to residency curriculum using basically getting suggestions for content inclusion and instructional strategies, and then synopsis for end-of-course feedback, which I think is really, really important. You get all of this data and you want to distill it down to say, what are the key points from 165 students for this course? What are the positives? What are the negatives? What are the things that we need to work on?
We presented it at one of the AMA conferences. Gene Judy was the lead author on this, and one of our medical students, Hero Patel, was the lead author, and Gene was the mentor. But basically looking at the utility of a large language model in simulation-based medical education. So using ChatGPT, basically assessing whether ChatGPT can be used to create high-fidelity virtual clinical simulations for medical student education. And we showed that you can do this, and that it is actually very realistic as long as the inputs that you're putting in are there.
In terms of goals and objectives, you can see an example on this slide right here. I'm not going to read it all for you, but it basically will give you, using Bloom's taxonomy, the right kind of language for goals and objectives, and then really kind of characterize each of those specifically. So I want to move into three categories here, learner-oriented, AI and medical education, instructor-oriented, and institutional-oriented. When you think about learner-oriented, you're basically talking about intelligent tutoring systems where it can emulate a tutor. It interprets and responds to the student. This can be used for course content, strengths or gaps, automated feedback, or curated learning. And it can facilitate collaboration between learners. Your virtual patients, these are computer simulations that replicate scenarios and provide dynamic response to inputs. Students act as a healthcare professional. They can present challenges, real-life scenarios. You can work on communication.
and clinical reasoning. Gamification, customizable and can adjust based on performance. You can use data mining to improve knowledge and skills. And then adaptive learning systems, where spacing and sequencing is personalized and integrated with adaptive content. And this can be a powerful resource for individualized mastery. So instructor-oriented, we've used for curriculum development, as I mentioned. High yield summaries of core content, gaps in content, mesh terms to populate the curriculum map. We've used it for case studies or quiz developments to support the didactics, analyze our evaluation data.
This has all really kind of opened up a real wide range of ability to tailor the educational curriculum to individual students or groups of students that might need help in different ways. Institution-oriented AI admissions, so assessment of selection process to support holistic review, already being in use really kind of all over the place. But there was a big article out of UNC in Chapel Hill that they were using it for their admissions to pare down their application process. It's being used here on campus as well for our medical school admissions. Curriculum development, as we talked about, career exploration and advising. And this is a wide open...
field where you can take a lot of data, create a data lake from students from when they entered medical school, even before they were here at medical school. What they did during their time here, use historical data as you populate the data lake to say, if this person did X, Y, and Z and was able to match into this, what does that mean for all the other people that are coming behind them and what should they think about? So provide coaching for these patients and assess their experiences. Evaluation data, so again, simplification of the data to bullet points, positive, negatives, area for improvement. If you ask ChatGPT, how should medical schools use AI? This is what it comes up with. Curriculum development, clinical training, patient care, personalized education, assessment and feedback, and then resource management.
So as we think about this, we've got to be very cognizant of the ethical and regulatory considerations. So privacy, data entered into the AI chat becomes part of the data set used for future analysis, and there are obviously concerns with that. Bias, AI is built and trained on existing data, which may be flawed, and algorithms may benefit a certain group. And then transparency, when you think about educators and how we use this, we have to understand how AI works in order to validate our decisions. And so when we think about mitigating bias, I go by the premise of trust, but verify, and always suspect sabotage. You've got to regularly analyze your outputs and review your data that's used for training.
You've got to be aware of hallucinations or confabulations. So verify your citations. Make sure that if you're using AI for something when they ask the question, was AI used in this review or whatever it might be that you answer honestly. Keep academic honesty at the forefront. And then there's really the importance of ethical AI design and ongoing monitoring. And this is something that Nipin and I have talked a lot about. And we really are going to be focusing on to make sure that as we go down this pathway, that we're really thinking about this is done in the most ethical way as we think about this.
Precision education. So this is building on prior work that a good friend of ours, Sanjay Desai, is the chief academic officer of the AMA and colleagues at Advanced, which is a conceptual framework that precision education involves applying advanced analytics to multimodal data inputs to generate insights that can be used to drive precision interventions. And this academic medicine issue that I've highlighted on the screen here is a supplement from April of 2024 that offers a vision for the next era of medical education assessment that centers on why assessment is done, which we think is to make sure that learners achieve the learning outcomes that prepare them to provide high-quality, equitable patient care. But really, as you think about, you know, when you take an assessment, is the assessment just to see if you mastered the material? Or is it really in medicine, which is a field of lifelong learning? Is it to make sure that not only did you master the material at a level that we wanted you to, but then can we use the answers on that assessment to go back and hone the concepts that might need help, an individual learner might need help with? And I think that that's important to think about. On this next slide, it's not for you to memorize or anything, but there were links. If you go to this in AMA's website, there are links to each of these articles in the academic medicine supplement. But I wanted to highlight a few, whether it be thinking about
precision education and framework, demystifying AI and its current stated role in assessment, finding medicine's money ball, how lessons from Major League Baseball can advance assessment and precision education. The next era of assessment can ensure high quality, equitable patient care be the defining characteristic. When we think about precision medicine, it's all about being in the right place at the right time and the right care for the right patient. And I think that that's the ultimate goal. But we need to bring that down, bring that upstream to medical education to make sure that we are, in fact, achieving our goal by training precisely the next generation of learners. And so this requires a ton of data.
It requires inputs that then get analyzed, resulting in insights you plan, you do an intervention, there's learning that happens, you assess that, you see your outcomes, and then you basically keep going through this cycle here and adding to the pool of data that's going to help you determine what the next best steps are. So thinking about an intelligent tutoring system, this is basically providing human-like lessons without a human teacher, personalizing tutoring by learning about each student, and then informing teachers on students' learning needs. So this is really stuff that drew its foundation from cognitive psychology, computer science, and especially AI. And there was a shift in AI research that really moved from the logic focus of the previous decade to knowledge-based systems. And so ITS, or intelligent tutoring systems, I think are going to be a way to help individualize learning for a specific learner.
So how many in the room are familiar with Google's Notebook LM? Yeah, a few. So I'm going to stop sharing my screen for a second. All right, you should be seeing my main screen here. And I'm going to go to this bigger. Can you guys see that? No, I'm going to go over. You still can't see that? Can anyone? You guys can see it on Zoom. We can see it. All right, can you all see that? And can you see it on Zoom as well? All right, perfect.
If you have a Google account, this is free, open access for everyone. And what I wanted to do was I just threw two things in here. But what I wanted to show you was that you can basically put in the input. You select your source. So if you really like Dr. Seam's talk on introduction to mechanical ventilation, that's not what this one is. But you can basically put that in here. You can put in a bunch of different things as your inputs. Here I put the equation of motion to mechanical ventilation that was a talk that was from YouTube. And it will basically create an AI-generated podcast. It'll create a videocast. It'll create standardized questions. And you can see here, it gets a little bit old after a little while, but I'm going to play this for you.
This will create a 10, 15, 20-minute podcast that you can listen to on your drive with the input that you've put into it. You can create a video. And you can see here that I created a study guide where it gave me a quiz with 10 questions, the answer key, essay format questions, and glossary of terms as well from that input that I put in there. Is Notebook LM the be-all end-all? No. It has its limitations. It has its challenges. But these are the kinds of tools that are at our disposal now to augment our learning and to be able to say, what can we do differently?
I did this. I put in some just random things, respiratory physiology, AI, and medical education, just to say, what does this do for me? And you can really take this and kind of go bananas with it, really. You can put in as much as you want. And you control what the input is, because you're putting in the articles that you want to summarize. All right. Let's go back to our slides here. All right. So AI-powered learning tool. Generate podcasts. Ask questions about the material you've uploaded. Recap lengthy. Main ideas for lengthy lecture slides or main history. Practice questions or flashcards that help you remember things. Case-based simulation. This is a company I don't have any.
bias here, and Sharon's just one that I picked out. We don't use it in our medical school. But Sharon came up with really a platform to be able to have difficult conversations. How do you learn to have a conversation about breaking bad news? Or how do you tell a patient? We used to have a simulation, and Caleb probably was one of the people that went through it back in the day where we did difficult conversations. How do you learn to tell a family member that their loved one is brain dead? It shouldn't be the first time you're trying to use the terminology. In fact, now that it's been a few years since we've done that, I can highlight that somebody started to use the analogy of a broken down car.
and then realized kind of halfway through that when they started talking about the carburetor, that they were like, oh, this is not good. And they kind of were able to backtrack and get feedback and watch it themselves and realize. So using case-based simulation, I think, could be another way. In the clinical setting, nuance to basically scribe that's integrated with Epic. But again, remember that all of this has limitations. This was an editorial in the New England Journal that the chit-chat, the conversation that I had with the patient about how are things going, how's your dog, how's the grandchild, et cetera, was all cut out of the transcription. That's the stuff that I want in my note because it helps me, A, remember my patient, relate to my patient, build trust with my patient.
So in the hospital, this was at Penn, Caroline, integrating how Epic interacted with Caroline. And I don't rush out of my patient's room anymore because I know I can get my notes done half the time, using AI to really make our lives easier. Many people have used open evidence. Note writing or letter writing. You can use audio pen. You basically dictate into it and it'll write the letter for you. And it's in your own words. Just cleans up all the grammar. So these are all things that I think lead to future directions and opportunities. Establishing clear guidelines for the proper use of AI. EMR-based, HIPAA-compliant AI, allowing for seamless involvement in clinical education. Working together as a community to leverage tools and resources so that we can continue to provide the best education for the learner at the time that they need that to be able to provide good precision medicine to patients at the end.
So I'm going to now shift over to Dr. Seen. I'm going to fix this back. All right. Everyone, we're good now? OK. So Dr. Seen discussed a lot of effective uses of AI in Net-Ed. But I think one thing I want to think about is there's a lot of papers, a lot of opinion pieces. There's not really a lot of evidence. And so very recently, things have started to come out. And I want to walk through some recent general papers and ideas and kind of thoughts for what we do ..
Has anyone ever heard of the term of Pascal's wager? That was new to me. Anyone? Yeah? No. So I try to keep religion and politics out of talks, but humor me here because this is where Adam Rodman, who is a leader in the field of AI informatics and studying in medical education, I heard him speak about this. I thought it was actually very thoughtful. So Pascal, Blaine Pascal, was a 17th century philosopher. He talked about believing in God. So the core idea is actually very pragmatic, not a faith-based, but a risk-reward analysis. So the question is, does God exist on the top here, or does God not exist in your two columns?
If God exists and you believe, you gain heaven. If God exists and you don't believe, you risk hell. If God exists, it doesn't matter either way. So from a practical point of view, that's why you should believe in God was the argument of Pascal. So when we look at AI, you hear a lot of talk about it. And I guess that's where I look at when we think if there's superhuman AI, that will replace all of this. And so when I think about medical education, if we have superhuman AI, medical education working with AI, we're all going to lose our jobs. If we have no change in our approach to medical education, we're all going to lose our jobs.
but i think we hope for i think as we're getting to right now is very good it's just a really good tool it's not super human and i think that's kind of where we are today and it continues to improve and the rate which is improved we don't know but if we maximize this use in the medical education forum hope is that doctors will have jobs and that we can use these tools to help teaching but also of course the bottom line of what we do better patient outcomes And then if we don't change our approach and we're not taking advantage of some of these remarkable tools that Neurons give you several examples of, let's have doubts, but patients won't benefit from this tool that they're using in every sphere of life. So my focus has really been in that upper right box now over the last couple of years and trying to figure out how to do this thoughtfully and also collect data as we do it because there's lots of unintended effects and we're starting to see that. So if anyone read this, I don't think anybody read the entire paper, it was like 206 pages, but it was a preprint about your, and that's the title, Your Brain on ChatGPT.
So this is from MIT Labs. And it's a really interesting paper, because it took three groups, and they, over several months, had them write three different essays. One, used ChatGPT to help you write the essay. One group, now it's 18 people in that group. 18 people in a second group who used a search engine to help them out. And then the third group had to just write it on their own. These are kind of like SAT-level essays. And so they did three different essays. Then the interesting thing is, they made them write a fourth essay. But then the group that had only used LLMs had just used their brain. And the group that used their brain only were allowed to use LLMs for that final essay that they were evaluating.
And so during that fourth session, they studied brain activity by EEG. They talked a little bit about natural language processing. They scored with that. And they also did an actual scoring rubric for people who grade essays to see the quality of the essays. And so they were quite surprised, I think, and worried that the group that used the LLM for the first three.
They perform worse at all those levels, e.g., the neural level, the linguistic, and scoring level. And they really wanted to get it out as a preprint. It just came out in June before peer review because they're actually worried about more and more early childhood exposures to LMs as you're forming the ability to work. So now you see this. This was just someone shared to me in the New York Times just last week. I think more and more people and the companies are starting to do this. They'll notice, like in Google Gemini, they have a self-guided learning now.
where they don't give you all the answers at once. They fragment it because they're thinking about this. But this was an article, again, just last week. And I highlight a couple of paragraphs in the article. Because humans are pros when it comes to cognitive offloading. We free up our mental space. We avoid thinking, why use a map when you can use GPS? Students, obviously, will behave that way when there's a tool like chat. But when we've been teaching highly technical concepts like mechanical ventilation and physiology, we say something that learners hate, but the struggle is, the learning is in the struggle.
It really is, for things that are complicated. When you get over those points where you're very frustrated, it's when you really know it at a high level. And so this really echoed what we talked about. Sometimes the acts of thinking and struggling would foster critical thinking skills, and the brains like bodies develop as their use. And so as we are getting more and more early use of LLMs, that is impacting our brains. And there are studies looking at functional MRIs, for example, and showing decreased. So now moving to what we do in medicine. There was a nice series in New England Journal about AI for metabolic.
And this was a piece led by Raja Abdul Noor. And it's this figure in the piece that I thought was interesting. And it's a little busy, but I want to walk through some aspects because I think it's instructive. So on your x-axis here, you have time. And so this part of the box that I have highlighted here is pre-generative AI. And what you can see here is that if we take that left curve here for those in the room, that's somebody like me who went through medical training and developed their practice before generative AI is introduced. And what you can see here is that
Many of us, and I'll also say that it's not clear that any years of experience improved patient outcomes. So we shouldn't say the gold standard was the oldest. We're always trying to improve. But when you get to that scar, when you talk about breaking routine practice, it's like, OK, I've learned this. I've learned this. But now a patient presents, and it's a little bit different. How are we able to adapt? That's where critical thinking is. And so many people kind of plateau at automatic practice. And then, obviously, the true experts have developed adapt.
Now, all this is going to change in the world of generative AI. I think what's interesting is now we have that second group, the fellows, learners entering medical school pre-generative AI, and now we're learning. And then that right green curve there is the current medical students. And you think about each of them differently. And what we worry about is you get to a point of arrested development, right? And you see that scar there. You worry about, on every level, de-skilling. You learn something. You kind of practice a certain way. Then you start to rely on a new AI tool. You lose skills, and we'll talk about some data for that in a minute.
people having their development arrest in particular in the middle or middle at that time where they're learning they're learning they're learning now then they end up over relying on a tool that they're still developing and then there's de-scaling and then there's the thing of mis-scaling and mis-scaling is basically ai makes errors you don't know any better you don't challenge them and you implement that into your practice and then in a you know in a very early learners you have the concept of never scaling you never learn it you're already relying so i think these are really important areas i highlight all those those box those three areas at a level of ume gme and cme these are the areas we need to study this is going to be quite an experiment in history as we go through the next few years. And so there are now published examples of T-scaling. There was a really interesting study. Did anybody read this or heard about it? I'm sorry, question about the last slide. And I'm sorry if I missed this, but is this arguing that?
something about generative AI is actually going to decrease the ability of learners to gain more proficiency and more skills if they don't use it. No, if they, so the point would be, if they rely on it, it can cause miss-skilling, de-skilling, or never-skilling. But also, there's an opportunity, and we talk about precision medical education, to enhance the learning if we can use it the right way. I think these are the fundamental questions we are asking the next generation, and if you train current physicians, what you're going to see is unintended consequences are going to happen as we rely on some of these AI tools. So this was a study that was in Lancet GI Hepatology. This is very interesting. In Poland, there was a four-center trial where they introduced an AI detection tool for polyps during colonoscopies. And what they did is for the three months prior to the introduction of the tool, they looked at the adenoma detection rate. Then they implemented the tool. And then after a washout period, then they studied those same gastroenterologists for three months. But then on even days, they didn't have the tool. And on odd days, they did have the tool. So they want to see how did those endoscopists do in adenoma detection on those days they took the tool away.
You see, the rate before the AI introduction, the rate of adenoma detection was 28.4%. Then it dropped by 6% to 22.4%. After several months of the AI tool being in place, then you took it away on certain days and asked them to diagnose the adenomas, it dropped to 22.4%. Now, we need a lot more data than this. This was about 1,500 patients total in the pre and post. It's one of the first potential clinical examples of de-scaling when relying on AI tools. And we know where AI tools for bronchoscopy occur. So again, it's something that we have to think about. When to introduce them, how to support, and what happens if they go away for some reason.
It's effective on you. Now, there's only about 15 endoscopists here. They're showing the delta from the pre and post. You can see the four people had similar or higher detection rate in the post period when they took away the tool, but the majority of endoscopists had a lower detection rate when you took that tool away. So again, that answer question is a hypothesis the authors add is maybe the real expert endoscopists were already at that adaptive practice phase, and it didn't bother them when the tool was away, and it didn't decrease their impact.
So again, early days of this sort of work, but studying and understanding, even learning curves and de-skilling of experienced proceduralists and physicians are important things to understand. So I want to get all that framework, and this is kind of an abbreviated version of the talk I'm working on, to talk about what we're doing with mechanical ovulation and education. So this is a waveform we show during the summer and winter education blocks, and this is a patient who later had a cardiac arrest. We show that for a reason. We talk about death and torture. And as a profession, I think those of us in critical care, we see an opportunity here with AI amplification, because it is not a skill that's traditionally been taught very well across the world, and it's not resulted in consistent expertise. And I'm going to walk through some data rather than just asserting that and moving on. But here's a patient you see an effective trigger after the first and third breath on that left panel. You see the flows not going back to the last breath. So this is a patient that we need to recognize having auto-beam is hypotensive and needs to be managed instead of saying they have sepsis or PE or something, as we commonly see.
And so look, the milestones, mechanical ventilation is an entrustable professional activity. That means that experts and society feel that the public entrusts a position to perform management of mechanical ventilation independently. But we don't have curricular standards on how to teach it, and we don't have standards for how to assess it. So that makes it very challenging to say, okay, you're ready to go and you can do this effectively. And so what do we do? Traditionally, what we do, whether you're in secondary school or you're in clinical education, med school residency, or fellowship, you have traditional training because you have a fixed amount of time to complete a program. And then clearly, that means there'll be variability in each learner's ability to meet those objectives in that fixed amount of time. And then the question becomes, for foundational concepts that are the bedrock of your profession, is that adequate? And again, these are things that could kill you. So I would argue, no, that traditional approach is not adequate to provide, say, patient care.
So then, you know, the first question comes up when we talk about this. And again, pushback is, is there any evidence that traditional training is not a cheap master? And there's not that much. And I say that one, especially we have a lot of fellows in the room. It's very hard to do a study when you assess faculty. So that usually doesn't happen.
So, but this is a study from 2011, an Italian study by Colombo and colleagues in critical care medicine. What they did, they looked at 10 experts, and that was defined as three years of an ICU attending versus 10 residents. And they had reviewed pressure time, flow time curves, even had esophageal balloons in to very carefully characterize these asynchronies. And what they found is that the practicing ICU physicians were able to detect one third of these. Now, that was statistically much better than the 16% of the residents. So, you know, you can look at a glass half full or empty, but I'm not very, I think most would be very pleased with only recognizing that.
They might say, you know, they do some crazy stuff in Italy, and maybe that's not the way it is. Well, actually, our fellow who's going to be attending, Roxy just had twins. She led our group's work on looking at faculty. And so we'll have faculty from around the country who want to come and see our course. And so their clinician educators, very motivated, invested faculty who want to bring that course to their institution. And we said, all right, you can come and sit in the back, but you have to take our test. And so that was the way we got data on faculty.
And so this is in peer review. It was published as an abstract in the Blue Journal. But they took the same five-question exam that you all may have memories of during the summer education block. And what you can see is the attending scored 24 out of 100, fellow 16, 17 out of 100. The p-value is 0.05, 0.2. So if you want to say that was not different or different, depending on your point of view, you can do that. And you can see here, reporting it, people scored 0, 1 to 10, and then all the way up to 100. So it's similar data 14 years later, now in the US, compared to what Columbus saw.
And so then the next question I'll hear from, but you know, does it really matter? I'm not killing people. I'm not saying anybody's killing anybody. But I am saying that we know, and this is, there's work, there's data from Biel and Blanche, that if you have a high number of asynchronies in these papers, they use an asynchrony index of over 10%. So more than 10% of breaths have an asynchrony. Those patients in these two studies once showed a longer duration of mechanical ventilation or higher incidence of tracheostomy. So we always look at that. Actually, those are very important endpoints. And then we need to be very careful to make those.
Yeah. What was used as the gold standard for whether someone detected to synchronize asynchronies or not? So if faculty, like only 24% or 33% of faculty came up to the right answer, what determined the right answer? So even in the Colombo study, so this was expert vetted review of the questions. And then in the Colombo study, there were 43 pressure time and flow time, and then the experts agreed on the correct answers. Good question. Sorry. But in this study, it's just looking at the number of asynchronies.
So I work with a group of colleagues on a review paper. So the question is, OK, what is the standard for teaching this? And there's actually not many studies out there. You can see there's only two studies on waveform analysis and patient ventilator dyssynchrony. And you can see in the lower box there that even these public curricula, they typically are less than eight hours, one day. And so basically, and you hear that anecdotally, most fellows are getting a series of few lectures and simulations over the course of their training. So I think we believe in our group that you have to train fellows in mechanical ventilation. And I think in every subspecialty you have.
or a thing that is fundamental. And you need more time than traditionally we provide that maybe six to eight hours mix of lectures and sims. So we should really take more of a mastery approach. And so if you're familiar with the concept of mastery learning, it's going to take a variable amount of time for each learner to meet. But what you said is you're pre-specified learning. You want to have all of your learners meet that. And you can't do that for everything. We're not going to have indefinite fellowships. But there are certain foundational things that begin all of what we do. That's very important.
And so you all know about our course, the two days, two boluses of education, four days in the summer, two days in the winter. We do this across the country. And again, that's much more time, probably six times the amount of time is the average fellow across the country. And then we looked at how learners who took our course did compared to traditionally trained. So what we did is we did that waveform exam before fellows, of course, in July and after, and before and after in January. But we also, we went to other centers, and we gave talks. We asked their fellows to take the exam too. And so we did that for 20 other fellows. So what you see here is that the traditionally trained fellow at other institutions on the left, you can see at the end of the first, second, and even at the end of third year, end of fellowship, fellows are getting about a third of the questions right. Again, similar to that Colombo paper number.
After the VENT course, fellows and you were typically first-year fellows are getting almost 50%, which is not bad. And we actually thought that's enough. Then you go to the bedside, you look at waveforms, and you continue to improve. Surprisingly, the scores went down by January. But then it got to 80% by the end course. We feel like that was really pretty good, that you got to a point where you feel like that saved practice. So we found that traditionally trained fellows did not develop expertise. A single course did not prevent decay. And then the second course was very important.
Now the question is, more time with space learning. Is that it? Is that adequate? And so he built up a preceptorial course. And myself, Mike Keller, a lot of us have helped iterate that through.
through the years. And the question he went back to, what he learned about, there's an Oxford preceptorial method. And so it seems like some of you are familiar with it. Basically, what you do is you go to the local pub, you have a pint with your preceptor, and you teach them the content. And so many people, many faculty have gone through this already. And the question for us was, OK, we've done that winter course, but what happens for the rest of the time? And a lot of fellowships are structured with like 18 months clinical and 18 months research. So what happens when you're a Nazi?
So we based this course, the precept total course, in addition to that 50 hours of training. And you all know, I think the chapters here, we go through, you have to pass one chapter to advance to the next. And then there's a very rigorous multi-hour final assessment with three objective preceptors. So it's quite an enterprise. But there are no surprises in the curriculum. It's all there. You get a reading material, the concepts and cases to be reviewed, and a checklist for what success looks like. They're all provided in advance to the learner. And so, again, if the learner hasn't prepared, it's going to be a long 90 minutes because the preceptor is just going to sit there. That has happened before.
So then we looked at Mike Keller, our back reader. We looked at the preceptorial group, and we compared them to the group that just took the summer and winter course. And what you can see here in the green, that group, remember, we finished the course at six months. But you can see over the three years of fellowship, their scores and their ability to diagnose as secretaries remained stable, whereas after six months, after the winter course, there was decay down to less than 50% of the rest of the groups. So, and remember, that's still higher than the people who are traditionally trained getting, say, six to eight hours a week. So when we look at deliberate practice and mechanical ventilation, I want to go through this kind of quickly because we're at time here. We know that fellows, like all of us, our skills will decay without use. And we know that fellowships are structured with large blocks of times that are non-clinical to work on research problems. So how can we scale? This is an incredibly labor-intensive program for both teachers and learners to a large group to learn. And so AI is kind of where we're very excited about this.
so we can conduct research on deliberate practice. Again, I think this can be done across different specialties or different areas of your interest. But the problem you have is it's very hard to scale because you need subject matter experts and you need excellent medical educators. That intersection is not that many people. And so one of the things we're trying to do, and Preet, one of our fellows, has gotten a very prestigious ATS, CHESS, APCCM, PE grant, $30,000 to do this project with collaborators we have at the University of Cincinnati. Use this tool called Two Sigma that walks through clinical cases with learners and provides individualized feedback.
And so they've never done this with waveforms. It's typically been text-based, walking through a case. They say what they're going to do. We provide a rubric instructor for what they get back. Now we're doing this with waveforms. And so this is something we are developing now. And the goal is to, you get certain questions wrong, you get self-directed feedback on resources for you to learn so we can scale this program. So again, the potential benefits here, again, I talk about this as a model for, I think, where this field is going. We talk about precision medicalization. You can certainly get enhanced feedback assessment. Learners want more and more assessments. And we don't share the assessments.
need to have a rigorous validated assessment. So that becomes a big problem. And we can also provide personal learning. If you're stuck here, we can help you. Here are some resources that you can go to to fix that problem. We have a high dropout rate. It's a lot of work, and it can be complicated. And I felt that the educators are just taking time out of their own .. So we find ways of crisis. So the scale is real awkward.
And so our goals are to create an adjudicated waveform database we're working on. We are close to develop launching that AI coaching tool. And then, you know, Sammy Safadi, one former fellow here, he's developed NetSim, that's a simulator. We're going to integrate the simulator for self-direct learning, and we're going to study the... So in conclusion, I think we see the promise of AI, but over-reliance on it, when you develop foundational skills, may prevent... Then expertise that people already obtain may decay with over-reliance. And so I think the opportunity here for us, we think about scale complex. But we need research as we study any best approach, because we don't know...
So sorry for going all the time. Let's stop there. Thank you.