Welcome, everybody. I'm filling in for Dr. Andrea Levine today, who couldn't be here with us, but we are really lucky to have our guest speaker today, Dr. Jonathan Elmer. Dr. Elmer is an associate professor of emergency medicine, critical care medicine, and neurology at the University of Pittsburgh Medical Center.
He received his bachelor's in biochemistry from Swarthmore and then his medical degree from Mount Sinai School of Medicine. After that, he completed his residency in emergency medicine at Massachusetts General Hospital and Brigham and Women's Hospital, followed by fellowship in critical care medicine and neurocritical care at the University of Pittsburgh. He's been at the University of Pittsburgh ever since, where he got his master's in clinical research and has been funded with two K awards, one from the NHLBI and one from NIMD. And today, the topic of this talk is awakening and self-fulfilling prophecies after cardiac arrest, a pretty mystical title. And we're all looking forward to hearing what you have to say.
Dr. Almar, thank you for joining us. It's all yours. Thank you for having us. Thank you all for joining. I should say there's probably about 40 or 45 minutes total of slides. So for those who feel like interrupting and asking questions, please feel free to unmute midstream and ask questions. Otherwise, there will be plenty of time for questions at the end. I am assuming based on the background I heard that I am speaking primarily to an inpatient critical care crowd. And so I'm going to frame the talk like that. Apologies if there is like a clinic doctor in the audience who's thinking that this does not apply to his or her clinic.
It actually probably does. I'm going to focus primarily on cardiac arrest because that's my research area of interest. But I think I will try to tie this more generally to critical care and prognostication in the ICU because I don't think anything that I say, except maybe a brief mention of specific prognostic modalities, really has anything specifically to do with cardiac arrest. So there was a long title in the intro. I tried to boil that down a little bit. So why does it matter? And it matters because we prognosticate a lot in the ICU for everyone. Sepsis and brain injury and pancreatitis and all these things. We predict outcomes routinely. It's part of what we do.
And hopefully I will persuade you by the end of this talk that that is a little bit of a perilous thing that we do. It needs to be approached with a little bit of caution. I have some disclosures, nothing financial or commercial, but I have some grants in spaces that are related to this. As a brief overview, we will talk initially just like about what prognostication is, try to define some sort of common terminology. So we're all speaking the same language. Turn a little bit to the specific example of cardiac arrest, highlight some ways in which our current approaches are not working.
for us as clinicians or for our patients and their families, and then talk about some paths forward. This was a slide that was developed by one of our neurology residents, actually, who went into neurocritical care a couple of years back. And I think sort of highlights visually in a nice way, like why this matters. This is sort of how I feel on rounds many days, right? It's like you walk around the ICU and you see the patient and you predict an outcome. And often that predicted outcome, particularly when it's poor, leads to a treatment decision to withdraw or...
limit or not escalate life-sustaining therapies. And when we engage in all the nice, fair decision-making with families, and we relay that prognosis, and they choose to withdraw life-sustaining therapies, our patients will die. And so although we don't typically frame it this way, I think our prognostication in the ICU really is sort of a life-or-death decision-making process. It's at incredibly high stakes. It turns out it's also quite common. This was a study that was done by a mentee of mine, Alexis Steinberg, when she was a fellow and junior faculty member. And she assembled a cohort.
patients who died in the hospital at multiple hospitals across the country. Some of these hospitals were large veterinary care centers like you and I practice at, some were small community hospitals. And she simply asked for a structured, simple chart review that said, you know, when these patients died, was there a decision to withdraw life-sustaining therapies? And if so, what informed that decision? And what she found, I think, was quite striking. More than half of inpatient deaths were preceded by a decision to withdraw life-sustaining therapies. That's probably not so surprising. I mean, I think we all kind of have that sense in the ICU that...
It is more common that there is a planned death and an unplanned death. We sort of view it as a failing when patients die despite maximal support. We don't necessarily do it as such. Maybe we even do it as a victory when patients die after a thoughtful decision to withdraw life-sustaining therapies. I think a little bit more surprising to us was that withdrawal for perceived poor neurologic prognosis was about half of those cases or projected out nationally about 150,000 deaths in the U.S. each year. And strikingly, many of those deaths for perceived poor neurologic prognosis occurred in settings.
where there was no neuroprognostic expertise. There was no neurologist. There was no intensivist. There was a hospitalist who maybe didn't do any diagnostic tests at all to formulate a neurologic prognosis, rendered a dismal prognosis, and then had a treatment decision to withdraw life-sustaining therapies. So all of that to say, I think prognostication in the acute care setting, in the ICU setting where we live, is high stakes. It's life or death. It happens a lot and most often is driven by a neurologic prognosis as opposed to an overall assessment of multisystem organ failure or comorbidities or all the other reasons why we might withdraw life-sustaining therapies.
Okay, a brief review of what we've all seen probably many times at this point. How do we evaluate a test, a diagnostic strategy, a prognostic strategy? Typically, we use two by two tables. On the rows here are what are my prediction? What do I think the patient will do? On the columns here are what is my actual observed outcome? This is like the ground truth that you would use, you know, if you had an EKG and wanted to predict acute MI, you have this tool that you're evaluating, you have the ground truth outcome. And in cardiac arrest and most acute brain injury research, we have this sort of goofy framing of it.
which is grounded in the fact that historically we have focused on predicting poor outcomes as opposed to predicting a favorable recovery. And so we consider a true positive case to be a case where you think that they have a bad outcome and they actually have a bad outcome. And a false positive would be a false prediction of poor outcome and so on and so forth. So there's a couple of quantities that we think about when we're evaluating a prognostic test or a diagnostic strategy. One of them is the false positive rate, which is sort of akin to like the specificity of the test predicting poor outcome. But what that boils down to is like the proportion of patients.
who ultimately have a good outcome, whatever that means, you can talk more about that later, who are falsely predicted by your prognosis, by your test result, by your algorithm, by your ML tool, by your whatever, to have a poor outcome. Now, I already said, and I think this sort of resonates with you, that often when we predict a poor outcome, that leads to a decision to withdraw or withhold life-sustaining therapies. And so typically we aim to have a very, very low false positive rate because framed another way, a false positive rate basically says what proportion of patients who would have recovered Had you continued therapy, are you comfortable allowing to die comfortably based on your false prediction of poor outcome? And we're not in person, so I won't sort of ask for a show of hands, but I will ask you to sort of think to yourselves, you know, what percentage of the ICU patients you round on who are dependent on life-sustaining therapies, are you comfortable allowing to die when they otherwise would have recovered because you thought that they probably weren't going to do well? I think, you know, we have some data that suggests that for most providers internationally, that number is somewhere between like a tenth of a percent and one percent. You know, that means like maybe every week or two you're rounding.
It doesn't happen, but every month or two year rounding, it happens maybe once. And obviously the motivation for that is not that we want to withdraw life-sustaining therapies from patients with recoverable problems. It's that there's also a cost of never withdrawing life-sustaining therapies, which I don't need to describe to you. You know it well. And we obviously can't trach and peg and send to a long-term care facility all hundred plus thousand people who currently have withdrawal of life-sustaining therapies every year in the United States. It would break the healthcare system. It would be incredibly undignified, costly, emotionally burdensome for families. There's lots of reasons not to do it. Okay. What about the flip side of that?
The other thing we care about sort of quantitatively is the sensitivity of our test, which again would be essentially the proportion of those who will go on to have a bad outcome that your prognostic test or your tool correctly identifies. So what makes a good test? I think this, again, will resonate with this audience who lives in the ICU. Well, earlier is better. If you have someone with a traumatic brain injury who comes in and you can predict accurately that that injury is irrecoverable on the first day, as opposed to a month later, it can have some benefits for the patient and for the family and for the health system. It saves money and suffering and emotional anguish for families and all of these things that we try to avoid. But you also want it to not falsely predict poor outcome for the reasons we talked about. You want it to be reasonably sensitive and you want it to have pretty high confidence.
You know, if you sit down with a family and you tell them, well, I saw this twice before and it didn't go well, so I don't think it's going to go well. Yeah, your point estimate might be 0% for recovery potential, but having seen it twice before doesn't give you a lot to go on compared to when you sit down with families and you say, well, we've seen this hundreds or thousands of times before and it's never gone well. You can make a much more confident prediction. That's like a plain language conceptual explanation of like what a confidence interval looks like and what it conveys. So pivoting a little bit to cardiac arrest as a specific example of a severe acute illness that I focus on, but I think you could substitute again trauma or sepsis or anything else here in this slide and come up with the same conclusions. You all know you live in this world. Cardiac arrest is common and deadly. 600,000 plus cases in the U.S. annually. It's a leading cause of death and disability. And among those who survive the hospital care, only about 40 percent will be alive at hospital discharge. As you know, well, most patients who are resuscitated from cardiac arrest are initially comatose.
Some will wake up and enjoy a favorable recovery. Others will not. And the tools we have to tell the difference between those two groups, those with recoverable and those with irrecoverable brain injury on arrival, are poor, especially in the first days after cardiac arrest. In pictorial form, four in ten, again, same statistic, different way of presenting it, will recover. Those are the green people.
Of the 60 or the six in 10 who will not recover, two will die from things other than withdrawal for perceived poor neurologic prognosis. Those are the patients who re-arrest, who die from multi-system organ failure, who had an advanced directive to discover that they would never want critical care or progress to death by neurologic criteria. So that's what the breakdown looks like multiplied out by many tens of thousands of patients. As you know, because you use these tools, our tests are imperfect. We try to piece together a prognosis based on a combination of factors, clinical exam, MRI, biomarkers, SSEPs, EEG, all of these things. Time, we observe, we repeatedly reassess. And none of these tools in isolation has those performance characteristics that I described that one might want for a prognostic test. The low false positive rate, the reasonable sensitivity, the timeliness or the confidence. It doesn't work. And so one thing that we try to do to overcome the limitation of individual tools is we piece them together. So this is an algorithm.
for how you might combine neuroprognostic tests after cardiac arrest. This is from the European Resuscitation Council. U.S. groups have similar things. And so we hope by getting multiple diagnostic tests that we can increase our confidence in a poor outcome, that we can lower the false positive rate, the false prediction of poor outcome. Unfortunately, that also comes at costs like the sensitivity or the timeliness of the test. And so this ends up kicking the can down the road by several days at a minimum. And really only 30, 40, maybe 50% of patients overall meet these criteria.
and end up in that green poor outcome likely box. At least half, if not more, of patients end up in this totally unhelpful box of observe and reevaluate. So how are we doing today? Not very well. I'll give you a couple examples, I think, to try to convince you of that point. These are data from a large claims-based study that we did here using the Optum data set. Optum is an all-age, all-payers database that about a third of U.S. hospitals contribute to. We found about 35,000 patients who were resuscitated from cardiac arrest who died in the hospital. And we just looked, you know, who had what sorts of prognostic care that was rendered. And it turns out hospitals are very good at submitting claims for tests that they do because they want to be reimbursed.
This was a pretty dismal result, I think. One in 10 had a head CT. One in 20 saw a neurologist. Only 3% had an EEG. One percent had a brain MRI. So most of the people who were dying in the hospital had little or no neurodiagnostic testing that was done. Now, I already showed you data that of deaths, most are for perceived poor neurologic prognosis. Not all are, but most are. So already that's sort of alarming. We tried to call that a little bit further and refine the cohort.
and say, well, how would we take out patients who died from multisystem organ failure, who rearrested in the hospital in an unplanned way and couldn't be resuscitated, who progressed to death by neurologic criteria? So we did a sub-study that looked at those who survived at least two days in the hospital and never got a vasopressor, never had a bolus of vasopressor that looked like ACOS, never had a vasopressor infusion. You know, you all know you take care of patients who herniate routinely. They typically have a period of hemodynamic instability. Certainly those who died from multisystem organ failure will typically have pressors. Again, it's claims-based. It's not perfect, but the numbers are the same. Even for those who were hemodynamically stable, who lived for several days in the hospital, but then died.
Very, very few, a small minority, less than 10% had prognostic testing of any kind, less than 2% had guideline concordant multimodal testing. So that I think should alarm us that, you know, usual wild type care in the US, our post arrest patients and probably our brain injured patients in general are dying without guideline recommended prognostication. There's a bigger problem, though, which will inform our discussion for the rest of this talk. When we evaluate a prognostic test or tool or the association of some clinical factor with outcome, let's say I want to know the probability of recovery if a patient has had unwitnessed asystole, as a specific example. You know, that's the quantity that we're interested in, the probability of the outcome, why, given whatever it is, the test characteristic or the algorithm or the clinical finding or whatever else that I have observed clinically.
I think that makes sense. You see a patient, you understand what's going on clinically with them. You want to know what their probability of recovery will be. Unfortunately, what we actually observe is the association of the outcome of interest, given those clinical characteristics, and then a myriad of treatment decisions, which, as we've discussed at length already, includes withdrawal of life-sustaining therapies. So the risk you run is that you see that clinical characteristic X, say, unwitnessed asystole.
You conclude as a clinician that that's a bad prognostic sign based on your beliefs, the literature, etc. You recommend withdrawal of life-sustaining therapies, which is deterministic of that outcome. You recommend that the patient has withdrawal of life-sustaining therapies, the patient dies. The observed outcome becomes poor, and you're not really able to understand the relationship of that clinical characteristic with outcome by itself. Instead, what you can understand is the characteristics of the clinical characteristic.
or the association rather of the clinical characteristics and the treatment decision with outcome. So this is something true. This is not just true of cardiac arrest. This is all ICU literature. This is true of the Apache scores, of the ICH scores, of all of our risk stratification tools, of every prognostic test. They are all affected by this phenomenon, which creates this risk of self-fulfilling prophecies. The doctors see a sign or a symptom or a test result. They think it's bad. They modify their care based on that information, which that's just good medicine, right? Like that's why we exist in the ICU is to interpret data and to act on it.
But we then bias the results that the patient actually can experience. And then we go back and we study that cohort and we see these biased associations and it just gets perpetuated. It's a self-fulfilling prophecy that's propagated forward, both in the literature and also internally in us and our cognitive heuristics through which we make the vast majority of our clinical decisions. It creates these self-reinforcing behaviors in us and self-reinforcing literature. This is not just a theoretical problem. So this is a study we did that looked at patients in the Resuscitation Outcomes Consortium, 251 hospitals in the U.S. treated after cardiac arrest over 10 years. We looked at the decedents. And again, the most common day for withdrawal for perceived poor neurologic prognosis was the day that the patient was admitted far earlier than we can reliably prognosticate, followed by the second day, followed by the third day. That dashed line there is the soonest the guidelines think you might possibly be able to prognosticate. So most treatment decisions were made before accurate information were available.
those patients look a lot like patients who actually do okay. So overall survival in this cohort was like 35, 40%. 25% predicted survival for those who had withdrawal of life-sustaining therapies, had life-sustaining therapies not been withdrawn. So we're probably affecting a very large number of our observed outcomes with our treatment decisions. This is a different look at the same phenomenon. These are the last thousand patients who presented comatose after cardiac arrest at my hospital, and just a bar plot of the day that they awakened from coma.
It's rare for folks to wake up the first day, even though that's the day that we are most commonly withdrawing life-sustaining therapies. And 44% of those who eventually go on to awaken will do so after the guidelines recommend you can make a treatment decision in some cases. So again, our treatment decisions affect our ability to observe real ground truth, credible outcomes. So that sort of begs the question, so what do we do about this? I think the sort of knee-jerk response that most people have initially is, you know, well, we probably shouldn't learn from these cases. So why don't we, for example, do cohort studies in countries where withdrawal of life-sustaining therapy is either viewed as culturally unacceptable or as illegal? And, you know, I think you all know of countries internationally where this is true, parts of Italy, Japan, Korea, Brazil. I think my problem and sort of collaborated with colleagues in all of these countries with this strategy.
is that there are still treatment decisions that occur. These countries have found solutions to this quote-unquote problem of continuing long-term care in the face of irrecoverable injury. So in some of these countries, it is not legal to withdraw life-sustaining therapies, but you can see someone in the emergency department, you can look at their head CT, shows a big bleed, they're intubated already, you can't pull the tube out, but you can admit them to a non-monitored floor bed and then go back and check on them a couple of days and see how they're doing. You know, that's an example. You know, again, I say that as a valueless description.
of a potential system of care. I think the ethics of that are deeply rooted in local culture and values. But those sorts of treatment decisions turn out to be very, very insidious and hard to measure. They can be as deterministic of outcome as an actual decision to withdraw, but they're rarely recorded in the data sets that we can analyze. So another thing you could do is you could try to enroll everyone in a prospective trial where you get them to sign on the dotted line or their proxy, I suppose, when they roll through the emergency department with a bad problem and you say, you know, we'd like you to sign up for this trial where we will never withdraw life-sustaining therapies. This will let us observe the natural history of your loved one's illness. That raises, I think, obvious concerns, at least in the U.S. It would be extraordinarily costly. I think that it would be difficult to convince families not to opt out of the trial after a certain period of time. And the patients and families who said yes to that trial, I do not think.
would be reflective of the patients we see in our routine clinical practice. There'd be this huge selection bias that would limit your ability to draw conclusions. You could also exclude cases where there was withdrawal of life-sustaining therapies from your development of an algorithm or a model or your evaluation of a prognostic test. And there's two problems with that. One, again, these cases are systematically different, so your conclusions probably wouldn't be generalizable to the overall cohort. The other bigger problem is that we do this research not for the sake of science, but because we want to develop a new tool that we could use prospectively, something that would be clinically useful.
And when you're creating your cohort based on some future event, and then you try to figure out if it applies to the patient in front of you, without knowing the future of what will happen to your patient, you can't really figure out if the data apply to your patient. So it really sort of prohibits prospective use. If there are statistically minded folks in the audience. Causal inference is a frequent answer that people initially say. Causal inference is a way of trying to disentangle the relationship, in this case, the association of the clinical finding with outcome from the treatment effect or the confounders. And one of the ways it does that is to try to sort of reconstruct the counterfactual. What would the probability of the outcome have been had the treatment decision flipped? And the reason you can't do that here is because the withdrawal of life-sustaining therapy's treatment decision is deterministic of outcome. And you can't predict something statistically if it's never observed.
So you have to have both kinds of outcomes with both sorts of treatments in order to do that. That's called a common support or the positivity assumption. And you can't actually do the math to do the causal inference when there's a deterministic or an invariant relationship of a treatment with the outcome. One thing you can do, which was, I think, embedded in the title of this talk, is you could censor data after withdrawal of life-sustaining therapies, doing something like a survival analysis. So you say, well, I watched this patient for three days. That patient didn't recover in those three days. Then I withdrew life-sustaining therapies. I'm very confident in what I observed, that the three days nothing happened. And I'm just going to say, I don't know what would have happened after I withdrew life-sustaining therapies. I'm going to treat the ground truth outcome in that case as censored. This is like when you read an outpatient trial of, say, breast cancer that's looking for recurrence.
When someone is lost to follow up in that longitudinal trial, they're not treated as recurrence. Yes. Or recurrence. No, they're treated as lost to follow up and censored. And to analyze those sorts of data, we use survival analysis. So we did that in this paper. This was a cohort study that was conducted at two large academic medical centers here in Pittsburgh and University of Alabama. And, you know, this study started with, I think, a couple of fairly straightforward suppositions that I think, given what we've talked about so far, will resonate with you.
We believe that we probably could learn something from observational data, even in the face of withdrawal of life-sustaining therapies, that we shouldn't throw up our hands and say, all hope is lost, give up research, it's too hard, don't try, but that we would need to take some particular precautions. And specifically, the precaution we took is by considering death after that treatment decision, that decision to withdraw life-sustaining therapies as an unreliable outcome. We said it would not be a credible ground truth. We don't know what would have happened to those patients. But there are other patients, like those who died despite full support, progressed to death by neurologic criteria, or recovered.
where there were credible observations. And moreover, in cases of withdrawal, it's not that we can't learn anything from them. It's just that we don't know what their eventual outcome would be. We still know what happened during the period of time while they were under observation. So this was less of a sort of clinically useful study and more of a proof of concept, I think. And so the hypothesis we tested is that for patients at high risk of withdrawal of life-sustaining therapies, the sort of usual sort of statistical models that take that poor outcome as reliable would be systematically more pessimistic than models that censored that true outcome and just said, we don't know for those cases.
There were about 1,250 patients who we analyzed, 900 or so at our hospital, 300 or so at Alabama, were comatose and had all the sort of usual post-arrest care, hypothermic temperature control and EEG and multimodal prognostication. We extracted, I can't remember, something like 200,000 early clinical features from their health record and from their quantitative EEGs. And there's sort of three key models that I already alluded to. One would be the typical bread and butter statistical model, which comprises the vast majority of the literature in this space, which would be a logistic regression predicting a binary outcome. Either they got better or they didn't get better.
One would be the survival analysis, which would say, you know, how long did it take them to get better? And then censored the observation of the decision to withdraw life-sustaining therapies. And then the last simply predicted how high risk is a particular patient at withdrawal of life, of undergoing withdrawal of life-sustaining therapies. This is sort of the slide for the analytically minded people that shows how we broke things up and did training and tests and whatnot, but probably not of super interest in a clinical audience. What was interesting is both models performed pretty well. Both the traditional regression and the survival analysis had a pretty good area under the curve.
um averaged across the test set actually an identical area under the curve several decimal places um but when you had patients um who were at high probability of withdrawal of life-sustaining therapies over here the survival analysis was systematically more optimistic by a couple percent than the logistic regression when you had patients at low probability the reverse was true um and this was statistically significant. Now, I think the question I would ask if I were in the audience looking at this spot where the line's not quite flat, but it only goes up a couple percent, is like, so what is the difference of a couple percent in your outcome prediction you mean? And I think in this context, at least in this disease state, a couple percent makes a big difference. This is a graph of the outcome probabilities of a different but related cohort of patients based on clinical data observed through the first two hours post-arrest.
And what you see even two hours post-arrest is there's this big clumping of patients who are at particularly low survival probabilities. I guess this was awakening and survival in this study. When you go out to six hours, that becomes much more dramatic. And suddenly a good third or more of your cohort is here in this less than 5% recovery potential bin. And so when you move your prediction or your threshold for a treatment recommendation or decision, even a couple percent one way or the other in these patient populations, you end up with very large swings in things like the sensitivity and the false positive rate. So not in every study, but in prognostic studies of cardiac arrest care, the difference of a few percent in your outcome prediction can make a vast difference.
in the performance of your overall model. So hopefully I've convinced you by now that we prognosticate a lot. You knew that already before you came to this talk, that it's really high stakes. You knew that already too, that the tools we have are pretty poor and there's this significant risk of self-fulfilling prophecies, which bias our ability to interpret the literature, bias our clinical decision-making, and ultimately probably are costing patient lives. So I think this brings me to the last thing I want to talk about with the group, and that is an ongoing project we have to try to create methods that are better. And so I'm going to start again with a couple sort of broad concepts that I think will resonate with you. Number one, I've been pretty down on clinical decision making so far, but I do think that we, you and I, as clinical providers, as doctors, we're mostly pretty good at what we do, right? Like when we prognosticate and we withdraw, we integrate a lot of information and a lot of clinical expertise and a lot of biomedical knowledge.
And most of the time, our conclusions are probably okay. Not always. That's a problem, but most of the time. I also think that many patients from whom we withdraw life-sustaining therapies had sort of universally, obviously recognizable irrecoverable organ failures. So in the case of cardiac arrest, this is the patient who comes into the ED. They have loss of gray-white differentiation on their head CT, sulcal effacement, effacement of the basal cisterns. The read comes back that the CT shows herniation. They, you know, like maybe have a little infracranial flow if you chose to look for it.
But this is someone who clearly everyone would agree would progress to death by neurologic criteria if supported. You talk to the family for whatever reason. They don't want to wait for brain death. They withdraw life-sustaining therapies. That patient, I kind of feel okay learning from that outcome, even though there was a treatment decision to withdraw life-sustaining therapies. The problem is that some patients also seem like they're recognizable as potentially recoverable. And telling the difference between those two patients is something that our current tools can't do.
So our premise then is that if we incorporate expert knowledge, that's number one, that's the doctors who are mostly pretty good at what we do. into our algorithmic predictions, this may be a route towards breaking the cycle of self-fulfilling prophecies and developing decision support tools, prognostic algorithms, et cetera, which perform the way we want them to, which are accurate as opposed to simply learning to replicate the errors that we as a specialty have historically made. So the NIH is supporting this work. It has a sort of goofy acronym of CORCA. I am like notoriously bad at coming up with catchy acronyms for my studies. But what we're trying to develop is...
some sort of interpretable and trustworthy algorithmic approaches to outcome prediction from the really sort of complex time series data that we generate through our usual clinical care. Meds and vitals and EEG and image dicoms and all this stuff. And as I've already sort of alluded to, we'll treat those patients who get better or those who die despite maximal support as sort of credible ground truth outcomes that we can learn from. And in all the other cases, cases where there was a treatment decision, whether that was a treatment decision to limit or withhold cardiopulmonary support in the case of, say, worsening shock. or re-arrest or to withdraw life-sustaining therapies based on a perceived core neurologic prognosis. For all of those cases where there were treatment decisions that may have affected the outcome, we are reviewing them in exhaustive detail with at least three experts internationally in post-arrest care. And these are like really sort of the who's who of post-arrest prognostication. And we go in exhaustive detail. They have full access to all the available clinical information, everything that the doctors and nurses knew when they were taking care of the thing. They can look at the images. They can review the EEGs. They can ask questions if something wasn't clearly documented. They have an opportunity to discuss, to dialogue, to do whatever else.
And then we ask them to independently estimate, absent the treatment decisions, what would have happened with that patient? What would that counterfactual outcome have been, the one that can't be estimated statistically, but which doctors can try to reconstruct using their knowledge on the clinical scenario? We are nearly done. We've been doing this for about 18 months, 1,400 cases. We're reviewing over many hundreds of hours with these experts. We have another month or so to go.
And the sort of high-level sneak peek at the results to date, in about a third of cases, experts agree that the prospects were dismal. This was a little lower than we were expecting, but not vastly lower. In about 15% of cases or so, experts agree that there was a lot of recovery potential. And a lot of those are cases where, you know, you've seen the family and you say, gosh, I think that they could go home eventually, but they'll probably have some medical needs and they'll probably leave here for a nursing home. And together with sort of appropriate shared decision-making, family decides that that would not be an acceptable outcome. And so we withdraw.
I think these are rarely cases, I hope, although I'm biased because these are mostly cases that we've treated here. These are rarely cases where the doctors just ran amok. It's just all of these sort of other intangibles that lead to an observed poor outcome in a case where doctors actually think that the patient might have been able to recover, at least to some degree, with a different treatment decision. So what about those other half of cases? Half of cases our experts don't agree. Some think that there's a lot of recovery potential. Some think that there's very little. Most of those cases of disagreement are attributable to some serial optimists.
everyone on this call either is one or knows one. We have these clinicians who tend to be eternal optimists. They see things that everyone else feels bleak and nihilistic about, and they say, you know, we got a chance here. Let's push forward. I think that was reflected in our data. This is one visual look at those sort of respondent-level behaviors. So the plot on the left of your screen is how often was this individual expert, each column here is an expert, how often were they more pessimistic?
or more optimistic than the median consensus of the experts on the case. And I don't know who's who. This is all blinded. It could be me. It could be someone else. We got one sort of serial pessimist who 75% of the time is more pessimistic than the average. And then these two sort of serial optimists who more than 75% of the time are more optimistic than the average expert at the level of the case. And this is just the magnitude of the deviation. And again, you know, this pessimist is like two standard deviations away from the cohort. And these two optimists are two standard deviations away from the cohort.
This, interestingly, I think this is more of a reflection on like decision science and psychology, but there's really sort of only two phenotypes of doctors, at least in our expert study. There's the ones who are pessimistic and very confident that they're correct, and the ones who are optimistic and very uncertain that they are correct, which I don't know what to do with, but I think is sort of an interesting commentary on how we think about prognostication. So I think then the question becomes like, so what do you do with this data? Each of these clinicians, each of these researchers, is a true world expert in post-arrest prognostication. They wrote the guidelines, they did the studies, they ran the trials, they do this in their clinical practice. And in a lot of cases, they disagree.
In fact, when you look at individual predictors across respondents, you know, there's some like degree of cerebral edema here. You don't need to be able to read the chart to sort of get the message where they all agree it's probably bad to have cerebral edema. But then there's like some EEG findings, initial lactate, your lactate clearance, where the experts are all over the place. You know, some view the bad EEG findings as very informative of poor prognosis. Others think they're uninformative. Some think the higher your lactate is, the worse it is. A bunch don't think it really matters that much in this patient population. You know, our experts have really fundamentally different worldviews, not just that sort of optimistic, pessimistic look.
but also the broader sort of framing of how they incorporate clinical information and diagnostic test results. So what do we do with that then? This is where we'll wrap up and sort of pivot to questions. You know, I think one thing you could do is sort of like the wisdom of crowd stuff in the ML literature, where you could just try to figure out what is the average expert opinion? And you could assume that there are some cognitive biases that make some of us more optimistic, some of us more pessimistic. And really, those are just our own little foibles and risk aversion and things like that. And so what you really want to do is just figure out like, what does a middle of the road common sense expert think? And average out all those other sort of differences as white noise. The problem is when you start trying to average together.
decisions or models that have fundamentally different worldviews. It doesn't totally make sense. You know, by analogy, there are six big models for predicting what the global temperature will be 100 years from now. And they have these, again, sort of fundamentally different assumptions. You know, we'll either make a great leap forward in battery technology that allows airplanes to have batteries and all vehicles to be electric, or we won't. And if you average those two models together, you don't get to anything that's sort of necessarily more sensible. What you care about is like, what are the spread of values across which?
our predictions are robust despite these uncertainties, these different worldviews, these different assumptions. So in econometrics and forecasting, that's been sort of handled in a couple of ways. We don't need to sort of get into the details here, but basically the way to think about it is, you know, you make a model that understands my decision-making and then you make a model that understands Dr. Morris's decision-making because he's the first box in the corner here and then the next expert and then the next expert and then the next expert. And then what you return to the clinician at the bedside is this is sort of the range of what experts would think in this case.
Maybe this is sort of the interquartile range. You don't care about those extremes. But this is kind of the range recognizing that experts have legitimately different worldviews and belief about the significance of particular prognostic tests. There are modeling strategies that are kind of in the middle. Conformal prediction is one. I don't know that this audience is especially interested in sort of the nuts and bolts to kind of take sets of information and predict sets of information. It's like a hybrid of those two. This is stuff we're working through now and are going to spend sort of the next three years of this grant trying to work out on an analytical perspective.
Hopefully I've convinced you, I've been trying to hammer home the point repeatedly, maybe ad nauseum, that prognostication in our world in the ICU, both in brain injury and otherwise, is incredibly common. It is the highest stakes decision we make. It is literally life or death. That good prognostic tools and strategies should be unbiased. They should not perpetuate these self-fulfilling prophecies that our current tools do. And they should be accurate and they should be timely and they should do all the things that we would want in a diagnostic test or a tool.
that our current practice, our current standard of care is far from any of these metrics. It is none of these. And so together as a scientific community, we have to identify a fundamentally different approach to associating our clinical observations with eventual outcomes that stops perpetuating these self-fulfilling prophecies over and over again, both in the literature and in our models, but also cognitively and in our heuristics and our decision-making internally so that we stop making the same mistakes over and over again in our practice without recognizing it. So I'm going to stop there. And I think we have a few minutes for questions.