Evaluation level up : Measuring what matters

🎯 Evaluation! A make or break thing in the digital health. I sat down with Dr Shay Soremekun from the London School of Hygiene & Tropical Medicine to talk about the very hot topic of evidence generation and evaluation. Everyone's talking about LLM evals, technical performance, benchmarking. But ultimately people care about impact. Yet impact is rarely a neat, linear path to a yes/no answer. Anyone who's actually implemented something in the field knows about all the other contributing factors, the daily challenges and how hard it is to move the needle on big clinical outcomes with robust clinical evaluation.To understand why it improved care, if it did, was as important, if not more important, than understanding that it did. Because that will help us to understand how we need to potentially modify or adapt the intervention either in the same place or in future places to be able to achieve the same success. 🌟 Who will benefit from listening to this episode?Digital health companies : Founders, data scientists, AI/ML scientists, PMs, designers, clinical evaluation/study teams, cliniciansFunders : donors, investorsGlobal and public health professionalsImplementersResearchersRegulatorsShay explains the use of Program Theory and Logic models to visually connect your intervention, all the intermediate steps (not just technology, but people and change) to outputs, outcomes and long term impact.  Making space to observe other levers in the system you didn't initially anticipate. The work that Shay and her team have done with The Malaria Consortium and LSTM's Centre for Evaluation looking at a digital health tool for community health workers in villages and facilities showed this perfectly. Because they were intentional about observing the whole system, they discovered other factors contributing to impact and could redirect efforts accordingly.In this age of tightening budgets, and pressure to show clean shiny KPIs, how do you make room to observe these things? We also discussed decolonizing evaluation: capturing what's locally valuable, not just paradigms of success developed in comfortable offices and ivory towers.So many learnings here for understanding good co-design, implementation, and how to measure what matters.Chapters00:00 Intro04:15 Mozambique digital health tool for CHWs : context14:19 Tsking a system lens approach16:25 Measuring What Matters in Health Interventions23:02 Resourcefulness in Low-Resource Settings27:25 Challenges and Successes in Digital Interventions29:34 Proactive Measurement and Adaptation31:20 Scaling Up Successful Interventions32:59 Revolutionizing Evaluation Practices39:38 Incorporating Indigenous Knowledge in Evaluations44:57 Key Recommendations for Measuring What Matters 💡 Keep learning!If you found this episode helpful listen to:Episode 15 : Implementation 101 and how to Fail well with Caroline Perrin Episode 12 : Health First, Innovation second, with Smisha AggarwalEpisode 5 : What is the right approach for regulation and evaluation of digital health technologies?I have also written a Substack article summarising what I learned from this episode. Check it out for easy reference to the concepts outlined. Found it useful? Know someone who is struggling with this very thing? Share and keep raising the profile of people impacting underserved community globally. References and links to things we spoke aboutinSCALE papers:https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000235https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000217 Link for upSCALE  (roll-out of project):https://www.malariaconsortium.org/projects/upscalehttps://www.malariaconsortium.org/resources/integrating-upscale-into-the-ministry-of-health Study protocol for Uganda and Mozambique:https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0657-6 Links to Chilisa Bagele – one of the most foremost voices on decolonial evaluation methodology so a good start for anyone interested (talk at LSHTM):https://www.lshtm.ac.uk/newsevents/events/decoloniality-and-indigenous-methods-global-health-evaluation-professor-bageleCentre for Evaluation Website https://www.lshtm.ac.uk/research/centres/centre-evaluation About Dr Shay SoremekunDr Shay Soremekun is an epidemiologist and co-deputy Director of the Centre for Evaluation at the London School of Hygiene and Tropical Medicine.Her research in child and adolescent health and development covers trials of low-cost disease prevention programmes, and identification of risk factors and mitigation strategies for poor developmental and economic outcomes in this group. She is a member of the UK Government Evaluation and Trial Advice Panel (ETAP), and sits on the steering committee for the John Snow Society. She lectures at postgraduate level on topics of evaluation, epidemiology and public health, and has developed and organises an MSc module in Study Design. LSHTM page: https://www.lshtm.ac.uk/aboutus/people/soremekun.shay