#6 - AI Chatbots Gone Wrong

What if a chatbot designed to support recovery instead encouraged the very behaviors it was meant to prevent? In this episode, we unravel the cautionary saga of Tessa, a digital companion built by the National Eating Disorder Association to scale mental health support during the COVID-19 surge—only to take a troubling turn when powered by generative AI.At first, Tessa was a straightforward rules-based helper, offering pre-vetted encouragement and resources. But after an AI upgrade, users began receiving rigid diet tips: restrict calories, aim for weekly weight loss goals, and obsessively track measurements—precisely the advice no one battling an eating disorder should hear. What should have been a lifeline revealed the danger of unguarded algorithmic “help.”We trace this journey from the earliest chatbots—think ELIZA’s therapeutic mimicry in the 1960s—to today’s sophisticated large language models. Along the way, we highlight why shifting from scripted responses to free-form generation opens doors for innovation in healthcare and, simultaneously, for unintended harm. Crafting effective guardrails isn’t just a technical challenge; it’s a moral imperative when lives hang in the balance.As providers eye AI to extend care, Tessa’s story offers vital lessons on rigorous testing, transparency around updates, and the irreplaceable role of human oversight. Despite the pitfalls, we close on a hopeful note: with the right safeguards, AI can amplify human expertise—transforming support for vulnerable patients without losing the empathy and nuance only people can provide.Reference:National Eating Disorders Association phases out human helpline, pivots to chatbot Kate Wells NPR, May 2023An eating disorders chatbot offered dieting advice, raising fears about AI in health Kate Wells NPR, June 2023The Unexpected Harms of Artificial Intelligence in HealthcareKerstin Denecke Guillermo Lopez-Compos, Octavio Rivera-Romero, and Elia Gabarron Studies in Health Technology and Informatics, May 2025Credits: Theme music: Nowhere Land, Kevin MacLeod (incompetech.com)Licensed under Creative Commons: By Attribution 4.0https://creativecommons.org/licenses/by/4.0/