Hallucinations in the Courtroom: Why We Can’t Trust AI with the Law

When a generative AI tool "hallucinates" a recipe, it’s a funny anecdote. When it hallucinates a legal precedent, people go to jail. In this episode, Kimberly and Jessica talk with Rebecca Fordon—law librarian, professor, and board member at the Free Law Project—to discuss why the legal system is uniquely vulnerable to AI hype. From the "duopoly" of legal publishing to the 500+ documented cases of AI-generated legal errors, we look at what happens when the law meets Large Language Models.Why This Matters Legal research requires 100% precision, but generative AI is built for probability, not fact. We explore the "Shifting Baseline Syndrome" in research: as we move toward a world where machines take the "first pass," how do we ensure we aren't settling for "80% certainty" in a field where mistakes have life-altering consequences?Key TopicsThe Hallucination Tracker: How legal professionals are documenting the 500+ (and counting) cases of AI-invented precedents.Privilege & Privacy: The hidden risk of waiving attorney-client privilege by feeding data into general-purpose LLMs.The Westlaw/Lexis Duopoly: How the Free Law Project is fighting to make primary legal materials accessible and transparent for the public.The Expert Pipeline Crisis: If AI replaces the "grunt work" of junior associates, how will the next generation of attorneys learn to think like lawyers?Certainty Amplification: Why the "confident" tone of AI is at odds with the strategic nuance required in legal advocacy.Notable Quotes"I’m a little bit worried that we might be getting to a place where, if AI can do it in a quarter of the time and get to 80% certainty, we might decide that’s 'good enough.' That really bothers me as an attorney." — Rebecca Fordon"In ecology, 'Shifting Baseline Syndrome' means each generation accepts a new 'normal' as the baseline. We are shifting into a world where the machine is the first pass, and the human is just an error-checker. That is a dangerous new baseline for research." — Kimberly Becker🔗 Featured Links & ResourcesRESOURCE: Free Law Project (CourtListener & RECAP)COMMUNITY: AI Law LibrariansBLOG: Musings about Librarianship by Aaron TayCRITIQUE: Refusing GenAICONCEPTS: KL3M (The "copyright clean" legal data model)Why shouldn't lawyers use ChatGPT for legal research? Standard generative AI models like ChatGPT are prone to "hallucinations," where they confidently invent legal citations, cases, and precedents that do not exist. In the legal field, using these outputs without verification can lead to sanctions, loss of attorney-client privilege, and significant ethical violations. Professional legal research requires domain-specific tools and human oversight to ensure 100% accuracy.Leave us a comment or a suggestion! Support the showContact us: https://www.womentalkinboutai.com/