#40 - How Two Fake Medical Papers Tricked AI

What happens when fake science looks real enough for AI to believe it? “Bixonimania,” a completely invented eye disorder, was introduced through a pair of bogus medical preprints filled with absurd acknowledgements and fabricated claims. It should have been easy to dismiss. Instead, chatbots began repeating it with confidence, describing symptoms, risk factors, and even suggesting users see an ophthalmologist. When health information is only a prompt away, a polished falsehood can quickly become a real problem.We unpack why this hoax was so effective. The papers mimicked the tone and structure of legitimate scientific writing, preprints carried the appearance of credibility, and online systems rewarded fast answers over careful verification. We compare how clinicians and attentive readers catch inconsistencies, missing context, and obvious warning signs, while large language models process text differently. Because LLMs are built to predict likely sequences of words rather than confirm truth, they can turn something obviously fake into something that sounds entirely plausible.From there, we widen the lens to the broader challenges of AI safety and AI security in healthcare. From data poisoning to prompt injection to the feedback loop created when AI-generated content reinforces other AI-influenced material, the risks extend far beyond one invented diagnosis. This episode explores why trustworthy AI depends on more than technical performance alone. It requires human oversight, stronger vetting of what enters the information ecosystem, and real accountability for what gets published, amplified, and repeated.Reference:Scientists invented a fake disease. AI told people it was realStokel-WalkerNature News Feature (2026) Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com)Licensed under Creative Commons: By Attribution 4.0https://creativecommons.org/licenses/by/4.0/