Why Language Models Hallucinate

This new OpenAI paper explores the phenomenon of "hallucinations" in large language models (LLMs), where they generate plausible but incorrect information. The authors attribute these errors to the training and evaluation processes, arguing that these systems are rewarded for guessing rather than admitting uncertainty. They propose a statistical framework that connects these generative errors to misclassification rates in binary classification, suggesting that hallucinations are a natural consequence of current training objectives, even with error-free data. Furthermore, the paper highlights how post-training evaluations, often using binary scoring, perpetuate hallucinations by penalizing expressions of uncertainty, effectively keeping LLMs in a "test-taking" mode. To mitigate this, the authors advocate for modifying existing benchmarks to explicitly incorporate confidence targets and credit for acknowledging uncertainty, rather than solely introducing new hallucination-specific evaluations.