Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say

Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say Source: https://arxiv.org/abs/2605.09195 Paper was published on May 09, 2026 This episode was AI-generated on May 12, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. Every hallucination detector we have fails at coin-flip accuracy on one specific kind of error: confidently wrong answers about facts that were true when the model was trained. A new paper argues this isn't an engineering miss — it's geometry. The staleness signal lives on its own axis inside the model, perpendicular to the directions current detectors are listening to, and a tiny linear probe can read it with ninety-percent accuracy. Key Takeaways: - Why temporal knowledge drift sits on a representational axis that's roughly orthogonal to both correctness and uncertainty — and what five convergent tests show about that independence - How the cross-cutoff experiment uses byte-identical prompts on differently-aged models to prove the probe is reading internal knowledge state, not properties of the question - Why retrieval circuits in the MLP layers produce nearly identical dynamics for stale recall and outright confabulation, which is exactly why confidence-based gating can't separate them - The deployment hole this exposes: at standard entropy thresholds, more than half of stale answers slip through, and many are more confident than the median correct answer - Where the paper's framing reaches further than its evidence — narrow Wikidata-shaped facts, mid-scale models, and a supervised probe that needs labeled drift data - The broader interpretability question the result raises: how many other useful signals are encoded inside models but never consulted at output time? 00:00 - The detector gap nobody noticed: Setting up the puzzle: every existing hallucination detector sits at chance on stale facts, while a simple linear probe hits roughly ninety percent. 03:20 - Three independent axes in the residual stream: The conceptual core — staleness, wrongness, and uncertainty appear to be three separable directions the model can vary independently. 06:41 - Null-space projection and the orthogonality evidence: Walking through the cleanest of five tests showing drift information survives even after scrubbing out correctness and uncertainty directions. 10:01 - Why the retrieval circuit can't tell the difference: Activation patching reveals that stale recall and confabulation produce nearly identical MLP dynamics, explaining why confidence-based detection is blind. 13:22 - Latent but activatable: the dormant gauge: Causal steering shows the staleness signal is wired up correctly and causally meaningful, but the model's default answer pathway doesn't route through it. 16:42 - The cross-cutoff experiment: Time-capsule twin models receiving identical prompts give different probe verdicts ninety-eight percent of the time, isolating internal knowledge state as the cause. 20:03 - Where the framing outruns the evidence: Pushback on dataset narrowness, model scale, and the supervised-probe requirement that limits practical deployment. 23:23 - A new taxonomy of being wrong: Why the lasting contribution may be conceptual — that 'wrong' is not a single kind of thing inside a model, and what that implies for future detectors. Recommended Reading: - The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets: The prior 'Geometry of Truth' work the episode credits for establishing the linear-probe move that this paper extends to a third axis. (https://arxiv.org/abs/2310.06824) - Detecting hallucinations in large language models using semantic entropy: The semantic entropy detector that this episode benchmarks as sitting near coin-flip accuracy on stale facts — useful for understanding what the new probe is being compared against. (https://doi.org/10.1038/s41586-024-07421-0) - Locating and Editing Factual Associations in GPT (ROME): The canonical activation-patching study of MLP-based fact retrieval circuits that underpins the episode's discussion of why stale recall and confabulation look identical inside the model. (https://arxiv.org/abs/2202.05262) - Discovering Latent Knowledge in Language Models Without Supervision (CCS): Introduces the contrastive probing approach the episode lists among the existing detectors that fail on temporal drift, and gestures at the unsupervised direction a follow-up drift probe could take. (https://arxiv.org/abs/2212.03827)