One in Four NeurIPS Papers Cites a Reference That Doesn't Exist

One in Four NeurIPS Papers Cites a Reference That Doesn't Exist Source: https://arxiv.org/abs/2607.00738 Paper was published on July 01, 2026 This episode was AI-generated on July 6, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A Microsoft team audited 2.5 million citations across four top AI conferences and found phantom references — works that simply don't exist — scattered through as many as one in five accepted papers. The twist: peer review is structurally blind to them, reviewer scores carry zero signal about them, and the fix costs about four cents a paper. Key Takeaways: - Why 'under 1% of references' and 'one in five papers' are the same dataset — phantoms scatter one per bibliography, below any reviewer's detection threshold - How RefChecker's funnel design (six catalogs first, caged LLM last) audits an entire conference for about $157, or four cents a paper - The sharpest number in the paper: ICLR 2023 accepted vs rejected papers had nearly identical phantom rates (16.0% vs 16.9%) despite a two-point gap in reviewer scores - Why the quotable 'one in four' figure is the least reliable one, and the defensible number is 5.1% of NeurIPS 2025 papers carrying two or more phantoms - How authors traced their fake citations to LLM tools that polished fuzzy memories into pristine BibTeX — often at camera-ready, after review had finished - The false-positive problem: the tool flagged the Adam optimizer paper because PDF extraction mangled the title, and most hand-inspected flags weren't real hallucinations 00:00 - Reading the wrong denominator: The cold open reframes a sub-one-percent citation defect rate as touching nearly one in five accepted papers at a single conference. 01:21 - Why reviewers never catch it: The case that expert review should catch fakes collapses on two cracks: reviewers don't check references, and phantoms scatter one per bibliography. 04:25 - Only the phone numbers count: The authors deliberately audit only mechanically verifiable citations, counting just two failure types and logging everything else as ordinary drift. 05:41 - The caged funnel that costs four cents: RefChecker clears most references with cheap deterministic catalog lookups and sends only the suspicious residue to a constrained LLM that never gets the last word. 07:46 - The dark tail and the ChatGPT timeline: The distribution's tail — one paper with twenty phantom references — is where the authors are most confident, and affected rates climb post-ChatGPT with authors blaming LLM bibliography tools. 09:53 - The home inspector who skips the wiring: Three tests show review scores carry no signal about phantoms, culminating in accepted vs rejected papers being flagged at near-identical rates. 12:53 - How often is a flag actually real?: Most hand-inspected flags turned out to be false positives from mangled metadata — including the famous Adam optimizer paper — while true phantoms look immaculately formatted. 14:43 - Quote the conservative number: Tyler argues the quotable 'one in four' has no measured precision, so the defensible figure is the two-phantom bucket — 5.1% at NeurIPS 2025. 16:26 - Run the four-cent check, then decide: The fix is cheap automated verification at submission and camera-ready, with flags opening a conversation rather than firing automatic desk rejections. Recommended Reading: - How Language Model Hallucinations Can Snowball: Explores how a model's plausible-but-false outputs compound and get committed to, the mechanism behind the polished-BibTeX phantom citations this episode dissects. (https://arxiv.org/abs/2305.13534) - SciFact: Fact or Fiction — Verifying Scientific Claims: The episode draws a sharp line between checking a citation ('a phone number') and adjudicating a claim; this is the canonical work on the harder problem they deliberately avoided. (https://arxiv.org/abs/2004.14974)