AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review

AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review Source: https://arxiv.org/abs/2607.01507 Paper was published on July 01, 2026 This episode was AI-generated on July 3, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. One paragraph stating a political belief was enough to make AI analysts reach opposite conclusions from identical data — and 86% of those biased analyses passed hostile expert review, because nothing in any single report was actually wrong. A Stanford team's fix is a new statistic, a sibling of the p-value, that measures whether a finding was fished from the extreme edge of everything the data could have said. On its first deployment, the instrument built to catch AI bias caught the humans instead. Key Takeaways: - How a single persona paragraph made coding agents reproduce 72% of the ideological gap found across 42 human research teams — with a claimed-significance gap nearly 9x the human one - Why peer review structurally can't catch this bias: 86% of analyses passed a cross-model AI audit and 78% passed blinded human PhD statisticians, with skeptics' work exactly as clean as believers' - The two mechanisms visible for the first time in agent logs — exploration bias and selection bias — including two agents reading the same negative estimate as 'evidence' vs 'a flaw to fix' - The m-value: the p-value's mirror statistic, measuring how often re-running the analysis (not re-collecting the data) would produce a result that extreme — and why analyst choice moved answers 2.8x more than noise - What happened when the instrument was pointed at the human teams: 40% of their statistically significant results sat in the most extreme 5% of the analysis space - The steelman that survives the episode: extreme is not the same as wrong — the m-value measures typicality, not quality, and can't distinguish a grandmaster's move from a fished result in any single study 00:00 - Forty accountants, forty answers, all legal: The cold open: AI agents given identical data but one paragraph of political belief reached opposite conclusions — and most of those analyses passed expert review. 01:43 - Can one paragraph bend a rigorous analysis?: The experiment design — four contested questions, believer vs skeptic personas, real datasets — plus the human foil of 42 research teams and the permuted-data control that rules out honest ambiguity. 04:21 - Why hostile review couldn't find the bias: A cross-model AI audit and blinded human statisticians grade the biased analyses — and pass them — leading to the episode's core reframe: the bias lives in which path through the garden of forking paths got walked, not in the path itself. 06:14 - Watching bias enter, decision by decision: Because agents log every step, we watch exploration bias and selection bias emerge in real time — including two opposing agents interpreting nearly the same regression estimate in opposite ways, and a classifier predicting conclusions from methods alone. 09:04 - 4,400 defensible answers to one question: Pooling every review-surviving specification produces a full map of what the data could defensibly say — a spectrum from strongly negative to strongly positive, built for about a hundred dollars. 10:13 - The p-value's missing sibling: The formal core: the m-value measures fragility to analyst choice rather than data noise, made practical by the Agentic Bootstrap — and on the immigration question, analysis choice moved the answer 2.8x more than the data's noise did. 13:10 - The instrument turns on the humans: Applying m-values to the 42 human teams' 897 reported specifications reveals they pile into the extremes — with 40% of significant human results sitting in the most extreme 5% of the analysis space, leaning in belief-consistent directions. 14:46 - But extreme isn't the same as wrong: The steelman critique: like a grandmaster's bizarre-but-best chess move, the right analysis can be an outlier, so the m-value measures typicality rather than quality — a limit the authors half-concede, leaving it a population-level diagnostic rather than a single-study verdict. Recommended Reading: - The garden of forking paths: Why multiple comparisons can be a problem, even when there is no 'fishing expedition' or 'p-hacking': The Gelman & Loken essay that coined the metaphor at the heart of this episode — how defensible-in-isolation analytical choices can bias conclusions without any single visible flaw. (http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf)