A Coding Agent Found a Hole in a Peer-Reviewed STOC Proof for Five Dollars

A Coding Agent Found a Hole in a Peer-Reviewed STOC Proof for Five Dollars Source: https://arxiv.org/abs/2606.31134 Paper was published on June 30, 2026 This episode was AI-generated on July 1, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. An off-the-shelf coding agent on a $200-a-month subscription read a proof that already cleared peer review at one of theory's most selective conferences, tried to make it machine-checkable, and got stuck on one line that turns out not to follow — handing back a counterexample you can check by hand. The trick is treating a math proof like a software project, with data types, unit tests, and a manager who can order a rewrite. You'll come away seeing why the real bottleneck in math is no longer writing proofs but trusting them. Key Takeaways: - Why general-purpose coding models have quietly overtaken specialist Lean-tuned models at formalizing math - The reframe at the heart of the paper: treat a proof like software, with types, unit-test lemmas, and an orchestrator that can backtrack and refactor - How the system caught a genuine gap in a 2025 STOC proof — and returned a hand-checkable counterexample of one triangle and ten dots - Why the '$5 per problem' and '91%' headline numbers are softer than they sound — subscription pricing arbitrage and a statistical floor from just 32 problems - Where the guarantee actually lives: the Lean kernel proves the proof, but only AI judgment guarantees the formal statement means what the paper said - The reframing of AI-for-math from theorem discovery to tireless, literal-minded refereeing 01:08 - Why trusting a proof is the new bottleneck: Sets up the stakes: AI can now generate proofs faster than any human can check them, and a beautiful proof can still hide an uncaught error. 01:30 - The escape hatch that can't be argued with: Explains Lean 4 and its paranoid kernel — the difference between a persuasive argument and a program that either compiles or fails. 02:22 - Two walls that block autoformalization: Lays out why the problem isn't solved: specialists lost to generalists, and Mathlib lacks the vocabulary of cutting-edge research. 04:05 - What if a proof were a software project?: The core reframe: types, unit-test lemmas, and an orchestrator that backtracks — how the system tests meaning it can't directly check. 06:20 - Proving the parent before the children: Walks through the two assembly lines and the backwards proof-tree strategy that checks the argument's interface before its internals. 10:03 - The line the machine couldn't force: The system proves every lemma but one, checks the failing step against the paper's own definitions, and returns a hand-checkable counterexample. 12:21 - Green nodes, orange nodes, and honesty: The axiom ledger as an honest readout of how self-contained each paper is — from all-green proofs to labeled citations to the one gap. 13:38 - The numbers that oversell themselves: The 91% solve rate and $5-per-problem cost, and why both are softer than the headline — a statistical floor and subscription pricing arbitrage. 15:47 - The compiler proves; only AI judges meaning: The steelman critique: small sample, AI checking AI on faithfulness, narrow scope — and where the guarantee genuinely does and doesn't hold. 17:52 - Would you trust the machine referee?: The bigger claim — bridging persuasion and certainty on a consumer subscription — and the open question of whether the AI-judged loop is too circular. Recommended Reading: - Autoformalization with Large Language Models: The foundational demonstration that general-purpose LLMs can translate informal math into formal statements, directly setting up the autoformalization problem this episode's system reframes as software engineering. (https://arxiv.org/abs/2205.12615) - The Lean 4 Theorem Prover and Programming Language: The system paper for Lean 4, the language whose kernel provides the 'compiles-or-it-doesn't' ground truth the whole episode hinges on. (https://doi.org/10.1007/978-3-030-79876-5_37) - DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data: A representative example of the specialized Lean-expert models that this episode argues have been quietly overtaken by general-purpose coding agents. (https://arxiv.org/abs/2405.14333) - PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition: The competition-math benchmark behind the episode's contested '91 percent' figure, useful for readers wanting to judge the solve-rate and cost comparisons themselves. (https://arxiv.org/abs/2407.11214)