When 85% on SWE-bench Turns Into 58% Under Proof

When 85% on SWE-bench Turns Into 58% Under Proof Source: https://arxiv.org/abs/2609.21190 Paper was published on September 18, 2026 This episode was AI-generated on September 21, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. Claude scores 85% on SWE-bench Verified. Run those same patches through an adversarial proof auditor hunting for one input where they disagree with the maintainers' own fix, and it drops to 58 — GPT-5.5 falls from 81.2 to 33.4. The surprise isn't that models can't prove their code correct; proving turns out to be nearly free. What they can't do is state what 'fixed' was supposed to mean. Key Takeaways: - Why the median SWE-bench Verified instance is decided by exactly one newly-passing test — and what that means for the word 'resolved' - How Benchproofer builds a formally verified twin of a real GitHub issue using axioms: assumed facts about unformalized dependencies that are allowed to understate but never overstate - The three mechanical checks (buggy version must fail, mutants must fail, adversarial models attack) that keep a specification from being technically true and completely useless - Why structured plain-English specs (EARS) collapse — Claude loses 17 points, GPT loses 46 — while real formal specs cost under a point - The Django date-picker failure where the agent printed 'no crash' as proof it won, and that transcript was the evidence it lost - The steelman: the 85→96% jump comes from specs built with the gold patch and test list in hand — calibrated against the answer key 00:00 - One test decides whether resolved means resolved: The cold open: Claude's 85% on SWE-bench Verified falls to 58 under adversarial proof audit, and the reason is that the median instance turns on a single newly-passing test. 01:47 - A spelling quiz versus the whole dictionary: Why proofs differ from tests in kind rather than degree, and why nobody had run this on real code before — real fixes live inside hundred-thousand-line codebases with no specification anywhere. 02:44 - The box the size of the universe: Using a real matplotlib bounding-box bug, Paige shows that a specification needs both 'every point is inside' and 'something touches each edge' — and that the missing half is where everything goes wrong later. 03:51 - Subcontracting a bolt you never inspect: How axioms let the pipeline formalize only the changed code, why an axiom may understate but never overstate a dependency, and how fuzzing weakens ones that don't survive. 04:56 - Three checks and an adversarial tiebreak: The mechanical gauntlet every specification must survive — buggy version must fail, mutants must fail, adversarial models attack — and the hidden-test tiebreak that distinguishes a loose spec from a second correct answer. 06:51 - GPT falls further, but what did we learn?: GPT-5.5 drops from 81.2 to 33.4, and Eric presses on what 'overturned' actually means — divergence from the gold patch on one input, not proof the code breaks in production. 07:26 - Why 'shall' statements are decoration: Structured plain-English EARS specs cost Claude 17 points and GPT 46, while real formal specs cost under a point — and handing a model a correct spec lifts Claude to 96 and GPT to 94. 08:28 - Proving isn't the wall. Stating the target is.: When agents must write their own specifications, the 11-to-15 point gain vanishes — and the failure is almost always one thing: the spec doesn't cover enough of what the fix touches. 10:58 - The transcript that proved it had lost: A Django date-picker bug where the agent's patch verifies, passes all fifteen tests, and prints 'no crash' — while missing the gold fix's return of '0-0-0' that two hidden tests depend on. 12:20 - The number built with the answer key: Eric's steelman: the ground-truth specs were written with the gold patch, buggy code, and required test flips in hand — so the 11-to-15 point gain may be measuring a formalized grading rubric, not a spec an engineer could write cold. 12:22 - What 'resolved' is actually measuring: Proving costs about a point, only a handful of 50,000+ attempts hit their step limit, and the closing takeaway: the ceiling isn't proving anymore, it's stating precisely what the code was supposed to do. Recommended Reading: - SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: The original benchmark whose 'resolved' label this episode interrogates — worth reading to see exactly how test-flip criteria were defined in the first place. (https://arxiv.org/abs/2310.06770) - Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (EvalPlus): The closest prior demonstration of the episode's core claim — that passing a benchmark's tests overstates correctness — using vastly expanded test inputs rather than proofs. (https://arxiv.org/abs/2305.01210)