How One Researcher Beat GPT-5.2 and Gemini 3 by Judging Their Answers, Not Improving Them

How One Researcher Beat GPT-5.2 and Gemini 3 by Judging Their Answers, Not Improving Them Source: https://arxiv.org/abs/2606.31543 Paper was published on June 30, 2026 This episode was AI-generated on July 1, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A solo researcher outscored the flagship configs of GPT-5.2 Pro and Gemini 3 Pro on the hardest reasoning benchmark by more than eighteen points — using those exact models, without training anything smarter. The trick: on genuinely hard puzzles the popular answer is almost always the trap, so the whole game is selection, not generation. You'll come away with a concrete rethink of what test-time compute should actually buy you. Key Takeaways: - Why majority voting fails hardest exactly on the puzzles that matter — the crowd converges on the same tempting wrong assumption, so more votes buries the lone correct answer - How treating problem modality (text, image, code) as the axis of diversity beats simply sampling one model hot many times — and why the renders are deliberately blurred - What 'holistic judging' does: reading all candidates' full reasoning traces side by side recovered 7 minority answers for only 13% of total system cost — the cheap phase does the decisive work - The counterintuitive prompting finding: every attempt to structure or template the reasoning made it worse, a 'compliance tax' that collapses the diversity the system depends on - Where the evidence is soft — the '+7 from judging' comes from re-scoring one run, not a head-to-head, and the component attributions are educated inference, not proof - A striking infrastructure reality: 84% of the GPT-5.2 API calls failed, roughly doubling the cost through wasted retries 00:31 - Why the popular answer is a trap: Lays out the eighteen-point result and the counterintuitive core claim: these models already produce right answers but can't tell which one it is. 01:58 - The spaceship puzzle nothing could solve: Explains what makes ARC-AGI-2 unmemorizable and walks through the spaceship puzzle that all twenty-nine candidates failed. 04:00 - A whodunit where the crowd is the red herring: Makes the minority-report argument for why the correct answer on hard puzzles is almost always a minority opinion that voting crushes. 06:03 - Three specialists, one puzzle, blurry images: Describes phase one — generating diverse candidates across text, image, and code modalities, and why the renders are deliberately degraded. 08:45 - The jury that reads every argument: Covers the two selection methods that failed and the holistic judge that reads full reasoning traces together, recovering minority answers cheaply. 13:15 - When zero candidates got it right: The synthesis case where no candidate produced the answer, yet the judge assembled a correct grid from broken partial insights. 14:47 - Why structuring the prompt made it worse: The prompting heresy — templates and step-by-step scaffolding degraded performance by taxing reasoning and collapsing diversity. 17:59 - How much of the story actually holds up: The steelman critique: the component attributions come from re-scored single runs without confidence intervals, so the architecture convinces more than its internal credit split. 22:57 - What test-time compute should really buy: The durable reframing — spend inference budget on diversity plus a cheap judge, not more votes — and where the pattern might transfer beyond one benchmark. Recommended Reading: - On the Measure of Intelligence: François Chollet's paper introducing the ARC benchmark and its founding thesis — measuring skill-acquisition on novel tasks rather than recall — which the episode leans on to explain why ARC-AGI-2 punishes frontier models. (https://arxiv.org/abs/1911.01547) - Self-Consistency Improves Chain of Thought Reasoning in Language Models: The canonical majority-vote-over-samples method that this episode argues fails precisely on hard puzzles where the correct answer is a minority opinion. (https://arxiv.org/abs/2203.11171) - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: Foundational work on using models as evaluators, relevant to the episode's holistic-judging phase and its unaddressed self-preference and position-bias risks. (https://arxiv.org/abs/2306.05685) - Large Language Models are Zero-Shot Reasoners: The 'Let's think step by step' result on minimal prompting, a useful counterpoint to the episode's claim that prescriptive scaffolding imposes a 'compliance tax' on hard reasoning. (https://arxiv.org/abs/2205.11916)