Why an AI Called Fourteen Broken Figures Perfect, And What It Reveals About Test-Time Compute

Why an AI Called Fourteen Broken Figures Perfect, And What It Reveals About Test-Time Compute Source: https://arxiv.org/abs/2607.11598 Paper was published on July 13, 2026 This episode was AI-generated on July 14, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. An AI judge looked at fifteen figures with titles stacked on top of each other and content running off the page, and rated fourteen of them flawless. It wasn't lazy or biased — it literally couldn't see the defects, and a new paper argues that same blind spot is the reason a whole third way of spending compute has stayed half-invisible to the field. You'll come away understanding why internal effort plateaus, why grounding has to hold on both the fixing and the scoring side, and where the authors' own evidence stops short. Key Takeaways: - Why re-reading and best-of-N both plateau: they only reshuffle information already inside the frozen weights, and can't manufacture what was never there - The 'interaction scaling' third axis, where an external instrument observes what the model actually did and imports information the weights never held — climbing to 100% on coding tasks where reasoning-only capped at 73% and a perfect judge capped at 87% - The coverage principle: a grounded tool helps only as far as it can see — a real linter misses runtime bugs, and a screenshot reviewer actually makes slide layouts worse - How the standard screenshot-based metric hides real improvements, rating 14 of 15 figures perfect while a geometry tool found only 3 clean - The honest weak point the authors name: the same instrument both drives the fix and scores the result, so some gains are mechanically guaranteed and there's no human-preference study - Why the perfect coding scores (15 tasks) and the curated 14-of-15 figure set are existence proofs, not measures of how often the blindness bites in normal use 00:57 - You can't send the student back to school: Sets up test-time compute and the two familiar ways to spend it — think longer, or try more attempts and keep the best. 01:48 - Why re-reading can't add a chair: Explains the information-theory result behind why self-correction plateaus — reprocessing your own output can't create new information. 03:03 - The third axis nobody counted: Introduces interaction scaling and its bare three-player setup: a proposer, an instrument that executes or measures, and a reviewer that turns the report into concrete defects. 04:07 - Why a perfect judge still hits a ceiling: The coding experiment where reasoning caps at 73%, best-of-N at 87%, and the interaction loop climbs to 100% with zero run-to-run variance. 05:38 - The linter that can't see the knife: The coverage principle: a grounded linter misses runtime bugs like a metal detector misses a ceramic knife, and a screenshot reviewer of slides actually increases defects. 08:03 - Fourteen perfect, three actually clean: The blind-inspector problem: a screenshot judge rates 14 of 15 figures perfect while a bounding-box tool finds only 3 clean, and grounded scoring reveals real fixes the screenshot could never detect. 10:50 - What I don't buy yet: The steelman critique: same instrument on both ends makes some gains mechanically guaranteed, there's no human-preference study, and the coding and figure results are small, curated existence proofs. 12:25 - Grounding on both sides, or nothing moves: Wraps the reframing — compute has three axes, interaction is the only one that imports new information — and asks whether reported gains should require a grounded instrument. Recommended Reading: - Large Language Models Cannot Self-Correct Reasoning Yet: The empirical case for why internal self-correction plateaus without external feedback — exactly the 'reprocessing adds no information' claim this episode builds its whole argument on. (https://arxiv.org/abs/2310.01798) - Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters: The canonical treatment of the 'think longer vs. try more' test-time compute axes that this episode reframes into a third, interaction-based axis. (https://arxiv.org/abs/2408.03314) - Large Language Monkeys: Scaling Inference Compute with Repeated Sampling: The deep dive on best-of-N sampling, sharpening the episode's point that even a perfect judge can only pick from candidates the model actually drew. (https://arxiv.org/abs/2407.21787)