Why You Can't Fine-Tune Foresight Into an AI Agent

Why You Can't Fine-Tune Foresight Into an AI Agent Source: https://arxiv.org/abs/2606.27483 Paper was published on June 25, 2026 This episode was AI-generated on June 29, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A team taught a language model to forecast the future before acting — and it learned the format flawlessly while learning none of the actual skill, slapping a confident 100% on plans that were vague and contradictory. The unsettling claim: fine-tuning can install a perfect imitation of an ability with nothing underneath it. This episode unpacks the format-capability gap, a three-stage fix that makes a model's self-confidence honest, and exactly where the evidence outruns the story. Key Takeaways: - Why teaching a model the shape of foresight produces a confident lie instead of an actual plan — the 'format-capability gap' between eliciting an ability and installing one - The three-stage apprenticeship that injects the capability first (200B tokens of trajectories), teaches the format second, and makes the confidence number honest third - How a 'verbalized Q-value' plus a Brier-score calibration reward locks the model's self-confidence honest so it can't be gamed - Why confidence that drops to 5% on an unsolvable problem is more useful for deployed agents than confidence that stays high and wrong - The honest limits: the full fix runs on one proprietary 2B model, the gains over a simpler baseline are about two points, and the calibration evidence is hand-picked rather than measured - Why skipping the supervised warm-start collapses the whole pipeline — reward can sharpen a skill but can't conjure one 00:00 - Perfect form, empty content: The cold open lays out the central failure: a model that learned the foresight format almost perfectly while producing hollow, confidently-wrong plans. 01:48 - Why the next agents live or die on this: Eric and Cassidy frame why a trustworthy confidence signal matters for long-running agents, and trace the idea of mental rehearsal back to Kenneth Craik's 1943 work. 02:38 - The format-capability gap: The naive fix — fine-tuning on look-ahead blocks — fails the same way across three models, illustrating that post-training elicits abilities but can't install new ones. 05:11 - A world model written out loud: The pivot to a folded-in, verbalized world model and the 'Q-value spoken out loud' framing that makes the model's own estimate gradeable against reality. 07:43 - Build the skill before the report: The three-stage apprenticeship — capability injection via teacher-written future blocks, format teaching, and a reinforcement-learning stage with stacked grounding, calibration, and task rewards. 12:53 - Watching the confidence wobble: Confidence trajectories that rise, dip on caught errors, and stay low on unsolvable problems — the behavior the design is aiming for, with a flag that these cases are hand-picked. 14:56 - The headline is smaller than the story: The benchmark results: modest ~two-point gains, real payoff on multi-hop reasoning, near-free efficiency, and a collapse when the warm-start is skipped. 17:36 - Where the ideas outrun the evidence: The steelman critique — incremental gains, a single unreproducible proprietary model, anecdotal calibration, leaky teacher blocks, and a loosely-used Q-value label — set against the well-documented diagnosis. Recommended Reading: - Dream to Control: Learning Behaviors by Latent Imagination: The Dreamer line of work the episode contrasts against — a separate latent world-model simulator the agent dreams inside, exactly the 'bolted-on module' framing this paper deliberately refuses. (https://arxiv.org/abs/1912.01603) - ReAct: Synergizing Reasoning and Acting in Language Models: The reactive step-see-step paradigm the episode positions as the status quo the paper is trying to move agents beyond. (https://arxiv.org/abs/2210.03629) - Language Models (Mostly) Know What They Know: A direct empirical counterpoint to the episode's skepticism about self-graded confidence, asking whether a model's own calibration signal can be trusted across many episodes. (https://arxiv.org/abs/2207.05221)