Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer

Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer Source: https://arxiv.org/abs/2607.01232 Paper was published on July 01, 2026 This episode was AI-generated on July 2, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. Train just ten layers of a 36-layer model with reinforcement learning and you beat training all 36 — because the improvement doesn't spread across the network, it concentrates in a handful of middle layers. This episode traces where, physically, RL adaptation lands inside a transformer, why a zero-cost 'just train the middle' heuristic beats the standard recipe on math, and where the headline overreaches the evidence. Key Takeaways: - Why RL improvement concentrates in a small set of middle layers rather than spreading evenly — a clean inverted-U across 36 floors that repeats across seven models, two families, three algorithms, and three task domains - The 'door' dissociation: middle layers matter not because they move more (weight change is roughly uniform) but because of leverage — the quality of a layer's parameter subspace, not the distance it travels - A zero-cost heuristic — train the geometric middle layers by position alone, no profiling — that beats full-parameter training and recovers ~21% of the total RL gain for free - That the important layers are fixed during pretraining and portable across tasks (rankings correlate ~0.59 across math and code), so RL just moves into a room that was already built - The steelman critique: 'one layer is enough' is softer than the title — many single-layer wins sit at the edge of noise, and the training strategies were only validated on math - Why a panel of seven layer-specialists (34% answer overlap) beats sampling one model seven times — structural diversity over sampling diversity 00:00 - Fewer moving parts, better score?: The counterintuitive opening result — ten trained layers beating all thirty-six — and the claim that RL improvement is concentrated, not smeared across the network. 01:46 - What RL is actually sharpening: Sets up the transformer as a stack of floors, the pretraining-then-post-training split, and how GRPO races a model's own answers against each other. 03:06 - One floor at a time: The experimental design — freeze 35 layers, train one, repeat 36 times — and the subtle point that frozen layers still shape the feedback. 04:23 - A ruler for the hill climb: The contribution metric explained as a hill climb, with real spreads including a layer that overshoots full training and one that goes negative. 06:03 - The hump in the middle: The inverted-U shape of layer contribution that repeats across seven models, two families, three algorithms, and three domains. 08:01 - Is the finding real, or lucky?: How the authors stacked the deck for full training, ruled out learning-rate rescue, and showed the strong layers generalize out of domain. 09:41 - The same floors, a different job: Evidence that layer rankings are portable across tasks and fixed during pretraining rather than chosen by the RL objective. 11:15 - Distance or leverage?: The dissociation that all layers move about the same distance yet contribute wildly differently — the door analogy of leverage over force. 13:46 - Skip the scan, train the middle: Three exploit strategies, culminating in a zero-cost position heuristic that beats full training, plus the panel-of-specialists voting result. 17:08 - How much of this really holds?: The steelman critique — small absolute gains, math-only validation of the strategies, and the unanswered question of why the middle matters. 20:15 - Remember the door: The durable idea that RL reshapes a fixed, pretrained functional geography rather than the whole network, and the question posed to listeners. Recommended Reading: - LoRA: Low-Rank Adaptation of Large Language Models: The canonical parameter-efficient fine-tuning method that this episode's 'train fewer of the right layers' recipe is implicitly competing with and complementing. (https://arxiv.org/abs/2106.09685) - DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models: Introduces GRPO, the exact RL algorithm the episode uses to sharpen math performance before asking which layer absorbed the gain. (https://arxiv.org/abs/2402.03300) - The Unreasonable Ineffectiveness of the Deeper Layers: A companion perspective on where capability lives in the transformer stack, showing which layers can be pruned with little loss — a mirror image of the episode's middle-layer leverage claim. (https://arxiv.org/abs/2403.17887) - Self-Consistency Improves Chain of Thought Reasoning in Language Models: The sampling-diversity majority-vote baseline the episode's 'panel of layer-specialists' explicitly beats, making it the natural point of comparison for structural vs. sampling diversity. (https://arxiv.org/abs/2203.11171)