EPISODE · Apr 18, 2025 · 13 MIN
Concise Reasoning via Reinforcement Learning
from Best AI papers explained · host Enoch H. Kang
This paper explores the relationship between the length of reasoning in large language models and their accuracy, arguing that longer responses are not inherently better and often arise from the reinforcement learning training process. The authors demonstrate mathematically how the PPO algorithm can incentivize longer or shorter responses based on reward signals and the GAE parameter λ. They propose a two-phase RL training strategy: first enhancing reasoning capabilities on challenging problems, then enforcing conciseness on occasionally solvable ones. Experimental results on math and STEM benchmarks show that this approach can significantly reduce response length while maintaining or improving accuracy and robustness, even with minimal training data.
What this episode covers
This paper explores the relationship between the length of reasoning in large language models and their accuracy, arguing that longer responses are not inherently better and often arise from the reinforcement learning training process. The authors demonstrate mathematically how the PPO algorithm can incentivize longer or shorter responses based on reward signals and the GAE parameter λ. They propose a two-phase RL training strategy: first enhancing reasoning capabilities on challenging problems, then enforcing conciseness on occasionally solvable ones. Experimental results on math and STEM benchmarks show that this approach can significantly reduce response length while maintaining or improving accuracy and robustness, even with minimal training data.
NOW PLAYING
Concise Reasoning via Reinforcement Learning
No transcript for this episode yet
Similar Episodes
Mar 31, 2026 ·54m
Mar 27, 2026 ·14m
Mar 24, 2026 ·42m
Mar 20, 2026 ·42m
Mar 17, 2026 ·41m
Mar 13, 2026 ·44m