Beyond Reward: Limits of RL in LLM Reasoning

This academic paper critically re-evaluates the widespread belief that Reinforcement Learning with Verifiable Rewards (RLVR) enhances the fundamental reasoning capabilities of large language models (LLMs) beyond their initial base models. The authors employ the pass@k metric across various benchmarks, including mathematics, code generation, and visual reasoning, to assess the boundary of reasoning capacity by allowing models multiple attempts to solve problems. Surprisingly, the study finds that while RLVR training improves sampling efficiency (better performance at smaller k values), it does not introduce novel reasoning patterns; instead, the reasoning paths of RL-trained models are already present within the base models' output distributions, and RLVR even reduces the overall scope of solvable problems at larger k values. The research concludes that distillation, unlike RLVR, can genuinely introduce new knowledge and expand a model's reasoning boundary, suggesting a need for alternative training paradigms to truly advance LLM reasoning.