Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning

This academic paper introduces Trajectory Bellman Residual Minimization (TBRM), a novel value-based reinforcement learning algorithm designed to enhance the reasoning capabilities of large language models (LLMs), particularly in mathematical problem-solving. Unlike prevailing policy-based methods like PPO and GRPO, TBRM streamlines the training process by eliminating the need for critics, importance sampling, or clipping mechanisms, requiring only a single rollout per prompt. The authors present theoretical evidence showing TBRM's convergence to a near-optimal policy using off-policy data and empirical results demonstrating its superior performance and efficiency compared to baselines on several math benchmarks. The findings suggest that value-based approaches, like TBRM, offer a promising and efficient alternative for improving LLM reasoning.