Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

The academic paper introduces Supervised Reinforcement Learning (SRL), a novel training framework for Large Language Models (LLMs) developed by researchers from Google Cloud AI Research and UCLA to address the difficulty of multi-step reasoning. SRL reformulates problem-solving as a sequence of logical actions, providing dense, step-wise rewards based on the similarity between the model's generated actions and expert trajectories, which contrasts with the sparser, final-outcome rewards used in Reinforcement Learning with Verifiable Rewards (RLVR). The framework trains models to generate an internal reasoning monologue before committing to an action, encouraging flexible and sophisticated reasoning patterns like interleaved planning and verification. Extensive experiments on challenging mathematical reasoning and agentic software engineering benchmarks demonstrate that SRL significantly outperforms baseline methods like Supervised Fine-Tuning (SFT) and RLVR, especially when used to initialize training before subsequent RLVR refinement.