Rethinking the Trust Region in LLM Reinforcement Learning

This research paper introduces Divergence Proximal Policy Optimization (DPPO), a novel reinforcement learning framework designed to improve the fine-tuning of Large Language Models (LLMs). The authors identify a structural flaw in the standard PPO algorithm, noting that its ratio-clipping mechanism incorrectly penalizes rare tokens while failing to stop destabilizing shifts in common tokens. To resolve this, DPPO replaces heuristic clipping with a principled trust region constraint based on direct estimates of policy divergence. Because calculating exact divergence is memory-intensive, the study proposes Binary and Top-K approximations to maintain efficiency without sacrificing performance. Theoretical proofs and empirical tests demonstrate that DPPO achieves significantly better training stability and efficiency than existing methods like GRPO. Ultimately, the framework provides a more robust foundation for aligning LLMs with complex reasoning tasks and human preferences.