General Preference Reinforcement Learning

This paper introduces General Preference Reinforcement Learning (GPRL), a novel post-training framework designed to align large language models with complex human values. Traditional methods often rely on a scalar reward model, which frequently leads to "reward hacking" as the model exploits a single quality dimension at the expense of others. To resolve this, the authors utilize a General Preference Model (GPM) that embeds responses into multiple subspaces, representing quality as a multi-dimensional, structured signal. GPRL estimates advantages for each dimension independently, ensuring that no single axis can dominate the learning process through normalized scaling. The system also features a closed-loop drift monitor that detects and corrects single-axis exploitation in real-time by reweighting dimensions and tightening trust regions. Experimental results show that GPRL significantly outperforms existing methods like DPO and GRPO on benchmarks such as AlpacaEval 2.0 and Arena-Hard by resisting stylistic drift. Ultimately, the research suggests that the future of open-ended alignment lies in the mathematical shape of rewards rather than just their strength.