Reinforcement Learning from Human Feedback (RLHF)

A topic overview of RLHF: how human comparisons become preference data, how reward models and cautious optimization steer assistant behavior, why the PPO pipeline and DPO represent a genuine method war, and where feedback loops can be gamed or go brittle. Sources: • Proximal Policy Optimization Algorithms: https://arxiv.org/pdf/1707.06347 • Learning to Summarize with Human Feedback: https://arxiv.org/pdf/2009.01325 • Training Language Models to Follow Instructions with Human Feedback: https://arxiv.org/pdf/2203.02155 • Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback: https://arxiv.org/pdf/2204.05862 • Constitutional AI: Harmlessness from AI Feedback: https://arxiv.org/pdf/2212.08073 • Direct Preference Optimization: Your Language Model is Secretly a Reward Model: https://arxiv.org/pdf/2305.18290 • RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs: https://arxiv.org/pdf/2404.08555 • Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning: https://arxiv.org/pdf/2504.03784 • Reinforcement Learning from Human Feedback: Progress and Challenges: https://eecs.berkeley.edu/research/colloquium/230419-2/