Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Maya and Leo unpack Direct Preference Optimization, the 2023 paper whose napkin-worthy algebra showed the reward model was hiding inside the language model all along. They walk the old two-stage RLHF pipeline, then the substitution that cancels the reward variable and leaves a supervised-looking classification loss, the implicit reward you can read off the tuned model's margin over its reference, and the mooring dial that still governs drift. Then they stage the method war the paper ignited: DPO as the stable default for offline preference pairs versus the RL camp's case for online sampling, auditable reward artifacts, and long-horizon feedback — a fight the rest of the topic keeps re-litigating. Sources: • Direct Preference Optimization: Your Language Model is Secretly a Reward Model: https://arxiv.org/pdf/2305.18290 • Proximal Policy Optimization Algorithms: https://arxiv.org/pdf/1707.06347 • Constitutional AI: Harmlessness from AI Feedback: https://arxiv.org/pdf/2212.08073