Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

This research paper provides a theoretical and empirical comparison between Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). The authors identify a performance gap between the two methods caused by model mis-specification, where the intended reward or policy cannot be perfectly captured by the chosen model classes. Their analysis reveals that RLHF maintains a structural advantage when policy models are limited, whereas DPO performs better when reward models are restricted. Furthermore, the study highlights a statistical efficiency gap, demonstrating that RLHF requires significantly fewer samples than DPO to recover effective rewards in sparse data environments. Ultimately, the source offers a framework for selecting the superior alignment strategy based on specific computational constraints and data availability.