Distortion of AI alignment revisited: RLHF is a decent utilitarian aligner

This paper provides a fine-grained theoretical analysis of Reinforcement Learning from Human Feedback (RLHF), specifically examining its performance in pluralistic settings with diverse user preferences. The authors challenge previous assertions that RLHF inherently suffers from exponential distortion, demonstrating instead that such degradation is primarily a result of a distribution mismatch between the preference data and the reference policy. By establishing tight upper and lower bounds, the study proves that RLHF remains a utilitarian aligner that can reasonably maximize average utility when this mismatch is controlled. The findings suggest that on-policy data collection or specific pre-training fine-tuning can significantly mitigate alignment errors. Ultimately, the paper reconciles the gap between pessimistic theoretical models and the empirical success of large language models like GPT-4.