Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models

This research provides the first **rigorous theoretical framework** for Self-Rewarding Language Models (SRLMs), explaining how they achieve alignment through **iterative self-training** without human feedback. The authors identify a **single-step update limit**, proving that non-iterative methods are highly vulnerable to failure if the **initial model quality** is poor. To address this, they derive **finite-sample error bounds** demonstrating that an iterative approach progressively diminishes the influence of a weak starting point at an **exponential rate**. By introducing the **Policy Condition Number**, the study quantifies a model's suitability for self-alignment and shows how repeated updates steer the system toward **internal consistency**. Their analysis further extends to **linear softmax models**, utilizing effective dimension to prove that these models can overcome the "curse of dimensionality" during the alignment process. Ultimately, the work confirms that **iterative dynamics** act as a stabilizing force, transforming self-rewarding into a robust statistical learning problem.