How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics

This research investigates how response sampling and reference policies influence the alignment of large language models with human preferences. Using the Identity Preference Optimization (IPO) and Direct Preference Optimization (DPO) frameworks, the authors demonstrate that instance-dependent sampling can improve ranking accuracy, whereas skewed on-policy sampling often leads to undesirable model concentration. The study particularly highlights the risks of iterative alignment, where models are repeatedly trained on their own generated data. Their theoretical analysis and experiments reveal that these feedback loops can trigger persistent oscillations or a total entropy collapse, where the model's output variety vanishes. By characterizing these long-term dynamics, the paper provides mathematical regimes that ensure system stability during the training process. Ultimately, the work suggests that treating alignment as a dynamic system rather than a static task is essential for maintaining reliable model behavior.