Alignment from Demonstrations for Large Language Models

The provided text is a research paper introducing Alignment from Demonstrations (AfD) as a novel method for aligning large language models (LLMs) using high-quality demonstration data. It identifies limitations in current preference-based alignment techniques and proposes framing AfD within a reinforcement learning framework, specifically inverse reinforcement learning, to address these shortcomings. The paper explores trajectory distribution matching as a core objective, demonstrating how supervised fine-tuning relates to minimizing forward KL divergence. Furthermore, it introduces a computationally efficient algorithm based on reward model extrapolation to enhance alignment, validated through experiments on harmlessness and helpfulness tasks.