Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

This paper introduces a framework for fine-tuning large language models (LLMs) by leveraging user edits found in deployment logs, such as those from writing or coding assistants. Unlike traditional methods that rely on expensive manual labeling, this approach treats user modifications as a rich, multi-dimensional source of preferences, supervision, and cost feedback. The researchers provide a theoretical analysis of how these different feedback types impact model performance and demonstrate that individual algorithms have distinct trade-offs. To address these variations, they propose ensemble procedures that combine multiple learning methods to achieve more robust results. Empirical evaluations on summarization and email writing tasks show that ensembling consistently outperforms single-method approaches. Finally, the study highlights how these techniques can effectively personalize LLMs or adapt them to broader user distributions during real-time interaction.