Fine-Tuning Strategies for Preserving In-Context Learning in Linear Attention

This research examines the tension between in-context learning (ICL) and fine-tuning in Transformer-based models, specifically using linear attention to provide a theoretical foundation. While fine-tuning is often employed to enhance zero-shot performance on specific target tasks, the authors demonstrate that updating all attention parameters can inadvertently damage the model's ability to learn from demonstrations. They identify a superior strategy: restricting updates to the value matrix, which improves task-specific accuracy while maintaining the model’s original few-shot capabilities. The study further explores the use of an auxiliary few-shot loss, finding that it boosts performance on the target task but reduces the model's ability to generalize to out-of-distribution tasks. These theoretical insights are validated through both mathematical proofs and empirical experiments on the MMLU benchmark. Ultimately, the work provides a framework for optimizing language models without sacrificing their inherent flexibility as in-context learners.