Understanding Prompt Tuning and In-Context Learning via Meta-Learning

The academic paper investigates prompt tuning and in-context learning through a meta-learning and Bayesian lens, positing that optimal prompting can be understood as conditioning Bayesian sequential predictors. The authors detail how meta-trained neural networks, like LSTMs and Transformers, function as Bayes-optimal predictors and explore the theoretical limitations of prompting, particularly for complex, multimodal target task distributions. Empirical experiments on coin-flip sequences confirm these theories, demonstrating that Soft Prompting—using sequences of real-valued vectors—is significantly more effective than hard-token prompts, even showing surprising efficacy in fine-tuning untrained networks. Ultimately, the research provides a fundamental conceptual framework for understanding the mechanisms and constraints of prompt optimization.