Strategy Coopetition Explains the Emergence and Transience of In-Context Learning

This academic paper explores the emergence and transience of in-context learning (ICL) in transformer models, revealing a dynamic interplay with another strategy, context-constrained in-weights learning (CIWL). The authors term this phenomenon "strategy coopetition," where ICL and CIWL both cooperate by sharing underlying neural circuits and compete for dominance during training. While ICL appears earlier, it is ultimately superseded by CIWL, yet its initial emergence is facilitated by the simultaneous development of CIWL. The research also presents a mathematical model to explain these interactions and demonstrates how specific data properties can be manipulated to make ICL a persistent learning strategy.