Extrapolation by Association: Length Generalization Transfer in Transformers

This academic paper explores length generalization transfer in Transformer language models, investigating their ability to extrapolate knowledge from shorter inputs to longer, unseen ones. The authors demonstrate that training a model on a related "auxiliary task" with longer inputs can significantly improve the generalization of a "main task" trained only on shorter examples, across diverse domains like arithmetic, string manipulation, and maze navigation. This transfer effect is also observed in pretrained language models, suggesting they develop reusable computational frameworks. Furthermore, the research provides mechanistic evidence that this transfer correlates with the shared use of attention heads between related tasks, indicating a compositional reuse of inductive structure.keepSave to notecopy_alldocsAdd noteaudio_magic_eraserAudio OverviewflowchartMind Map