EPISODE · Mar 7, 2026 · 23 MIN
Learning without training: The implicit dynamics of in-context learning
from Best AI papers explained · host Enoch H. Kang
This research explores the mechanisms of in-context learning (ICL) in Large Language Models, proposing that transformers learn by implicitly updating their internal weights during inference. The authors demonstrate that a transformer block effectively transforms prompt examples into a rank-1 weight update of the model's MLP layer. This process allows the model to adapt to new patterns without permanent training, mathematically mirroring stochastic gradient descent as tokens are processed. Theoretical formulas are provided to map these context-driven adjustments exactly, showing that MLP layers are naturally structured to absorb and store contextual information. Experimental results on linear regression tasks confirm that modifying model weights using these formulas produces identical predictions to providing the original in-context prompt. The study ultimately unifies ICL with model editing and steering vectors, offering a principled framework for understanding how LLMs reorganize their internal representations dynamically.
What this episode covers
This research explores the mechanisms of in-context learning (ICL) in Large Language Models, proposing that transformers learn by implicitly updating their internal weights during inference. The authors demonstrate that a transformer block effectively transforms prompt examples into a rank-1 weight update of the model's MLP layer. This process allows the model to adapt to new patterns without permanent training, mathematically mirroring stochastic gradient descent as tokens are processed. Theoretical formulas are provided to map these context-driven adjustments exactly, showing that MLP layers are naturally structured to absorb and store contextual information. Experimental results on linear regression tasks confirm that modifying model weights using these formulas produces identical predictions to providing the original in-context prompt. The study ultimately unifies ICL with model editing and steering vectors, offering a principled framework for understanding how LLMs reorganize their internal representations dynamically.
NOW PLAYING
Learning without training: The implicit dynamics of in-context learning
No transcript for this episode yet
Similar Episodes
Mar 31, 2026 ·54m
Mar 27, 2026 ·14m
Mar 24, 2026 ·42m
Mar 20, 2026 ·42m
Mar 17, 2026 ·41m
Mar 13, 2026 ·44m