EPISODE · Aug 3, 2025 · 41 MIN
In-Context Learning: Implicit Weight Dynamics
from Neural intel Pod · host Neuralintel.org
This academic paper explores In-Context Learning (ICL) in Large Language Models (LLMs), a phenomenon where models learn new patterns from prompts without explicit weight updates. The authors propose that a transformer block implicitly modifies its internal weights during inference, specifically the Multi-Layer Perceptron (MLP) layer, as context is consumed. They introduce the concept of a "contextual block" to generalize this mechanism, demonstrating theoretically and experimentally that context is transformed into a low-rank weight update for the MLP. This work suggests that ICL behaves like an implicit gradient descent, with each token influencing the model's effective weights, providing insights into the mysterious emergent properties of LLMs.
What this episode covers
This academic paper explores In-Context Learning (ICL) in Large Language Models (LLMs), a phenomenon where models learn new patterns from prompts without explicit weight updates. The authors propose that a transformer block implicitly modifies its internal weights during inference, specifically the Multi-Layer Perceptron (MLP) layer, as context is consumed. They introduce the concept of a "contextual block" to generalize this mechanism, demonstrating theoretically and experimentally that context is transformed into a low-rank weight update for the MLP. This work suggests that ICL behaves like an implicit gradient descent, with each token influencing the model's effective weights, providing insights into the mysterious emergent properties of LLMs.
NOW PLAYING
In-Context Learning: Implicit Weight Dynamics
No transcript for this episode yet
Similar Episodes
Mar 14, 2026 ·23m
Mar 11, 2026 ·16m
Feb 28, 2026 ·14m