EPISODE · May 4, 2026 · 18 MIN
EP006 — A Small Loop That Acts Like a Deep Model (Looped Reasoning)
from Machine's Learning
The dominant recipe for building capable language models is depth — more layers, more parameters, more distinct transformer blocks. A new mechanistic interpretability paper from Nam, Gromov, Yaida and colleagues looks at what happens when you take a much smaller model and run the same block over and over again in a loop. Inside, the internal state moves through cyclic trajectories that settle into fixed points; the attention heads stay consistent across iterations; each pass through the loop is doing the work that a separate layer would do in a traditional deep model. The cross-domain parallel: an assembly line versus a solo sculptor walking around the same piece of marble many times. The forward question is where capability actually lives — in the parameters, in the computation graph, or in the trajectory through state-space.
What this episode covers
The dominant recipe for building capable language models is depth — more layers, more parameters, more distinct transformer blocks. A new mechanistic interpretability paper from Nam, Gromov, Yaida and colleagues looks at what happens when you take a much smaller model and run the same block over and over again in a loop. Inside, the internal state moves through cyclic trajectories that settle into fixed points; the attention heads stay consistent across iterations; each pass through the loop is doing the work that a separate layer would do in a traditional deep model. The cross-domain parallel: an assembly line versus a solo sculptor walking around the same piece of marble many times. The forward question is where capability actually lives — in the parameters, in the computation graph, or in the trajectory through state-space.
NOW PLAYING
EP006 — A Small Loop That Acts Like a Deep Model (Looped Reasoning)
No transcript for this episode yet
Similar Episodes
Apr 21, 2026 ·13m
Apr 19, 2026 ·16m
Apr 17, 2026 ·13m
Apr 13, 2026 ·11m
Apr 11, 2026 ·16m