A Mechanistic Analysis of Looped Reasoning Language Models

This paper provides a mechanistic analysis of looped language models, which reuse specific Transformer layers in a recurrent cycle to increase computational depth without adding parameters. The authors demonstrate that these models frequently converge to cyclic fixed points, creating stable, repeating trajectories in latent space that maintain consistent attention patterns. Crucially, the research reveals that these recurrent blocks self-organize into "stages of inference"—such as information mixing and compression—that closely mirror the behavior of standard feedforward models. The study further identifies how architectural choices like input injection and normalization determine whether a model remains stable when extrapolated to higher recurrence counts during inference. These insights suggest that looped architectures naturally replicate the computational hierarchies of larger models, offering a path toward more efficient design for complex reasoning tasks.