Linear Transformers Implicitly Discover Unified Numerical Algorithms

The academic paper introduces a study on training a linear transformer to perform masked-block completion tasks on low-rank matrices, which simulates complex numerical problems like Nyström extrapolation. Surprisingly, the transformer implicitly discovers a single, unified, iterative numerical solver, termed EAGLE (Emergent Algorithm for Global Low-rank Estimation), despite being trained only on input-output pairs under a mean-squared loss objective. This discovered algorithm is robustly the same across three distinct computational constraints: centralized (full visibility), distributed (restricted communication), and computation-limited (low-dimensional attention) settings. Theoretically and empirically, EAGLE exhibits second-order convergence, which is significantly faster in terms of iteration complexity than classical first-order methods like Conjugate Gradient or Gradient Descent, positioning it as an efficient, resource-adaptive solver for prediction, estimation, and completion tasks.