Linear Attention Transforms RNNs and Accelerates Autoregression

The provided text is an excerpt from a research paper titled "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention," which focuses on addressing the quadratic computational complexity of traditional Transformer models, especially when processing long sequences. The authors introduce a "linear transformer" that reduces the complexity from $O(N^2)$ to $O(N)$ by expressing the self-attention mechanism as a linear dot-product of kernel feature maps. This new formulation allows for an iterative implementation that dramatically accelerates autoregressive prediction and reveals the relationship between transformers and recurrent neural networks (RNNs). Experimental results demonstrate that these linear transformers maintain performance comparable to standard softmax attention but are up to 4000x faster for tasks like image generation and automatic speech recognition inference. The paper details the mathematical derivations and presents empirical evidence across various synthetic and real-world tasks, showcasing the model's improved memory and time efficiency