A Comprehensive Survey of Efficient Transformer Models

The provided text is an excerpt from a comprehensive survey titled "Efficient Transformers" published in ACM Computing Surveys, which addresses the challenges and innovations surrounding the original Transformer architecture. The survey focuses on the quadratic complexity of the self-attention mechanism and how various "X-former" models, such as Reformer and Longformer, aim to improve computational and memory efficiency across domains like language and vision. The authors present a detailed taxonomy of these efficient Transformer models, categorizing them based on core techniques like Fixed Patterns, Learnable Patterns, Low-Rank methods, and the use of Neural Memory. Additionally, the paper discusses the nuances of model evaluation and design trends, while also giving a technical background on the standard Transformer block and orthogonal efficiency efforts like parameter sharing and quantization. Ultimately, the work serves as a guide for researchers navigating the rapid development of more efficient deep learning models