The Evolution and Scaling of Google’s TPU Supercomputers

This paper details the eight-year progression of Google’s Tensor Processing Units from the second generation through the latest Ironwood architecture. Despite a rapidly shifting AI landscape dominated by Transformers, the TPU has maintained a stable underlying design while achieving a 3600x increase in supercomputer performance. Key innovations such as optical circuit switches and SparseCores have enhanced system resilience and efficiency, allowing for massive scaling to over 9,000 nodes. The authors emphasize a shift toward power efficiency and sustainability, introducing Compute Carbon Intensity as a holistic metric for environmental impact. By prioritizing hardware-software codesign and architectural longevity, these chips have successfully navigated the decline of Moore’s Law to power modern AI workloads. Overall, the text positions the TPU as a foundational model for the future of AI supercomputing.