Do We Still Need GPUs? Rethinking AI with Matrix-Enhanced CPUs

This episode explores whether GPU dominance in AI computing could be challenged by matrix-enhanced CPUs, examining a paper by Jack Dongarra, Torsten Hoefler, and Satoshi Matsuoka from the University of Tennessee, ETH Zurich, and RIKEN. It traces why GPUs became essential for AI—citing AlexNet's 2012 breakthrough, NVIDIA's introduction of high-bandwidth memory with the P100, and tensor cores in Volta—before unpacking the two architectural bets the paper makes: on-package HBM (which physically stacks DRAM dies for a 1024-bit-wide interface versus 64 bits on conventional memory) and CPU-integrated matrix engines like ARM's SME or Intel's AMX combined with mixed-precision arithmetic. The discussion highlights Fugaku's A64FX chip as real-world proof that the bandwidth side of this equation already works, having topped the Top500 and memory-bound benchmark lists from 2020 to 2022, while noting the matrix-engine half remains a projection the paper tests on a trillion-parameter Kimi-K2 model at 256K-token context. Listeners interested in AI hardware economics will find this compelling for its rare rigor: the hosts stress that the authors clearly separate measured hardware results from projected estimates rather than blending speculation with data. The episode also breaks down the prefill-versus-decode distinction in LLM inference—compute-bound versus memory-bandwidth-bound—as the key lens for understanding where CPU architecture could realistically compete with GPUs. Sources: 1. Do We Still Need GPUs? Rethinking AI with Matrix-Enhanced CPUs https://podcast.do-not-panic.com/uploaded-pdfs/2026-07-09T15-03-00-780Z-need-gpus.pdf 2. A 1.2V 8Gb 8-Channel 128GB/s High-Bandwidth Memory (HBM) Stacked DRAM with Effective Microbump I/O Test Methods Using 29nm Process and TSV — D. U. Lee, K. W. Kim, K. W. Kim, et al. (SK Hynix), 2014 https://scholar.google.com/scholar?q=A+1.2V+8Gb+8-Channel+128GB%2Fs+High-Bandwidth+Memory+%28HBM%29+Stacked+DRAM+with+Effective+Microbump+I%2FO+Test+Methods+Using+29nm+Process+and+TSV 3. Co-Design for A64FX Manycore Processor and 'Fugaku' — Mitsuhisa Sato, Yutaka Ishikawa, Hirokazu Tomita, et al. (RIKEN/Fujitsu), 2020 https://scholar.google.com/scholar?q=Co-Design+for+A64FX+Manycore+Processor+and+%27Fugaku%27 4. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures 5. Bandwidth-Optimized Sapphire Rapids HBM (Xeon CPU Max Series) Technical Overview — Intel Corporation architecture team, 2023 https://scholar.google.com/scholar?q=Bandwidth-Optimized+Sapphire+Rapids+HBM+%28Xeon+CPU+Max+Series%29+Technical+Overview 6. DeepSeek-V3 Technical Report (and DeepSeek-V3.2 sparse-attention follow-up) — DeepSeek-AI, 2024-2025 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report+%28and+DeepSeek-V3.2+sparse-attention+follow-up%29 7. Kimi K2 Technical Report — Moonshot AI / Kimi Team, 2025 https://scholar.google.com/scholar?q=Kimi+K2+Technical+Report 8. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Y. Li, F. Wei, C. Zhang, H. Zhang (EAGLE line of work), 2024-2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 9. In-Datacenter Performance Analysis of a Tensor Processing Unit — N. Jouppi et al., 2017 https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit 10. NVIDIA Grace Hopper / Grace Blackwell Superchip architecture whitepapers — NVIDIA, 2022-2025 https://scholar.google.com/scholar?q=NVIDIA+Grace+Hopper+%2F+Grace+Blackwell+Superchip+architecture+whitepapers Interactive Visualization: Do We Still Need GPUs? Rethinking AI with Matrix-Enhanced CPUs