The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention

Episode title: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention A two-paper deep dive framed around the Dwarkesh Patel x Reiner Pope blackboard lecture on training and serving frontier LLMs. The hosts work through "Unified Scaling Laws for Routed Language Models" (Clark et al., DeepMind 2022, arXiv 2202.01169) for the mixture-of-experts side and the DeepSeek sparse-attention paper (arXiv 2512.02556) for the attention side, treating Pope's blackboard framing on the podcast as the pedagogical lens. The episode separates what the papers establish from what Pope's practitioner intuition adds on top, with particular attention to how MoE on the FFN side and sparse attention on the QK side attack independent cost pools and can compound rather than compete. Sources: arXiv 2202.01169 — "Unified Scaling Laws for Routed Language Models" https://arxiv.org/pdf/2202.01169 arXiv 2512.02556 — DeepSeek sparse-attention paper https://arxiv.org/pdf/2512.02556 Dwarkesh Podcast — "Reiner Pope: The math behind how LLMs are trained and served" (April 29 2026) https://www.dwarkesh.com/p/reiner-pope Transcript: https://gist.github.com/dwarkeshsp/79100f0fdeed69d76241903bb0604dbe Older MoE context: GShard (arXiv 2006.16668), Switch Transformer (arXiv 2101.03961) Chinchilla scaling laws (arXiv 2203.15556) — referenced in the Pope episode Interactive Visualization: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention