How LLMs Got 3× Faster Without Getting Smarter: Speculative Decoding, Explained

Two language models running side by side are faster than one. A 60-million-parameter model drafting tokens for an 11-billion-parameter model gave Google a 2-to-3× speedup with mathematically guaranteed identical output. The smaller model is wrong about a third of the time. The bigger model only verifies in parallel. And somehow you come out ahead. That's speculative decoding. The original paper landed the same day as ChatGPT — November 30, 2022. Today it runs inside Google Search, vLLM, TensorRT, and every major LLM serving stack on the planet. This episode is the sequel to "How LLM Inference Actually Works." The mechanism. The four-line proof that says you cannot lose quality, ever. The Leviathan formula — three numbers (acceptance rate, draft length, cost ratio) that determine the speedup. Plug them in and you get the answer. The architecture progression: small-LLM drafts (2022) → MEDUSA (2024, prediction heads on the target) → EAGLE (2024, predict feature vectors) → EAGLE-3 (2025, multi-layer feature fusion, 3.0-6.5×) → Lookahead Decoding (no draft model at all). Block Verification (ICLR 2025) — the original inventor still evolving the algorithm. The honest production reality. Research papers say 5-6×. vLLM at production concurrency reports 1.2 to 2.5×. The Red Hat gpt-oss-120B benchmark hits +9.5 to 20.7 percent throughput improvement, not 3×. Acceptance rate below 0.55 turns the technique net-negative. Math at 0.518 actively hurts; code above 0.8 hits 6×+. Two case studies: Cursor's 13× speedup from using the file you're editing as the draft (not a draft model, structural prior). Morph Fast Apply at 10,500 tokens per second on a 7B model. The whole AI-code-editor category runs on this trick. MagicDec — counterintuitive long-context exception where speculative decoding helps MORE at larger batch. Five testable predictions. Closing thesis: two LLMs running together are faster than one. The math is as old as ChatGPT itself. And it is the reason your AI is faster every six months. RELATED EPISODES How LLM Inference Actually Works — the mechanism this episode optimizes The AI Chip War — H200/MI300X hardware substrate the benchmarks run on The Real Cost of AI — economics of why 3× cheaper inference matters Computer Use 45× — the cost layer above inference RAG in Production — retrieval layer on top of the inference stack CHAPTERS 00:00 Cold open — Two LLMs faster than one 01:10 EP2 recap — memory-bound inference 02:04 The mechanism — draft + verify 04:20 The four-line proof — why it's lossless 06:03 The Leviathan formula 07:26 Architecture progression: small-LLM → MEDUSA → EAGLE → EAGLE-3 → Lookahead 09:33 Block Verification (ICLR 2025) 10:07 Production reality — research vs serving 11:15 SpecDecode-Bench falloff + MagicDec exception 12:41 The α=0.55 floor + domain spread 13:12 Cursor 13× (file-as-draft) + Morph 10,500 tps 14:28 What spec decoding enabled (Realtime Voice, AI-code-editor) 14:59 The frontier — SSD, DFlash, speculative cascades 16:30 Five predictions 17:49 Closing thesis SOURCES Nov 30 2022 — Leviathan, Kalman, Matias (Google) 'Fast Inference from Transformers via Speculative Decoding' Feb 2 2023 — Chen et al. (DeepMind) 'Accelerating LLM Decoding with Speculative Sampling' Jan 2024 — MEDUSA paper (multiple decoding heads) Jan 2024 — EAGLE paper (feature-level autoregression) Mar 2025 — EAGLE-3 (NeurIPS 2025, multi-layer feature fusion) Nov 2023 — Lookahead Decoding (LMSYS / Hao AI Lab) ICLR 2025 — Block Verification (Leviathan co-authored) Aug 2024 — MagicDec long-context paper Dec 2025 — Google DFlash (block-diffusion on TPU v5p) Apr 2026 — Red Hat gpt-oss-120B production benchmark on H200 Oct 2024 — vLLM speculative decoding blog (2.8× CNN/DailyMail at QPS=1) May 2024 — Cursor 'Editing files at 1000 tokens/sec' Berkeley EECS-2025-224 — Liu, 'Efficient LLM System with Speculative Decoding'