How LLMs Got 3× Faster Without Getting Smarter: Speculative Decoding, Explained

Two language models running side by side are faster than one. A 60-million-parameter model drafting tokens for an 11-billion-parameter model gave Google a 2-to-3× speedup with mathematically guaranteed identical output. The smaller model is wrong about a third of the time. The bigger model only verifies in parallel. And somehow you come out ahead.That's speculative decoding. The original paper landed the same day as ChatGPT — November 30, 2022. Today it runs inside Google Search, vLLM, TensorRT, and every major LLM serving stack on the planet.This episode is the sequel to "How LLM Inference Actually Works." The mechanism. The four-line proof that says you cannot lose quality, ever. The Leviathan formula — three numbers (acceptance rate, draft length, cost ratio) that determine the speedup. Plug them in and you get the answer.The architecture progression: small-LLM drafts (2022) → MEDUSA (2024, prediction heads on the target) → EAGLE (2024, predict feature vectors) → EAGLE-3 (2025, multi-layer feature fusion, 3.0-6.5×) → Lookahead Decoding (no draft model at all). Block Verification (ICLR 2025) — the original inventor still evolving the algorithm.The honest production reality. Research papers say 5-6×. vLLM at production concurrency reports 1.2 to 2.5×. The Red Hat gpt-oss-120B benchmark hits +9.5 to 20.7 percent throughput improvement, not 3×. Acceptance rate below 0.55 turns the technique net-negative. Math at 0.518 actively hurts; code above 0.8 hits 6×+.Two case studies: Cursor's 13× speedup from using the file you're editing as the draft (not a draft model, structural prior). Morph Fast Apply at 10,500 tokens per second on a 7B model. The whole AI-code-editor category runs on this trick.MagicDec — counterintuitive long-context exception where speculative decoding helps MORE at larger batch.Five testable predictions. Closing thesis: two LLMs running together are faster than one. The math is as old as ChatGPT itself. And it is the reason your AI is faster every six months.RELATED EPISODESHow LLM Inference Actually Works — the mechanism this episode optimizesThe AI Chip War — H200/MI300X hardware substrate the benchmarks run onThe Real Cost of AI — economics of why 3× cheaper inference mattersComputer Use 45× — the cost layer above inferenceRAG in Production — retrieval layer on top of the inference stackCHAPTERS00:00 Cold open — Two LLMs faster than one01:10 EP2 recap — memory-bound inference02:04 The mechanism — draft + verify04:20 The four-line proof — why it's lossless06:03 The Leviathan formula07:26 Architecture progression: small-LLM → MEDUSA → EAGLE → EAGLE-3 → Lookahead09:33 Block Verification (ICLR 2025)10:07 Production reality — research vs serving11:15 SpecDecode-Bench falloff + MagicDec exception12:41 The α=0.55 floor + domain spread13:12 Cursor 13× (file-as-draft) + Morph 10,500 tps14:28 What spec decoding enabled (Realtime Voice, AI-code-editor)14:59 The frontier — SSD, DFlash, speculative cascades16:30 Five predictions17:49 Closing thesisSOURCESNov 30 2022 — Leviathan, Kalman, Matias (Google) 'Fast Inference from Transformers via Speculative Decoding'Feb 2 2023 — Chen et al. (DeepMind) 'Accelerating LLM Decoding with Speculative Sampling'Jan 2024 — MEDUSA paper (multiple decoding heads)Jan 2024 — EAGLE paper (feature-level autoregression)Mar 2025 — EAGLE-3 (NeurIPS 2025, multi-layer feature fusion)Nov 2023 — Lookahead Decoding (LMSYS / Hao AI Lab)ICLR 2025 — Block Verification (Leviathan co-authored)Aug 2024 — MagicDec long-context paperDec 2025 — Google DFlash (block-diffusion on TPU v5p)Apr 2026 — Red Hat gpt-oss-120B production benchmark on H200Oct 2024 — vLLM speculative decoding blog (2.8× CNN/DailyMail at QPS=1)May 2024 — Cursor 'Editing files at 1000 tokens/sec'Berkeley EECS-2025-224 — Liu, 'Efficient LLM System with Speculative Decoding'