How LLM Inference Actually Works

Every token from ChatGPT, Claude, or Gemini runs through the same chain: tokenization, embeddings, attention, KV cache, quantization, hardware. Each step is the bottleneck somewhere. Hero stat: GPT-4-equivalent capability fell from $60 per million tokens (Nov 2021 GPT-3) to 7 cents (April 2026). A 300X collapse in under four years. This episode walks the full mechanism, then the economics. Tokenization: Llama 3's vocab is 128,000 subword pieces. The tokenizer is trained once and frozen, which is why Chinese, Japanese, and Thai fracture into 2-4X more tokens. Same content, way higher API cost. Attention: every token checks against every other — N-squared in sequence length. 100,000 tokens → 10 billion comparisons. The KV cache: a 70B model at 100K context needs 32 GB just for cache, nearly half an H100's VRAM, for a single request. PagedAttention (vLLM) drops waste from 80% to under 4% — 24X throughput. FlashAttention: zero writes of the attention matrix to HBM. Quantization: FP32 → INT8 → INT4, 8X memory cut. Speculative decoding: a small draft model runs ahead, the big model verifies — 2.3X faster, mathematically lossless. Continuous batching pushes GPU utilization from 30-40% to 80-90%. Hardware: NVIDIA's H100/H200/B200 lineage. AMD MI300X (specs beat H100, software stack doesn't). Groq's all-SRAM LPU — 230 MB on-die at 80 TB/s, 24X an H100's bandwidth. Cerebras WSE-3 — 4 trillion transistors, 21 PB/s on-chip. Plus the Christmas Eve 2025 NVIDIA–Groq $20B licensing deal and the January 2026 OpenAI–Cerebras $10B compute deal. The cost timeline: GPT-3 at $60/million → GPT-4 at $30 → GPT-4o-mini at 15 cents → GPT-4-class at 7 cents. Accelerating at ~200X per year after January 2024, driven by DeepSeek and Google undercutting on Gemini. Three dated predictions for 2026-2028, ending on the line that ties the whole stack together: memory bandwidth — not compute — is the actual bottleneck of modern inference. RELATED EPISODES The AI Chip War — hardware story under inference: NVIDIA / Groq / Cerebras + export controls Speculative Decoding — the next chapter on inference speed, deepened The Real Cost of AI — the economics of running inference at scale RAG in Production — the retrieval layer on top of the inference stack Computer Use 45X — operational cost layer once inference is cheap enough to loop CHAPTERS 00:00 Cold open — the 300X cost drop 01:44 Intro and roadmap 02:08 Tokenization — Llama 3's 128K vocab, the Chinese-fragmentation tax 04:39 Embeddings — 8,192-dim vectors, ~2B params before processing 07:38 Attention — Q/K/V, N-squared compute, the long-context cost 11:23 Model scale — Llama 3 8B/70B/405B, DeepSeek V3's MoE 13:19 KV cache — 32 GB per request, half an H100's VRAM 16:44 PagedAttention — 80% waste to under 4%, 24X throughput 17:55 Quantization — FP16 → INT8 → INT4, AWQ vs GPTQ 19:49 FlashAttention — zero writes to HBM 21:02 Speculative decoding — 2.3X faster, lossless 24:37 Continuous batching and disaggregated inference 26:14 Silicon — H100/H200/B200, the arithmetic intensity gap 28:32 Groq's all-SRAM LPU + the $20B NVIDIA deal 30:34 Cerebras WSE-3 + the $10B OpenAI deal 32:58 Current pricing across Claude Opus 4.7, GPT-5, Gemini 2.5 Pro 34:12 The 300X cost-decline timeline, 2021-2026 35:58 On-device inference — Apple Foundation Models, MLX 42:04 Three dated predictions for 2026-2028 43:43 Closing — memory bandwidth is the bottleneck SOURCES PagedAttention / vLLM — Berkeley (2023) FlashAttention-3 — Tri Dao et al. (2024) EAGLE-3 speculative decoding — NeurIPS (2025) Llama 3 + DeepSeek V3 technical reports (Meta / DeepSeek) NVIDIA H100 / H200 / B200 datasheets Groq LPU architecture (Groq engineering blog) Cerebras WSE-3 / CS-3 specs; AMD MI300X spec sheet NVIDIA–Groq $20B licensing deal — CNBC (Dec 24, 2025) OpenAI–Cerebras $10B compute deal (January 2026)