How LLM Inference Actually Works

Every token from ChatGPT, Claude, or Gemini runs through the same chain: tokenization, embeddings, attention, KV cache, quantization, hardware. Each step is the bottleneck somewhere.Hero stat: GPT-4-equivalent capability fell from $60 per million tokens (Nov 2021 GPT-3) to 7 cents (April 2026). A 300X collapse in under four years.This episode walks the full mechanism, then the economics.Tokenization: Llama 3's vocab is 128,000 subword pieces. The tokenizer is trained once and frozen, which is why Chinese, Japanese, and Thai fracture into 2-4X more tokens. Same content, way higher API cost.Attention: every token checks against every other — N-squared in sequence length. 100,000 tokens → 10 billion comparisons. The KV cache: a 70B model at 100K context needs 32 GB just for cache, nearly half an H100's VRAM, for a single request. PagedAttention (vLLM) drops waste from 80% to under 4% — 24X throughput.FlashAttention: zero writes of the attention matrix to HBM. Quantization: FP32 → INT8 → INT4, 8X memory cut. Speculative decoding: a small draft model runs ahead, the big model verifies — 2.3X faster, mathematically lossless. Continuous batching pushes GPU utilization from 30-40% to 80-90%.Hardware: NVIDIA's H100/H200/B200 lineage. AMD MI300X (specs beat H100, software stack doesn't). Groq's all-SRAM LPU — 230 MB on-die at 80 TB/s, 24X an H100's bandwidth. Cerebras WSE-3 — 4 trillion transistors, 21 PB/s on-chip. Plus the Christmas Eve 2025 NVIDIA–Groq $20B licensing deal and the January 2026 OpenAI–Cerebras $10B compute deal.The cost timeline: GPT-3 at $60/million → GPT-4 at $30 → GPT-4o-mini at 15 cents → GPT-4-class at 7 cents. Accelerating at ~200X per year after January 2024, driven by DeepSeek and Google undercutting on Gemini.Three dated predictions for 2026-2028, ending on the line that ties the whole stack together: memory bandwidth — not compute — is the actual bottleneck of modern inference.RELATED EPISODESThe AI Chip War — hardware story under inference: NVIDIA / Groq / Cerebras + export controlsSpeculative Decoding — the next chapter on inference speed, deepenedThe Real Cost of AI — the economics of running inference at scaleRAG in Production — the retrieval layer on top of the inference stackComputer Use 45X — operational cost layer once inference is cheap enough to loopCHAPTERS00:00 Cold open — the 300X cost drop01:44 Intro and roadmap02:08 Tokenization — Llama 3's 128K vocab, the Chinese-fragmentation tax04:39 Embeddings — 8,192-dim vectors, ~2B params before processing07:38 Attention — Q/K/V, N-squared compute, the long-context cost11:23 Model scale — Llama 3 8B/70B/405B, DeepSeek V3's MoE13:19 KV cache — 32 GB per request, half an H100's VRAM16:44 PagedAttention — 80% waste to under 4%, 24X throughput17:55 Quantization — FP16 → INT8 → INT4, AWQ vs GPTQ19:49 FlashAttention — zero writes to HBM21:02 Speculative decoding — 2.3X faster, lossless24:37 Continuous batching and disaggregated inference26:14 Silicon — H100/H200/B200, the arithmetic intensity gap28:32 Groq's all-SRAM LPU + the $20B NVIDIA deal30:34 Cerebras WSE-3 + the $10B OpenAI deal32:58 Current pricing across Claude Opus 4.7, GPT-5, Gemini 2.5 Pro34:12 The 300X cost-decline timeline, 2021-202635:58 On-device inference — Apple Foundation Models, MLX42:04 Three dated predictions for 2026-202843:43 Closing — memory bandwidth is the bottleneckSOURCESPagedAttention / vLLM — Berkeley (2023)FlashAttention-3 — Tri Dao et al. (2024)EAGLE-3 speculative decoding — NeurIPS (2025)Llama 3 + DeepSeek V3 technical reports (Meta / DeepSeek)NVIDIA H100 / H200 / B200 datasheetsGroq LPU architecture (Groq engineering blog)Cerebras WSE-3 / CS-3 specs; AMD MI300X spec sheetNVIDIA–Groq $20B licensing deal — CNBC (Dec 24, 2025)OpenAI–Cerebras $10B compute deal (January 2026)