How RAG Actually Works (and Why Most Production Systems Are Broken)

Retrieval-Augmented Generation is in every production LLM application now. Most of them fail in similar, specific ways — and the fixes are mostly not about the LLM. This episode walks through the pipeline layer by layer, from chunking to embeddings to vector indexes to hybrid retrieval to reranking, with empirical numbers from two production RAG systems built for this show — including the one that caught two real factual errors in an already-published episode.The thesis: the 80/20 of RAG quality lives in retrieval, not in the language model at the end. Anthropic's Contextual Retrieval reduced retrieval failure rate by 67 percent without touching the LLM. That's the shape of the problem.What's actually covered. The Lewis et al. 2020 paper that named RAG, and how modern production diverges from it. Why your cosine-similarity thresholds are probably wrong (empirical distribution on text-embedding-3-small: off-topic 0.10 to 0.25, narrative match 0.50 to 0.65, sequel-grade overlap 0.65 to 0.70 — set thresholds from observed distribution, not textbook defaults). HNSW, IVF, Product Quantization — when each wins at scale, and why a billion-vector index needs 6 terabytes of RAM at full precision. Hybrid retrieval with BM25 plus dense embedding, plus reranking — Anthropic's 5.7 to 1.9 percent failure cascade as the cleanest published demonstration.Then the production failure modes. Junk retrieval. Missing context. Hallucination on grounded generation. Stale data. Multi-document reasoning failures. Lost in the middle. And the seventh: wrong-topic evidence retrieval. The "Cheng versus Costello" pattern — the verify-claims-rag system flagged a script claim as wrong, citing evidence about a different study. The retrieval surfaced a related-but-different paper and the judge couldn't tell. Demonstrated live on the script for this episode.RAG versus long context. Claude 4.7 at 1 million tokens. GPT-5.5 at 1 million. Gemini 2 at 2 million. The 2024 question — is RAG obsolete — has a clearer 2026 answer. No. But the line moved. RULER showed the headline 1 million-token context claims drop to roughly 60 percent effective recall on real long-document tasks even when Needle in a Haystack says 99 percent. The 2026 default architecture is compound: long context for cross-document reasoning, RAG for fresh data and citation, light fine-tuning for output format.Plus five predictions on where the field is going through end of 2026. Companion to the show's "How LLM Inference Actually Works" — same shape, different layer.RELATED EPISODESHow LLM Inference Actually Works — companion technical-mechanism episode (same shape, different layer)Speculative Decoding — companion inference-optimization episodeWarm AI Sycophancy — LLM reliability lensCerebras IPO — the case study (this episode caught two errors there)CHAPTERS00:00 Cold open — two errors caught in a published episode01:16 Today's pipeline03:11 Chunking05:57 Embeddings and the threshold table08:57 Vector indexes11:35 Hybrid retrieval and reranking14:12 What breaks in production17:51 Cheng-vs-Costello pattern + EP34 catches19:26 RAG vs long context21:17 The frontier and predictions24:43 Closing — the trust layerSOURCESLewis et al. 2020 — RAG (arxiv 2005.11401) + Karpukhin DPR + Khattab ColBERTMalkov & Yashunin — HNSW (arxiv 1603.09320) + Cormack RRFAnthropic — Contextual Retrieval (anthropic.com/news/contextual-retrieval)Asai Self-RAG + Yan Corrective RAG + Edge GraphRAGLiu Lost in the Middle + Hsieh RULER (arxiv 2404.06654)Databricks — Long Context RAG Capabilities (Oct 2024)Cheng 2025 sycophancy follow-up (N=1,604) + Costello 2024 DebunkBot (Science, N=2,190)Notion — Turbopuffer migration + Klarna 2024/2025 walkback + MongoDB-Voyage AI