Stochastic KV Routing for Cache Sharing

This episode explores Apple’s Stochastic KV Routing approach for reducing transformer inference costs by letting some layers reuse key-value caches from earlier layers instead of storing separate caches at every depth. It explains why KV cache memory becomes a major bottleneck for long-context autoregressive decoding, and why depth-wise cache sharing is a different idea from token eviction or temporal compression. The discussion connects the paper to Grouped Query Attention and other prior cache-sharing methods, highlighting Apple’s main argument: train models with stochastic cross-layer routing so they can tolerate many cache-retention layouts and then expose a practical serving-time knob for trading memory use against model quality. A listener would find it interesting because it ties a concrete systems problem in LLM deployment to a training strategy that could make large models more flexible under real hardware constraints. Sources: 1. Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing — Anastasiia Filippova, David Grangier, Marco Cuturi, João Monteiro, 2026 http://arxiv.org/abs/2604.22782 2. Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, 2019 https://scholar.google.com/scholar?q=Fast+Transformer+Decoding%3A+One+Write-Head+is+All+You+Need 3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023 https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints 4. Layer-Condensed KV Cache for Efficient Inference of Large Language Models — Haoyi Wu, Kewei Tu, 2024 https://scholar.google.com/scholar?q=Layer-Condensed+KV+Cache+for+Efficient+Inference+of+Large+Language+Models 5. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models — Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, Bohan Zhuang, 2024 https://scholar.google.com/scholar?q=MiniCache%3A+KV+Cache+Compression+in+Depth+Dimension+for+Large+Language+Models 6. Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing — Anastasiia Filippova, David Grangier, Marco Cuturi, Joao Monteiro, 2026 https://scholar.google.com/scholar?q=Stochastic+KV+Routing%3A+Enabling+Adaptive+Depth-Wise+Cache+Sharing 7. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan-Kelley, 2024 https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention 8. XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference — Joao Monteiro, Etienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian, 2024 https://scholar.google.com/scholar?q=XC-Cache%3A+Cross-Attending+to+Cached+Context+for+Efficient+LLM+Inference 9. KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing — not identifiable from snippet, recent https://scholar.google.com/scholar?q=KVSharer%3A+Efficient+Inference+via+Layer-Wise+Dissimilar+KV+Cache+Sharing 10. Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity — not identifiable from snippet, recent https://scholar.google.com/scholar?q=Compressing+KV+Cache+for+Long-Context+LLM+Inference+with+Inter-Layer+Attention+Similarity 11. CaR: An Efficient KV Cache Reuse System for Large Language Model Inference — not identifiable from snippet, recent https://scholar.google.com/scholar?q=CaR%3A+An+Efficient+KV+Cache+Reuse+System+for+Large+Language+Model+Inference 12. Compute or Load KV Cache? Why Not Both? — not identifiable from snippet, recent https://scholar.google.com/scholar?q=Compute+or+Load+KV+Cache%3F+Why+Not+Both%3F 13. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp3 14. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp3 15. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3 16. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp3 17. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp3 18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3 Interactive Visualization: Stochastic KV Routing for Cache Sharing