Decode-Branch Transformers

In this episode: • The Inference Bottleneck: Introduction to the podcast and the fundamental asymmetry between compute-bound prefill and memory-bound decoding in LLMs. • Enter the Decode-Branch: Linda introduces the Decode-Branch Transformer, explaining how it separates prompt processing from continuation prediction without duplicating the KV cache. • Asymmetric Attention & Shared Weights: A deep dive into the architectural details, including asymmetric shared-KV attention and how the decode branch leverages grouped execution. • Mixture of Experts and Router Replay: Discussing the integration of the architecture with MoE models, introducing router replay, and exploring expert allocation trade-offs. • Training Costs and Takeaways: Norris and Linda analyze the training compute overhead, evaluate the empirical results, and share final thoughts.