Kimi Linear: An Expressive, Efficient Attention Architecture

Topic 1 closes with the 2017 paper's confession answered. Kimi Linear, from Moonshot AI's Kimi Team, claims a first: a mostly-linear attention architecture that beats full attention under matched training runs. Maya builds the machine in four landmarks — the Board (a fixed-size memory that never grows), the Eraser (the delta rule's overwrite-don't-pile update), the Knobs (a learned forgetting dial per feature channel, which doubles as the position signal), and the Floorplan (three linear layers for every one full-attention layer, with the full layers carrying no position encoding at all). Leo brings the history of linear-attention promises and prosecutes the fair-fight claim — author-chosen benchmarks, 48-billion-parameter scale, full attention 'kept on retainer' — before conceding what the architecture genuinely buys: up to seventy-five percent less KV cache and roughly six-times-faster decoding at a million tokens. The running law-firm deal-room assistant grounds the stakes, and the hosts land on what would actually settle the argument: replication outside the lab. Sources: • Kimi Linear: An Expressive, Efficient Attention Architecture: https://arxiv.org/pdf/2510.26692 • Attention Is All You Need: https://arxiv.org/pdf/1706.03762 • Gated Delta Networks: Improving Mamba2 with Delta Rule: https://arxiv.org/pdf/2412.06464 • KDA kernel in flash-linear-attention (open-source implementation): https://github.com/fla-org/flash-linear-attention/tree/main/fla/ops/kda • Kimi-Linear-48B-A3B-Instruct model checkpoint: https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct