“In modern LLMs, a lot of the compute and memory is tied up…” — Y Combinator Startup Podcast at 8:12
“In modern LLMs, a lot of the compute and memory is tied up in the KV cache, and so V3 makes use of MLA, which compresses keys and values into a smaller latent space before caching them, then decompresses them during inference. Although MLA is a bit more complex to implement, the previous DeepSeq V2 paper found it…”
0:00
12:31