“and rethink attention, but instead thought up various…” — Latent Space: The AI Engineer Podcast at 2:47
“and rethink attention, but instead thought up various quite useful, but ultimately workarounds, right? Band-Aid solutions, things like windowed attention. So this is very common. Anybody who says they're using a transformer with a context length of, you know, 256,000 or more, they're not using a true transformer.…”
0:00
32:59