Language Models Can Control Their Own Attention
from Best AI papers explained Hosted by Enoch H. Kang
Researchers have introduced Declarative Attention (DA), a protocol that enables large language models to autonomously manage their own focus during long-context tasks. Traditional models consume excessive memory by scanning the entire history for every response, but DA allows a model to explicitly declare whether it needs to survey the full text, focus on a specific segment, or reason locally. By parsing these text-based declarations into dynamic attention masks, the system can skip irrelevant data and significantly reduce the number of tokens processed. Experiments on Gemma and Qwen models show that this approach cuts attention costs by up to 52% with only a minor impact on accuracy. This method effectively transforms selective attention from an internal calculation into a legible, instruction-driven process that scales efficiently with longer documents.
Embed this episode
No transcript yet.