Retrieval Transformer

The sources describe RETRO (Retrieval-Enhanced Transformer), a language model that enhances its performance by retrieving information from a large database. RETRO uses a key-value store where keys are BERT embeddings of text chunks and values are the text chunks themselves. When processing input, it retrieves similar text chunks from the database to augment the input, allowing it to perform comparably to much larger models. By incorporating this retrieved information through a chunked cross-attention mechanism, RETRO reduces the need to memorize facts and improves its performance on knowledge-intensive tasks. The database contains trillions of tokens.