Continuous-Query Limited Memory Language Models

In this episode: • Welcome & The Parametric Memory Bottleneck: Professor Norris and Linda introduce the episode's topic, discussing the fundamental flaws of storing facts in model weights. • From Relational to Continuous Queries: Linda explains the limitations of prior Limited Memory Language Models (LMLMs) and introduces Co-LMLM's continuous-query approach. • Under the Hood: Training and The Token: The hosts dive into the technical details of how the model uses hidden states as retrieval queries via a special token. • Scaling Up: The Annotation Pipeline: Discussion on how the researchers scaled their training data to 100 billion tokens using a clever distillation pipeline. • Results, Unlearning, and Wrap Up: Reviewing the model's performance on factuality benchmarks and its elegant solution to machine unlearning.