How Much Do Language Models Memorize?

This research paper investigates language model capacity by introducing a new method to measure how much a model truly memorizes versus what it generalizes. The authors distinguish between unintended memorization, which is specific data storage, and generalization, which is the understanding of broader patterns. By testing the GPT family, they determine these models possess a storage capacity of approximately 3.6 bits-per-parameter. The study reveals that the double descent phenomenon occurs specifically when a dataset's size surpasses the model's total bit capacity. Furthermore, the researchers established scaling laws to predict the success of membership inference attacks, which identify if a specific datapoint was used in training. Their findings suggest that modern models are trained on so much data that standard membership inference is increasingly difficult for average samples.