Weight Decay Improves Language Model Plasticity

In this episode: • Introduction and The Premise: Professor Norris and Linda introduce the episode's focus: a new paper showing how weight decay impacts language model plasticity. • Challenging the Validation Loss Paradigm: Linda explains how optimizing for pretraining validation loss can actually hurt downstream fine-tuning performance. • The Mechanics of Plasticity: A deep dive into why weight decay helps, covering linearly separable representations, attention matrix rank reduction, and the prevention of overfitting. • Trade-offs and Practical Implications: The hosts discuss the stability-plasticity dilemma and how these findings should change the way industry teams tune hyperparameters. • Conclusion: Professor Norris concedes his skepticism and the hosts wrap up the episode.