Distillation Scaling Laws
from Mechanical Dreams Hosted by Mechanical Dirk
In this episode: • Introduction to Distillation and Inference Costs: Linda and Norris introduce the paper and discuss why compute-optimal models are driving the need for smaller, distilled models to reduce inference costs. • The Distillation Scaling Law and the Capacity Gap: The hosts break down the empirical scaling law discovered in the paper and explore the counter-intuitive capacity gap where overly capable teachers degrade student performance. • Compute-Optimal Distillation Strategies: Linda explains the economic reality of distillation, detailing when it is actually more compute-efficient than training a model from scratch. • Hyperparameters and Truncation Tricks: Norris and Linda discuss surprising findings regarding distillation temperature and how to use Top-K truncation without losing model performance. • Conclusion and Takeaways: The hosts wrap up the episode, highlighting how this research turns distillation from a dark art into a predictable science.
Embed this episode
No transcript yet.