Pushing LLM Pretraining Scales
from Mechanical Dreams Hosted by Mechanical Dirk
In this episode: • Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about dethroning AdamW. • The Batch Size Wall and MoEs: The hosts discuss why AdamW struggles at massive batch sizes, particularly for the dense layers in Mixture of Experts models. • Muon and Spectral Descent: Linda explains how Muon orthogonalizes momentum using Newton-Schulz iterations to maintain stability at scale. • Fixing SOAP's Slingshot Instability: The conversation shifts to SOAP, diagnosing the stale preconditioner problem and how real-time eigenbasis updates and KL-Shampoo fix it. • Systems Engineering and Final Verdict: Professor Norris and Linda discuss the layer-wise distributed optimizer built for Megatron-LM and conclude which optimizer wins.
Embed this episode
No transcript yet.