EPISODE · May 9, 2026 · 7 MIN
🚀 Scaling LLMs is no longer just about more GPUs—it's about the geometry of the cluster-Stanford's CS336 Lecture 8
from Steven AI Talk · host Steven
🚀 Scaling LLMs is no longer just about more GPUs—it's about the geometry of the cluster.In Stanford's CS336 Lecture 8, we dive deep into the parallelization strategies that make training trillion-parameter models possible. From Zero Redundancy Optimizers (ZeRO) to 4D parallelism, the complexity is staggering.Key Takeaways: 🔹 ZeRO-3 (FSDP) allows sharding parameters "almost for free" on high-speed networks. 🔹 Tensor Parallelism is mandatory for intra-node scaling but relies on massive bandwidth. 🔹 Pipeline Parallelism is the bridge for cross-node training, now improved with "Zero-Bubble" techniques. 🔹 Expert Parallelism (MoE) decouples MLP layers for sparse routing efficiency.The golden rule? Use all sharding methods until the model fits in memory, then scale with Data Parallelism.Check out the full technical summary and transcripts in our repo! All my links: https://linktr.ee/learnbydoingwithsteven#learnbydoingwithsteven #LLM #AIResearch #DistributedComputing #MachineLearning #DeepLearning #StanfordCS336 #GPU #TPU #ModelParallelism #DataParallelism
What this episode covers
🚀 Scaling LLMs is no longer just about more GPUs—it's about the geometry of the cluster.In Stanford's CS336 Lecture 8, we dive deep into the parallelization strategies that make training trillion-parameter models possible. From Zero Redundancy Optimizers (ZeRO) to 4D parallelism, the complexity is staggering.Key Takeaways: 🔹 ZeRO-3 (FSDP) allows sharding parameters "almost for free" on high-speed networks. 🔹 Tensor Parallelism is mandatory for intra-node scaling but relies on massive bandwidth. 🔹 Pipeline Parallelism is the bridge for cross-node training, now improved with "Zero-Bubble" techniques. 🔹 Expert Parallelism (MoE) decouples MLP layers for sparse routing efficiency.The golden rule? Use all sharding methods until the model fits in memory, then scale with Data Parallelism.Check out the full technical summary and transcripts in our repo! All my links: https://linktr.ee/learnbydoingwithsteven#learnbydoingwithsteven #LLM #AIResearch #DistributedComputing #MachineLearning #DeepLearning #StanfordCS336 #GPU #TPU #ModelParallelism #DataParallelism
NOW PLAYING
🚀 Scaling LLMs is no longer just about more GPUs—it's about the geometry of the cluster-Stanford's CS336 Lecture 8
No transcript for this episode yet
Similar Episodes
No similar episodes found.
Similar Podcasts
No similar podcasts found.