Advanced Distributed Training: Overcoming Bottlenecks

Topic 3 opens with a map of the bottlenecks that decide whether a hundred-billion-parameter model can be trained at all: model-state memory, activation memory, GPU-to-GPU communication, pipeline bubbles, and the data movement inside a single chip. Maya and Leo stage the field's real argument — partition the model versus shard the redundant states — introduce the four levers (copy, slice, split, shard), and set up the diagnostic habit behind GPipe, Megatron-LM, ZeRO, FSDP, and FlashAttention: find the bottleneck first, then choose the technique. Sources: • GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism: https://arxiv.org/pdf/1811.06965 • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism: https://arxiv.org/pdf/1909.08053 • ZeRO: Memory Optimization Towards Training Trillion Parameter Models: https://arxiv.org/pdf/1910.02054 • Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs: https://engineering.fb.com/2021/07/15/open-source/fsdp/ • Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing: https://dl.acm.org/doi/pdf/10.1145/3728725.3728812 • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: https://arxiv.org/pdf/2205.14135 • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning: https://arxiv.org/pdf/2307.08691 • Embarrassingly Simple Self-Distillation Improves Code Generation (SSD): https://arxiv.org/pdf/2604.01193 • Test-Time Scaling Makes Overtraining Compute-Optimal: https://arxiv.org/pdf/2604.01411