Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Episode two of Topic 3 goes inside the layer. When a single Transformer layer is too big for one chip, no pipeline schedule can save you — so Megatron-LM cuts the matrix multiplications themselves across GPUs, column-wise then row-wise, with an all-reduce 'huddle' only where partial results must meet. Maya and Leo walk the feed-forward and attention splits with their sixty-four-GPU team, then swap chairs and restage the tensor-versus-pipeline fight from the other side: no bubbles and graduate-student-readable code versus a conference call that never hangs up and a ceiling at the server chassis. Plus the four dials to check when tensor-parallel throughput disappoints. Sources: • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism: https://arxiv.org/pdf/1909.08053 • GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism: https://arxiv.org/pdf/1811.06965