GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

The first deep dive of Topic 3 takes on the bluntest bottleneck: the model does not fit on one device. Maya and Leo unpack GPipe's move — slice the layer stack into stages, stream microbatches through them like trays down a sandwich line, and re-materialize activations instead of storing them — then stage the field's real argument between pipeline and tensor parallelism: idle bubbles versus constant communication, reach across servers versus fully busy chips. Plus the trap of equal-layer splits, and the four measurements that tell you whether a pipeline is actually parallel or only looks that way on a diagram. Sources: • GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism: https://arxiv.org/pdf/1811.06965 • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism: https://arxiv.org/pdf/1909.08053