ZeRO: Memory Optimization Towards Training Trillion Parameter Models

Episode three of Topic 3 stops cutting the model. After two episodes of pipeline and tensor surgery, ZeRO — the Zero Redundancy Optimizer — asks the almost-rude question: what if the real waste was never the math, but the copies? Maya and Leo open in a town where eight libraries stock identical collections and ledgers fatter than the books, then follow the sharding ladder — optimizer states, gradients, finally the parameters themselves — as memory turns from possession into scheduling. They stage the split-the-model versus shard-the-states argument with concessions on both sides, land on the paper's own answer (compose them), and close with the support-call diagnostic: six memory tenants, measured separately, and the question of whether the savings just moved into network time. Sources: • ZeRO: Memory Optimization Towards Training Trillion Parameter Models: https://arxiv.org/pdf/1910.02054 • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism: https://arxiv.org/pdf/1909.08053 • Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs: https://engineering.fb.com/2021/07/15/open-source/fsdp/