Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

Episode four of Topic 3 is the sequel to ZeRO's argument: what happens when sharding wins and moves into PyTorch as Fully Sharded Data Parallel. Maya and Leo open in a machine shop where no parts live at the bench — crates arrive exactly when a job needs them — and follow FSDP's loop of gathering full parameters for one wrapped block, computing locally, and letting the copy go. Then the four negotiations that decide whether the run gets faster or merely fits: crate size (wrapping policy), departure time (gather and prefetch), the recompute bargain (activation checkpointing), and the overflow annex (CPU offload). They weigh general-tool automation against hand-built parallel layouts, and close on the support-ticket diagnostic: memory measured by category, gathers checked against compute overlap, and the hard question of whether the pain just moved into network time. Sources: • Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs: https://engineering.fb.com/2021/07/15/open-source/fsdp/