EPISODE · Jun 11, 2026 · 21 MIN
EP2:
from AI Without Illusions · host csbaby
In this episode, we tackle the hardcore systems engineering challenge of physically fitting massive Large Language Models onto GPU clusters. Stepping away from heavy math and using relatable analogies—like a factory assembly line—we break down the four key dimensions of distributed training: data parallelism, tensor parallelism, pipeline parallelism, and ZeRO/FSDP sharding. We explain exactly what each method shards across your hardware and the practical scenarios for when to reach for them, providing a clear, systems-level framing for engineers looking to train at scale.
What this episode covers
In this episode, we tackle the hardcore systems engineering challenge of physically fitting massive Large Language Models onto GPU clusters. Stepping away from heavy math and using relatable analogies—like a factory assembly line—we break down the four key dimensions of distributed training: data parallelism, tensor parallelism, pipeline parallelism, and ZeRO/FSDP sharding. We explain exactly what each method shards across your hardware and the practical scenarios for when to reach for them, providing a clear, systems-level framing for engineers looking to train at scale.
NOW PLAYING
EP2:
No transcript for this episode yet
Similar Episodes
Mar 31, 2026 ·54m
Mar 27, 2026 ·14m
Mar 24, 2026 ·42m
Mar 20, 2026 ·42m
Mar 17, 2026 ·41m
Mar 13, 2026 ·44m