OAPN #5: Distributed Training 101

In OAPN #5, we introduce the basics of distributed language model training.Using educational material from Nathan Lambert’s RLHF Course and Stanford’s CS336, we walk through the foundations of conventional model training and explain the core concepts in a way that connects directly to what OpenAgents is building.We then introduce DiLoCo, or Distributed Low-Communication Training, and discuss how projects like Bittensor, Templar, and Prime Intellect have approached distributed training, along with how the Psionic and Pylon approach differs.This episode is meant to help people understand what is actually happening under the hood when their machines participate in networked training and why that matters for the future of open AI infrastructure.If you want to understand what runs on your computer to earn bitcoin as part of a distributed training network, this is the place to start.Links:Website: https://openagents.comDocs: https://docs.openagents.comGitHub: https://github.com/OpenAgentsInc/openagentsPsionic: https://github.com/OpenAgentsInc/psionicX: https://x.com/OpenAgentsInc This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit openagents.substack.com