PodParley PodParley
PodParley
Dion: Distributed Orthogonal Updates for Scalable AI Training episode artwork

EPISODE · Aug 4, 2025 · 5 MIN

Dion: Distributed Orthogonal Updates for Scalable AI Training

from Intellectually Curious · host Mike Breault

An exploration of the Dion optimizer (Distributed Orthogonal Updates) and how it tackles the scalability bottlenecks of training giant models. We break down why orthonormal updates matter, why Muon’s dense-matrix approach struggles with sharded, multi-GPU deployments, and how Dion uses amortized power iteration with QR and Cholesky on distributed shards to deliver fast, communication-efficient updates. Learn about integration with PyTorch DDP, FSDP2, and tensor parallelism, rank-fract compression with error feedback, and the empirical gains in wall-clock time over AdamW and Muon at scale—plus what this could unlock for the future of AI training.Note:  This podcast was AI-generated, and sometimes AI can make mistakes.  Please double-check any critical information.Sponsored by Embersilk LLC

Episode metadata supplied by the publisher feed · Published Aug 4, 2025

What this episode covers

An exploration of the Dion optimizer (Distributed Orthogonal Updates) and how it tackles the scalability bottlenecks of training giant models. We break down why orthonormal updates matter, why Muon’s dense-matrix approach struggles with sharded, multi-GPU deployments, and how Dion uses amortized power iteration with QR and Cholesky on distributed shards to deliver fast, communication-efficient updates. Learn about integration with PyTorch DDP, FSDP2, and tensor parallelism, rank-fract compres...

PodParley-generated summary based on available episode metadata and transcript content.

NOW PLAYING

Dion: Distributed Orthogonal Updates for Scalable AI Training

0:00 5:24

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

Chosn Conversations: Beyond the Journal Chosn AI Journal Welcome to Chosn Conversations: Beyond the Journal, where your AI hosts explore the transformative power of conversational journaling and emotional intelligence. Each episode takes you beyond traditional journaling methods, diving deep into voice journaling techniques, mental wellness strategies, and the science behind AI-supported emotional health. We share inspiring user stories, analyze the latest research in digital mental wellness, and provide practical guidance for incorporating journaling into your self-care routine. Whether you're curious about AI therapy alternatives, looking for mental health support tools, or wanting to optimize your journaling practice, our conversations extend beyond the written page into meaningful audio experiences that offer evidence-based insights in an accessible, compassionate format. Join us as we navigate the intersection of technology and mental well-being, helping you track your emotional journey and build lasting resilience through the power of Vampires of the Paper Flower Consortium Elizabeth Guizzetti Come for the evening, stay for eternity! Paper Flower Consortium is a podcast from the largest vampire coven in Seattle. Their stories are told by Loretta Fabron Onfoy, coven historian and librarian, in the hope that the modern vampire's way of life is not lost during the next great language transformation. Some tales in this anthology are horrific, some are droll, some are filled with misadventure--just like any eternal existence. Episodes sponsored by the Paper Flower Consortium's Business Community. The history is followed by questions from curious initiates. Want to ask Lady Loretta a question about vampirism? Have a topic you want to see discussed? Email [email protected] Awaken With JP Sears Show JP Sears Comedian, Life Coach, and curious student of life, JP Sears shares connected conversations with high level, inspiring, authentic, wickedly fascinating guests. Just being himself on the show, JP combines the humor that’s garnered him over 300 million online video views with his insight from over 15 years of being a life coach. This injection of empowerment will help you overcome challenges, uplevel your thinking, find more passion and purpose, and leave you entertainedAF! Ask your doctor if this podcast is right for you. But first, ask your chiropractor if asking your doctor is right for you. The All In One Show 239resulthomecare.com The All In One Show dives deep into the raw, real-life experiences of healthcare, exploring the highs, lows, and everything in between. From heartwarming success stories to the gritty, sometimes gory realities behind the scenes, this podcast is your unfiltered look at life in the healthcare field. Whether you're in the industry, curious about what goes on beyond the hospital doors, or simply drawn to real stories that inspire and challenge, this show offers something for everyone. Join me as I share personal insights, unforgettable moments, and the lessons learned from a career that’s as unpredictable as it is rewarding.

Frequently Asked Questions

How long is this episode of Intellectually Curious?

This episode is 5 minutes long.

When was this Intellectually Curious episode published?

This episode was published on August 4, 2025.

What is this episode about?

An exploration of the Dion optimizer (Distributed Orthogonal Updates) and how it tackles the scalability bottlenecks of training giant models. We break down why orthonormal updates matter, why Muon’s dense-matrix approach struggles with sharded,...

Can I download this Intellectually Curious episode?

Yes, you can download this episode by clicking the download button on the episode player, or subscribe to the podcast in your preferred podcast app for automatic downloads.
URL copied to clipboard!