Skip to content
Defeating Nondeterminism in LLM Inference episode artwork

Sep 23, 2025 15 min

Defeating Nondeterminism in LLM Inference

from Personal Podcast Hosted by John Doe

The text from the Thinking Machines Lab explores the challenge of achieving reproducible results when using large language models (LLMs), a problem termed nondeterminism in LLM inference. While a common explanation points to floating-point non-associativity combined with concurrent execution on GPUs, the authors argue that this "concurrency + floating point" hypothesis is incomplete, as core LLM operations are often run-to-run deterministic. The true culprit behind the lack of reproducible results from a user's perspective is the lack of "batch invariance" in crucial kernels like RMSNorm, matrix multiplication, and attention, meaning an individual request's output depends on the variable, non-deterministic batch size imposed by concurrent user load. To defeat this nondeterminism, the article proposes and demonstrates the implementation of batch-invariant kernels and shows that this approach allows for truly on-policy reinforcement learning by ensuring numerical consistency between sampling and training. ============== Code content percentage: 4.661% Total text length: 37718 characters 🔗 Original article: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ 📋 Monday item: https://omril321.monday.com/boards/3549832241/pulses/10045252264

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Defeating Nondeterminism in LLM Inference

0:00 15:39

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Personal Podcast?

This episode is 15 minutes long.

When was this Personal Podcast episode published?

This episode was published on September 23, 2025.

Can I download this Personal Podcast episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.