Skip to content
LLMs are Bayesian, In Expectation, Not in Realization episode artwork

Mar 1, 2026 19 min

LLMs are Bayesian, In Expectation, Not in Realization

from Best AI papers explained Hosted by Enoch H. Kang

This research explores the discrepancy between transformer in-context learning and Bayesian inference, arguing that models are Bayesian in expectation rather than through every individual realization. While previous studies used martingale diagnostics to question the Bayesian nature of these models, this paper identifies positional encodings as the primary factor that breaks the required exchangeability. By accounting for how architectural design prioritizes sequence order, the authors prove that transformers still achieve near-optimal compression and information-theoretic efficiency when performance is averaged across different orderings. Empirical tests on black-box LLMs and controlled ablations demonstrate that order-induced variance exists but predictably decays as context length increases. Ultimately, the study suggests permutation averaging as a practical and effective method for reducing uncertainty and improving the reliability of model outputs in tasks with exchangeable data.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

LLMs are Bayesian, In Expectation, Not in Realization

0:00 19:12

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Best AI papers explained?

This episode is 19 minutes long.

When was this Best AI papers explained episode published?

This episode was published on March 1, 2026.

Can I download this Best AI papers explained episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.