EPISODE · May 12, 2025 · 16 MIN
Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective
from Best AI papers explained · host Enoch H. Kang
This academic paper explores whether the in-context learning (ICL) process in Large Language Models (LLMs) behaves like Bayesian inference. The authors use the martingale property, a key characteristic of Bayesian systems with exchangeable data, as a framework for their analysis. They demonstrate that violations of this property and deviations in how LLMs' uncertainty scales with more data provide evidence that ICL is not Bayesian. The findings suggest that LLMs lack a principled understanding of uncertainty in exchangeable scenarios, raising concerns about their reliability in applications where data order doesn't matter or where quantifiable uncertainty is critical.
What this episode covers
This academic paper explores whether the in-context learning (ICL) process in Large Language Models (LLMs) behaves like Bayesian inference. The authors use the martingale property, a key characteristic of Bayesian systems with exchangeable data, as a framework for their analysis. They demonstrate that violations of this property and deviations in how LLMs' uncertainty scales with more data provide evidence that ICL is not Bayesian. The findings suggest that LLMs lack a principled understanding of uncertainty in exchangeable scenarios, raising concerns about their reliability in applications where data order doesn't matter or where quantifiable uncertainty is critical.
NOW PLAYING
Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective
No transcript for this episode yet
Similar Episodes
Mar 31, 2026 ·54m
Mar 27, 2026 ·14m
Mar 24, 2026 ·42m
Mar 20, 2026 ·42m
Mar 17, 2026 ·41m
Mar 13, 2026 ·44m