Textual Bayes: Quantifying Uncertainty in LLM-Based Systems

This paper titled "Textual Bayes: Quantifying Uncertainty in LLM-Based Systems," available on arXiv. This paper addresses the critical challenge of quantifying uncertainty in large language model (LLM)-based systems, which is crucial for their application in high-stakes environments. The authors propose a novel Bayesian approach where prompts are treated as textual parameters within a statistical model, allowing for principled uncertainty quantification through Bayesian inference. To achieve this, they introduce Metropolis-Hastings through LLM Proposals (MHLP), a new Markov chain Monte Carlo algorithm designed to integrate Bayesian methods into existing LLM pipelines, even with closed-source models. The research demonstrates improvements in predictive accuracy and uncertainty quantification, highlighting a viable path for incorporating robust Bayesian techniques into the evolving field of LLMs.