Conformal Tail Risk Control for Large Language Model Alignment

This paper introduces Conformal Bayesian Optimization (Conformal BayesOpt), a novel approach designed to enhance Bayesian Optimization (BayesOpt) by integrating conformal prediction sets. Traditional BayesOpt often faces challenges like unreliable predictions due to model misspecification and covariate shift, particularly when selecting new data points. Conformal BayesOpt addresses these issues by directing queries towards regions where model predictions are statistically guaranteed to be valid, even with imperfect models, and includes a mechanism to correct for covariate shift. The research demonstrates that this method significantly improves the reliability of query outcomes while maintaining comparable sample-efficiency in various optimization tasks, including drug design.