FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch

This paper introduces FunBO, a novel method utilizing Large Language Models (LLMs) to discover and refine acquisition functions (AFs) for Bayesian Optimization (BO). The core idea is to treat the discovery of effective AFs as an algorithm discovery problem, leveraging FunSearch, an LLM-based approach for mathematical sciences. FunBO iteratively generates and evaluates candidate AFs written in code, aiming to improve BO's sample efficiency and performance across diverse optimization problems, outperforming existing general-purpose and some function-specific AFs while offering interpretability through code. The research demonstrates FunBO's effectiveness on various benchmarks and hyperparameter optimization tasks, highlighting its ability to generalize well in and out of the training distribution of functions.