IPO: Interpretable Prompt Optimization for Vision-Language Models

This paper details an innovative method for improving vision-language models (VLMs) by leveraging large language models (LLMs) to optimize the text prompts used in tasks like image classification. Current methods for prompt learning in VLMs can suffer from issues like lack of interpretability and overfitting. The proposed approach, termed Interpretable Prompt Optimization (IPO), uses an LLM as a parameter-free optimizer that iteratively refines prompts based on performance feedback and historical data, including image descriptions generated by a large multimodal model (LMM). Experiments across various datasets demonstrate that IPO produces human-interpretable prompts and achieves stronger generalization to novel classes compared to existing gradient-based methods. The study highlights the effectiveness of this task-agnostic LLM-driven optimization in enhancing VLM capabilities, particularly in few-shot scenarios, while acknowledging the computational cost challenges with larger datasets.