Interplay of LLMs in Information Retrieval Evaluation

This paper, authored by researchers at Google DeepMind, investigates the impact of using large language models (LLMs) in various roles within information retrieval (IR) systems, specifically focusing on their use as rankers and judges for evaluating search results. The paper examines potential biases that can arise from LLMs interacting in these roles, including a bias observed in LLM judges favoring results from LLM rankers. Through experiments on standard IR datasets, the authors analyze the discriminative ability of LLM judges and find they may struggle to differentiate between systems with subtle performance differences. The work also considers the influence of AI-generated content on LLM evaluation, although their preliminary findings did not indicate a strong bias against it. Ultimately, the document provides initial guidelines for using LLMs in IR evaluation and outlines a research agenda for better understanding these complex interactions to ensure reliable assessment.