Research paper of the week: Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels

The podcast discusses a research paper from Google for improving zero-shot LLM rankers by incorporating finely granular relevance labels into the prompts. Instead of binary labels (“Yes”/“No”), multi-level labels (e.g. “Highly relevant”, “Somewhat relevant”, “Not relevant”) are used to increase the accuracy of the ranking. Experiments on eight BEIR datasets show significant performance improvements compared to conventional binary approaches. Different strategies for aggregating the probabilities of the labels are compared, with the use of the highest probability being highlighted as particularly efficient. The results emphasize the potential of fine granular relevance labels to improve the performance of LLMs in the field of information retrieval. https://www.kopp-online-marketing.com/patents-papers/beyond-yes-and-no-improving-zero-shot-llm-rankers-via-scoring-fine-grained-relevance-labels