CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

What this episode covers

CORAL, a novel benchmark dataset for evaluating Retrieval-Augmented Generation (RAG) systems in a multi-turn conversational setting. The authors highlight the limitations of existing datasets in assessing conversational RAG and detail CORAL's unique features, including open-domain coverage, knowledge intensity, free-form responses, topic shifts, and citation labeling. They explain how CORAL is derived from Wikipedia, automatically converting its content into conversational formats, and outline the three core tasks it supports: conversational passage retrieval, response generation, and citation labeling. The authors present a unified framework for evaluating conversational RAG methods and report on experiments conducted on CORAL, showcasing the performance of different conversational search and generation models.

PodParley-generated summary based on available episode metadata and transcript content.