ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models — Elangovan et al. (2024)
from 1 More Paper (Brief)
In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertak- ing that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the ex- perimental design and results are reliable. The conclusions from these evaluations, thus, must consider factors such as usability, aesthetics, and cognitive biases. We highlight how cog- nitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. Furthermore, the evaluation should dif- ferentiate the capabilities and weaknesses of increasingly powerful large language models – which requires effective test sets. The scala- bility of human evaluation is also crucial to wider adoption. Hence, to design an effec- tive human evaluation system in the age of generative NLP, we propose the ConSiDERS- The-Human evaluation framework consisting of 6 pillars – Consistency, Scoring Critera, Differentiating, User Experience, Responsible, and Scalability. 1 This Brief reads the authors' abstract in the authors' own words. Abstract source: pdf-extraction Authors: Aparna Elangovan, Liu Ling, Xu Lei, Sravan Bodapati, Roth Dan Publication date: 2024 Read the paper Narration: Read verbatim by an AI-generated voice. The authors and publisher do not sponsor or endorse this recording. Full episode
Embed this episode
No transcript yet.