EP 37 | CME295: LLM Evaluations
from AI Bites: The Academic Series Hosted by Jack Lakkapragada
If an AI can write a poem, code a website, and pass the bar exam, how do we actually measure its performance? This episode tackles the notoriously difficult science of LLM Evaluation. We look at why standard testing benchmarks are breaking down and how researchers are trying to keep up. Key Topics: The Benchmark Problem: Why traditional multiple-choice tests are saturating and failing to capture true model intelligence. LLM-as-a-Judge: The growing trend of using powerful models (like GPT-4) to grade and evaluate the outputs of other models. Data Contamination: The massive challenge of testing a model when its training data essentially includes the entire internet—did it reason through the test, or just memorize the answer key? Note: This is an AI-generated study resource created via NotebookLM based on the Stanford CME295 curriculum and personal study notes.
Embed this episode
No transcript yet.