Episode 60: 10 Things I Hate About AI Evals with Hamel Husain

from Vanishing Gradients · host Hugo Bowne-Anderson

Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems.Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust.We talk through: The 10(+1) critical mistakes that cause teams to waste time on evals Why "hallucination scores" are a waste of time (and what to measure instead) The manual review process that finds major issues in hours, not weeks A step-by-step method for building LLM judges you can actually trust How to use domain experts without getting stuck in endless review committees Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agentsIf you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap.LINKSHamel's website and blog (https://hamel.dev/)Hugo speaks with Philip Carter (Honeycomb) about aligning your LLM-as-a-judge with your domain expertise (https://vanishinggradients.fireside.fm/51)Hamel Husain on Lenny's pocast, which includes a live demo of error analysis (https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill)The episode of VG in which Hamel and Hugo talk about Hamel's "data consulting in Vegas" era (https://vanishinggradients.fireside.fm/9)Upcoming Events on Luma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk)Watch the podcast video on YouTube (https://youtube.com/live/QEk-XwrkqhI?feature=share)Hamel's AI evals course, which he teaches with Shreya Shankar (UC Berkeley): starts Oct 6 and this link gives 35% off! (https://maven.com/parlance-labs/evals?promoCode=GOHUGORGOHOME) https://maven.com/parlance-labs/evals?promoCode=GOHUGORGOHOME🎓 Learn more:Hugo's course: Building LLM Applications for Data Scientists and Software Engineers (https://maven.com/s/course/d56067f338) — https://maven.com/s/course/d56067f338 Get full access to Vanishing Gradients at hugobowne.substack.com/subscribe

NOW PLAYING

0:00 1:13:16

1×

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

When Control Looks Like Love - Abuse, Technology, and Breaking the Cycle | Montgomery County Family Justice Center Ep. 171

May 6, 2026 ·31m

Don’t Miss This Year’s Derby Party | Live Auction, Prize for Best Dressed & So Much More! Ep. 170

Apr 22, 2026 ·9m

Gaia Elisa Rossi

Apr 16, 2026 ·53m

Ben Seidman #3

Apr 7, 2026 ·53m

Doctor to Patient: A Rare Cancer (LMS) Diagnosis & Advocacy | Derby Party Beneficiary The National Leiomyosarcoma Foundation Ep. 169

Apr 3, 2026 ·24m

It Started With 15 Meals, Now It's 2 Million | From Meal Service to Culinary Career Training: Shepherd's Table Ep. 168

Mar 26, 2026 ·28m

Similar Podcasts

Vanishing Postcards Evan Stern Vanishing Postcards is a documentary travelogue that invites listeners on a road trip exploring the hidden dives, traditions, and frequently threatened histories discovered by exiting the interstates. Named one of the Best Podcasts of 2022 by Digital Trends. True Crime Mystery Hour Anthony Okoye True Crime Mystery Hour, Unraveling the Darkest Mysteries, One Case at a Time.Step into the shadowy world of True Crime Mystery Hour, where unsolved cases, chilling disappearances, and criminal enigmas take center stage. Each episode delves deep into the most baffling mysteries, cold cases, and infamous crimes that have captivated investigators and the public. From notorious serial killers to mysterious vanishing acts, we explore every twist, turn, and hidden clue. Please tune in for a gripping journey through the unknown as we seek to uncover the truth behind some of the darkest stories ever told. Prepare for suspense, intrigue, and a deep dive into the macabre. Most Terrifying Places in America Travel Channel On ‘The Most Terrifying Places in America’, hear real about ghost stories at America's most-famous haunted landmarks. With direct audio from the TV show, a team of ghost hunters, psychic mediums and historians take you around the U.S. to find out why these paranormal hot spots deserve their reputation.Also, go back and listen to episodes of These Woods Are Haunted and The Alaska Triangle. On These Woods Are Haunted, hear true accounts of people who ventured deep into the forest only to come screaming out with stories that defy reality. And on The Alaska Triangle, hear how experts and eyewitnesses attempt to unlock the mystery of the Alaska Triangle, a remote area infamous for alien abductions, Bigfoot sightings, paranormal phenomena and vanishing airplanes. Hosted on Acast. See acast.com/privacy for more information. The Insider Vanishing Inc. Magic Vanishing Inc. is proud to share with you our magic podcast, The Insider. Interviews with the finest magicians in the world.

Frequently Asked Questions

How long is this episode of Vanishing Gradients?

This episode is 1 hour and 13 minutes long.

When was this Vanishing Gradients episode published?

This episode was published on September 30, 2025.

What is this episode about?

Can I download this Vanishing Gradients episode?

Yes, you can download this episode by clicking the download button on the episode player, or subscribe to the podcast in your preferred podcast app for automatic downloads.

URL copied to clipboard!