The Evaluation Crisis - NeurIPS 2025 by Basis Set

What this episode covers

"Passes the bar exam" doesn't mean AI can practice law. "Beats humans on ImageNet" doesn't mean it understands images. You'll learn why most AI benchmarks are fundamentally broken through the cautionary tale of the "infant morality study"—researchers thought babies preferred moral helpers, but they just liked bouncing balls. The Clever Hans effect is alive and well in 2025. If you're evaluating AI products, making purchasing decisions, or relying on AI benchmark claims, this episode gives you the critical thinking tools to cut through the nonsense. Topics Covered - Construct validity: Are we testing what we think we're testing? - The anthropomorphism trap: projecting human limitations onto AI - Why "passing the bar exam" doesn't mean AI can practice law - The Clever Hans problem in modern AI - EU AI Act and regulatory approaches - Testing AI like we test babies and animals (alien intelligence framework)

Share this episode

Similar Episodes

I'm ok

Mar 26, 2026 ·1m

REMIX: Why we over-shop and compulsively acquire, and how to stop, with Dr Jan Eppingstall

Jan 9, 2026 ·61m

REMIX: OCD and hoarding disorder with Jenna Overbaugh

Jan 2, 2026 ·47m

REMIX: Therapy and hoarding disorder - what are the options? With Dr Jan Eppingstall

Dec 26, 2025 ·78m

REMIX: ADHD and hoarding disorder with Professor Sharon Morein

Dec 21, 2025 ·46m

#207 13 actionable pieces of mental health advice from six former podcast guests

Dec 12, 2025 ·53m

Similar Podcasts

MG Show MG Show The MG Show, hosted by Jeffrey Pedersen and Shannon Townsend, is a leading alternative media platform dedicated to uncovering the truth behind today’s most pressing political issues. Launched in 2019, the show has grown exponentially, offering unfiltered insights, comprehensive research, and real-time analysis. With a commitment to independent journalism and factual integrity, the MG Show empowers its audience with knowledge and encourages active participation in the political discourse. That Hoarder: Overcome Compulsive Hoarding That Hoarder Hoarding disorder is stigmatised and people who hoard feel vast amounts of shame. This podcast began life as an audio diary, an anonymous outlet for somebody with this weird condition. That Hoarder speaks about her experiences living with compulsive hoarding, she interviews therapists, academics, researchers, children of hoarders, professional organisers and influencers, and she shares insight and tips for others with the problem. Listened to by people who hoard as well as those who love them and those who work with them, Overcome Compulsive Hoarding with That Hoarder aims to shatter the stigma, share the truth and speak openly and honestly to improve lives. Flottengeflüster ALD Automotive Österreich | LeasePlan Beim Flottengeflüster powered by ALD Automotive | LeasePlan präsentieren Jörg Janik und Peter Gutenbrunner alle zwei Wochen spannende Informationen rund um das Thema nachhaltige Mobilität. Beide beschäftigen sich schon lange mit der Thematik und bringen umfangreiches Fachwissen mit. Sollten sie aber doch einmal nicht weiter wissen, werden unsere Expert*innen hinzugezogen, die ihnen gerne mit Rat und Tat zur Seite stehen. The Small Business Startup School – Business Notes | Financial Literacy | Retail Psychology – For Professionals & Entrepreneurs The Small Business Startup School Inc. Starting or buying a small business? While personal circumstances may vary, business patterns remain timeless. On The Small Business Startup School, we explore strategies, insights, and practical solutions to help entrepreneurs confidently navigate their journey.Hosted by Ola Williams—a retail entrepreneur, fintech founder, and financial coach with over two decades of experience—this podcast marries financial awareness and retail psychology with optimism to deliver actionable takeaways.Join us to learn, grow, and connect as we uncover the keys to business success.Let’s continue to learn together and be encouraged to keep on connecting!

Frequently Asked Questions

How long is this episode of NeurIPS 2025 by Basis Set?

This episode is 15 minutes long.

When was this NeurIPS 2025 by Basis Set episode published?

This episode was published on November 26, 2025.

What is this episode about?

"Passes the bar exam" doesn't mean AI can practice law. "Beats humans on ImageNet" doesn't mean it understands images. You'll learn why most AI benchmarks are fundamentally broken through the cautionary tale of the "infant morality...

Can I download this NeurIPS 2025 by Basis Set episode?

Yes, you can download this episode by clicking the download button on the episode player, or subscribe to the podcast in your preferred podcast app for automatic downloads.