PodParley PodParley
PodParley
Is AI Benchmarking Broken? The Truth Behind "con@64" Revealed Brought to you by Avonetics.com episode artwork

EPISODE · Feb 20, 2025 · 9 MIN

Is AI Benchmarking Broken? The Truth Behind "con@64" Revealed Brought to you by Avonetics.com

from Beaker Banter · host Beaker Banter

Discover the controversial "con@64" technique, where AI models are prompted 64 times to reach a consensus answer. Is this a legitimate way to reduce variance or a sneaky trick to inflate benchmark scores? Dive into the heated debate on whether this practice skews real-world performance comparisons and unfairly impacts perceptions of model capabilities. Learn why some accuse XAI engineers of overhyping AI and how differing "con" values could be misleading the industry. For advertising opportunities, visit Avonetics.com.

What this episode covers

Discover the controversial "con@64" technique, where AI models are prompted 64 times to reach a consensus answer. Is this a legitimate way to reduce variance or a sneaky trick to inflate benchmark scores? Dive into the heated debate on whether this practice skews real-world performance comparisons and unfairly impacts perceptions of model capabilities. Learn why some accuse XAI engineers of overhyping AI and how differing "con" values could be misleading the industry. For advertising opportunities, visit Avonetics.com.

NOW PLAYING

Is AI Benchmarking Broken? The Truth Behind "con@64" Revealed Brought to you by Avonetics.com

0:00 9:25

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

#69 - MET CALL: Under Pressure (Hypotension)

Mar 30, 2026 ·52m

# 68 - Macrodata Refinement (Part 2)

Jul 2, 2025 ·54m

#67 - Macrodata Refinement (Part 1)

Jul 2, 2025 ·45m

#66: Only 2% Shall Pass (Immune Organs: Part 2)

Apr 22, 2025 ·73m

#65: Only 2% Shall Pass (Immune Organs: Part 1)

Mar 30, 2025 ·63m

#64: Periodically Absent (Part 2: MRKH)

Jan 16, 2025 ·29m

Similar Podcasts

Scar(r)ed For Life pjwrench A nostalgic, banter-full and analytical delve into all the TV shows, movies, games, books, and other media that scared (and scarred!) us s***less when we were children in the 90s. Ranging from the infamous to the obscure, from The Wizard of Oz to The Demon Headmaster, from Tomb Raider to fairytales, we’ve offer up 2 classic pieces of entertainment to reminisce and wax lyrical. Enjoy! We Call Bank Joey Hobbs 4 Best Friends Sharing Their Opinions On The Latest Sports News. NBA/NFL & More.All Friendly Banter Around These Parts! The Acquired Taste Podcast Whitney Diggs and Jadea Jackson Amazing things arise when two friends can come together to create a safe space to have deep conversations, laugh, and enjoy each other’s company through their unique storytelling. Join Whitney and Jadea in their new and exciting podcast, Acquired Taste, as they cover a wide range of topics including pop culture, personal experiences, and thought-provoking discussions. With their unique perspectives and hilarious banter, you will not want to miss out on the chance to connect through the power of audio. Join us every fourth Sunday for your monthly dose of an Acquired Taste! Baltimore Ravens Podcast Network Baltimore Ravens Stop in “The Lounge.” Every week, Ryan Mink and Garrett Downing will take you inside team HQ to chat (argue) about all things Ravens, dive into ridiculous topics and hang out with players who drop in. This lounge is dry, but the banter is not. Got a comment or suggestion? Email the show at [email protected] and tell us what you like (or don't like).

Frequently Asked Questions

How long is this episode of Beaker Banter?

This episode is 9 minutes long.

When was this Beaker Banter episode published?

This episode was published on February 20, 2025.

What is this episode about?

Discover the controversial "con@64" technique, where AI models are prompted 64 times to reach a consensus answer. Is this a legitimate way to reduce variance or a sneaky trick to inflate benchmark scores? Dive into the heated debate on whether this...

Can I download this Beaker Banter episode?

Yes, you can download this episode by clicking the download button on the episode player, or subscribe to the podcast in your preferred podcast app for automatic downloads.
URL copied to clipboard!