The three questions every CIO should ask about a vendor accuracy claim

from Agent Mode AI · host Agent Mode AI

Episode 9 of Agent Mode AI. Abby and Avery walk AM-146, the claim that vendor "ready-to-run" positioning without named task, named baseline, and named methodology is procurement-deck noise rather than procurement evidence. The procurement-grade reference shapes in 2026 are the academic-benchmark layer (CRMArena-Pro 35% multi-step reliability, CMU TheAgentCompany 30-35% reproduction range, WebArena ~36% browser-agent ceiling, SWE-bench Verified for code generation) and the Anthropic Claude for Chrome disclosure pattern (23.6% pre-mitigation, 11.2% post, 0% on URL-injection variants after patches). A third class — the named-customer audited deployment, with McKinsey Lilli, JPMorgan, BT Now Assist, and UK Government Digital Service as the canonical references — sits alongside. Sources cited: - CRMArena-Pro paper, Salesforce AI Research, August 2025 - Carnegie Mellon TheAgentCompany academic benchmark - WebArena academic benchmark - SWE-bench Verified - Anthropic published security disclosure on Claude for Chrome, 26 August 2025 - McKinsey internal Lilli platform deployment data - JPMorgan Chase 2023 AI value disclosure - BT Now Assist deployment, Hena Jalil - UK Government Digital Service Q4 2024 Claims tracked: - AM-146 — Three accuracy-disclosure questions for procurement — agentmodeai.com/holding/?claim=AM-146 - AM-009 — Claude for Chrome procurement-grade disclosure pattern — agentmodeai.com/holding/?claim=AM-009 - AM-140 — Procurement-committee pre-pilot questions — agentmodeai.com/holding/?claim=AM-140 Newsletter and the full Holding-up ledger: agentmodeai.com

Episode metadata supplied by the publisher feed · Published May 10, 2026

Embed this episode

Attribution link and audio player

What this episode covers

Vendor "ready-to-run" agentic AI positioning is not procurement evidence unless three questions are answered: accuracy on which task, against which baseline, measured how. AM-146 walks the 2026 procurement-grade reference shapes — CRMArena-Pro, CMU TheAge

Distinct summary based on available episode metadata or transcript content.

NOW PLAYING

The three questions every CIO should ask about a vendor accuracy claim

0:00 12:14

1×

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

CISO Perspectives (public) N2K Networks This season on CISO Perspectives, host Kim Jones explores some of the challenges of leading through uncertainty. We explore the complexity of the changing nature of regulation and working with the federal government, the evolution of privacy and fraud, and how emerging technologies like AI and quantum computing are changing cyber. When you don’t know what questions to ask, you’re afraid to ask, or don’t know who to ask, CISO Perspectives provides the foundation for learning in this brave new world.

Frequently Asked Questions

How long is this episode of Agent Mode AI?

This episode is 12 minutes long.

When was this Agent Mode AI episode published?

This episode was published on May 10, 2026.

Can I download this Agent Mode AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.

URL copied to clipboard!