Skip to content
S6E9 - Your A/B Test Dropped Conversion 8% in 6 Days… Now What? episode artwork

Jun 3, 2026 33 min

S6E9 - Your A/B Test Dropped Conversion 8% in 6 Days… Now What?

from Experiment Nation: The Podcast Hosted by Rommil from Experiment Nation

What do you do when an experiment tanks conversion rate by 8% in less than a week?In this episode, Gerda and Rommil break down real CRO and experimentation scenarios submitted by the community, including failed tests, executive pressure, guardrail metrics, knowledge management, developer friction, and why experimentation teams always seem to end up back in spreadsheets.Check out: http://katsed.com/We also discuss: * Root cause analysis for failed experiments * How to calm leadership during testing disasters * Guardrail metrics and control limits * Why experimentation knowledge banks become messy * Airtable vs spreadsheets vs enterprise tools * Why developers and designers struggle with experimentation culture * How to onboard technical teams into testing workflows A practical conversation for experimentation leads, growth teams, product managers, CRO specialists, and anyone building an experimentation culture.Chapters00:00 Burger Menu Debate: Left vs Right01:28 Mystery Meat Navigation & UX History03:03 Why Modern Phones Break UX05:08 Community CRO Scenarios Begin06:18 Experiment Tanks Conversion by 8%07:33 Root Cause Analysis & Leadership Pressure09:17 QA, Tracking Errors & Failed Tests10:03 Building a CRO Analysis Tool12:18 Probabilistic Root Cause Detection14:26 Managing Panic During Bad Experiments15:21 Guardrail Metrics Explained16:40 Using Control Limits in Experimentation18:18 Building an Experimentation Knowledge Bank20:05 Why You Don’t Need AI for Everything21:08 Why Process Changes Create Admin Pain23:08 Migrating Between Experimentation Tools24:31 Why Teams Always Go Back to Spreadsheets26:50 CRO Teams vs Developers & Designers29:09 Why Early Experiments Should Be Simple30:41 Why Developers Hate “Throwaway Code”32:16 Designers vs Experiment Constraints33:02 Outro & Next Community Questions This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

S6E9 - Your A/B Test Dropped Conversion 8% in 6 Days… Now What?

0:00 33:21

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Experiment Nation: The Podcast?

This episode is 33 minutes long.

When was this Experiment Nation: The Podcast episode published?

This episode was published on June 3, 2026.

Can I download this Experiment Nation: The Podcast episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.