How Your A-B Test Results Disappear After Ship

In this episode of CRO with Fexingo, Lucas and Luna dig into the phenomenon of vanishing lift: why an A-B test that shows a clear winner in the experiment often fails to move the needle after the winning variant goes live. Drawing on a case study from Etsy, where a button redesign showed a 5 percent relative lift in the lab but zero impact in production, the hosts explore three root causes: novelty bias, audience mismatch, and the Hawthorne effect. They walk through specific fixes, including holdout groups, staggered rollouts, and pre-registering secondary metrics. If you've ever shipped a winner only to see flat conversion rates, this episode explains why and what to do about it. #A-BTesting #CRO #Marketing #Etsy #ConversionRateOptimization #NoveltyBias #HawthorneEffect #AudienceMismatch #HoldoutGroups #StaggeredRollout #VanishingLift #ExperimentDesign #DataDriven #StatisticalValidity #FexingoBusiness #BusinessPodcast #MarketingPodcast #CROStrategy Keep every episode free: buymeacoffee.com/fexingo