Why A-B Tests Need a Holdout Group

Lucas and Luna dig into a subtle but catastrophic mistake in A-B testing: launching the winning variant to 100 percent of users without a holdout group. Using examples from Skyscanner, Etsy, and LinkedIn, they show how novelty effects, seasonal spikes, and metric degradation can make a test winner look real when it's not. They explain the holdout group method—keeping 5 percent of users on the old version to measure long-term lift—and why companies like Google and Booking.com treat it as non-negotiable. Two concrete rules for saving your next experiment from false positives. #A-BTesting #ConversionRateOptimization #HoldoutGroup #NoveltyEffect #Skyscanner #Etsy #LinkedIn #Google #Bookingcom #FalsePositive #LongTermLift #CROStrategy #DataScience #ExperimentDesign #FexingoBusiness #BusinessPodcast #Marketing #StatisticalValidity Keep every episode free: buymeacoffee.com/fexingo