The Sample Size Fallacy That Ruins Your A-B Tests

Most marketers think bigger sample sizes always mean better A-B test results. In this episode, Lucas and Luna break down the sample size fallacy — why too large a sample can detect statistically significant but practically meaningless effects, and how Booking.com discovered this lesson the hard way when a test with 2 million users showed a 0.1 percent lift that vanished on rollout. They walk through the math behind minimum detectable effect, explain why a test with 500,000 users can be worse than one with 5,000, and share a simple framework for choosing sample size based on business impact rather than statistical purity. Plus: how Netflix avoids this trap by pre-registering effect sizes before every test. If you run A-B tests, this episode will save you from wasting months chasing irrelevant wins. #A-BTesting #SampleSizeFallacy #CRO #ConversionRateOptimization #Marketing #Bookingcom #Netflix #MinimumDetectableEffect #StatisticalSignificance #DataScience #BusinessStrategy #Experimentation #Analytics #MarketingPodcast #FexingoBusiness #BusinessPodcast #CROStrategy #ABTestMistakes Keep every episode free: buymeacoffee.com/fexingo