Why Your A-B Test Loses Statistical Significance

Lucas and Luna explain 'peeking,' the practice of checking A-B test results early, which inflates false positive rates. They break down why stopping a test the moment it hits 95% confidence is a statistical trap, using examples from Optimizely and booking sites. The episode explores the concept of sequential testing and the 'always valid' p-value method, then discusses how persistent peeking led one travel company to implement stricter governance. They also share a simple rule: decide the sample size and duration before the test starts. A short donation segment highlights listener support via buymeacoffee.com/fexingo. #Peeking #A-BTesting #StatisticalSignificance #FalsePositive #Optimizely #SequentialTesting #PValue #ConversionRateOptimization #CRO #Marketing #DataScience #Experimentation #BusinessPodcast #MarketingPodcast #FexingoBusiness #LucasAndLuna #SampleSize #ExperimentDesign Keep every episode free: buymeacoffee.com/fexingo