VARIANCE

In this special second episode, the roles reverse as I share some stories from my extensive experimentation background across tech giants like eBay, Apple, and Yahoo. This is a cut from my first episode with Ishan where Ishan asked me about my experience. Discover the statistical challenges that plagued major experimentation platforms - from misaligned alpha thresholds and rampant false positives to the complexities of handling extremely skewed revenue metrics. Learn how I rebuilt trust in A/A testing through blind studies, applied multiple comparison corrections, and navigated the challenges of KPI engineering for metrics with distributions spanning from $1 to $8,000. This candid behind-the-scenes look into real-world experimentation reveals why understanding statistical concepts is crucial for accurate decision-making in tech. Perfect for data scientists, analysts, and product managers working with experiments and causal inference at scale.

Welcome to the first episode of Variance, a podcast where I explore experimentation, basic statistics, and data science career through conversations with industry experts. In this episode, I sit down with Ishan Goel, former Associate Head of Data Science at VWO (Visual Website Optimizer), where he led data science research and helped build VWO's new stats engine.Topics covered include:🟣 Transition from computer science to data science🟣 The role of experimentation in modern businesses🟣 Statistical concepts and methodologies🟣 Career advice for aspiring data scientists🟣 Common challenges in A/B testingConnect with Ishan:LinkedIn: https://www.linkedin.com/in/i-goel/Weekly Experimentation Webinars: Every TuesdayUpcoming Course on Experimentation Leadership (stay tuned)Connect with Pritul:LinkedIn: https://www.linkedin.com/in/pritul-patel