IntelliJAMS EP 058: Outputs vs. Outcomes: A Better Way to Think About A/B Testing

Running 10 tests a month sounds productive, but if none of them are tied to a real business question, you're just taking swings for the sake of swinging. In this episode Adam and Alex dig into why "how many tests should I run?" is often the wrong question, and what to ask instead when you're building an experimentation program on Shopify.In this episode of IntelliJAMS, Adam and Alex explore the difference between test output and test outcomes, how to run concurrent tests without confounding your results, and why elevating the conversation from "number of tests" to "strategic priorities" changes everything.Timestamps:0:00 - Intro0:27 - The question everyone's asking right now0:52 - Why "how many tests" is the wrong question2:23 - Outputs vs. outcomes: reframing productivity3:10 - Test the plan, don't plan the tests3:51 - You can't predict how many tests it'll take4:54 - Running concurrent tests without confounding results5:53 - Scoping metrics to the right part of the funnel6:48 - Every test needs a hypothesis7:30 - One test often leads to five new questions8:15 - Real example: ending a checkout upsell test early9:11 - How to push back on "I need 10 tests this month"11:49 - Elevating the conversation from output to strategy13:28 - Final thoughts: see the forest for the treesTopics covered:Why counting tests is an output metric, not an outcomes metricHow to reframe testing around strategic business goalsRunning concurrent A/B tests on Shopify without confounding dataSegmenting tests by funnel stage and visitor typeHow one test can lead to five new questionsPushing back when stakeholders demand a test quotaReal-world example: killing a checkout upsell test 24 hours inThe "test the plan, don't plan the tests" frameworkWant to sharpen your experimentation skills? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535Website: https://intelligems.ioBlog: https://intelligems.io/blogLinkedIn: https://www.linkedin.com/company/intelligems