IntelliJAMS EP 057: Calling out the anxieties of A/B testing

A/B testing can feel high-stakes, especially in e-commerce. You launch a test, check it an hour later, and either think you've broken your store or discovered a goldmine. Neither is true yet. This episode is a therapy session for anyone who's felt the anxiety of running A/B tests on their Shopify store.In this episode of IntelliJAMS, Alex McEachern and experimentation expert Ally Petretti walk through the most common anxieties of A/B testing and how to manage them with better process instead of more stress.Should you check your A/B test results every day?Checking your test daily is like weighing yourself multiple times a day on a diet. The number isn't wrong, but it's not ready to be meaningful yet. Check to make sure the test is collecting data and firing correctly, but don't read into the directional results until you've hit your planned runtime and traffic thresholds.What is statistical significance and when should you end an A/B test?Statistical significance is often treated as a finish line, but reaching stat sig on day one doesn't mean you have a winner. Stat sig only looks at the math. You also need time (to account for day-of-week behavior changes, promotions, and the novelty effect) and volume (small sample sizes are fragile and can skew results dramatically). A test typically needs to run for at least one to two full weeks to account for these variables.What is the novelty effect in A/B testing?The novelty effect happens when returning visitors see something different on your site and react to the change itself rather than the actual experience. In price testing, this can look like a customer treating a different price as a discount and converting out of urgency. The novelty effect fades after a week or two as your sample includes more new visitors and repeat visitors adjust.What is metric shopping in CRO?Metric shopping is when you pick a winning metric after a test is already running, instead of sticking to the success metric defined in your hypothesis. For example, if your hypothesis targeted conversion rate but AOV happened to spike, declaring a win based on AOV is metric shopping. It's what Ally calls "the silent killer of CRO." The right move is to note the unexpected metric change, form a new hypothesis around it, and run a separate test.How should you communicate A/B test results?Don't just drop a list of metrics. That's left open to interpretation. Data storytelling means framing results around your original hypothesis, explaining what customers did and why, and tailoring the narrative for your audience. A CFO needs a different story than a CMO. Use tools like the Intelligems Slack bot or MCP integration to help frame results for different stakeholders.How do you handle test ideas that aren't backed by data?Ideas from teammates or Twitter threads aren't bad, but they need a hypothesis before they become tests. A strong hypothesis includes a data-backed problem, a proposed solution (the test idea), and a success metric. If an idea doesn't have that, either formulate the hypothesis yourself from existing data or add it to the backlog until you can.How do you coordinate A/B testing across teams?CRO teams and paid media teams often test independently, which creates noise in each other's data. The key is sharing hypotheses (not just test plans), coordinating testing calendars, and recognizing that every team is working toward the same growth goal. Regular cross-team syncs where you discuss challenges, upcoming campaigns, and testing angles can turn conflict into collaboration.Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-aca...Connect with Ally Petretti:https://www.linkedin.com/in/ally-petretti-kuhn-47a27014/ Connect with Intelligems:Website: https://intelligems.ioBlog: https://intelligems.io/blog----0:00 - Intro: Why we need a CRO therapist1:05 - The #1 stressor in testing: urgency and timing1:44 - "I'm a genius" vs. "I'm killing the business" (early test reactions)2:33 - Peeking at tests: checking vs. reacting4:51 - Pre-test analysis: setting guardrails before you launch5:30 - Statistical significance isn't the finish line6:13 - The novelty effect and why it matters for price testing7:47 - Using the time series chart to read test stability8:39 - When to call a test early (the right way)10:12 - No test losers: every result is a learning10:59 - Hypothesis discipline and metric shopping12:18 - What metric shopping is and why it's the silent killer of CRO13:25 - Communicating test results: data storytelling15:57 - The three E's framework: Explore, Experiment, Extend17:25 - Understanding your customers through testing18:20 - "Feelings over data": handling test ideas without a hypothesis21:19 - Cross-team testing: getting paid media and CRO on the same page25:42 - Wrap-up and where to connect with Ally