IntelliJAMS

How much traffic do you need to A/B test? And is your brand actually big enough to run a real experimentation program? These are two of the most common questions Shopify merchants ask before getting started, and most of the time, the answer is "less than you think."In this episode of IntelliJAMS, Alex and Adam break down the actual thresholds that determine whether A/B testing makes sense for your store.They get specific: what order volume you need to reach statistical significance, how your AOV changes that math entirely, what annual revenue level makes an experimentation program pay for itself, and what to do if you're not quite there yet.Here's the short version: if you're doing 500–700 orders a month, you likely have enough volume to run meaningful Shopify A/B tests — especially if you're testing high-traffic pages like checkout and cart instead of niche landing pages. And if you're at $3–5M in annual revenue, a well-run experimentation program that delivers a 3–5% revenue lift will typically pay for itself within a few months.But traffic and revenue aren't the only objections. A lot of brands ask "is my brand big enough to A/B test?" not because of traffic, but because they don't have the team — no dedicated CRO specialist, no data analyst, no developer on standby for test builds. Adam makes the case that AI has largely closed that gap. What used to take four people can now run with one operator and the right tools.They also dig into the AOV problem: a furniture brand doing $5M a year on $4,000–5,000 couches might only process 80–100 orders a month. At that volume, reaching statistical significance on an A/B test takes so long it stops being useful. The math on "how much traffic do I need to A/B test" isn't just about visitors — it's about conversions, and AOV determines how many conversions you're working with.If you've been putting off testing because you assumed you weren't ready, this episode is worth watching before you make that call.Timestamps:0:00 - Intro0:27 - The three reasons brands think they're not ready to test1:55 - Two different problems: not qualified vs. not resourced2:45 - Why AI closes the "I don't have a team" gap3:10 - The 3–5% revenue uplift math (and when it pencils out)5:54 - Order volume vs. visitors: what actually drives stat sig6:30 - Why AOV changes everything (the $5K couch problem)8:00 - The 500–700 orders/month rule of thumb8:32 - You don't need to be Wayfair to run meaningful experiments9:28 - Take bigger swings when you have lower trafficTopics covered:How much traffic you actually need to A/B test on ShopifyWhy order volume matters more than visitors for statistical significanceHow AOV affects whether your brand is big enough for A/B testingThe revenue threshold where experimentation starts paying for itselfWhy AI has replaced much of the traditional CRO teamWhat to test first when you're under 1,000 orders/monthReady to start testing? Join GEM Academy for free courses and a community of Shopify brands sharing what works: https://www.skool.com/intelligems-academy-1535Connect with Intelligems:Website: https://intelligems.ioBlog: https://intelligems.io/blogLinkedIn: https://www.linkedin.com/company/intelligems

Your product page probably does a decent job explaining what you sell. But is it answering the questions customers are asking themselves and never typing into a chat widget? Adrian Stewart, co-founder of Scale Messaging, has a framework for finding those gaps, and it can change the way you approach A/B testing your messaging.In this episode of IntelliJAMS, Alex and Adrian dig into the "Frequently Unasked Questions" framework: four categories of questions shoppers silently ask themselves while browsing your site. They cover where most Shopify brands fall short on messaging, why reducing friction is easier than building motivation (but both matter), when urgency tactics actually work versus when they erode trust, and how to build hypotheses around messaging that you can test and learn from.Timestamps:0:00 - Intro and the Frequently Unasked Questions framework0:44 - Why "unasked" questions matter more than FAQs1:57 - The four categories: understanding, motivation, difference, trust4:26 - Motivation vs. friction and how they drive behavior6:35 - Why motivation is harder to build than friction is to reduce7:40 - Urgency as a bonus category (and when it gets hacky)9:56 - Fake scarcity vs. real scarcity: the windscreen wiper example11:22 - Difference: competing within your range, against competitors, and against inertia14:09 - How to figure out which unasked questions to prioritize17:13 - Message, expression, and placement: the three layers of testing19:02 - Why one test usually leads to five more questions20:15 - Where to start: trust is the fastest win, difference is the biggest opportunity24:07 - Recap and where to find AdrianTopics covered:The Frequently Unasked Questions framework (understanding, motivation, difference, trust)Why "difference" is the most overlooked messaging gap on product pagesThe motivation-to-friction ratio and how it affects conversionWhen urgency and scarcity tactics help vs. hurt your brandHow traffic source (paid social vs. search) should shape your messaging strategyBuilding messaging hypotheses you can A/B testMessage vs. expression vs. placement as three layers of experimentationWhy trust is the fastest win for most e-commerce brandsReady to start testing your messaging? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535Connect with Adrian / Scale Messaging:Website: https://scalemessaging.comLinkedIn: https://www.linkedin.com/in/adrianjstewart/Connect with Intelligems:Website: https://intelligems.ioBlog: https://intelligems.io/blogLinkedIn: https://www.linkedin.com/company/intelligems

Post-purchase upsells sit at the exact moment a customer has already committed — credit card swiped, conversion done. That means there's zero risk of hurting conversion and pure upside potential for your AOV. In this episode, we break down why post-purchase might be the highest-leverage Shopify A/B testing opportunity most brands haven't explored yet.In this episode of IntelliJAMS, Adam and Alex explore the economics of post-purchase upsells — from a CBD brand that grew AOV 20% overnight to the consumption psychology that makes "buy more now" actually better for long-term LTV.Timestamps:0:00 - Intro0:22 - Why post-purchase upsells are worth your attention1:04 - "Be close to the money" — does post-purchase qualify?1:52 - What brands are doing with post-purchase today2:11 - The CBD gummies case study: 20% AOV lift overnight3:32 - Why there's zero downward pressure on conversion5:05 - The consumption psychology surprise: more supply = faster consumption6:09 - Scarcity vs. abundance mindset and second-order effects8:04 - Three post-purchase upsell strategies that work9:35 - Matching upsell strategy to your margin profile10:26 - Using post-purchase for inventory clearance and sell-through11:02 - How Intelligems handles post-purchase testing and measurement12:28 - Why post-purchase tests can run in parallel with everything elseTopics covered:The economics behind post-purchase upsells (zero incremental CAC, no conversion risk)A real-world CBD brand case study: 50% off second pack, 40% take rate, 20% AOV increaseThree post-purchase strategies: same product discount, complementary products, and clearance/inventory sell-throughWhy consumption scales with supply — and what that means for repurchase ratesHow to match your upsell strategy to your margin profileRunning post-purchase tests in parallel without interaction effectsHow Intelligems measures incrementality, revenue, and profit on post-purchase offersWant to start testing post-purchase upsells (or anything else)? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535Connect with Intelligems:Website: https://intelligems.ioBlog: https://intelligems.io/blogLinkedIn: https://www.linkedin.com/company/intelligems

Running 10 tests a month sounds productive, but if none of them are tied to a real business question, you're just taking swings for the sake of swinging. In this episode Adam and Alex dig into why "how many tests should I run?" is often the wrong question, and what to ask instead when you're building an experimentation program on Shopify.In this episode of IntelliJAMS, Adam and Alex explore the difference between test output and test outcomes, how to run concurrent tests without confounding your results, and why elevating the conversation from "number of tests" to "strategic priorities" changes everything.Timestamps:0:00 - Intro0:27 - The question everyone's asking right now0:52 - Why "how many tests" is the wrong question2:23 - Outputs vs. outcomes: reframing productivity3:10 - Test the plan, don't plan the tests3:51 - You can't predict how many tests it'll take4:54 - Running concurrent tests without confounding results5:53 - Scoping metrics to the right part of the funnel6:48 - Every test needs a hypothesis7:30 - One test often leads to five new questions8:15 - Real example: ending a checkout upsell test early9:11 - How to push back on "I need 10 tests this month"11:49 - Elevating the conversation from output to strategy13:28 - Final thoughts: see the forest for the treesTopics covered:Why counting tests is an output metric, not an outcomes metricHow to reframe testing around strategic business goalsRunning concurrent A/B tests on Shopify without confounding dataSegmenting tests by funnel stage and visitor typeHow one test can lead to five new questionsPushing back when stakeholders demand a test quotaReal-world example: killing a checkout upsell test 24 hours inThe "test the plan, don't plan the tests" frameworkWant to sharpen your experimentation skills? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535Website: https://intelligems.ioBlog: https://intelligems.io/blogLinkedIn: https://www.linkedin.com/company/intelligems

A/B testing can feel high-stakes, especially in e-commerce. You launch a test, check it an hour later, and either think you've broken your store or discovered a goldmine. Neither is true yet. This episode is a therapy session for anyone who's felt the anxiety of running A/B tests on their Shopify store.In this episode of IntelliJAMS, Alex McEachern and experimentation expert Ally Petretti walk through the most common anxieties of A/B testing and how to manage them with better process instead of more stress.Should you check your A/B test results every day?Checking your test daily is like weighing yourself multiple times a day on a diet. The number isn't wrong, but it's not ready to be meaningful yet. Check to make sure the test is collecting data and firing correctly, but don't read into the directional results until you've hit your planned runtime and traffic thresholds.What is statistical significance and when should you end an A/B test?Statistical significance is often treated as a finish line, but reaching stat sig on day one doesn't mean you have a winner. Stat sig only looks at the math. You also need time (to account for day-of-week behavior changes, promotions, and the novelty effect) and volume (small sample sizes are fragile and can skew results dramatically). A test typically needs to run for at least one to two full weeks to account for these variables.What is the novelty effect in A/B testing?The novelty effect happens when returning visitors see something different on your site and react to the change itself rather than the actual experience. In price testing, this can look like a customer treating a different price as a discount and converting out of urgency. The novelty effect fades after a week or two as your sample includes more new visitors and repeat visitors adjust.What is metric shopping in CRO?Metric shopping is when you pick a winning metric after a test is already running, instead of sticking to the success metric defined in your hypothesis. For example, if your hypothesis targeted conversion rate but AOV happened to spike, declaring a win based on AOV is metric shopping. It's what Ally calls "the silent killer of CRO." The right move is to note the unexpected metric change, form a new hypothesis around it, and run a separate test.How should you communicate A/B test results?Don't just drop a list of metrics. That's left open to interpretation. Data storytelling means framing results around your original hypothesis, explaining what customers did and why, and tailoring the narrative for your audience. A CFO needs a different story than a CMO. Use tools like the Intelligems Slack bot or MCP integration to help frame results for different stakeholders.How do you handle test ideas that aren't backed by data?Ideas from teammates or Twitter threads aren't bad, but they need a hypothesis before they become tests. A strong hypothesis includes a data-backed problem, a proposed solution (the test idea), and a success metric. If an idea doesn't have that, either formulate the hypothesis yourself from existing data or add it to the backlog until you can.How do you coordinate A/B testing across teams?CRO teams and paid media teams often test independently, which creates noise in each other's data. The key is sharing hypotheses (not just test plans), coordinating testing calendars, and recognizing that every team is working toward the same growth goal. Regular cross-team syncs where you discuss challenges, upcoming campaigns, and testing angles can turn conflict into collaboration.Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-aca...Connect with Ally Petretti:https://www.linkedin.com/in/ally-petretti-kuhn-47a27014/ Connect with Intelligems:Website: https://intelligems.ioBlog: https://intelligems.io/blog----0:00 - Intro: Why we need a CRO therapist1:05 - The #1 stressor in testing: urgency and timing1:44 - "I'm a genius" vs. "I'm killing the business" (early test reactions)2:33 - Peeking at tests: checking vs. reacting4:51 - Pre-test analysis: setting guardrails before you launch5:30 - Statistical significance isn't the finish line6:13 - The novelty effect and why it matters for price testing7:47 - Using the time series chart to read test stability8:39 - When to call a test early (the right way)10:12 - No test losers: every result is a learning10:59 - Hypothesis discipline and metric shopping12:18 - What metric shopping is and why it's the silent killer of CRO13:25 - Communicating test results: data storytelling15:57 - The three E's framework: Explore, Experiment, Extend17:25 - Understanding your customers through testing18:20 - "Feelings over data": handling test ideas without a hypothesis21:19 - Cross-team testing: getting paid media and CRO on the same page25:42 - Wrap-up and where to connect with Ally

Conversion rate went up, but did profit? Most testing programs optimize for one metric in isolation, and that's exactly where they go wrong. In this episode, Amanda Siegel (Director of E-commerce at Scale Media) breaks down why she stopped calling what she does "CRO" five years ago and started treating testing as a tool to validate business strategy, not just chase website wins.In this episode, Alex and Amanda explore what happens when you balance conversion rate, AOV, and subscription adoption in every test instead of picking one, why the old UX tricks (spin-to-wins, slashed prices, CTA color swaps) are losing their edge, and how to get buy-in for short-term metric dips that lead to long-term growth.Timestamps:0:00 - Intro and why Amanda calls herself "a little crazy"1:20 - The limits of traditional CRO and flat testing3:00 - What's changed in the last 10 years of e-commerce UX4:17 - Consumers are catching on to optimization tricks6:10 - Why reducing discounts is actually improving conversion7:36 - Shifting from product sales to business strategy (bundles, subscriptions)9:02 - No silver bullets: what works for your business vs. someone else's10:20 - Conversion rate is one-dimensional and easy to game11:43 - The growth experiment mindset: testing for business growth13:58 - Intentional trade-offs and why short-term losses can be worth it15:10 - The two most important questions in commerce16:19 - Getting buy-in when a metric temporarily looks bad19:15 - "Testing is a tool, not a job title"21:14 - Diminishing returns of small, isolated tests22:58 - "Don't plan your tests. Test your plan."24:03 - Amanda's advice: be bold, back-plan from ideal outcomes25:39 - Why testing has to be cyclical, not one-and-done27:06 - Where to connect with AmandaTopics covered:Why conversion rate optimization (CRO) is too narrow for modern e-commerceBalancing conversion rate, AOV, and subscription adoption in a single testHow consumer behavior has evolved and why old UX tricks are losing effectivenessThe counterintuitive finding that reducing discounts can improve conversionUsing A/B testing and experimentation to validate business strategy, not just website changesGetting executive buy-in for short-term metric trade-offsThe growth experiment mindset for Shopify brandsWhy testing needs to be cyclical and revisited over time---Want to think bigger about your testing program? Join GEM Academy for free courses and a community of e-commerce operators sharing what actually works: https://www.skool.com/intelligems-academy-1535Connect with Amanda Siegel: https://www.linkedin.com/in/amandadsiegel/Connect with Intelligems:Website: https://intelligems.ioBlog: https://intelligems.io/blogLinkedIn: https://www.linkedin.com/company/intelligems