Outperform

Why does one of the world’s leading business podcasts (and media companies) employ a full-time Head of Experimentation and Failure?Ask Grace Miller, who holds the title at FLIGHTSTORY, the company behind The Diary of a CEO. On today’s episode, Grace shares how her team built an experimentation engine to drive growth across podcasts, product, paid media, and marketing.Grace shares how her work spans the entire business - from optimizing things like add-to-cart placement and email design, to podcast-specific experiments like trailer hooks, audience retention, ad placement, and YouTube thumbnails. She also discusses how FLIGHTSTORY takes experimentation offline through live events and community dinners to strengthen the audience bond. If you want to understand how modern media companies test, learn, and grow faster, this episode offers a behind-the-scenes look at experimentation in action.In this episode, we discuss:Why FLIGHTSTORY built a culture around experimentation and why failure is part of the processHow testing small changes (like podcast hooks, thumbnails, and ad placement) can significantly impact audience growthWhat it takes to build an experimentation engine across podcasts, product, marketing, and paid mediaHow live events and community experiences help strengthen audience connection beyond digital channels

What’s next for experimentation in 2025? We’re breaking down insights from the biggest names in tech—Netflix, Amazon, Meta, and more.Fresh from top experimentation conferences, we’ve identified three major trends set to shape the future of experimentation. In this Episode:Shifting toward self-serve capabilities for less-technical end usersThe rise of automated "set it and forget it" experimentsRethinking statistical guidelines to accelerate innovationConnect with us: Ryan on LinkedInChetan on LinkedIn

Eppo Head of Product Eric Metelka took to the stage in San Francisco at the 2024 Lenny & Friends Summit to share some big insights on where product teams are making mistakes in their experimentation programs.Drawing on case studies from industry leaders like Netflix, MSN and Booking.com, Eric explores how to rebuild trust in your data, accelerate experiment velocity, and foster a culture that thrives on bold, high-impact innovation. In this Episode:Discover how rigorous science ensures reliable results and builds trust with stakeholders.Learn the secret to running faster, more impactful experiments using self-service tools.Explore why bold changes drive growth—and how to manage risks with guardrails.

Data-driven is the buzzword of the day—but what are the strategies that separate top-tier, data-driven companies from the rest? From chaotic, one-off tests to fully automated systems, we share the four stages that drive scalable growth, speed up learning, and make decision-making a breeze.In this episode, Lukas Goetz-Weiss, joins Ryan Lucht to explore each maturity stage, the common roadblocks, and the practical steps any business can take to level up.In this Episode:Why most companies start with isolated, intuition-driven tests and the pitfalls of this chaotic approach.Why companies gain structure but struggle with resource bottlenecks and decision-making delays.How to build trust in experiment results, empower teams, and avoid pitfalls like p-hacking and underpowered tests.Examples of companies that turn experimentation into a seamless, strategic advantage.

"When in doubt, test it out!"In today's episode of Outperform, we uncover why the best teams thrive on a culture of testing everything— powerful metrics like segment-level differences and the dynamic balance between best practices and bold experiments, showcasing why both small tweaks and big swings matter. Plus, how a well-balanced experiment portfolio and savvy management can supercharge your path to strategic innovation!In this Episode: Discover five unconventional methods for generating powerful A/B test hypotheses.Learn how to harness historical company insights, customer feedback, and UX best practices to drive impactful business outcomes.Explore the significance of a balanced experimentation portfolio, blending big and small bets to effectively transform your business and maintain a competitive edge.

The power of experimentation is never clearer than when small changes spark massive breakthroughs. But how do you get your teams convinced to start testing many small changes in the first place? We explore some common roadblocks that stand in the way of world-class cultures of experimentation. Ryan, Che, and Sven share insider insights from companies like Airbnb and Stitch Fix to give you actionable ideas and strategies in building your own experimentation program - and career.In this Episode: Taking calculated risks and consistently experimenting are key to achieving breakthroughs and driving success.Standardizing processes and metrics minimizes friction, making it easier for teams to run experiments independently.Leadership is essential in building a culture of experimentation, where successes are celebrated and failures are seen as valuable learning opportunities.

AI is evolving at lightning speed - the shift from custom-built models to pre-trained large language models (LLMs) is driving rapid adoption from businesses.But how do we know if all these models are actually driving positive business outcomes? As the New York Times put it, “AI has a measurement problem”. In this Episode:Why most current AI model evaluation methods fall shortHow to truly measure AI effectiveness with real-world data and experimentation

Imagine a world where your AB tests run themselves—faster, smarter, and more dynamic than a simple 50/50 split that takes weeks to measure. Today’s episode breaks down how multi-armed bandit algorithms could change when and where you use AB testing. But don’t be fooled—there are no free lunches and no cure-alls.. We’ll show you why understanding your use case, keeping the stakes low, and having human oversight are the keys to getting the most out of these powerful algorithms.In this Episode:The potential of multi-armed bandit algorithms to supplant traditional AB testingCritically assessing the limitations and complexities involved in applying bandit algorithms to real-world data and marketing contexts.The role of these algorithms in personalizing customer interactions and automating low-risk experiments.