Ep. 261 | Andy Pierce: False Confidence at Machine Speed: How to Make Synthetic Customers Useful

Episode 261: What if you could use AI-generated customer twins to test value propositions, pricing, and demand before you go to market? That is the idea my Bain colleague Andy Pierce is exploring. Synthetic customers can help teams move faster, pressure test offers, and simulate trade-offs much faster and at far lower cost than traditional research alone. Says Andy, "You can do things at half the time and a third the cost. You can be a hero internally by helping your business functions get to success better, faster, cheaper." But synthetic customers can also mislead you if you treat them like magic. Left ungrounded, large language models tend to be overly positive, can drift over time, and may reflect bad data rather than real human behavior.  To put theory into practice, we'll explore a 24-month telco case study that ran in parallel with a synthetic panel and hit ~85% overlap with human survey responses after confronting both bias and brown-nosing behavior. Guest: Andy Pierce, Partner, Bain & Company Host: Rob Markey, Partner, Bain & Company Give Us Feedback: Help us improve the podcast here: https://bit.ly/CCPodcastFeedback Time-Stamped Topics: 00:00 — Value proposition basics and the design target question 00:05 — Pricing as the most natural lever and why finance often doubts research 00:09 — Early LLM experiments 00:10 — Telco case study 00:12 — Over-positivity and tuning to an ~85% survey overlap 00:14 — Quarter-by-quarter improvement 00:35 — Rational vs. experiential vs. emotional  Time-Stamped Quotes: [32:00] "We've already started to see a world where I can train an LLM on a new idea, develop a new value proposition—or an extension to an existing value proposition—and instead of trying to predict the research outcome, I'm trying to predict actual sales in the marketplace." [37:00] "It's not good enough to just create the persona … you still have to test against real humans." [38:00] "Synthetic customers are here to stay. It's not a fad. And clients are already using them to build a competitive advantage." Resources Referenced on Today's Show: Generative Agent Simulations of 1,000 People — https://arxiv.org/abs/2411.10109 Simulating Human Behavior with AI Agents — https://hai.stanford.edu/assets/files/hai-policy-brief-simulating-human-behavior-with-ai-agents.pdf How Synthetic Customers Bring Companies Closer to the Real Ones — https://www.bain.com/insights/how-synthetic-customers-bring-companies-closer-to-the-real-ones/ UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents — https://arxiv.org/html/2504.09407v2 The Rise of Synthetic Respondents in Market Research — https://nielseniq.com/global/en/insights/education/2024/the-rise-of-synthetic-respondents Synthetic Users: If, When, and How to Use AI-Generated "Research" — https://www.nngroup.com/articles/synthetic-users/ A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas — https://arxiv.org/html/2505.07850v1 Luxury in Transition: Securing Future Growth — https://www.bain.com/insights/luxury-in-transition-securing-future-growth/