Episode 35: Beyond Online Experimentation: Generative Software That Optimizes Itself

Martin Tingley, Head of Windows Experimentation at Microsoft and former Head of the Experimentation Platform Analysis Team at Netflix, talks about why humans are the bottleneck in experimentation, and how a five-level maturity framework points the way toward self-optimizing software. Our conversation traces the path from basic hypothesis testing to a frontier where Generative AI creates, evaluates, and refines product variants in a closed loop. We explore the architectural shift required to move from testing single variants to optimizing entire parameter spaces, and how startups are already using AI to generate production-ready landing pages for Fortune 500 companies in hours rather than weeks. Tingley also shares a strategic lens on "experimentation programs," explaining how plotting the distribution of treatment effects across different product areas can serve as a powerful tool for capital allocation and high-level strategy. LINKS Martin on LinkedIn Want Your Company to Get Better at Experimentation? by Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit and Martin Tingley (Harvard Business Review) Avoid the Pitfalls of A/B Testing by Iavor Bojinov, Guillaume Saint-Jacques and Martin Tingley (Harvard Business Review) Martin & Co.'s Seven Part Blog Series on Experimentation at Netflix Roberto Medri (Meta) on High Signal: The Incentive Problem in Shipping AI Products — and How to Change It Tim O’Reilly on High Signal: The End of Programming As We Know It Watch the podcast episode on YouTube Delphina's Newsletter