Why You Shouldn’t Let Perfect Be the Enemy of Experimentation — Dan Pannasch, RevenueCat

On the podcast I talk with Dan about how to design experiments that answer the right questions, common A/B testing pitfalls to avoid, and how a simple checklist might just save your complex experiment.Top Takeaways🍞 Conclusions from tests sometimes go stale faster than you realize👌 Minimizing the cost of running tests will improve decision making🤪 Check your sanity — or don’t live and die by statistical significanceAbout Dan Pannasch👨‍💻 Senior Product Manager at RevenueCat💪 Dan saw what experimentation looked like across a portfolio of app businesses when his previous company TelTech’s success led to an acquisition by IAC. He joined RevenueCat in May 2022 and leads the Experiments project.💡 “You could change the color [of the buy button in A/B testing and] release it in the new application. And if you can't tell which one won [with users], then you learned that it doesn't matter. You didn't learn which one won, but you did learn that it doesn't matter for you right now.”👋 Twitter | LinkedInLinks & Resources‣ Join the RevenueCat team‣ Sub Club interview with Blinkist’s Jaycee Day‣ RevenueCat’s Experiments toolFollow us on Twitter‣ David Barnard‣ Jacob Eiting‣ RevenueCat‣ Sub ClubEpisode Highlights[2:18] Experimentation: What is app experimentation and why should you do it? The right decision making, considering impact on variables, and risk mitigation are everything when it comes to user experience.[9:04] Taking a page from DuoLingo’s playbook: Product strategy and intuition naturally limits possibilities — and it’s not the place for A/B testing. Microdecisions within deliverables are testable, and then it’s just cost-benefit analyses. [14:04] The early days: The cost-benefit analysis should pervade every stage of the process, from early growth and beyond. Trying to design the perfect A/B test isn’t always possible when customers are begging you for.[19:20] Paywall plays: Where you put the paywall is a tough decision. But there are strategies for implementation and risk mitigation.[24:35] Testing 101: Be sure to write down the hypothesis before testing so that you can measure impact. Unexpected results — where you learn the most about variables — depend on it.[28:05] Follow it up: Dan shares his thoughts on user follow ups to boost quantitative data with qualitative data. Sometimes talking to users can be very powerful.[31:13] Sanity check: How to do a testing plan, as done by Dan during his time as a PM at TelTech. Plus, an explanation of statistical significance.[39:53] Impact and intuition: To understand user experience impact and product intuition, it’s critical to ensure the design aligns with the value proposition.[42:22] Actual testing: There are pitfalls and screw-ups to watch out for when testing (and even before).[46:33] Analyzing the results: Dan provides his overview for analyzing the results after running the experiment. Second and third order effects are important but not always immediately obvious. [48:41] The Experiments product: RevenueCat’s new tool enables easy A/B testing for two offerings. The data helps you analyze the full subscription lifecycle to understand which variant is producing more value for your business.[55:55] Bugs: No product will ever be perfect, but Experiments offers app developers the tools and confidence to make sure it’s at least most of the way there.