Season 6, Episode 16: Can an LLM evaluate ad creative? (with Luca Fiaschi)

My guest on this episode of the podcast is Luca Fiaschi, a machine learning expert who previously held executive data science roles at MistPlay, StitchFix, and HelloFresh. Luca is now a Partner for the Generative AI vertical at PyMC Labs, a consultancy that specializes in the application of Bayesian methods to business problems and which maintains the open source PyMC library for Bayesian statistical modeling as well as the open source PyMC Marketing media mix modeling library. The subject of my discussion with Luca is PyMC Labs' recent paper, LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings. I found the paper fascinating and wrote an overview on LinkedIn; the paper's authors (from PyMC Labs as well as Colgate) use an LLM to score product concepts, finding that the distribution of LLM-produced scores can be comparable to that produced by human panels.Among other things, the podcast episode covers:Background on the paper, and how the partnership with Colgate-Palmolive came about.An overview of the LLM querying methodologies used in the paper and how they produced score distributions.The approaches used in the paper for calculating similarities between the LLM-produced and human-produced scores.The conclusions / findings of the paper.What the implications of the paper are on marketing creative ideation and the use of LLMs for evaluating product and advertising concepts.Thanks to the sponsors of this week’s episode of the Mobile Dev Memo podcast:Xsolla. With the Xsolla Web Shop, you can create a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards, and analytics.INCRMNTAL⁠⁠. True attribution measures incrementality, always on.Universal Ads is Comcast’s self-serve TV ads platform that lets you launch campaigns in minutes across premium inventory from NBC, Paramount, Warner Bros. Discovery, Roku, and more.Interested in advertising in the Mobile Dev Memo newsletter or podcast? ⁠Get in touch⁠.