Episode 29: Lessons from a Year of Building with LLMs (Part 1)

Hugo speaks about Lessons Learned from a Year of Building with LLMs with Eugene Yan from Amazon, Bryan Bischof from Hex, Charles Frye from Modal, Hamel Husain from Parlance Labs, and Shreya Shankar from UC Berkeley.These five guests, along with Jason Liu who couldn't join us, have spent the past year building real-world applications with Large Language Models (LLMs). They've distilled their experiences into a report of 42 lessons across operational, strategic, and tactical dimensions (https://applied-llms.org/), and they're here to share their insights.We’ve split this roundtable into 2 episodes and, in this first episode, we'll explore:The critical role of evaluation and monitoring in LLM applications and why they're non-negotiable, including "evals" - short for evaluations, which are automated tests for assessing LLM performance and output quality;Why data literacy is your secret weapon in the AI landscape;The fine-tuning dilemma: when to do it and when to skip it;Real-world lessons from building LLM applications that textbooks won't teach you;The evolving role of data scientists and AI engineers in the age of AI.Although we're focusing on LLMs, many of these insights apply broadly to data science, machine learning, and product development, more generally.LINKSThe livestream on YouTube (https://www.youtube.com/live/c0gcsprsFig)The Report: What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/)About the Guests/Authors (https://applied-llms.org/about.html) Get full access to Vanishing Gradients at hugobowne.substack.com/subscribe