The PM Pod by Dan

Discover how fine-tuning allows you to take powerful, pre-trained AI models and tailor them for specific tasks and domains, unlocking their true potential for cutting-edge applications. We'll explore advanced techniques like Instruction Fine-tuning, which uses explicit instructions to guide model behaviour for more tailored results.Learn about the Hugging Face Hub, a crucial platform that makes hosting, sharing, and accessing AI models easy.Find out how it integrates with various libraries and provides the foundation for building diverse AI capabilities.

Mastering AI Evals: A Guide for Product Managers and Engineers, Get insights gained from helping over 30 companies, highlighting that unsuccessful AI products almost always fail due to a lack of robust evaluation systems. Successful teams, in contrast, obsess over measurement and iteration, enabled by these evaluation systems. Discover how evals are a critical element of any AI initiative, preventing product failure and accelerating iteration velocity.Explore the AI Evals Flywheel, presented as a virtuous cycle crucial for differentiating great from mediocre AI products, connecting evaluation, debugging, and changing product behaviour. The discussion covers the three essential levels of AI evaluation:Level 1: Unit Tests (Assertions): These are fast and cheap, ideal for running on every code change to get quick feedback. They should be organised beyond typical unit tests and frequently updated based on observed failures.Level 2: Model & Human Eval: This level is for deeper validation, requiring logging traces and human feedback. Learn the importance of removing friction from looking at data, using binary ratings for simplicity, and tracking correlation between model and human evaluation to decide how much you can rely on automation.Level 3: A/B Testing: This is the most costly level, typically reserved for more mature products and significant changes to ensure the AI product drives desired user outcomes or behaviours.Learn about the more effective bottom-up approach to AI Eval metrics, focusing on discovering domain-specific failure modes by looking at actual data and letting metrics naturally emerge, rather than starting with generic top-down metrics. Hamel uses real-world examples, like Rechat's AI assistant Lucy and NurtureBoss, which used a bottom-up approach to identify key issues accounting for over 60% of their problems.Finally, uncover the three free superpowers that robust evaluation systems unlock: Fine-Tuning (primarily via preparing high-quality data), Data Synthesis & Curation (leveraging existing eval infrastructure to filter and curate data, often synthetically generated using LLMs), and streamlined Debugging (due to the significant overlap between the infrastructure needed for evaluation and debugging). Tune in for practical takeaways, including tips on simplifying your approach, looking at lots of data, and using LLMs to generate tests, synthetic data, and critiques. This episode provides essential insights for anyone building AI products, focusing on the most impactful investment you can make: your evaluation system.

In this episode, we dive into the OpenAI Cookbook’s latest guide on the GPT-4.1 model family. Discover how this upgraded model pushes the boundaries in coding, instruction following, and handling complex tasks. We explore practical prompting strategies tailored for developers—covering agentic workflows, long context management, and effective tool use. You’ll hear real-world examples, including a SWE-bench verified agentic prompt and a customer service use case. Plus, we break down how to structure chain-of-thought prompts and apply file diffs in code workflows. Whether you’re building AI agents or just optimizing your prompts, this episode is packed with insights you can apply right away.

In this episode, we zoom in on the art and science of crafting effective AI prompts—through the lens of a real-world Google case study. Learn how top product managers at Google are using prompt engineering to accelerate workflows, enhance product research, and prototype faster with AI tools like ChatGPT and Gemini.Dan breaks down prompt strategies tailored for PMs, including frameworks for writing clear, goal-oriented prompts, common mistakes to avoid, and how to iterate toward better outputs. Whether you're writing specs, user stories, or customer messages, better prompting means faster, smarter outcomes.What you’ll learn:How Google PMs structure prompts to drive resultsPrompt patterns for product discovery, analysis, and UX writingWhy prompt literacy is becoming a core PM skill in the AI ageWho should tune in:Product managers, tech leads, and startup builders looking to level up their AI game.

In this episode, Dan unpacks OpenAI’s comprehensive 32-page guide on Agentic AI—a powerful new paradigm where AI systems don’t just respond, but take initiative, pursue goals, and operate with autonomy. We explore what it means to build agentic systems, their potential applications in product development, and the implications for product managers and technology leaders.Expect key takeaways from the guide, including design principles, safety considerations, and practical examples that show how agentic AI is reshaping workflows, decision-making, and customer experiences.Perfect for:Product leaders, AI/ML enthusiasts, and technologists curious about the next evolution in applied AI.

In this episode, we dive deep into Vertical Operating Models—a game-changing approach for scaling product teams and aligning technology with business outcomes. Whether you’re a product manager looking to drive more ownership or a tech leader aiming to streamline cross-functional collaboration, this episode breaks down how vertical structures can increase autonomy, accelerate delivery, and foster innovation.Join Dan as he explores real-world examples, key frameworks, and the trade-offs of vertical vs. horizontal operating models. Learn how to rethink your team structures to better serve users and drive impact.Who should listen:Product managers, engineering leads, CTOs, and anyone building or scaling digital products.