All the Hard Stuff with LLMs in Product Development // Phillip Carter // #170

MLOps Coffee Sessions #170 with Phillip Carter, All the Hard Stuff with LLMs in Product Development. We are now accepting talk proposals for our next LLM in Production virtual conference on October 3rd. Apply to speak here: https://go.mlops.community/NSAX1O// AbstractDelve into challenges in implementing LLMs, such as security concerns and collaborative measures against attacks. Emphasize the role of ML engineers and product managers in successful implementation. Explore identifying leading indicators and measuring ROI for impactful AI initiatives.// BioPhillip is on the product team at Honeycomb, where he works on a bunch of different developer tooling things. He's an OpenTelemetry maintainer -- chances are, if you've read the docs to learn how to use OTel, you've read his words. He's also Honeycomb's (accidental) prompt engineering expert by virtue of building and shipping products that use LLMs. In a past life, he worked on developer tools at Microsoft, helping bring the first cross-platform version of .NET into the world and grow to 5 million active developers. When not doing computer stuff, you'll find Phillip in the mountains riding a snowboard or backpacking in the Cascades.// MLOps Jobs board jobs.mlops.community// MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related Links⁠Website: https://phillipcarter.dev/https://www.honeycomb.io/blog/improving-llms-production-observabilityhttps://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llmhttps://phillipcarter.dev/posts/how-to-make-an-fsharp-code-fixer/The "hard stuff" post: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llmOur follow-up on iterating on LLMs in prod: https://www.honeycomb.io/blog/improving-llms-production-observability--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Phillip on LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/Timestamps:[00:00] Phillip's preferred coffee[00:33] Takeaways[01:53] Please like, share, and subscribe to our MLOps channels![02:45] Phillip's background[07:15] Querying Natural Language[11:25] Function calls[14:29] Pasting errors or traces[16:30] Error patterns[20:22] Honeycomb's Improvement cycle[23:20] Prompt boxes rationale[28:06] Prompt injection cycles[32:11] Injection Attempt[33:30] UI undervalued, charging the AI feature[35:11] ROI cost[44:26] Bridging ML and Product Perspective[52:53] AI Model Trade-offs[56:33] Query assistant[59:07] Honeycomb is hiring![1:00:08] Wrap up