Skip to content
Collaboration & evaluation for LLM apps episode artwork

Jan 23, 2024 46 min

Collaboration & evaluation for LLM apps

from Practical AI Hosted by Daniel Whitenack and Chris Benson

Small changes in prompts can create large changes in the output behavior of generative AI models. Add to that the confusion around proper evaluation of LLM applications, and you have a recipe for confusion and frustration. Raza and the Humanloop team have been diving into these problems, and, in this episode, Raza helps us understand how non-technical prompt engineers can productively collaborate with technical software engineers while building AI-driven apps. Sponsors: Read Write Own – Read, Write, Own: Building the Next Era of the Internet—a new book from entrepreneur and investor Chris Dixon—explores one possible solution to the internet’s authenticity problem: Blockchains. From AI that tracks its source material to generative programs that compensate—rather than cannibalize—creators. It’s a call to action for a more open, transparent, and democratic internet. One that opens the black box of AI, tracks the origins we see online, and much more. Order your copy of Read, Write, Own today at readwriteown.com Changelog News – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today. Fly.io – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. Featuring: Raza Habib – LinkedIn, X Daniel Whitenack – Website, GitHub, X Show Notes: Humanloop Upcoming Events: Register for upcoming webinars here!

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Collaboration & evaluation for LLM apps

0:00 46:14

Chapters

  1. 0:06 Welcome to Practical AI
  2. 0:42 Origin of Human Loop
  3. 5:22 Types of designers
  4. 8:05 Tech & non-tech worklfow
  5. 11:47 What am i building?
  6. 13:18 Sponsor: Read Write Own
  7. 14:39 What can Human Loop do?
  8. 17:22 In-production feedback
  9. 18:31 Fine-tiuning jargon
  10. 21:42 Fine-tuning trends
  11. 24:10 Proliferation of open models
  12. 26:49 Sponsor: Changelog News
  13. 28:20 Different roles in the HL system / Collaborating in Human Loop
  14. 32:04 Production framework
  15. 34:39 Importance of evaluation
  16. 38:13 Surprising usecases
  17. 42:42 Exciting things that are happening
  18. 45:27 Outro

No transcript yet.

Requests help us decide what to transcribe next.

Share this episode

Similar podcasts

CISO Perspectives (public) N2K Networks This season on CISO Perspectives, host Kim Jones explores some of the challenges of leading through uncertainty. We explore the complexity of the changing nature of regulation and working with the federal government, the evolution of privacy and fraud, and how emerging technologies like AI and quantum computing are changing cyber. When you don’t know what questions to ask, you’re afraid to ask, or don’t know who to ask, CISO Perspectives provides the foundation for learning in this brave new world.

Frequently asked questions

How long is this episode of Practical AI?

This episode is 46 minutes long.

When was this Practical AI episode published?

This episode was published on January 23, 2024.

Can I download this Practical AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.