Skip to content
Metrics Driven Development episode artwork

Aug 29, 2024 42 min

Metrics Driven Development

from Practical AI Hosted by Daniel Whitenack and Chris Benson

How do you systematically measure, optimize, and improve the performance of LLM applications (like those powered by RAG or tool use)? Ragas is an open source effort that has been trying to answer this question comprehensively, and they are promoting a “Metrics Driven Development” approach. Shahul from Ragas joins us to discuss Ragas in this episode, and we dig into specific metrics, the difference between benchmarking models and evaluating LLM apps, generating synthetic test data and more. Sponsors: Assembly AI – Turn voice data into summaries with AssemblyAI’s leading Speech AI models. Built by AI experts, their Speech AI models include accurate speech-to-text for voice data (such as calls, virtual meetings, and podcasts), speaker detection, sentiment analysis, chapter detection, PII redaction, and more. Featuring: Shahul Es – GitHub, LinkedIn, X Daniel Whitenack – Website, GitHub, X Show Notes: Ragas Upcoming Events: Register for upcoming webinars here!

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Metrics Driven Development

0:00 42:12

Chapters

  1. 0:00 Welcome to Practical AI
  2. 0:43 What is Ragas
  3. 5:18 General LLM evaluation
  4. 10:09 Current unit testing workflow
  5. 14:36 Metrics driven development
  6. 17:20 Sponsor: Assembly AI
  7. 20:59 Most used metrics
  8. 26:27 Data burdens
  9. 35:49 Exciting things coming
  10. 40:48 Thanks for joining us!
  11. 41:25 Outro

No transcript yet.

Requests help us decide what to transcribe next.

Share this episode

Similar podcasts

CISO Perspectives (public) N2K Networks This season on CISO Perspectives, host Kim Jones explores some of the challenges of leading through uncertainty. We explore the complexity of the changing nature of regulation and working with the federal government, the evolution of privacy and fraud, and how emerging technologies like AI and quantum computing are changing cyber. When you don’t know what questions to ask, you’re afraid to ask, or don’t know who to ask, CISO Perspectives provides the foundation for learning in this brave new world.

Frequently asked questions

How long is this episode of Practical AI?

This episode is 42 minutes long.

When was this Practical AI episode published?

This episode was published on August 29, 2024.

Can I download this Practical AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.