Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents, so you can launch your app near your users. Learn more at Fly.io.
Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard, and I'm really excited to talk to another founder today in the AI space. Today we have with us Shahul from Ragas. He's one of the co-founders. Welcome, Shahul. How are you doing? Hey, I'm doing good. Hi, Daniel. Hey, folks. What are you doing with me?
Yeah, yeah. Well, thanks for joining at a late hour in India. Appreciate that. But yeah, I would love to hear a little bit, maybe for those that aren't familiar with what you're doing, maybe share a little bit about what that is and also maybe how you came upon the types of problems that you're solving with your current work. Sure. That's a very good place to start. So Echo Brothers is an open source library for evaluating LLM applications.
And what we are going to do with Ragas as an open source library is to provide the developers for AI engineers who are building LLM applications, the tools, and the workflows that are necessary to automate or partially automate the process of evaluation through using different techniques, different methods and techniques that we bring out through Ragas. And how we came up to the idea of Ragas is basically me and my co-founder, Jayden, we have been working in LLM for the past six, seven years. And when LLM came out, we were already working with natural-made models. Jayden mostly worked on the inference and infrastructure point of it. And myself, I was working as an applied researcher. So we were practically, we loved it. We were practically doing a lot of experiments with it. We were part of different open source initiatives, building LLMs and also using LLMs to build different frameworks at that point. Even Langchain and Labindex was coming out as one of the earlier frameworks at that point. This is around early 2023. And we were also working with different applications and stuff. And RAG was one of the most popular applications that went into production very easily with LLMs. And this is one of the first things that LLMs actually opened up as a possibility for enterprises to build something on top of them.
which can save a lot of time and money for them. So we were also building RACs. And after a couple of experiments with some clients, we found out that, OK, we are able to build these LLM applications, this RAC application using any LLM. It has different moving components. And going forward, there will be a lot more moving components to any LLM applications. It will be something like a common system where there are, as you can see, now it's not only RAC, now it's like two cases. There are different versions that constitute a common system kind of stuff.
And we thought, okay, you know, why don't we build something, some evaluation metrics, view evaluation metrics that can be used to understand the quality of any application that any AI engineer is building. And we ourselves found out that, you know, going through these answers or going through these intermediate results manually is not a scalable approach. I could rephrase it to say that it's a very important thing to do and nobody is really keen to do it. This is something everybody will push into someone else's responsibility, but it is something very important too. So we thought of identifying methods or coming up with solutions that could help developers do the evaluation.
but also save them a lot of time while doing it. So it should not be something that, you know, it is basically an evaluation of an application, regardless of RAG or any agent workflow. It's basically a very tedious process that takes up a lot of time. If you go through manually, you want to make sure that this tedious process, this manual process that takes up a lot of time, is cut down into one by a tenth of the time, and it should get you the same insights as you do manually. So that's how we came with our initial MVP of progress, and we have released an open source library link.
mid of 2023 and since then we have been continuously integrating and we have been getting organic growth and usage from that point from there and that's that's what we're going to do Yeah, that's awesome. I noticed that you very specifically refer to the evaluation of LLM applications, not necessarily the evaluation of LLMs. Could you explain like, what might be the difference between those mindsets for people that are maybe getting into this? Maybe they're looking at, they have looked at benchmarks, let's say for like a leaderboard for LLMs, right? And there's a certain level of, you know, evaluation or benchmarking there. But then you're talking about...
the evaluation of LLM applications. So could you help us understand kind of some of the differences there, some of what needs to be thought of at the application level versus at the model level? This goes to the ideology of AA as a consumer product as of now, because pre-LLMs, most of the enterprises or most of the startups who are building AA-powered applications used to have their own models. And they obviously also have a team of designers building and managing these models. So in that spectrum...
the building of the LLM itself or the model itself and also the evaluation of the model itself was the responsibility of the researcher, the engineer, slash data scientists who were working on it. But now it's not that people are building an LLM application, so AA applications is not really building their own models. They are consuming models from an external endpoint or even open source models that are being released and building applications on top of it. Now this actually forms a wide spectrum where at the left end of the spectrum, there are people building the LLMs itself. They have a separate list of targets, loss functions, metrics, and benchmarks to evaluate one.
And towards the right end of the spectrum, there are people who doesn't really care about how the LLM is built itself. They only care about, you know, can this thing, which I'm considering using an API, do what I'm trying to do here, using this AI as a technology. Now, when a researcher or an organization who builds LLM, evaluates LLM, they don't really know what is the exact use case for which this AI or LLM is going to be used against, or the type of data this will be used against. When the evaluation is done at the researcher or an LLM builder part of the spectrum, they are limited. to the evaluation or testing of this LLM's capabilities on a general corporate basis. They are not really tailoring this evaluation or testing to your application because they don't really have control over what you're trying to build with it. They can only say that, okay, this is a general capability of the model. And we know that even the general capabilities of the models and benchmarks are highly leaked into the training data. And even that is questionable, but that's a different question. This is what a research over LLM builder does at this end. But when it comes to LLM application builder, he can't really, if someone builds a rag or a tooling agent on top of an LLM, he can't really say that, okay,
The LLM Builder has said X and X accuracy, so I should get X and X. That would be a while I should make. So what we are going to do here is giving that user, AA application builder, who is at the right end of the spectrum, the power, the virtual tools to evaluate this application without knowing so much about ML or getting into so much jargon or anything. You want to make it as easy as possible, so that as intuitive as possible, because most of the people who are building AA applications are not from an ML background, they are from a software engineering background, our application building, application background, their speciality is building and scaling this application, not building and scaling the models. Yeah, that makes a lot of sense. And also I'm realizing there's this kind of spectrum that I was thinking about while you were talking on the one side, you have kind of data scientists or researchers who are building out benchmarks or metrics for models. On the other end of the spectrum, you have maybe software engineers who are used to writing unit tests or integration tests for their software.
And then like what we're really talking about is integration of LLMs into software applications or into certain workflows. You talked a lot about kind of this distinction between LLM benchmarks and evaluating LLM applications. Could you talk a little bit about the differences? Maybe there are software engineers in the audience that maybe they're used to writing unit tests and integration tests for their software. Now, as a consumer, like you said, they're integrating some LLM functionality or maybe a reasoning, you know, chain of reasoning with LLMs into their software from a practical standpoint.
What are the new types of things that they might need to consider that are different maybe from the way that they've unit tested in the past or written tests in the past now that they're working with these LLM workflows? Sure, that's a very interesting question. So when it comes to the application builder slash software engineer who's building with AI, as you said, most of them are already familiar with unit tests and integration tests that they regularly write for their software softwares. Now, the major difference is that I also have observed many of the software engineers who are relating to evaluation using this analogy. Okay, this is testing.
I will learn or understand evaluation using my understanding of the testing. So the thing is that this is a fundamental thing about analogy itself. So now you are using analogy to understand one thing, but it could be that these two things can have different physical and chemical properties here. For example, when it comes to evaluation and versus traditional software testing, traditional software testing is mostly a discrete space where you have an input, you have an expected output, and if you give this input, you are supposed to get this expected output from the software. There is no variation in that that satisfies.
the test now you know if the logic is basically to add one plus one you should get if the whole logic is basically addition if you give input as one plus one the output should be two there is no other possibility that access that would you know satisfy that this case is correct but when it comes to natural language there is more of a continuous space like if you have an LLM that does the same thing let's say one plus one or you know addition LLM could even say two in natural language or two in decimal point numbers, right? So both are actually correct. So here, there are like a continuous space where the output cannot be exactly matched against, but you should have an understanding. But there is a, in this continuous space of outputs that are possible, there's obviously a subset that is actually, can be regardless, correct. Obviously, if the LLM gives an answer as three in natural language or something, that means, that is out of the bay, right? So what software engineers should really understand is that...
When you deal with ML, when you integrate A into your application, you should try to think of what it has in a continuous space rather than a very discrete space or in a black and white manner. It could be that it is in the middle of it. There are a lot of space in the gray area. That is what a software engineer should understand. Then there is also a non-deterministic part of the whole AI thing. Basically, if you have a system which is part of core and part of AI, the system is going to be somewhat non-deterministic, whatever you do.
So this is also something that traditional software is not like that. So traditional software is very deterministic. So you have an input, you will have an output. So here again, you have this deterministic thing to take out for. So that's also one thing. These are the two major differences that exist between software testing, traditional software testing, and this A application or combine A application testing that makes it different from each other. Yeah, super interesting. And I know in the software world, there's all of these sort of...
frameworks for development around test-driven development or data-driven development or these different things. I notice in Ragas, in your core concepts and your documentation, you talk about metrics-driven development for these LLM applications. So for those that are maybe developing LLM applications out there, could you describe a little bit the mindset of metrics-driven development, what you mean by that? And then maybe we can get into a few more of the details of Ragas itself and how you enable that framework. Yeah. Metrics-driven development is a concept from the test-driven development itself. So the idea we want to educate developers and software developers who are more familiar with metrics, but they are familiar with testing to what we're trying to do here. So as I said, metrics versus test is like metrics is something that delivers a value or that helps you understand the performance of some application in a scale of, let's say, 0 to 1 or something. Now, whenever you have an application, if you want to iterate or change, let's say you are bumbling and pull an application that comes in.
different agentic workflows plus iraq workflow, etc. And now let's say you want to change one single prompt or one single function code or something, or even the retriever, how would you understand the effect of this change in your pipeline? It's a question that matrix-driven development tries to answer. So if you have a metric, if you have a way to objectively quantify or understand the performance of your system before and after this change, you can also understand and analyze the systems or responses or the behavior of the system using these numbers. For example, if you are switching out the retriever and let's say you add a set of metrics that...
effectively quantifies the performance of your system, you could actually switch on the retriever, or switch on any function code or something, and then run the end rebugning once again for the given test set and understand the change in those metrics in particular dimensions. And, okay, let's say once you observe this change, you could again easily deep down, okay, you could easily fetch these samples for which the change is reflected to maximum, and then you could easily analyze and understand the matter. That actually helps and reduces a lot of time when it comes to debugging and testing these applications. So that's the idea of metrics being relevant. What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Assembly AI is where you can turn voice data into insights, chapters, transcripts, summaries, and so much more with their leading speech AI models. So Dylan, give me a glimpse into what you're doing with speech AI models at Assembly AI. So at Assembly, we're building industry-leading speech AI models for various tasks like speech-to-text, streaming speech-to-text, speech understanding to help developers easily convert voice data, whether it's live or pre-reported, into super accurate text, and then to help developers extract a ton of information and metadata around voice data or even around the text.
that they just were able to convert from that audio data. So these are things like picking out entities or PII that was spoken in voice files or summarizing voice and audio data down into custom summaries. It's things like being able to detect how many speakers spoke and who said what and what the names of different speakers were. So we bundle all those things into a super simple API with really great docs that developers can just sign up to for free to start, use the API, build into their apps, and then build these really cool AI apps and products and workflows and automations on top of voice data with. I dig it. Okay. Can you take me a little deeper into the opportunity for developers? Because it seems like there's a lot of voice data out there and there's a lot of trapped value in that voice data. There's so much voice data being created on the internet now. Podcasts, videos, phone calls, voice messages, audio books, virtual meetings. It's crazy. And you can now transform and understand all this voice and audio data in ways that were not even possible a year, 18 months ago. So what we're seeing with the help of these new AI models that we're creating at Assembly, developers and organizations are just racing to build all these new applications, workflows, automations that leverage the voice data they have either with their...
organization or within the product to build really cool new products and services workflows that are just like taking off in the market. So at assembly, we're building the industry leading models for all those different apps and workflows, whether it's speech to text or speaker diarization or speech understanding capabilities to summarize voice data or extract entities or voice data or mask PII from phone calls for various types of automations that might be built. And we're exposing that through a super simple, super scalable API that's just constantly being updated and constantly getting better. And so we're seeing a crazy amount of developers and companies just build really cool apps and services on top of our API every day. It's really only just getting started, especially with the model.
updates that we have planned over the second half of the year that are coming out. They're really excited to launch to the developers on our API. Okay. Constantly updated speech AI models at your fingertips. Well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser, or you can get started with a $50 credit at assemblyai.com slash practical AI. Again, that's assemblyai.com slash practical AI.
There's a lot that you've already integrated into Ragas, which are really interesting and maybe some people have heard of or some people haven't, like faithfulness and context recall, noise sensitivity, aspect critique, summarization score, and more that I'm not listening. I'm wondering if you could maybe just share a couple of those that you think are maybe most utilized or maybe interesting from your standpoint, just to give a sense of people, like what types of metrics are we talking about here? I think it would be beneficial to answer that in an abstract level. For example, if I say metrics, the whole concept of we devising or giving the developers these metrics is that it's not extremely hard to come up with these metrics, but it is when a software developer or an application developer thinks about evaluation, because he's already familiar with, you know, this concept of putting up metrics, deciding the right flow and everything, it could take him like two to three days to figure out, you know, the right metrics. So what Rados does is, if you are an application developer, okay, you are developing this particular type of application.
you can come to Ragas, and you can go to Ragas Metrics, and there you will find enough workflows, enough parts of the documentation that can help you navigate according to use cases, according to requirements, that can land you up in the right metrics that you should use. And the documentation will also provide an intuitive understanding of how this metrics is calculated underneath. So these are the two value props that we provide there. It's not that a developer can easily make up these metrics as they don't, but sure, it is mostly, if you are a software developer application there, it could be very hard for you to figure out, you know. So we are effectively taking that load off of you and providing you the ways to easily navigate and understand the right metrics to evaluate your application, tell it to your application. Now we are also expanding the restore metrics to two use cases and identity workflows, and we are also going to reformulate the whole load on that particular so that people can easily find the metrics. What kind of metrics can be? These are all, you know, whenever a developer comes to robust, he obviously asks a lot of questions, or even think about metrics, even asking a lot of questions, like, for example, what is my use case?
what are the parts in my application that I want to evaluate? And then what should I go for an LMS matrix, which has high correlation with human judgment, but also has its own lack of its own issues, you know, non-determinism and everything, or non-LMS matrix, which are traditional, it has less correlation with human judgment, but it is, you know, more reproducible. So all these questions, all these doubts will pop up in mind when I think about matrices. We are trying to abstract all that into Raga's matrixes and provide the developer a way to think about matrices. And with matrices, they also, when they adopt a matrix on Raga, they will also have related...
list of items or features that they can use because they adopted the metrics. For example, let's say you designed a metrics for language English. Now, let's say you are evaluating for Portuguese or in Spanish, you'll have to convert it to other language or whatever. Whenever you're evaluating different language, you'll have to do that. Now, if you're using a metric from Raghans, Raghans will take care of that. Then there are also issues, like if you're building an LN-based metric, which is a trend here, and most of the Raghans are LN-based metrics, because that has high correlation with human judgment.
Okay, let's say the LLM, you know, you are evaluating the LLM-based metrics and you are finding that the LLM-based metrics is actually performing. It's actually not performing better enough in your case. You should have a way to allay in these LLM. You know, the people will have some expectations for different metrics. For example, let's say you are trying to do something like truthfulness and truthfulness is basically the amount of hallucination or that is happening in the generated answer given a context. And if you look at it, different developers and different domain has different strictness towards hallucination. I could have a statement like, I have a blue car.
and versus i have a car and i could for me maybe in my domain i could say that these are two equal signals but for someone who is working in fintech or something it might not be closed so there's also a domain kind of bias when it comes to matrices or in rejectments because developers from the different domain expects different level of strictness and everything so bringing out this alignment with these metrics depending upon your domain is also one extra thing that you're trying to tackle here which is called basically we call it a metric element that is trying to align large language model judges to your specific attachments using the feedback that you can give to Edelands. It is also an upcoming feature in Ragas. I want to just dig into a couple of those things. So one is you talk about alignment, but also there's the idea of it's very important in this case for me to be thinking about data and examples from my domain in terms of how I'm evaluating these. So I'm just looking through just to make things concrete for people, looking at your example around answer relevance. And there's data samples in there that have a question, an answer, and then a set of contexts. And then you can evaluate with Ragas using the metric answer.
relevancy. So obviously there's a data component there. And in that component, there's, you know, answers that are there and contexts that are there. So one question would be like, how much and what type of data will people need to configure such that they can appropriately evaluate their LLM applications? And are there metrics that require sort of data upfront?
or metrics that maybe are reference-free that wouldn't require data up front? What's the perspective there in terms of kind of, I guess, the cold start? Like if people are starting an LLM application from scratch, they haven't run it in production, what's the data burden and the path towards both getting this data in place and getting the alignment in place? So regarding the data itself, the number of sample people generally used to evaluate is around 100 to 500 when it comes to offline evaluation. So it really depends on how you formulated your test data. So for example, you could have the same results or same kind of ranking when you have 100.
example data point, or in many use cases, even 100 may not be enough to include all the kind of use cases or all the kind of distributions that you see in production. So it really depends on the variety of items that you see in production, that you expect in production. Then let's say you have a very niche use case and the variety is very skewed, your test data can be very small, yet still serve the purpose. But if your use case is very broad and you see a wide variety of users coming into your application or a wide variety of requests, your test data has to be broad enough to include all these different distributions in the data set. So that's about the number.
And it's not really about the concrete number, but again, basically making sure that you understand what are the different distributions of queries coming into your system, and also making sure these distributional queries are also represented in the test data set so that you know when you change a probe or when you change a probe, which type of queries are being affected. It could be that when you change a particular retriever or a particular tool or something, when you start out a particular tool or something, it could be that a specific set of queries are being affected. So over a pipeline almost never gets affected if you want to do a huge. change. Therefore, the small changes, it's mostly a subset of queries, a subset of distribution, and it is very important that you identify these items. So that's about the, you know, how to form data set itself for evaluating applications. And regarding the second question of metrics and reference-free and the reference metrics, we usually provide both reference-free and reference-with reference metrics, but there is a big shortcoming to reference-free metrics itself because reference-free metrics is obviously, you know, you know, we could estimate things there, but there is an error to estimation with reference-free metrics. For example, answer relevancy or, you know.
you could estimate with some way that if the given answer was correct or something, but if you don't really have the exact thing, it's really possible to say that, okay, the NLM application arrived at the right thing at the end of the day. So there, what we provide, even with production data, it's really hard to accurate a test data set because, again, with production data, data is very, very easy. It's a common thing that people, if you ask any researcher or a guy who has put MLS into production, MLS into production, and the method of accurate test data set, obviously they are going to look at production data, but the way in which test data set are being accurate is... There is a long way to go from production data to formulating test data because production data is incredibly messy because it's an uncontrolled environment. It's not a human-controlled environment. People can come and say anything there, and they have zero consequences. So the thing is that production data becomes incredibly messy depending upon the application. If your application is serving an internal set of users like an internal company, employees or something, still, you have an amount of control. But if your application is strictly B2C and you are opening up the whole application to anyone using your application, there can be users who are basically trolls.
Just use your system and leave out full speed data. So production data, being incredibly messy, there is a long way from going from production data to a good test data set. So there again, what we are trying to do is providing a way to synthetically create these test data sets. Now, these synthetic creation of test data sets will be grounded on things like production data. Then there are your internal documents that should be taken into account when creating a synthetic test data set. Because, again, you're trying to create a test data set that's really tailored to use, because that's not a generic one. So there are two points where we ground it, basically the set of internal documents or whatever you have.
that you ground your application, and also the production data where the user engages with your application. And this is, again, one of the upcoming features. The test data set generation is already there, but we are also trying to extend it to where that we could ground these into the from production data. Basically, we call it seeding from production data. Basically, if a user has been already using our test generation, but he wants to take motivations from production to imitate more behavior in the test data set, that is also happening in production. Now, there is a lot of things that has to be done to make sure that to first.
understand what is happening in production. You know, there can be different distributions, again, as I said, different set of users. We have to understand different set of queries that's coming in, different set of interactions that are happening, and everything. And once then we understand that, we could also have a way to synthesize these type of data points using a lot of data. Now, the developers will not be to annotate these data points, but to verify these data points. Once this data or test data is synthesized by RALOS, you could export it to a simple, you know, UI tool. And then you could simply, or even Excel sheet, because most of the users could export this data to an Excel sheet, and basically, they go through it.
And then once they go through it, they can easily cut out the bad data points that they think is a little messed up or something. Because again, we can't really guarantee 100% efficiency while we're synthesizing these data points. So what we are trying to do is improve the efficiency. Let's say if you generate 100 data points, our goal is to make sure that all 100 data points are equally valid and good. But it might not happen like that in every case. So the other person can take 10 minutes of time and go through this data set manually. It is a fairly quicker process than annotating or creating these data sets. Because creating these data sets will be easy to take a day or two or even a lot of money when you give it to other human annotators.
So the other part basically can go through this data points that's been synthesized and then out of the points that he thinks that is not viable. So again, that is the real reason in which we save a lot of developer time evaluation. Because again, it's a very, you know, it's a very time consuming process that nobody wants to do. And it's mostly one other thing to do it and it's a messy process. So these are the type of innovations we are trying to bring in the evaluation space as well.
to save a lot of time for the developer. And this ideology or this philosophy is why we have been getting this organic growth. Because it is very hard to come up with these kind of solutions. But when we come up with these kind of solutions, there are a lot of developers who are, who wants, who badly wants it. And they are the people who motivate us to continually bring these kind of innovations to the innovation space. Yeah, that's great. I love the innovations around synthetic data use and evaluation.
utilizing kind of the LLMs and these models to help in the evaluation process, but in a way that's validated and still fitting with improvement over time. So yeah, as we kind of close out here, this has been a fascinating discussion, but maybe just to close out here, what are some of the things that you're excited about moving into the next six months or so to either explore, or maybe it's things that you see happening in the AI ecosystem more broadly? What really excites you about the direction that people are going with their LLM applications? So with LLM applications, With early 2023, RAG was a big thing. And if you remember the time when RAG became popular, there was again a lot of limitations. There were only 4K contacts or LNM was everything and everything. I think with agent decode tool use cases, we are at the same level as of now. We have been trying to bring more and more tool use cases.
tool use cases are actually incredibly useful when it comes to building a whole LLMA application experience because combined with internal knowledge. Internal knowledge is what you can infuse with Rack and then taking action scan is something that infuses tool use cases, tool bindings. So if in a tool binding, I'm very excited to see the next class of models performing better on tool use cases. Again, now even these recent models have... I've been bringing abstractions or using toolbinding to facilitate this, but still it's a little bit shaky. But the next class of models, I didn't expect it to be better. And then the whole thing with Rack plus 2 use case, we'll see no more enterprises adopting and adopting and using other applications at different capacities. And it's saving a lot of time and resources for both people who are at the back of these applications and other people who are interacting with these applications. So that's something I'm really excited when it comes to the next six months of catalog applications.
And also, I think when it comes to frameworks or libraries that are being built around applications, I'm seeing more and more better structures these days when it comes to these frameworks and libraries. Because people now have almost one and a half year of building with another application experience. Now that experience is yielding more and more understanding of what is the best instruction to be used to build these applications and everything. And then there's an overall agreement that is coming up with how to format outputs, how to build these combo systems and stuff. Earlier, the first year in 2020, people were really, really confused on how to build these applications, how to use it and everything. Now that clarity, I can see that more clarity is happening at that end too. So at the model building stage, again, the model building spectrum, we are again having more clarity on things like the data of the models, the type of data, more and more papers and more and more research happening at the data processing and pre-processing stage, what is the type of data needed to train the higher quality and best quality analysis.
we have mostly settled on the architecture itself from that point of development. Now, the main thing people are working on this data. Again, when it comes to data, people are, again, now mostly we will finish up the free data that's available on the internet very quickly, and now it's, again, synthetic data that has a really good chance of improving these models and stuff. So the idea of models output being used to feed models and improving the models itself for different use cases is itself a fascinating thing regarding the model building stage. When it comes to evaluation, there needs to be more and more innovations.
happening in that evaluation space as an open research or as an open approach to, you know, bringing all the time involved in testing and assessing these applications. Because that's one big pain point on why the enterprises or, you know, big companies could, a big barrier for these big companies to adopt LLM applications, LLMs or AAs in their system. Because these people, they have high responsibility. So again, with high responsibility, you have to do these testing and evaluations. And there is a very good way of doing it and a great way of doing it.
it becomes a pain point. And that's something we are also trying to bring in. You know, we are trying to, you know, formulate all the research, all the innovations that are happening in the evaluation space to come together and build an on-source standard for evaluating LLM applications. So there is an agreement between everyone on how to evaluate LLM applications. And that's the long-term mission of the company. Awesome. Yeah, well, thank you, Shahul, for taking time to join and, again, joining at a late hour where you're at. I'm really excited about what you're doing with Ragas. And this is a really interesting space that we'll be interested to follow. So thanks for taking time and hope you can have a good rest of your week. Sure. Thanks, Daniel.
This was fun talking to you, and I hope your users learned something from this conversation. Yeah, thanks. Bye-bye. All right, that is Practical AI for this week. Subscribe now. If you haven't already, head to practicalai.fm for all the ways. And join our free Slack team, where you can hang out with Daniel, Chris, and the entire Changelog community. Sign up today at practicalai.fm slash community. Thanks again to our partners at fly.io, to our Beat Freaking Residence, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.