Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.
Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist and founder of a company called Prediction Guard. And I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing very well. Enjoying this fine springtime weather of LLMs. You have the spring LLM bloom, I guess. That's right. That's right. Well, I don't even think we can use the word bloom because that's loaded now. Yeah, I was going to say that has a whole different meaning. There's no word that's not loaded with some sort of AI meaning at this point. Yeah.
We should just go straight to our guest. Yeah, including llamas, which we're excited today to have with us Jerry Liu, who is co-founder and creator of Llama Index. Welcome, Jerry. Yeah, thanks, Daniel and Chris, for having me. Super excited to be here.
Yeah, I'm really excited because we've had a few conversations in the past and I've used Llama Index in some of my own work and also kind of tried some integration stuff with various data sources. So I'm really excited to hear a little bit more of the story and kind of the vision behind the project. If I'm just reading from the docs, Llama Index is about connecting LLMs or large language models with external data. So maybe a first question, kind of a general question, not specific to Llama Index necessarily, is like, why would one want to connect large language models with external data? Yeah, it's a good question. And so for those of you who are already in the space of LM application development, this might sound obvious to you, but for those of you who might be still somewhat unfamiliar, large language models have a lot of different sorts of capabilities that are really good at answering questions, doing tasks, being able to summarize stuff, basically anything you throw at it, like try to write a short story, write a poem, it can do. And the default mode of interacting with a language model like ChatGPT is that you would write stuff to it in a chat-like interface. This query would hit the model and you'd get back some output. I think one of the next questions that people will get into, especially as they're trying to explore building applications on top of large language models is...
how can this language model understand my own private data, right? Whether you're kind of like a single person or you're an entire organization. And these days, there's like a lot of different ways for actually trying to incorporate new knowledge into a language model. The models themselves are trained on just like a giant corpus of data. And so if you're like an ML researcher, your default mode is just, how can I train this model on more data so that I can try to memorize this knowledge, right? And the algorithm there is basically through some sort of like reading the sense or the weights or RLHF or any sort of like fancy ML algorithm that actually includes the knowledge and the weights of the model itself. I think one interesting thing about large language models these days is that instead of like training the model, you can actually take the model as is.
and just like figure out how to have it reason over new information and so for instance like use that input prompt as like the cache space to feed in new information tell to reason over that data and to answer questions over that data and i think that's very interesting because you can take the model itself which you know has been trained on a variety of data but doesn't necessarily have inherent knowledge about you know you as a person or your organization data but then you can tell it hey here's some new data that i have now given this data how can i answer the following questions and this is part of the stack that a lot of people are discovering these days where you can actually just use the language model itself as a pretrained service and then wrap that in this overall software system to incorporate your data with the language model. Cool. Yeah. And your project is called Llama Index. Now, before the past few months or six months or whatever, when I was thinking about indices or an index, one of the things that first came to my mind was like, oh, I have a database maybe. And there's an index that I use to query over that database. And some of that is a little bit like fuzzy magic to me in terms of how that actually works at the lower level in a database. But what is this idea of an index or indexing in the context of Llama Index or in the context of data augmentation for large language models?
It's kind of funny. I think when we first started the name, it was a bit more of like a casual naming convention. You know, it used to be called GPT Index, and I kind of made up the name because it sounded roughly relevant to what I was building at the time. But I think over time, especially as it's morphed into more of a project that people are actually using, this concept of an index has become a bit more concrete, and so I can articulate that a bit better. The idea of Llama Index is, you know, just to step back and talk about the overall purpose of the project, is to make it really easy and powerful and fast and cheap to connect your language models with your own private data. And we have a few constructs to do so within Llama Index. And so...
part of the way you can think about llama index is how can we build some sort of stateful service around your private data around something that at the moment is somewhat stateless. Like the language model call is a stateless service, right? Because you feed in some input and you get back some output. So how can we wrap that in a stateful service around your own data sources so that, you know, if you want to ask a question or tell the element to do something, it can reference that state that you have stored. And so if you think about any sort of like data system, there is the raw data that's stored somewhere in some storage system. There might be like indexes or views, like similar to like a database analogy where you can kind of look at the data in different ways. And I can talk a little bit about how that works.
And then there's usually some sort of like query interface, right, that you can actually query and retrieve the data. So if you look at like a SQL database, right, you have, you know, the raw data stored and some sort of tables. You can define different indexes over different columns. And then the query interface is like a SQL interface. You run SQL and then it will be able to execute the query against your database. And there's a lot of like kind of roughly similar concepts that apply to thinking about the llama index itself as this tool set. Because if we're going to build this like stable service, right, on top of that, it can integrate with large language models. By the way, to clarify, we're not like really solving the storage part, right, where we integrate with like a ton of different backup storage providers. We integrate with like other databases too.
But if you even think about us as like some sort of data interface or orchestration, you know, there's a raw data which needs to be stored somewhere. And so if you have like a bunch of text documents, you need to store that in like a vector database or MagoDB or S3, all those types of things. And then you can define these different indexes on top of this data. And the way we think about indexes is how do we structure your data in the right way so that you can retrieve it later for use without lens. And so then I can talk a little bit how this works, but the set of like indexes that you can define is actually pretty interesting. Basically, the set of data structures that offers like a view of your data in different ways. And then you wrap that in this overall query interface that can, you know, use these indexes on top of your data.
to do retrieval and lm synthesis and give you back a final answer and so i would look at this in terms of like the components of the overall system there's just like if you're building this like stateful service there's these three components uh how do you you know adjust the store of the raw data index it and then query it So I want to actually pull you back for just a moment as we're kind of learning this. And if you're an app developer and you're interested in creating a stateful service and you've started kind of going down the path about like, well, there's kind of the old school way of going and doing a SQL query and all that. And now we're using LLM models and adding our data to it. I know that we've kind of gone beyond that just a little bit, but if you can back up and talk a little bit about what are you getting? If you're the app developer and you're listening to this and you're trying to understand, like, why would I go down that path? I sense that there's value there, but we haven't talked about it versus a robust set of SQL queries on your own data. Why would you bring in that large language model in the beginning? What does it bring?
a bear that's worth all of that effort? Could you talk a little bit about that baseline value add to it? Yeah, that's a really good question. And I think I might have jumped the gun a little bit, so I appreciate you bringing me back. No worries. It's because you're excited, as are we. But I also want to make sure that people listening have a chance to truly understand it at the same way that you are. Definitely. I think one thing about language models that's very powerful is their ability to just comprehend unstructured text and also natural language. And so this matters in both ways in terms of how you can store the data as well as query the data. Because now, you know, let's say you're the end user, you can just type in a natural language like English question, right, ideally into this interface.
get back a response and so the setup is way easier than having to learn sql over some source of data or you know having to even code up like this very complex like pipeline to try to you know like parse the data in different ways um because you could treat the language model itself as a black box feed us something get something out right and so i think that by itself is a very very powerful tool i think these days people are trying to figure out what you can do with that tool so another kind of like illustrative example of like the power of language models using this as like intelligent natural language interface is you actually don't have to do a ton of data parsing when you actually feed in the data so for instance let's say you have a pdf document or, like, any sort of, like, Microsoft Word document or even an HTML web page, just copy and paste that entire thing, right? You know, just extract the text from it, dump it into the input prompt, and then just, like, tell the LLM, hey, here's just this, like, giant blob of text I copied over. Now, given this text, can you please answer this following question? And the crazy thing is the language model can actually do that, assuming, you know, it fits within the prompt space. And that's also very powerful because this kind of affects the way you do, like, ETL and data pipeline, right? In the traditional sense, if you had a bunch of this, like, unstructured text, you'd have to spend either manual effort or write a complicated program to pull out the relevant bits from this text and parse it into some table.
store it and then you know run sql or some other like query over this text whereas here you know with power of language models you can store this text in a bit more of like a messy unstructured format as like raw natural language and then still figure out a way to pull out this unstructured text, just dump it into the input prompt and ask the question over it. Is it conceivable with what you're saying, if I'm thinking as an app developer about diving into this, that I'm hearing you say you're going to do this, which is an additional thing to learn and be able to go, you know, it's an additional skill set that you're adding on. But I also hear you talking about other things that I used to have to do that maybe I don't have to do anymore. And to some degree, is it realistic to say from an effort standpoint, it becomes a wash once you have the skills a little bit, or maybe even you're gaining more power and doing less work along the way to do it so that it's kind of like, of course you would do it going forward. Is that a fair way of thinking about it?
Yeah, so it's an interesting way of thinking about it because I think the high level question is just like, you know, what parts have become easier and what parts have gotten harder once you have this language model technology? Because on one hand, things have gotten a bit easier and powerful to build these expressive like question answering systems with less effort. You know, you take in this giant blob of unstructured text, you know, figure out how to store it, you feed it into the language model. And then all of a sudden you can ask these questions over these like files that you couldn't really do before with more kind of like traditional AI technologies or just like manual programming. That said, I think this new paradigm kind of involves its own set of challenges that I'm happy to talk about.
I think there's a lot of, like, stacks emerging about how to make the best use of language models on top of your data. And there's some very basic stuff that's happening these days, but there's also kind of, like, more advanced stuff that we're working on. And I do think it's very interesting to think about what are the technological challenges that are preventing us from unlocking the full capabilities of language models. Because, again, with a very basic stack, and, again, like, you can see this if you just, like, play around ChatDBT, you can already get a ton of value from your data by just, like, doing some very basic processing on top of it, and you can start asking questions that you couldn't really ask before. But, you know, with some more advanced capabilities and some, once you're solving some more interesting technical problems...
What are kind of like the additional queries that can ask on top of your data that you also couldn't do before? before we jump into so i want to kind of dive into the weeds about like the two things that you talked about like how do i index my data how do i query my data all the goodness around that in llama index before we do that maybe just as also like a to set the stage for some people that are coming into this and maybe parsing some of the jargon that's thrown around so one of the other things that people are really kind of diving into is thinking about like how do i engineer my prompts how do i chain prompts together and all of that sort of thing and could you highlight because my at least the way i would phrase it like those two things are complementary with the things that you're doing with llama index but could you kind of help people understand like how do those pieces fit together into terms of like architecting one of these systems. I guess in the end, LM application development to put it in a very oversimplified view is just some fancy form of prompt engineering and prompt chaining, right? It's actually not super different with how we're thinking about building this interface with data. And so just as a very basic example, you know, if you're kind of coming into the space fresh, like a very basic prompt that you could put into a language model is something like the following. Here is my question, right? And then you put the question here and then you put in, here's some context. And in this context variable, you just dump all the context that could be relevant to the question, right? You copy and paste a blog post, you copy and paste like API documentation, just copy and paste it into the input prompt space. And then now at the bottom say,
given this context, give me the answers to this question. And you send it to a language model and then you get back an answer. So that's like the most basic question answering prompt that you could use to kind of perform some sort of like question answering over your data. It really is just prompting, right? Because you're putting stuff into the prompts. You have this overall prompt template and you have variables that you want to fill in. I think one interesting kind of like challenge that arises is how can you feed in context that exceeds the prompt window? Because for GPT-3, it's 4,000 tokens. For Anthropica, I guess it's like 100,000 tokens. But if you look at like an Uber SEC 10K filing, it's like 160,000 tokens, right? So if you want to ask a question, like what's the summary of this entire document?
Or like, what are the risk factors in this very specific section of the document? How do you feed that entire thing in so that you can basically answer the following question? And I think that's where things get a little bit more interesting because you can basically do one or more of the following things. One is you could have some external model, like something separate from the language model prompt, that's actually doing retrieval over your data to figure out what exactly is the best context to actually fit within this prompt space. Two is you can do some sort of synthesis strategies to synthesize an answer over long context, even if that context doesn't actually fit into the prompt. For instance, you could chain repeated LN calls over sequential chunks of data.
and then combine the answers together to actually give you back a final answer. That's one example. In the end, all this architecture is just kind of designed around being able to feed in some input to the LLM and get back some output. And the core of that really is a prompting, right? And so part of this is just like developing an overall system around the prompting.
Well, Jerry, you had mentioned kind of these three levels of integrating external data into your LLM application. There's sort of data ingestion and there's like indexing and query. I'm assuming data ingestion has to do with like, oh, I'm going to connect to the Google Docs API and like pull the data over and then indexing and query build on top of that. But before we dive into those second two phases, which is where I think a lot of the cool stuff that you're doing is found, what should we know about sort of data and the data ingestion layer in terms of relevance to how Llama Index builds on that and other things? The data ingestion side is just like the entry point to building a language model application on top of your own data. I think LMs are cool. I want to use it on top of some existing services. What are those services and how can I load in data from those services? One component of Lama Index is this kind of like community-driven hub of data loaders called Lama Hub, where we just offer a variety of different data connectors to a lot of different services. I think we have over like 90-something different data connectors now. And these include like connections to file formats. So for instance, like PDF files, HTML files, like PowerPoints, images even. They can include connectors to APIs like Notion, Slack, Discord, Salesforce. Actually, sorry, we actually don't have Salesforce yet. That's something that we want.
It would be very useful. If you're interested in contributing a Salesforce letter, please, I would love that. And then the next part is just like being able to connect to kind of like different sorts of multiple formats, like audio images, which I think I've already mentioned. So the idea here is you have all this data. It's stored in some format. It's unstructured. It could be text or it could even be like images or some other format. How do you just like load in this data in a pretty simple manner and just wrap it with some overall document abstraction? So there's not a ton of tech going on here. And the reason it's more just like a convenience utility for developers to just like easily load in a bunch of data. And again, going back to the earlier point, the reason there's not too much tech is.
LLMs are very good at reasoning over unstructured information, so you actually don't need to do like a ton of parsing on top of this data that you load to basically get some decent results from the language model. And so once you actually load in this data in a lightweight container, you can then use it for some of the downstream tasks like indexing and querying. Awesome. Yeah. And I see it like this is I think where things get super interesting, like I mentioned. So in Lama Index, I'm in the docs right now, like you mentioned list and table and tree and vector store and structured store and knowledge graph and empty indices. Could you describe like generally how to think about an index within Lama Index? And then like, why are there multiple of these? And what categories generally do they fit in? One way of thinking about this is just taking a step back at a high level. What exactly does the data pipeline look like if you're building an LLM application? So we started with data ingestion where you load in a document from some data source like a PDF document or API.
And now you have this unstructured document. The next step, typically, is you want to chunk up the text into text chunks. So let's say, naively, let's say you have just a giant blob of text from a PDF. You can split it, you know, every 4,000 words or so, or every 500 words, into some set of text chunks. This just allows you to store this text in units that are easier to feed it into the language model. And a lot of this is a function of the fact that the language model itself has limited prompt space, right? So you want to be able to chunk up a longer document into a set of smaller chunks. Now you have these chunks. They're stored somewhere. They could be stored, for instance, within a vector database. For instance, like a pinecone, VVA, Chroma. They could also be stored, for instance, like a...
of a document store, like MongoDB, or a certain file system on your local disk. Now they're stored. The next part is, how do you actually want to define some sort of structure over this data? A basic way of defining some sort of structure over this data, and this is where we got into indices, is just adding and embedding to each chunk. And so if you have a set of text, how do you define an embedding for each set of text? And that in itself could be treated as an index, right? An index is just like a lightweight view over your data. The vector index is just adding and embedding to each piece of text. There's other sorts of indexes that you could define to define this view over your data.
there's a keyword table that we have where you just have a mapping from keywords to the underlying text. You could have like a flat list where you just like basically store a subset of node IDs as like its own index. Before I get into the technicals of like the indexes and you know, like what they actually do, one thing to maybe think about is just like, what are the end questions that you want to ask? And what are some of the use cases that you'd want to solve? Before you dive into that, I was going to ask you really quick, could you define what an embedding is for those people who are learning large language models at this point, just so they'll understand what it is when you say you're defining that as the index? Embeddings is a part of kind of a very common stack emerging these days around this LLM data system that's emerging.
And so an embedding is just a vector of numbers, usually like floating point numbers. You could have like a hundred of them, a thousand of them. It depends on the specific embedding model. And the way an embedding works, it's just think about this list of numbers as a condensed representation of the piece of content that you have. You know, if you can somehow in a very abstract manner, take in some piece of context, let's say this paragraph is about the biography of like a famous singer, right? And then you get an embedding from that. It's a string of numbers. The embedding has certain properties such that this string of numbers is closer to other numbers that are semantically about like similar content and farther away from.
other strings and numbers representing texts that are farther away in terms of semantic content. So for instance, like if you look at the biography of a singer, it's going to be pretty close to a biography of another singer versus if it's about, I don't know, like the American Revolution or something like that. And that will probably be a little bit further away. And so it's a way of like condensing a piece of text into some vector numbers that has some mathematical properties where you can measure similarity between, you know, different pieces of content. Maybe this is another point of distinction. And I get all these questions very often. So I think it's useful to discuss them on the show. Like last week at ODSC, I got a lot of these sorts of questions. So like we're talking about bringing in data.
creating an index to access that data, that index might involve like a vector store or embeddings, but Llama Index is sort of not a vector store. Like it's cool to be a vector database company right now, but Llama Index is something different. And like, again, these are two things that are complementary, I think. Could you draw out that distinction a little bit just to help people kind of formulate those compartments in their mind? I think these days there's a lot of vector store providers and they handle a lot of the underlying storage components. And so if you look at like a Pinecone or Weavey, they're actually dealing with the storage of these unstructured documents.
one thing that we want to do is leverage these existing kind of like storage systems and expose query interfaces that i guess a broader range of query interfaces beyond the ones that are just like directly offered by a vector store and so for instance a vector store will offer a query interface where you can typically query you know the set of documents with an embedding plus like a set of metadata filters plus maybe some additional parameters but we're really trying to build this like broader set of like abstractions and tools through our indices our query interfaces plus like other abstractions we have under the hood to basically perform like more interesting and advanced operations and manage the interaction between your language model and your data and almost like be a data orchestrator on top of your existing storage solutions. And so we do see ourselves as separate because we're not trying to build the underlying storage solutions. We're more trying to provide a lot of this like advanced query interface capability to the end user using the power of language models on top of your data.
I think we got a little bit off track a little bit, but I think it was good. So kind of circling back to the indices that are available in Llama Index, and you've talked about this pipeline of processing and potentially one index being a vector store. And maybe listeners are a little bit more familiar with vector search or semantic search or that sort of thing with everything that's going on. But you have much more than that, like these other patterns and these other indices that enable other patterns. Could you describe some of those alternatives or additions to vector store index and when and how they might?
come into play? Yeah, that's a good question. And maybe just to kind of like frame this with a bit of context, I think it's useful to think about certain use cases for each index. So the thing about a vector index or being able to use a vector store is that they're typically well suited for applications where you want to ask kind of like fact-based questions. And so if you want to ask a question about like specific facts in your knowledge corpus, using a vector store tends to be pretty effective. For instance, let's say your question is, let's say your knowledge corpus is about like American history or something, right? And your question is, hey, like what happened, you know, in the year of 1780?
That type of question tends to lend well to using a vector store because the way the overall system works is you would take this query, you would generate an embedding for the query, you would first do retrieval from this vector store in order to fetch back the most relevant chunks to the query, and then you would put this into the input prompt of the language model. And so the set of retrieve items that you would get would be those that are most semantically similar to your query through embedding distance, right? So again, like going back to embeddings, like the closer different embeddings are between your query and your context, the more relevant that context is, and the farther apart it is, then the less relevant. And so you get back the most relevant context or query, feed it to a language model, get back an answer. There are other settings where standard top K embedding-based lookup, and I can dive into this in as much technical depth as you guys would want to, but there's a setting where standard top K embedding-based retrieval doesn't work well, right? And one example where it doesn't typically work well, and this is a very basic example, is if you just want to get a summary of an entire document or an entire set of documents. Let's say, you know, instead of asking a question about a specific fact, like what happened in 1776, maybe you just want to ask the language model, can you just give me an entire summary of American history in the 1800s? That type of question tends to not lend well to embedding-based lookup.
because you typically fix like a top K value when you do embedding-based lookup, and you would get back very specific context. But sometimes you really want the language model to go through all the different contexts within your data. So a vector index storing it with embeddings would create a query interface where you can only fetch like the K most relevant nodes. If you store it, for instance, with like a list index, you could store the items in a way such that it's just like a flat list, right? So when you like query this list index, you actually get back all the relevant items within this list, and then you'd feed it to our synthesis module to synthesize the final answer. So the way you do retrieval over different indices actually depends on the nature of these indices. Another just like a very basic example is that
We also have like a keyword table index where you can kind of like look up specific items by keywords, right, instead of through like embedding-based distance. Keywords, for instance, are typically good for stuff that requires high precision and a little bit like lower recall. So you really want to fetch like specific items that match exactly to the keywords. This has the advantage of actually allowing you to retrieve a bit more precise context than something that like vector-based embedding lookup doesn't. The way I think about this is like a lot of what Lana Index wants to provide is this overall query interface of your data. Given any class of queries that you might want to ask, whether it's like a fact-based question, whether it's a summary question,
or whether it's, you know, some more interesting questions. We want to provide the tool set so that you can answer those questions. And indices, like, defining, like, the right structure of your data is just one step of this overall process in helping us achieve this vision of, like, a very, like, journalizable query interface over your data. Some examples of, like, different types of queries that we support. There's the fact-based question lookup, which is, like, semantic search using vector embeddings. You can ask, like, summarization questions through, you know, using our list index. You could actually run, like, a structured query. So you could, if you have a SQL database, you could actually run, like, structured analytics over your database and do, like, Texas SQL. You can do, like, compare-contrast type queries where you can actually...
look at different documents within your collection and then like look at the differences between them you could even look at like temporal queries where you can reason about like time and then go forwards and backwards and basically kind of like say hey this event actually happened after this event here's the right answer to this question that you're asking about and so a lot of what uh llama index does provide is a set of like tools the indices the data adjusters a query interface to solve like any of these queries that you might want to answer So Jerry, you really got me thinking a lot about this. The possibilities of the query schemes is pretty darn cool. You know, we kind of started with ingest and moved into kind of indexing and now we're talking about queries. Could you kind of give me an example with the tool at a little bit more of a practical level? Because you kind of hit the concepts about like what the possibilities are. But as someone who hasn't used the tool myself, I'm trying to get a sense of what that workflow is like. Pick what would probably be like a really common query scheme that you're doing and dive into that just a little bit to give us a sense of a hands-on, practical, you know, fingers on keyboard sense of it. Because I'm trying to get a sense of where I'm going to go for playing after we get done with the episode. So I want to try.
100%. I think one thing that has popped up pretty extensively after talking to a variety of different users is actually financial analysis. I think looking at SCC 10K tends to be a pretty popular example. If you look at the Anthropic Cloud example, they also use SCC 10Ks. And I guess the reason it's popular is one, there's just like a ton of text and so it's just very hard to parse if you read as a human. Two, it's like a useful thing for people in like financial institutions like consulting because you want to like compare and contrast the performance of like different businesses, you know, and look at the performance across years. Believe it or not, I actually read 10Ks a lot and that would be a really useful example for me. Believe it or not, I'm not kidding you.
As a result, we've actually been playing around with a decent amount too. Some of the cool things that we're showing that LamaIndex can do on top of your 10Ks is, for instance, let's say you have two companies, let's say Uber and Lyft for the year 2021. You can actually ask a question like, can you actually compare and contrast the risk factors for Uber and Lyft or their quarterly earnings across these two documents? One is the Uber 10K, one is the Lyft 10K. This is actually an example where if you do just top K embedding-based lookup, the query fails because if you ask the question, you know, comparing and contrast Uber and Lyft and don't do anything to it, and let's say, you know, your Uber and Lyft documents are just in some one vector index, you...
don't really have a guarantee you're going to fetch the relevant context to this question to be able to answer this thoroughly, right? And then the model might hallucinate, you'll get back the wrong answer, and then, you know, it's just not a good experience. I think what you typically want to do is have some sort of like nicer abstraction layer on top of this query that can actually kind of map that query to some plan that would roughly be like how a human would think about like executing or answering this question. Let's say you want to compare and contrast the financial performance of Uber and Lyft in the year 2021. Well, first, okay, what was the financial performance of Uber in 2021? What was the financial performance of Lyft?
you'll break it down into those two questions. And then for each question, use it to kind of like look over your respective index. Let's say you have an index corresponding to Uber and an index corresponding to Lyft. Get back the answer, right? Get back the actual kind of like revenue, for instance, for Uber and Lyft. And then like synthesize both of them at the top level again. Be able to like pull in the individual components you extracted from each document and then synthesize the final response that's able to compare the two. So that's like an example of something that we can actually do pretty well with while I'm indexing. We have like a variety of tool sets for allowing me to do that.
and that's an example of a query that's kind of more advanced because it requires comparisons beyond just like kind of asking stuff over a single document another example just to kind of like take the 10k analogy further is let's say you have the yearly reports of the same company across different years let's say from like 2018 to 2022 you can ask a question like did revenue like decline uh go up or down in the last like three years and then you can actually do a very similar process where given the query interface that we provide break this question down into some questions over each year pull out the revenue and then basically at the end do a comparison step to see whether or not it increased or declined. Just as an aside to any listeners out there wondering why on earth somebody would read 10Ks, especially considering that our audience is focused on data and such as like, if you want to learn about another technology company and really understand what it does and be able to compare it, this is an example where you can gain tremendous intelligence on another company with publicly available information. And by comparing multi-year 10Ks, like you just said, you'll learn way more about that company than its own employees know about it. So anyway, just thought I'd mention that as an aside. Yeah, I look forward to hearing your success with speeding up your workflows around reading the 10Ks, Chris, with Lana Index.
I'm excited about this. This is going to save me a lot of time. Jerry, one of the things that we talked a little bit about in one of our previous conversations, which I know you've also thought very deeply about and even have a portion of the docs and functionality and llama index devoted to is evaluation, like query response evaluation. How do I know my large language model barfed up an answer based on some query? And I pulled in some external data and I inserted some context and maybe I strung a few things together.
How am I to evaluate the output of that? Could you give us maybe a high level from your perspective, how you think about this evaluation problem, and then maybe go into a little bit of some of the things that you're exploring in that space? Yeah, totally. Just to preface this, we are super interested in evaluation, or are more tailored towards this interface of your data with LMs. I can dive into that a bit more. And we have some initial evaluation capabilities, but we're super community-oriented. We'd love to just kind of chat with them. There's a lot of different tool sets out there that allow you to do different types of evals of your data and building nice interfaces for doing so. And so I think this is an area of active exploration and interest for us as well.
Just kind of thinking about this a little bit more deeply about, evaluation is very interesting because there is the evaluation of each language model call itself, and then there is the evaluation of the overall system. And so diving into this bit more, like at a very basic level, if you have a language model, you have an input, and you get back some output, you can try to validate whether or not that output is correct, right? Given a single language model call. Did the model actually give you the correct answer? Given the input, did it spit out garbage? Did it like hallucinate? That type of thing. The interesting thing about a lot of systems that are emerging these days is that they're really like systems around like a repeated sequence of language model calls. And this applies whether or not you're dealing with like...
more agent-based framework which you know you ask a question that can like just repeatedly kind of like do react chain of thought prompting or like be able to pick a tool but the end result is it's able to give you back a response another example this is like auto gpt where you just let it run for like five minutes and just like keeps on doing stuff like over and over again that's how it gives you back something or you know even in the case of retrieval augmented generation there's just like a fancy name for roughly like what we're doing with wama index which is like a query interface over your data like even within our system there could be a sequence of repeated lm calls but the end result is that you send in some input into the system and you get back some output you know given this high level system how do you evaluate the input and output properly i think in traditional machine learning typically what you want to have is you want to have like ground truth labels for every input that you send in. So if you like, for instance, ask a question, you want to know the ground truth answer and you want to compare the predicted answer to the ground truth answer and see how well the predicted answer matches up. This is still like something that people are exploring these days, even in the space of JavaScript AI and LLMs, you have ground truth like text and then you have predicted text and you want some way of scoring how close this predicted text is to ground truth text. I think the core set of eval modules that we have within Lama Index actually are ground truth free or label free. And that part in itself is actually very interesting.
because you have this input, you ask a question, you get back this predicted response. You also get back the retrieve like sources, like the documents themselves. What we found is that you can actually make another LLM call to just like compare the sources against the response and then also compare the query against the sources and the response to see how well all two or three of these components match up. And this doesn't require you to actually specify what the ground truth answer is. You just look at the predicted answer, see if it matches up to the query or the context in a separate LLM call. And it's interesting because one,
It makes use of LLM-based evaluation, which is kind of like an interesting way to think about it, basically using the language model to evaluate itself, right? I'm sure there's like downsides, which we can get into, but you know, a lot of people are doing it these days too. And then the second part is it doesn't require any ground truth because you're using the language model to evaluate the capabilities of its own answer plus context. You don't actually need to, as a human, feed on the actual answer. And the benefit of this is that it just saves a bunch of time and costs. You don't actually need to like label your entire data set to run evals. I still think this overall like space is probably relatively early. I think there's still some big questions around like latency and cost if you're trying to really do LLM-based evals more fully.
like using the LLM to evaluate a large data set takes a lot of time and costs a lot of money and so this is generally kind of like an area that we're still kind of like actively thinking about Yeah, that's awesome. As we kind of like get near to the end here, I know things are like progressing so quickly. I can't keep up with all of your tweet threads about like awesome new stuff that's happening in Llama Index, but I know there's a lot. As you look to kind of the next year and like where your mind is at, what you really want to dive into, and also what's really exciting to you about the field. There's a lot of people excited about a lot of different things, but from your perspective, having been in the trenches, building large language model applications, interacting with users of Llama Index, what is it that really excites you moving into this next year in terms of the possibilities, the real practical possibilities?
on the horizon and how kind of our development workflows will be changing in the near future. Yeah, totally. I think there's a few related components to this that I'm both excited for, as well as like, you know, the challenges that we're going to solve. Probably the first component is just being able to build this automated query interface over your data. You know, if you're looking at all the query use cases that we solved, one of the key questions that we keep going back to is here's a new use case on top of this data. And here's a new question that you'd want to ask. How do we like figure out how to best like fulfill a query request? Right. And I think especially as your data sources become more complicated, then it's just like, how do you think about like
how to index and structure the data properly, how do you think about interesting automated interactions that happen at the query layer between the language model and your data, how do we, like, basically just, like, make sure we solve this request. And then the second component to this is we want to make sure that we build this interface that can handle any type of query you throw at it. Now, how do we do this in a way that is, like, cheap, fast, and easy to use? For a lot of users, once they move on beyond the initial prototyping phase, they think about starting to minimize, like, class and latency, and then think about how do you, like, pick the best models for the job, right? There's, like, the OpenAI API, which works well, generally, you know, they're probably the best model out there, but it can also be quite slow. And then there's also these, like, open-source LMs popping up.
probably, like, a few new ones every week, and then how do users make the best decisions whether to use that over their data. And then I think the next part to this is, you know, a lot of LLM development is moving in this overall trend of, like, automated reasoning, right? If you look at, like, agents and tools and AutoGPT and all this stuff, it's just, like, how do you make, like, automated decisions over your data? And then I think, as a consequence, there's always going to be this trade-off between how few constraints can we give it and how many, like, like, should we give it more constraints or fewer constraints, right? Because the fewer constraints you give to something like this, it has more flexibility to potentially do way more things, but then also just, like, error, right? Like, it'll just, like, make mistakes, and then there's no, really no way to correct it easily.
and you can't really trust the decisions. Whereas if you kind of like constrain the outputs of these like automated decision makers or agents, then you can potentially get more interpretable outputs, maybe at the cost of like a little bit of flexibility in terms of functionality. And I think we've been thinking a lot about that with respect to like the data retrieval and synthesis space too, right? Like how can we give you back results that are expressive, but also, you know, like perform well and aren't going to make mistakes like a ton of time. Awesome. Yeah, well, I'm really happy that we had a chance to talk through all the great Llama Index things. And make sure if you're not following Jerry on Twitter, find him there. He posts a lot of great stuff that they're working on. And of course, you can find Llama Index. If you just search for Llama Index, there's like a great set of docs and all those things. We'll make sure we include those links in our show notes for people to find out about that and get links to their docs and their blog and all the good things. So check out Llama Index. And thank you so much for joining us, Jerry. It's been awesome.
Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already. And if you're a long-time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Change Talk podcasts. Check out what they're up to at Fastly.com and Fly.io. And to our Beat Freakin' Residence Breakmaster Cylinder for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.