 Hey Wiz, what do you think are the best adventure films out there? That's a tough question to answer, Greg. Yeah, yeah, yeah. What if I gave you some data? What if I said, well, between Lord of the Rings, The Hobbit, or Dune, maybe Harry Potter? How would you pick the best? it or Dune, maybe Harry Potter? How would you pick the best? Well, I'd probably think about maybe the ratings and, you know, who is in it and, you know, what happens in the movies, like how well I enjoy them. You know, it's still a tough question, though. So you do like some reasoning to try to figure out, well, maybe there's some sort of critic scores. Maybe those are quantitative. Maybe you might think about the plot and kind of the the details of the story, what people thought about that. There's essentially a lot of data you might try to consume to really answer this question best. Isn't that right? That's absolutely true. Yeah. Yeah. Yeah. So you kind of have to reason through different types of data, almost doing what we might call agentic behavior. Isn't that so? That's right, Greg. Yeah. Yeah. Yeah. Yeah. Well, I think you think we can build an AI to answer this question of what is the best of the best today? I think we can definitely try. Yeah. Yeah. All right. Well, that's what we're going to do. We're going to see you back when we're ready to start building. Welcome everybody today to Data Agents with Llama Index. My name's Greg, and that was Chris, aka Dr. Greg in the Wiz. We're from AI Makerspace. Thanks for taking the time to join us today. Today we're going to try to build an intelligent AI system that uses agentic reasoning behavior to discern. Well, what is the best couple of movies out there? You've all maybe recently seen the Dune series, and we're curious to see how it stacks up against some of the classics. And we've got the perfect tool in store today to be able to look at this. and we're curious to see how it stacks up against some of the classics. And we've got the perfect tool in store today to be able to look at this. We're going to use data agents with Lama Index. By the end of today, you'll understand the Lama Index core constructs of the framework and how they come together into data agents. We're going to do a deep dive on the technical architecture. We're not going to get too high level pie in the sky with the concepts today. We're going to spend a solid hour in Llama Index building this thing out. We hope you enjoy it. Let's get into it. If you have questions along the way, please drop them in the Slido link or directly in the YouTube live chat, and we will do our best to get to all of them today. All right. So as we align our AIM for the day, we really want to understand these core constructs. That's what we're going to build with. That's what we're going to build on. Then we're going to go ahead and build a complex RAG application that uses agentic reasoning. We're going to build indexes for both qualitative and quantitative data, meaning movie reviews and context about those movies. We're going to leverage metadata filtering to be able to move between these two indexes, and that is going to be the agentic reasoning piece of our application. And hopefully we can get an answer. What do you think the best adventure movies are out there? Maybe you can throw it in the chat. So today we're going to dive into the core constructs and then articulate how to build out the semantic and SQL-based separate pipelines that we can combine and navigate using metadata. All right, so LAM index, just super high level. What are we talking about? This is the data framework for LLM applications that benefit from context augmentation. This is the keyword they are using today. Now, don't get confused. Context augmentation is nothing more than retrieval augmented generation or augmenting what's in the context window. That's called in-context learning. Now, when we do fact checking, when we try to make sure we're not hallucinating, it's all about reference material. That's sure we're not hallucinating, it's all about reference material, that's what we're augmenting with here. So, you know, again, what's the best? We're going to need some fact checkable information to back up our response by the time we're done with this thing, right? And so we want to get good answers that are rooted in reality, rooted in facts. Overall, the RAG process is to ask a question, feed that question into an embedding model to get a space that is to get a representation in embedding space of our question. Then we're going to look for similar information within our vector store. We're going to set up a prompt template that's ready to receive reference material. And then we're going to inject that reference material back in natural language before we put everything into the LLM. Now, this process is the retrieval process. And this process really does lead the way because retrieval done well makes generation go better. It improves our ability to leverage context. And again, this RAG process is mostly completed at this point, but we haven't done the G, we, this is the next step towards production ready data, context, augmented LLM applications, production ready. And so this is very cool because they're doing a lot of cool things. But one of the things that they did is they said, well, the main abstractions are going to be in core. That's really what we want to focus on today. And then there are a lot of other things that are going on, a lot of things you can interface with. We'll see a few of those today. But we want to, again, focus on the core so we can really build up this idea of data agents. Now, when they released Lama Index v0.10, they sort of showed this image, and I really like it. We're going to focus on the core package today, but we're also going to see that we're leveraging a number of other pieces. We're leveraging the agents, we're leveraging LLMs and embeddings. Of course, Lama Index doesn't have LLMs or embeddings in its core because it doesn't create LLM and embedding models. It's a framework that allows us to leverage those LLMs and embedding models. We can also leverage indices from other companies. For example, we'll leverage Quadrant today. And VectorStore is a particular type of indice. That is the type we're going to use today, is the Vector Store. And then, of course, we can have access to tools. We'll see one tool that we'll use today to help us actually build that metadata filtering capability, namely the OpenAI functions API, the function calling API. But before all that, let's focus on the core of Lama Index. Let's focus on the core constructs. Let's walk through this step by step. The core constructs are associated with steps of the process, the same process we just diagrammed out. We need to load our data. we need to index all of the data, we need to store it, and then we can ask questions against it. The structures within LLAMA Index Core we need to be able to leverage our nodes, embeddings, indices, retrievers, query engines, and data agents. Let's walk through step by step. First, we can think about loading. And we're going to load with nodes. Loading with some noting. And loading is nothing crazy it's just ingesting your data but the actual name for the chunks that we're going to use in llama index are nodes so these are some nodes pictured within a database slow vector store here database. It's a little vector store here. So nodes are the way that data is ingested. It's the how of data ingestion. So nodes are very, very important in Lama Index and everything builds on nodes. They're nothing crazy. It's just a chunk, but it does have metadata associated with it. have metadata associated with it and metadata allows us to be more intelligent down the line because it's data about our data it's up a level of abstraction we can use that later as we layer in more and more complexity the way we get nodes is we parse up our data. We use generally a node parser. This does the chunking of our documents. Okay, not very complex here. Once we get data loaded in, we're ready to do some indexing and we need to leverage embeddings to do this. Now, the process of indexing is simply the idea of structuring our data so that down the line, we're going to be able to easily do retrieval. Okay? Remember, retrieval augmented generation. So indexing is very key because the better we structure, the more easy it is to retrieve. And the way this looks in Lama Index is we, of course, split our documents into chunks, nodes with our node parser, and then we're going to create embeddings for each chunk. Now, each chunk of document is a node. Now, that node might have metadata associated with it as well. Often it does. We're going to pass our chunks through an embedding model to get a representation of that language in embedding space. And all of this is going to allow us to get essentially a list of nodes. This list of nodes is sort of where our indexing process is completed. Well, what's next after we've done the indexing? Well, I mean, we don't want to do the indexing again every single time we have to build something. So after indexing, we want to go and we want to store our indexed nodes within an index. Indices just being the plural of index. Now, again, storing is just about avoiding re-indexing. We could run the entire thing every time, but that would be dumb because we would just be wasting compute to get representations we already had. So this use of embeddings that actually allows us to store our list of nodes goes directly into the vector store. So we take our list of nodes and this vector store is a specific type of index. It's the most common type. It's the type you're probably going to be building with, but there are other types of indices. We don't really need to worry about those today because we're going to use vector stores. But that's really all there is to it. Storing is putting stuff in a database. Once we have that stuff in a database, what's next? Well, we want to be able to wrap our vector store or our database in a way to get stuff out of it, in a way to actually do the retrieval. This is where we need a retriever to allow us to do our querying. And of course, this is just asking questions, but we're going to ask questions and get answers based on the LLMs and data structures that we are using through this process. So the way we did the embeddings, the way we set up the data structure, the way we decided on metadata, all of this matters when it comes to what is retrieved. And so the retriever is simply fetching stuff out of the vector database. It's getting nodes. And the nodes are then often brought back and synthesized using these synthesis modules in Lama Index. Now, it's important to understand, and we've talked about this quite a bit recently over the past few weeks, that when we think about this querying process and we think about retrieval, this is what actually gives us the context augmentation. And so both the chunking, AKA the creation of nodes and the retrieval, AKA the retriever affect this context augmentation. Because we're just wrapping the vector database in a way to find stuff that's similar to our question. Or in the case of a more complex system like we have today, route to the proper database to find stuff that's relevant to our question. And so when we think about kind of the way this is split up, we can think about there is the chunking and then there's the retrieval. And we want to optimize both of these. And it's hard to kind of find perfect ways to visualize this, but we talked last week, if you joined us for advanced retrieval, about the fine line between chunking and retrieval existing at about the vector store. This is where we're going to sort of dive in today. For more on that, check out our advanced retrieval video from last week. But sort of going back to where we started here we're just talking about this process now today we're going to do something a little bit more complex but let's stack this in terms of llama index core constructs let's return well after we've done loading indexing storingying, we're not going to be doing evaluation today. It's time to talk about the query engines. And the query engines are simply generic interfaces that allow you to ask questions over the data. They might be built on many indexes, but they're built on at least one. The index and retriever form the basis for our query engine. It's hard to visualize query engines, to be honest with you. I haven't found a great way to visualize it, but here's another way that we could visualize it, taken directly from a blog from Jerry, CEO of Lama Index, from last year. The query engine really is kind of the heart of what's going on because the query engine as a unit, as a sort of pattern that we can leverage, allows us to build very, very interesting, much more complex applications. Well, that's a good lead in to the idea that we've been trying to get to all along of data agents. And data agents take this idea of a query engine and expand it out. So data agents, I like the Lama Index sort of nomenclature here, they call them LLM powered knowledge workers. They're doing this automated reasoning or perhaps semi-automated reasoning and decision making to decide, okay, I'm going to do some reasoning to decide, do I go hit an API that I have access to, maybe a Gmail API, a Slack API, a SERP API for Google. Maybe I'm hitting DuckDuckGo. Maybe I'm hitting something else. Or maybe I'll just go ahead and tie a couple of query engines together. And now you see the query engine is contained all within a single block here. But there's a lot that goes into a query engine. In fact, there's potentially an entire rag application behind that query engine. And so what we're going to do today is we're going to sort of expand this out. And we're going to use what's called the OpenAI function agent in Lama Index. And you see straight from their docs, agents are, quote, a step beyond our query engines a layer of abstraction up from the query engine and it allows us to do reasoning it allows us access to tools to api's and so this is built on the open AI function API which allows us to sort of intelligently call functions which is great. It's fine tuned for that. And what we're going to do is we're going to leverage the OpenAI Functions API to build this OpenAI Function agent to allow us to look at two different query engines that we build up and we're going to decide which to use based on the metadata associated with those in short we're going to build a functional tool that helps us decide on which query engines we're going to use and And remember, query engines are built on one or more indexes via retrievers. That's why we call this the auto retriever functional tool. It's going to allow us to look at our question, look at the metadata associated with our query engines, select not only the right type of data, but exactly which aspect of that data, and then dig in and try to return the best possible context. So I'm beginning a little abstract. Let's sort of bring it down a little bit closer to earth here. So today's combined build in short is going to be up a layer of abstraction. We're going to take an input. We're going to go to a data agent. It's going to decide, should I use the semantic query engine or should I use the SQL based query engine? Should I use the quantitative data, the SQL based data, or should I use the qualitative data? And we're going to build these indices and retrievers out with different types of data. So our semantic rag pipeline is going to be based on Wikipedia data. We're going to take Wikipedia data on Dune films 1 and 2, Lord of the Rings films 1 and 2, Hobbit films 1 and 2, and Harry Potter films 1 and 2. one and two, Hobbit films, one and two, and Harry Potter films, one and two. Then we're going to create an index. We're going to prep the Wikipedia data. We're going to build a vector store index. We're going to build a vector index retriever and a retriever query engine. Look at that. All the keywords all at once. Then we're going to go and we're going to get some quantitative data. We're going to do this using IMDB ratings for Dune 1 and 2, Lord of the Rings 1 and 2, Hobbit 1 and 2, Harry Potter 1 and 2. And we're going to build out that index. This is pretty cool the way this works because we're going to download the data. It's going to be in CSV format. We're going to create data frames just using pandas. And then we're going to use the SQL database index that's built into LOM index. And then the very powerful and very cool natural language to SQL table query engine, where we can ask questions in natural language and query our SQL data directly. We're going to use OpenAI models to get it done today, including the new embedding model from OpenAI. And that's really all there is to it. So again, let us know. What do you think the best movies are? One and two, because it's not fair to all of them. Only doing one and two. Come out. Let's see what the AI thinks as we head over to Wiz for some agentic rag with Lama Index and data agents. Wiz, over to you, man. Yes. Okay. Hello. Thank you. So let's look at the notebook. Basically, we've got exactly what Greg said. The idea here is that we have this idea of, you know, these data agents, so many different agents, right, that we want to work together in some kind of, you know, in some kind of like way that makes sense. So what are we going to use? Well, first of all, we're going to use a few new terms that you might not have been so familiar with. And so Greg's already done a great job explaining those. They're explained a little bit more in the notebook, but that's fine. So first things first, we're going to need to do, this is a lot of images and a lot of stuff going on here, obviously. So we're not going to, you know, it's a lot simpler in the code than we're seeing it here, right? So let's first move to the main boilerplate. This is just necessary. We just have to do it. There's no way around that. You know, there's not really, unfortunately, there's nothing we can do about that. The next thing we need to do is get our libraries. Now, you'll notice we're getting Lama Index and OpenAI. So Lama Index makes sense. OpenAI makes sense. We're also going to grab Lama Index readers in Wikipedia. We're also going to use the QDRIM vector store today. So the QDRIM vector store is basically a really fantastic VectorStore that's going to help us to understand exactly what we're storing everything in. When it comes to VectorStores, and this is an important point, right? The actual VectorStore that we're using is pretty, it's pretty important, basically. And so the reason that is, is that when we think about storing all this information and thinking about like, you know, oh, well, we want, you know, we want some, you know, interesting information or data or something like that, right? Well, we want to be able to say, can we access that metadata cleanly, right? And so the way that we do that is we can use Qdrant. It's going to basically handle all of that for us. Very convenient. Obviously, it's a, you know, it's a great way to do it. So there you go. We're also going to use SQL alchemy and pandas. Now, SQL alchemy is going to be the thing that we basically just, we want to be able to, you know, have this in-memory vector store. That's pretty important, right? So the way that we're going to get that is by using SQL alchemy and pandas is going to help us to transition from our CSV to our actual SQL database. We also have this optional piece, which is optional. So you don't have to use it if you don't want to. You don't have to think about it if you don't want to. But we do have an optional piece where this is all going to be available to us through the, you know, through the actual 1B integration with Lama Index. So we can see all of our calls that we're making. We can keep a history of all the calls that we're making, which ones are failing, which ones are succeeding. But this is an optional piece that we can ignore if you would like to. So next up, we have our environment variables, and we can also set our 1B callback if we're going to use that. And then next we have our settings. So if you remember kind of, you know, these previous you know, these previous versions that we've used, right? So this is to say, say we're thinking about the idea of, you know, what LLM should we use if we don't specify an LLM, right? Well, the idea is that we actually want to, we would prefer to use that version, you know, of the model when we don't specify a default. So you can think of these as kind of like defaults, basically. So we've got a set up Lama index. We've indicated what things we want to use as our defaults. Now we have to create an actual index. Creating an actual index is not a difficult part. Basically, it is the way we can think about this is as follows. So we have this idea of vectors and vector stores and all this other stuff. vectors and vector stores and all this other stuff. And what we want to do is we want to make sure that we have this set up in a way such that we have the index created cleanly, right? So the way we're going to do this is we're going to use the LUM index core to do that. Now, before we can actually set up an index, we need some data. The data that we're going to use today, of course, is as described. This is just a bunch of different movies. These are the disambiguated Wikipedia titles that the Wikipedia tool succeeds. The big idea here is basically just fetch a lot of Wikipedia articles about these movies. Okay, first of all, that's done. Second of all, right, we need to set this up cleanly. We need to set it up so it works well. For this example, we're going to use the in-memory QGIT client. That in-memory client is very powerful, but it does, you know, come with the fact that it's running all in memory all the time. And it's in this session. So, you know, for production environment, of course, we want to set up a separate QGent client. But for this particular instance today, just so it runs nicely in the Colab, we're going to use the memory QGent client. The next section is just setting a collection. So a collection kind of is like metadata on top of metadata. So not only do we have metadata, but we also have collections. So we can store different kinds of information in different collections, and we can point those collections to different vector stories. Now, we're not going to do that today because we're already doing quite a bit, but it is a thing we can do and it is pretty dope. You'll notice that when we create our collection, we have to be manually setting our vector parameters. This is because Qt is going to expect that all of our vectors have this size. This size is based on the default embedding model that we chose, which if you remember, we set up here in settings, and that's text embedding three small. And so it just happens to have a dimension of 1536. Now that we have our collection and our client, we can create our QGRT vector store. Now our vector store is going to be powered by QGRT, right? So we talked about the benefits of that. Comes with all this metadata stuff out of the box, works really well, scales really well. You'll love to see it. We're then going to create our storage context. Our storage context is what it sounds like. It's context relating to our storage. I know I just said it backwards. But the idea is that we need to have some kind of way to tell Lama Index how it should interact with this vector store. Right? So, when we create our vector store index, we have a really detailed view of this vector store, right? So, that's what that storage content context does. It acts as a nice layer between the vector store index and the thing that's powering our vector store index. We're going to initialize this as empty, you might notice. So there's no docs here, which is, you know, not useful, right? We know we have a lot of docs, but there's none here. And we're going to go through the process of adding nodes manually, because we'd love to add this additional metadata tile, right? So the idea is we have some ingestion pipeline that's going to convert our documents into nodes, and then we're going to insert those nodes into our vector store. And the reason we're doing this manually is because we want to add some metadata. Now, of course, you can add whatever metadata you want, doesn't matter, but you have to follow this pattern if you want to add that additional manual metadata. Now, of course, you can add whatever metadata you want, doesn't matter. But you have to follow this pattern if you want to add that additional manual metadata. You'll also notice that our adjacent pipeline is going to convert our actual, you know, documents, as you can see here, into nodes, which is useful, because we need to convert them into smaller pieces, so that they're more compatible with our RAG pipeline. Okay. So now that we've inserted a bunch of nodes, and you'll see, just to be clear, right, we take each movie and its corresponding, this is the title from the movie, right, in the wiki doc, and we ingest it, we turn it into some nodes, and then for each node, we add this metadata, which is related to the title of the movie. And so now we have an index that has a bunch of nodes and the nodes are associated with their title. So now we can create a simple query engine, right? So we can see our simple rag is just our index.as query engine couldn't be easier. You might think, well, what LLM are we using? We didn't tell it what LLM to use and you're right, but we did. Remember, in our settings, we specified this default LLM, and that default LLM is going to be OpenAI. Pretty cool. All right. So now that we have that, let's check out what the actual prompt is. The prompt is pretty simple. Context information is below. Here's some context. Given the context information, not prior knowledge, answer the query, query, and then it's expecting some answer. It also has a refine prompt that can be useful if we want to leverage it. We won't be specifically leveraging it today, however. So we can ask questions like, who is the evil wizard in the story? And we get Lord Voldemort. Now, already we can kind of tell what a potential issue is right if we're dealing with like uh six different movies that all have evil wizards this is not necessarily the response that we were hoping for or looking for right uh now we could also ask questions like who are the giant beam beings that roam across the world and it can say uh you know stone giants are the beings that roam across the world but again i mean like dune has sand worms we've got ants you know so we're kind of missing this specific information based on the movie that we're providing which is uh you know not necessarily the most useful thing in the world so let's make that better by using the auto retriever function uh functional tool uh got an image here that you see using the auto retriever functional tool. Got an image here that you see in the slides. Basically all this is saying is we're going to do some filtering before we return our list of potential contacts. And this filtering is going to be based on our LLM's decision. So when we go to our actual implementation here, you'll see that we have this vector store info. This is a collection of metadata, which is going to include the title of all of our movies, right? to our actual implementation here, you'll see that we have this vector store info. This is a collection of metadata, which is gonna include the title of all of our movies. And it's basically just going to say, hey, based on the metadata, you're gonna want to have some awareness of what you can choose as part of the metadata. So if we tell it like, hey, you can choose as part of the metadata, right? So if we tell it like, hey, you can choose the title of a movie, but we don't tell it what movies that we have, that's not a very useful tool. So we have to, again, whenever we're doing AI engineer type of work, we're building RAG applications, we're working with Lama Index, we wanna be sure that we're thinking about how is the LLM gonna interpret this information, right? How is it gonna leverage this information to give us a sweet answer? And so in this case, you can see all we're gonna do is let it know the titles of the movies we have. And then in this next piece, I'm not gonna spend too much time on this, but the idea is that this is just describing the function that we want to be able to call that's going to filter our metadata, right? So we have here some query, we have some filter key list and some filter value list. And then we're going to build this auto retrieve function. Now this is a function that lives on our side. And basically all it does is it's going to do retrieval based on a set of filters. You'll see it takes query, filter key list and filter value list. That might seem familiar. We just built that above in Pydanta. It's going to build a set of exact match filters based on the key and values we've provided. In this case, we're keeping it simple. Title is our only metadata field. So if it needs to filter, it's going to use title. And then we're going to use our vector index retriever, similar to what we did before. We're going to set up a query engine about it using a retriever query, or sorry, our retriever query engine. And then we're going to go ahead and query it with our query. And that query is going to take into account, right, this actual metadata filter. So we'll see an example of how this works in just a second. But the idea is all that's left, right, is that we want this to be in some form agentic. And so what we're going to do is we're going to describe our tool and when we should use it, use this tool to look up non-review based information about films. And then we're going to have our auto retrieve tool take our function that we've created, right? And that function that we've created again is this function. And then it's going to describe the tool name. This should be descriptive, right? Because the LLM is going to use this information to determine when to use this tool. We're going to provide our description, which I mean, it makes sense, but it should be descriptive. And then we're going to use our auto retrieve model, which is this bit here. Now, the reason this is important, this auto retrieve model from Pydantic is because we need open AI in this this case to be able to call our function reliably. And this Pydantic model helps us describe in great detail to OpenAI how we call the function, right? You'll see we have query. It should be a string. Here's the description of how that string should look. We have our filter key list. It should be a list of strings. And here's what it should be, right? So this is the idea. We have to describe to OpenAI exactly how to call this function. And that's what we've done using this auto-retrieve model that we built in Pydanta. Very cool. Okay. All that's left to do now is let her rip, right? So we're going to create an OpenAI agent from tools. We're going to pass in this one tool. We're going to have verbose equal true, just so we can see the outputs. And then we're going to say something like who started in the 2001 film, right? Now check this out. When we say who started in the 2001 film, we get calling function, semantic film info with args, query, cast of Dune 2001 film, filter key list, title, filter value list, Dune 2001 film, right? So we're gonna actually filter out every other movie in our vector database. We're only going to care about just Dune 2001 and not at a like implicit level where we're relying on the similarity to only be closely related to what, you know, not like that, but in like a literal sense, we are only going to query documents related to what, you know, not like that, but in like a literal sense, we are only going to query documents related to Dune 2001. And then we get the answer of the people who starred in the movie. Let's go. And then we can say, who are those giant guys from Lord of the Rings that roam around the forest, right? So in this case, again, we're going to use that semantic film info tool. We're going to say characters from lord of the rings the two towers we're going to filter it on the title and the lord of the rings two towers and then we're going to get a bunch of dope characters right one of the characters that we're going to get is uh is the uh tree beard right and so the giant guys from lord of the rings the two towers are likely the ents also known as tree beard Treebeard and the Ents of Fangorn Forest. So our query is kind of bad, right? Like, this is not a good query. But it doesn't matter because because of the filtering, we're able to not just rely on the on our context potentially containing Lord of the Rings or Lord of the Rings adjacent material. We can guarantee it does by using that metadata filtering. Very cool. So next, we're going to do the same thing, but instead of just filtering on metadata and then relying on vector search, we're actually going to do some SQL queries, right? So we're going to import all of our movie review tables as CSVs, and then we're going to load them as data frames, and then we're going to push those into our in-memory SQLite database. Again, if this was a production environment, we would not want this to be an in-memory SQLite database. This would just be some database that we have, right? But you can see we can have a ton of different tables. Like even this toy example here, we have like, you know, a lot of different tables in our database. We're going to see how well it works given that. So now that we've created a database with tons of tables and those tables are all based on those review DFs, right? And the review DF here basically being just this exact, you know, CSV file but represented as a pandas DF and then converted into a SQLite table, we have the ability to load that SQL database as a kind of index, right? And so what this means is that we're going to give it an engine. The engine in this case is the thing that points to our SQLite database. And then we have our list of movies that it's going to care about. So say we had more tables, right? Say we had like movies that were a hundred fold more movies. We had every movie, let's say, right? Well, we could have come up with a list based on just what are adventure movies in this case, and only use those specific tables in this particular index. So again, we have the ability to even above a level still think about metadata. In this case, of course, we just have the movies that we do. And so we're going to use every table easy peasy. Now we're going to create a query engine and it's going to be called the NL SQL table query engine. It might make sense. What this does is it takes a NL or natural language query and it converts it to SQL and runs SQL on that engine that we provided. And then it returns us a result. Very cool. We can wrap that all up in a query engine tool where we describe exactly what's happening. We can indicate what movies we have in this database or what tables we have in this database to be more generic. And then we can say, you know, what they contain info about. And we can also specify when this tool should be used. We can set this up. And then we can do things like what is the average rating of the second Harry Potter movie and get a response like Harry Potter, the Chamber of Secrets is the average rating was about 7.54 out of 10. And then we can ask questions like which movie has better reviews, Lord of the Rings or Dune. And we can get responses that Lord of the Rings has about 9.87 out of 10. And that Dune has 8.34 out of 10. So, you know, Lord of the Rings has better reviews compared to the Dune series. So we're kind of on the way to answering that question Greg had, right? Which is the best adventure movie. So what if we wanted to use both? Well, if we want to use both, all we have to do is use the combined pipeline. You'll notice that we still have our OpenAI agent. All we do is we add an auto-retrieve tool, an SQL tool, and then we can ask questions like, what movie is about a chamber? And what is the average rating of the movie? Right? So in this case, we're going to see a lot of tool usage. We're going to see the semantic film with args. We're going to go ahead and it's going to filter by all of our movies in this case, because this is a potential, you know, it could be any of these movies. And so then it's going to give us the output that Dune sometimes is in, you know, has things about chambers. Okay. It's also going to say, hey, sequel query, you know, take the average rating from Harry Potter, the Chamber of Secrets film, right? And then we're going to finally say to our semantic film info, to think about just the Chamber of Secrets film. And we're going to finally get a response of Harry Potter Chamber of Secrets is about a chamber. The average rating of this movie is 7.54 out of 10 or 7.2 out of 10. So the idea is, again, that we have this ability to make multiple calls. We can also see it working with two different semantic query calls. What worlds do Lord of the Rings and Dune movies take place in? We get the response, Lord of the Rings movies take place in the world of Middle Earth. On the other hand, the Dune movies feature various worlds, including Caladan, Arrakis, Gator Prime, and Arakan. So Arakan, sorry. So very cool, very cool. The idea is we can use multiple tools to figure this out. Finally, we ask the all-important question. Which of the following movie series is considered the best? Harry Potter, Dune, Lord of the Rings, or The Hobbit? Based on your answer on both reviews and non-review information, we can see that we get the final answer of, among the movie series mentioned, Harry Potter is considered one of the best movie series. Additionally, Dune is also considered one of the best movie series, okay? So we kind of get a tie, I suppose. In the LLM's interpretation, sorry to disappoint you, Richard, who said that you will not trust Rag if it pipped anything else than Lord of the Rings. Harry Potter, dude, they're tied for the best movie um in its impression based on the data that it has access to so that's the uh that's the idea that's what we're doing with with all of this uh it it's basically just using the lm right to decide when to use tools to decide which tool is most relevant and use that information to, you know, to make sure that we're using the right tool at the right time. That's that data agent, right? We have many sources of data. We need something that can be smart about picking when it uses what. And, you know, thanks so much for watching the code demo. Now, I'm going to pass it back to Greg. Before I do, Thanks so much for watching the code demo. Now I'm going to pass it back to Greg. Before I do, got to tell you to like, subscribe and ring the bell. YouTube does help. I know it's kind of corny to say that, but we're here every Wednesday talking about cool stuff. So thanks so much. And back to Greg. Yeah. Awesome. I got one thing I just want to chat with you about for a second, just to make sure it's clear to the audience here. So we built up this data agent. And the thing that we did is we said, well, this is the auto retriever functional tool, right? And we said, this thing performs auto retrieval from a vector database, and then applies a set of filters. And you actually did this. On just the semantic query engine. And then you also did it on the semantic and sequel based query engine, right? So that sort of agentic behavior can happen in all sorts of ways, and we demonstrated a few today. Is that right, chris that's correct yeah yeah so there's there's many layers here when you sort of put this reasoning engine at the front you know even within a single semantic query engine we saw well some of the movies have similar monsters some of the movies have similar you know caverns and caves some of the movies you know adventure movies have these similarities in common so you really have to look at your data and decide on the right metadata decide on the way you're actually assigning the prompt to the agent but also actually doing the prompting in your own query, right? Like it all comes together into better answers, you know? Absolutely right. Absolutely right. Okay. All right. All right. Well, let's wrap up and then we'll jump into questions here in just a moment. So everybody that concludes our event for the day. We've got a number of great questions coming in the chat. We saw the core constructs of Lama Index V 0.10 from nodes to embeddings, to indices, to retrievers, to query engines, to data agents. And we saw that data agents really are just that next layer of abstraction above the query engine, allows us to leverage multiple query engines also gives us access to tools one of those tools could be the open AI functions API so that's kind of an interesting thing as well so there's a lot more good stuff when we talk about metadata filtering reasoning during retrieval processes and agentic systems that we look forward to bringing you guys on YouTube live as we continue into 2024, really the year of agents. So if you got questions, please go ahead and throw them in Slido or throw them in YouTube. We will triage by taking Slido questions first. So please upvote those. And let's get Wiz back on the stage. Let's go. We've got a bunch of good stuff coming in. Some of the classics as well, people are asking. But the first one that I saw a while ago on Slido is, does the embedding apply to the text chunk in the node and the metadata or just the text chunk for the default case just the text chunk um you can definitely set it up in a way where you combine the metadata uh or you you also embed metadata i mean mean, of course, the sky's the limit all the time, but by default, it's the text inside the node. Okay, okay. Next question. Another anonymous one here. How do the different response modes of the query engine affect the agents afterwards? Any recommendation? Yes. So assuming I understand the question correctly, how it can respond can affect the agents afterwards. So depending on what information it responds with, how much information you're basically sharing in that message thread can absolutely impact the performance of the agents afterwards. When it comes to like the best, it's just going to be a use case based, but I would say like sharing any and all information that is relevant to help your agent is the best way to go. All right. Next question. Another technical one here. Do we need the storage context with the ingestion pipeline? The example of ingestion pipeline and LOM index website does not use it. Yeah. So basically what we're going to do is we're going to, we're actually going to create the index first. And so that's what we use the storage context for. And then we are going to use our ingestion pipeline to build nodes, which we will then insert into that vector store index we created with the context. So the way that we did this today, you absolutely do need that step first basically that's telling llama index what they can expect from our quadrant um vector store index all right all right um okay quick uh let's do a couple quick ones here is nodes just a llama index term i mean basically yes uh it's synonymous with like uh or it's close to synonymous with things like document from, you know, from classic NLP. In Lama Index, obviously nodes mean a little bit more than just that, right? The idea is that they represent some standard first class object. Greg already kind of went through that. But the idea is that the way that they mean node, yes, it is a Lama index specific term. All right. And then on data agents, it says the data agent seems to rely on open AI function tool capability, at least the one we demonstrated today. Which other LLM has a similar capability that can replace this open AI function capability? Anthropic? I mean, yes. So there are actually a lot of LLM providers now that provide that kind of, you know, that kind of functionality. So the thing that I would say is like, check your LLM provider, make sure that, you know, if you need it or want it, you have access to some version of function calling, some of them even use the same API, you know, as as, as is obviously the case, just because of the way that it works, OpenAI kind of being quote, unquote, first on a lot of these things means that they actually had a lot of, you know, a lot of people are adapting to the way that they do things. And so you can use even the same thing with some other LLM providers. But I would be, I would always recommend you to check your LLM provider to see what capabilities it has and how it exposes those capabilities to you as a user. All right. All right. Yeah. So speaking of either or, we're getting a ton of questions about, well, what about Langchain versus Llama Index? Let's go ahead and start attacking some of these. Could we have used Langchain instead of Llama Index to do this? I mean, we could have. Yeah, for sure. Could we have done it in the way that we did it with Lama Index? No. So the way that I would think about this particular implementation with Lama Index is that we're basically, we have a bunch of different sources of data. And our OpenAI function calling agent, right, is just choosing which of those sources of data to use. So tools are more akin to sources of data in Lama Index that I would say like, Langchain has a very generic version of a tool whereas these data agents are meant to pick between different piles of data more cleanly, if that helps. So we could have asked the same questions, but the actual like specific details of the infrastructure and the routing of the prompt to different places is gonna be slightly different. That's correct, yes. Okay, all right. And presumably if we're using the same reference information, it's also gonna say Harry Potter's the best, maybe. reference information. It's also going to say Harry Potter's the best maybe. I I'm not going to make that. I'm not going to make that claim. There is there is perhaps some bias introduced by the prompt or you know, it could be anything but it probably would sorry to disappoint those in chat. So just double clicking in on this Richard asked earlier uh generally okay I got Langchain I got llama index I'm doing some agents um how do I choose if I can sort of ask the same questions I would say uh choose whichever one so stick with whichever one your organization is already using is kind of like the cop out here, right? If you are making a decision on which to use, I would say the decision largely comes to what is it you're trying to do? If what you're trying to do is organize some kind of querying system across many different, various data sources uh that that you know have representations in many different ways so databases uh you know pdf blah blah blah you get the point right um then i would stick with llama index absolutely yeah if you're if you're looking for a more generic or or uh kind of free form agent or less strictly, specifically to analyze information across many data sources, I might think about Langchain for that. Okay, next head to head here, Andres. Llama index NL to SQL versus Langchain SQL agent. Is it right to compare these? Can we say one is, are these doing the same thing? Is one better than another today? The answer to your question is actually, which model are you using? And how good is that model at that task? Because under the hood, we're relying on an LL to do this operation llama index and laying chain uh aren't doing anything other than asking the llm to do it and so um if you're you know it really that question is about which llm are you using and I would say like you know gbt35 Turbo or more uh quote unquote or more look at the leaderboards if uh if you're if you're interested what or more might look like but the idea is that's that's kind of the strength quote unquote of model i would recommend or class of models so your anthropics your coheres your open ais but the actual tooling is just asking the model to do it between both of them so uh no no no sincere difference there okay okay speaking of asking models to do it between both of them so uh no no no sincere difference there okay okay speaking of asking models to do things what about open source we saw you know snowflake this week crushing it with the embedding models we got some we got some open source lms that can compete yet with this agentic stuff no um unfortunately not uh when it comes to complex reasoning, you just need a very big model or a very well fine-tuned model. So some shops might have models that work, but generally, if you're going to be using this kind of agentic behavior, it's better to use one of your favorite closed source models. That's not to say it won't work most of the time using something like a Mixtrel or something like a Mistrel, but it is to say that if you need that reliability, if you need that accuracy and complex reasoning and understanding, we're still thinking about closed source today. All right, all right, all right. Well, that about wraps up the time we have today. It looks like we did get another request for the Ragus notebook that we owe folks from last week. So we'll go ahead and make sure that we get that out to you guys. Thanks for the nudge, folks. We definitely appreciate it. Gets busy over here at AI Makerspace. For those of you that had other questions in the Slido, I'm not sure I quite fully understood them. Feel free to drop those into the YouTube comments and we'll get to them asynchronously. But thank you so much for joining us live, everybody. And Wiz, thanks for joining us today for the Q&A and the code demo, per usual. All right, now that you've already liked and subbed, please go ahead and join the AI Makerspace community. If you're not there, we just passed over a thousand folks in Discord, and we'd love to have you join the party. There's lots of stuff going on all the time, and we're continuing to iterate more and more. By the way, we're doing an interview in about 30 minutes with one of our AIM community members that's been successful. You might want to come live and check that out. And if you're ready to start learning for free, a little bit deeper into all these concepts, you might start with our LLMs in Production Cohort 1 course that we open sourced not long ago. And if you want to really, really accelerate and give yourself many, many, many trail heads on which to learn about prototyping and production AI. Check out our AI engineering boot camp. That's our flagship course today. Although we've got a lot more courses in the works coming down the pipe as we stay out on the open source edge in 2024 and beyond. So thank you, everybody, for coming. If you have feedback, please let us know. I know we got a couple of suggestions on future events. Keep those coming, please. And that's it. If you're doing anything this week, don't forget to not only build and ship, but to share. Tag at AI Makerspace if we encourage you to do something this week. Tag me, tag Chris, tag folks in the community. We'd love to amplify you and help build your personal brand, help you achieve your goals as you move forward into your next phase of generative AI career. All right, everybody, that's a wrap. Till next time, keep building, shipping, and sharing, and we will do the same. See you on YouTube live again soon. Bye, guys.
Data Agents with LlamaIndex
Dive into the future of AI with our groundbreaking event on leveraging agents in LLM applications for 2024! Discover how to skillfully integrate agentic reasoning with advanced techniques like RAG and fine-tuning to architect applications that deliver both performance and cost-efficiency. This session offers an in-depth look at the innovative LlamaIndex v0.10 and its revolutionary approach to AI engineering with "LLM-powered knowledge workers." Learn how to construct complex RAG pipelines that masterfully navigate both structured and unstructured data, leveraging semantic pipelines, NL2SQL tooling, and OpenAI’s cutting-edge metadata filtering. Perfect for AI Engineers and Leaders aiming to enhance their projects’ bottom-line value through smart system architecture, this event is a must-attend to stay ahead in the dynamic world of AI. 

Click to join us and transform your understanding of AI application architecture!
https://lu.ma/llamagents

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/5959y4pV8cT79dKzBs42Hr

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/iqwYN9pEUAxUi4Av9