 Hey, whiz, as we've been saying in class, as goes retrieval, so goes generation when it comes to rag. Is there like a right way to do retrieval? I don't know about right way, but there are certainly lots of awesome ways. Yeah, so once we get like a RAG system set up, we want to take to the next level. And how exactly are we supposed to do that? Well, it depends a lot what you're trying to do, the kind of data you have, the kind of information you want to retrieve. It turns out there's a lot of awesome ways to do it. And as always, we got this performance versus cost thing. Methods, methods, methods, algos, algos, algos coming out all the time. Do we need to care about all of them? Or are there just a few that we really should be focused on today for our tool belt? I think, you know, as it stands right now, there's a few we should be focused on making sure we have in our tool belt. Absolutely. Rock it. Yeah. All right. That's what we're going to do today. We're going to try to break down exactly which of these you should know about. And we're trying to give you some context for you to optimize your own context. Sound good, Wiz? Sounds awesome. All right. We'll see you back in a little bit man so today we want to talk advanced retrieval everybody and welcome this is the sort of second step of our kind of advanced rag sequence we talked about chunking last week and you'll realize today if you joined us for that one retrieval really builds directly on the back of chunking i'm excited for this my name is greg i'm co-founder and ceo of ai makerspace that's chris the wiz alexia co-founder and cto of ai makerspace and we're pumped to have you with us today throughout today's presentation if you have any questions along the way, please throw them into the Slido link that will drop in the YouTube chat now, or just go ahead and smash the chat. Let us know what you're thinking, and we'll see if we can answer each and every question along the way today. We've got quite a bit of background to get into today, so let's go ahead and rock and roll. Advanced retrieval, that's the name of the game today. And as we align towards today, we want to understand that it's not exactly a figured out science, this idea of how do you do retrieval in any given context. So we want to understand, first of all, how retrieval in any given context. So we wanna understand, first of all, how retrieval fits into RAG, and we're kind of zooming in on specific RAG pieces, but we wanna have the big picture in mind when we do that. We wanna be able to look at the different algorithms and compare performance across them. And importantly, we want to be able to understand the fine lines between things like chunking and retrieval and ranking or re-ranking. There's quite a few methods, but we have to set the context of our discussion first. We're going to talk about those methods. We're going to show you re-ranking. We're going to discuss it a little bit. But really, at the end of the day, this is about trying to get the big picture and understand where advanced retrieval methods fit in. We're going to walk you through what you need to know to get there now. Let's start with RAG. One of the common threads that we're going to run through today is we're going to be dealing with one of our favorite types of data, some movie review data, and we're going to be doing John Wick movie reviews. Now, one of the things that you could easily do is you could, for instance, use a simple chat with your PDF style model, and you could upload, let's say, some Wikipedia data that you just printed from the top page on, say, John Wick 4, and you could ask some questions. wick for postponed due to covid and how long for instance and we can use this simple retrieval chain i don't have any any information was it postponed due to covid and we we can read the page and we can see that it actually was postponed was the movie postponed due to covid and we can we can sort of start to understand a little bit about how important retrieval is because we can look at john wick 4 and we can look at the wikipedia page and we can see like it clearly was postponed due to covid the original we're just not able to see the screen that you're sharing right now yeah thanks a lot man and so what we'll what we'll want to do is we'll want to go ahead and So what we'll want to do is we'll want to go ahead and do this full share here. So what we want to do is we want to sort of say, okay, if I want to do a little John Wick 4, If I want to do a little John Wick 4 upload here, and I want to say, okay, was the movie postponed due to COVID? We can sort of see, like, yes, indeed it was. And we can see this directly on Wikipedia, for instance. but we can also see it directly in our source data and we can go and look at the different sources now we got three we got four sources here and not all of them mention covid in fact only one of them the one at the end mentions covid this is important because we're still returning source 0, source 1, source 2, and the question is, are these really adding value? We can look at exactly what's going on in this application, and we can look and see, for instance, what exactly is the chunk size that we're using in this application. We can see it's actually 1,000 out of the box, in this application. We can see it's actually a thousand out of the box and it's a zero or it's a 100 overlap. So that's kind of, we talked about recursive character text splitter last time. The question is, is this really enough? And the answer is potentially, you know, no, it's not enough for any given context. And so what we want to do is we want to make sure that we're returning all the right information for any specific thing that we want to be able to do. And we want to make sure that all this is towards the end in mind we have of avoiding hallucinations, making our answers fact checkable, returning the right reference material, and improving our answers as a result. But we want to avoid redundant reference material. We want to avoid useless reference material. We want to avoid basically everything in our context window that's not relevant because we're just pumping tokens in that aren't going to help us, that's additional cost. And so to really understand this process, we want to make sure that we have the correct overview of exactly what RAG is and where advanced retrieval fits in. So we can break down RAG into two key components. The first one is dense vector retrieval. This is, of course, the retrieval piece. And the other one is in context learning. What exactly are we putting in the context window? Both of these matter when it comes to looking at advanced retrieval methods. because we ask a question, that question gets turned in to a representation and embedding space. We search for similar stuff in embedding space to our question, for instance, in the example we just saw, and the similar stuff is returned. Now, what does similar mean? Well, we saw not necessarily exactly what we might expect, although some of the chunks are very relevant, not all of them are. When we find those similar chunks, we can set up our prompt template and we can return the reference material from the chunks in natural language. So this right here is the retrieval process using dense vectors. The piece where we're actually improving our generation is the part where we augment our context, we augment our prompt. This is in context learning. So this is sort of the RAG process, but it all happens prior to the generation. And the generation is where we actually get our better answer and we can, you know, yay, be happy that RAG was the right solution for us. So as we think about this R in RAG, we think about this retrieval process, we really need to break this thing down. We ask a question, we search this vector store, this vector database, this index for stuff related to our question that's similar in vector embedding space, we return that stuff we found in natural language. And the first piece of this is the piece where we're actually chunking and building our vector store. Okay, we need to set this up so we can ask a question in the first place. When we look at vector stores, there's many of them. We're going to use Qdrant today, and it's one of the hottest ones out there. If you're going to pick one off the shelf, go ahead and pick this one. But regardless of which ones you pick, it matters what vectors you're storing and why. So to create any vector store, you're going to have to take your documents. You're going to have to chunk them. You have to create embeddings for each chunk. And those embeddings are what gets stored. Chunking is simply breaking things into small pieces. And as we've said before, the last event we did on semantic rag, it's so important to remember that whatever gets chunked gets retrieved. And this is where we need to understand the fine line that exists at the vector store. Because how we chunk is one piece of the puzzle, but what we know about the chunks is a piece we're going to continue to explore today. We talked about chunking methods previously. Today we're going to continue to explore today. We talked about chunking methods previously. Today, we're going to leverage the baseline recursive character text splitter best practice chunking strategy that really takes the fixed size approach, the chunk size and overlap, and augments it slightly. Augments it so that we can make sure that we're not stopping in the middle of words. Rather, we're more likely to stop on double new lines, which would be sort of a new paragraph, perhaps a new chapter in a novel or single new lines. Alternatively, we wanna chunk where we have a space. We wanna really avoid chunking mid-word on character count, if at all possible. This recursive text splitting allows us to get close to our chunk size, but do a little bit better job. So here's an example from an Alice in Wonderland book, Down the Rabbit Hole, and we see we chunk on a double new line here and we chunk on a space here. This is with a simple chunk size 200 and overlap zero example at the beginning of this novel. Now there are more sophisticated ways to do this chunking. And one of those ways is to look at the meaning of each chunk. The meaning of things semantically is something we want to be leveraging during chunking as well as during retrieval. If we can afford the additional computation cost to see that improvement in performance. We talked about semantic chunking last time. It worked very well. We talked about semantic chunking last time. It worked very well. What we're going to talk about this time is we're going to talk about this retrieval piece where we're returning the stuff. In the retriever that is essentially a wrapper for the vector store, we're returning natural language and the chunking happens upstream where retrieval happens downstream. We can kind of look at these two things as separate, although keep in, there is a fine line. And as we mentioned in the beginning here, as goes retrieval, so goes generation. And when we do retrieval right, we get better at generation. better at generation. That's because we're augmenting the context. We're giving the model more high-quality context to be able to do its generation with. This idea of in-context learning goes back to the GPT-3 paper called Language Models Are Few-Shot Learn learners. As we move from instruction only zero shot to one shot with big and performance enough models, we get a huge increase in accuracy for our generations across many different tasks. And so this idea of in-context learning really comes from scaling up the LLMs big enough to see prompting become a very powerful pattern that we can use without fine tuning. And of course, when we do prompt engineering, we want to make sure that we have those clear and specific instructions. We're giving it the best possible context. And this is our focus today. Specifically, we're focused on the retrieved context. And another way to think about this is as you prototype, you want to always start with prompting. You can always start with prompting, but as you then move beyond prompting, you're often picking up RAG as the next pattern to leverage. This is simply this context optimization that we're doing here. When we move from prompting to retrieval augmented generation. People often ask, well, do I even need RAG because I have these super long context windows? Well, what do you mean by optimization? Are you trying to optimize how much context you can actually put in? Or are you trying to optimize how high quality each piece of context is? And as a result, optimize the efficiency of your inference, as well as the cost, if it's, say, on a per-token basis. And so this context optimization, what we're focused on is we're focused on this piece specifically, getting this right. We're focused on this piece, We're focused on this piece, getting this right. And in fact, this is the whole shebang. Whether you talk to Langchain, who's really focused on enabling LLM applications that leverage context and reasoning, or Lama Index, who will tell you that there are data framework for LLM applications that benefit from context augmentation. Both chunking and retrieval, both of them affect this context. Now, what is good context? Well, there's a number of ways we can measure this, and we've talked about this before. This is a leading framework called the RAG assessment framework. We encourage you guys to check it out if you haven't yet. But ultimately, what good context means is your problem as the AI engineer building this system or the AI engineering leader. If you want to look at Ragas a bit closer, we'll see it in action today. We encourage you to really get this retrieved context right, we want to get the most relevant information. We want to minimize redundance. We want to make sure that it is the stuff that's going to help us do a better generation. It's meaningful and relevant. And this is what we're focused on when we do advanced retrieval. So today we're going to focus on looking at this through the lens of IMDB movie review data from the John Wick movies. We're going to take this data and we're going to create a vector store. Now, the way we do this kind of matters as well as the way we engage with it. And so let's talk about the different stages of retrieval. We've already discussed this, but this is sort of broken down to a slightly more zoomed in level. When we get our data from IMDB, we're going to load it, we're going to transform it. This chunking, this deciding how to tokenize, how to turn it into embeddings before putting it into the vector store, these are all key components. You might say chunking happens here and retrieval happens here. But we would put forth for you today that there is a fine line that exists at the vector store. What is this fine line exactly and how should we think about it? Well, this is where we can look at each of the most useful advanced retrieval methods and get some insight. So first off, simple, super naive, regular retrieval. Let's base on ourselves with that. Then let's look at parent document retrieval, followed by self-query retrieval, followed by time-weighted retrieval, followed by contextual compression retrieval. And hopefully we can start to see this fine line come into focus for us. When we're doing naive retrieval, all we're doing is we're finding nearest neighbors through a simple cosine similarity. It's a simple dot product. And we're returning naively anything that's similar in embedding space. As we saw in the John Wick example in the beginning, not everything is necessarily super duper relevant to the question that we asked. So maybe we can do better. And that really motivates this entire space. When you think of naive retrieval, there are a few things you can tweak, but it's kind of what Langchain calls vector store backed retrieval. It's just simple vector store, cosine similarity based. You can set the similarity threshold. You can also specify that you only want the top K results back. You don't want 50 or a hundred. You want, let's say the top five or 10. This is a great place to start. And if you're prototyping, you should start here. But after you've prototyped, after you've set the baseline for your evaluation, after you've started to make sure that you understand what the performance is in the current state, and you wanna take it to the next level, you can start investigating things like the parent document retriever. And the big idea here is that small docs, small chunks, they're pretty good because they're more precise. They really are a little bit more accurate in terms of semantic meaning and what is going on in that particular part of the document. But big chunks are good too, because, you know, there's a lot more context, you know, and we're back to this word here. So think about a sentence is always within a paragraph. A paragraph is typically within a chapter or a section, chapters within a novel. I mean, there's a hierarchy to this. And this idea of leveraging the hierarchy is what the parent document retriever is all about. And the technique here is pretty simple. You search for the small chunks and you return the big ones. So it's interesting here, right? Because we're still talking chunks. We're doing retrieval. I thought retrieval happened after chunking. Well, what needs to happen here is that during the chunking process, we actually have to generate metadata for each of the parent-child chunks. Specifically, the child chunks are the thing that are real. They're in the vector store. Parent chunks are held in memory. That's really a separate issue. But you want to think of the child chunk as knowing who his parents are. And so what you do is you look for the child chunks and you return the parent chunks. But you have to have the metadata created for each chunk at the point of creating the vector store. This metadata is essential. Having this metadata created at this point allows us to do an advanced retrieval technique. And so these two things are really inherently linked. As we can see in multiple methods, when we look at the self-query retrieval, for instance, self-query is essentially a way to think about this is to use this text-to-SQL type functionality, meaning kind of chat with your SQL database a little bit. You want to think about this self-query as being able to chat with your database a little bit. And the way that that's done is through, again, metadata filtering. What did Barr say about Foo, for instance? We're going to filter on some metadata we have. This is sort of too general to make it kind of useful to really understand in terms of what we're doing today. But what we're able to leverage in our example with the John Wick movies is if you look at a simple movie review, we have metadata on each of our chunks. Let's say each chunk is an entire John Wick review. They're not very long, so we can make them just one size. We also have star rating. We have the title. We have the reviewer, the date, the actual review text. That's our chunk. And then how many people found it helpful? There's lots of metadata that we can draw from. And so if we give our retriever permission and access to look at that metadata, we can potentially very much improve our results. And this is a very nice, very clean, sophisticated way to take retrieval to the next level. Again, it's all about that metadata that we're creating at the step that we're generating our vector store. Thinking about that fine line, we can also do things like time-weighted retrieval. Time-weighted retrieval out of the box essentially says, what was the date of the last lookup that was similar to this? So kind of the most recently accessed is the big idea. Is the most useful with the kind of the most recently accessed is the big idea is the most useful with the kind of idea that if you're accessing something very frequently it's probably pretty useful now for us in this movie review example we're gonna want to answer based on what movies came out most recently in the spirit of the most recently accessed data. So we're going to modify it a little bit and you'll see how this works. And that one's pretty straightforward. Time weighted retrieval. It's probably kind of what you thought. Maybe not exactly, but you're sort of looking at what is most recent to retrieve, potentially very useful for your application. And then finally, we can think about contextual compression retrieval. Now, this is exactly what it sounds like. We're compressing the context. And so when we think about this returning of the natural language, the thing that's interesting here is we're doing this to the natural language, the language that we return specifically, the reference material to the context. So, you know, you can think about compressing a paragraph down to a few words or many chunks down to just a few chunks. And this compression idea, especially when we think many chunks down to a few chunks, could start to really give us insight into this idea of what is the proper rank order of chunks and how many chunks should I have if I want to let's say If I want to, let's say, minimize redundancy, maximize signal to noise, and really get the best possible generation, best to be defined by you and your application. Well, this is kind of heading us down the path of the re-ranking algorithm. And re-ranking is potentially one of the easiest things to apply, and we would highly encourage you to try it out of the box, as we'll show you in today's build. Again, we're going to use the data from John Wick movies. We're going to use Langchain and a Qdrant vector store for this. We're going to use OpenAI models for both embeddings and for our chat model. And we're going to walk through each of the methods of retrieval from naive to parent document to self query to time weighted and ultimately to contextual compression and re-ranking get ready everybody it's time for advanced retrieval for rag with the whiz. Let's walk them through the notebook, my man. Oh, yeah. Okay. So advanced retrieval with laying chain is conceptually maybe difficult, but, you know, the actual code, thanks, laying chain, pretty straightforward. So what we're going to do today is we're going to do a couple things. We're going to look at this advanced retrieval system, how we, you know, use it, how we think about it. And then we're going to see it on a number of different RAG pipelines. Now, we're going to be exploring a lot of different options. We will be releasing as well a notebook that's going to talk about uh how to assess these things against each other we're not going to go over that in the uh in the demo today just because it's going to be rather dense but the idea is that if we pit all these things together we should be able to figure out which one is best for our use case the first thing we're going to want to do is just grab straight up uh dependencies. So we do need our dependencies. We're going to use Langchain, Langchain OpenAI, and Langchain Cohere. This is because we're going to be using OpenAI as our LLM and our embedding model, and Cohere as our re-ranker when we get to contextual compression. We're also going to be using Quadrant today as our vector DB just because I like it, and Lark because we need it for our self-query retriever. We're going to provide our open AI key and cohere API key. So far, so good. This is all the normal stuff. So now we're going to collect some data. And what data do we need? Well, we need John Wick review data. So we have available on our GitHub a bunch of different John Wick review CSVs, so JW1234 that you're welcome to use. You can create your own as well. If you do create your own, just make sure that the metadata is aligned properly. That will be a necessary step. So we get all this data. That's awesome. Okay, once we have all this data, we're gonna wanna do something with it. And what we're gonna do is we're gonna create documents out of it. Now we're gonna get kind of a little bit manual here in our document creation because we do want to make sure that we have all of the appropriate metadata. So what we're gonna do is we're going to do is we're going to add a bunch of metadata to our CSV documents. So we're going to leave the review itself as its own, you know, that's like the base content, right? Then we're going to add these metadata columns of review date, review title, review URL, author, and rating, right? And then we're going to add the movie title for each movie. Now we're going to be kind of naive about this. We're just going to call them John Wick 1, 2, 3, and 4. They technically have different names, but good enough for right now. We're also going to cast our rating to an int. If it exists, else we'll just make it a zero. If they didn't provide a rating, you can think about this a little bit more and make a different decision here, or maybe get the, did they like it or not from some kind of classifier. So we could build that and then determine the actual review score, but we're just going to set it to zero because it's straightforward. And then we're going to do this last accessed at. We're kind of cheating here, right? We're just kind of declaring when these were last accessed at. And we're going to say that the first movie was last accessed, you know, basically three days ago. And then the last movie or the fourth movie was accessed today or now. And so the reason we're doing this is to kind of illustrate what the time-weighted vector store is going to be doing. Obviously, this is not literally what it's for. It doesn't really care when the documents were added. It more cares when they're accessed last so that, you know, very, very frequently accessed documents are going to be kind of boosted. But we're just going to use this to illustrate what's happening behind the scenes. So let's just make sure that our documents worked out. And indeed, they worked out. We have our page content, which is our review. And then we have a bunch of metadata. We have the source. This is the CSV it's from. We have the row content, which is our review. And then we have a bunch of metadata. We have the source. This is the CSV it's from. We have the row it's from. We have the date that it's from. We have the title. We have the URL. We have the author. We have the rating. We've got the title of the actual movie. And we have this last access stat. So let's go. Next, we're going to set up our quadrant vector store. And I did confirm it is called Quadrant, not Qdrant. So I've been calling it Qdrant this whole time. It happens. We're going to realize that it's called Quadrant now and apologize deeply to the company. But other than that, we need an embeddings model. We're just going to use TextEmbedding3.small because it's small. It's cost effective. Easy peasy. Small, it's cost effective, easy peasy. Yes, Vignesh review title was part of the original data. So if we look at the kind of original CSV here, so if we go to the data, I'll zoom way in here so we can see it. You can see we have these columns that already existed. We have this row, which is our index. We have a review date, author, rating, review title, and review. All of those were already populated for us. And so we're just indicating in the CSV loader that we want those to be metadata. We do not want them to be part of what's called page content. So that's why we did that. Great question, though. So we're going to create our embeddings model and then we're going to create our vector store, which is going to be quadrant from documents. And it's easy enough, right? We're going to use the location being memory, right? So this is an in-memory vector store. You can use a hosted self or otherwise quadrant vector store, but we're just going to keep it simple with memory so that no one's got API issues. And we're gonna name the collection John Wick because why not? I mean, it's about John Wick. So the first chain we're gonna create is the naive rag chain. Since we're focusing on the R in rag today, we'll create our retriever first. We're gonna have a retrieval. We're gonna retrieve 10 documents. Now this is just gonna use the simple cosine similarity. That's it, right? That's all this is doing. It's looking at your query. It's looking at all the documents or a subset of the documents because it's using approximate nearest neighbors. And then it's saying, these are the 10 most related. Here you go. So easy peasy there. We love to see it. We are retrieving 10 documents, which is a lot of documents. And we're doing this because we know down the line we're going to use contextual compression to weed that down to the smallest number of documents. Right. So we're going to we're going to compress that 10 down to a to a smaller number. Then we put the A in RAG, which is just our prompt. We're just kind of sticking with a simple one, right? The star of the show today is R. So A, you know, just going to be kind of normal. And then G, the generator, going to be kind of normal, just using a GPT-35 turbo. It's a heck of a model. We're going to create our naive retrieval chain, and you'll see we've got this big old blob. This is a familiar blob to some of you if you've been following along with us as we've gone through everything. But, you know, the idea is, I've put on comments, I should help explain what this is doing. Because we're, because we care so deeply about retrieval, right? We really want to make sure that we are getting the documents that we are retrieving. So we have to make sure that we populate those documents at the end. We can't just get the output because that's only going to let us see what's happening end to end, which is useful, but not super useful. So we're going to need to do this kind of slightly more complicated pattern to make sure that at the end, we get both our response from the model and we get the context that we retrieved, right? Makes sense. We want to think about retrieval, so we're going to retrieve the documents and populate them. Now, we're going to see how this symbol chain does in a few different prompts, right? Put a little comment here. You might think that we've cherry-picked prompts that showcase the individual skill. We sure have, right? So that's, you'd be correct. The first query, did people generally like John Wick? And we're just going to look at the response because only Ragus really is going to care about the context in a way that's meaningful. So did people generally like it? And yeah, it seems like they did, right? Did any reviews have a rating of 10? If so, can I have the URLs to those reviews? And we get a rating of 10. We get a URL that has a rating of 10? If so, can I have the URLs to those reviews? And we get a rating of 10. We get a URL that has a rating of 10. That's great, right? Let's go. And then what happened in John Wick? And then we get the, basically, this is the plot of John Wick 1. Okay, so that's important to keep in mind when we get into a later retriever. This is the plot of John Wick 1. Okay, so that retriever is like, it's fine. It does the job. I mean, we're happy about it, right? Nothing's super odd. The results are not spectacular. All right, fine. So what about parent document retriever? Well, as Greg said, basically what we're going to do is we're going to take each raw document and or a large chunk of document, okay? Then we're going to store those larger chunks or raw documents in our memory store. So the memory store is not a vector store. It's just kind of chilling in memory, right? So we're not going to do any semantic similarity searching on those big chunks or those parent chunks. We're going to chunk each of those parent chunks into smaller documents, and we're going to associate them with the parents, and then store those in a vector store. Okay, so the idea is we're going to take one big document, turn it into a bunch of smaller documents, and associate it back to the big document. Okay, then we're going to put those into our vector store. So how this looks is we're going to search for those smaller chunks that are in our vector store, but we're going to return the parent chunk that they're associated with, right? So let's say we have a parent chunk that's been split into four different chunks, and our semantic similarity search returns that, you know, three of the four of those chunks are the most related documents. We're just going to return the one big parent document chunk, right? We don't have to return those smaller child chunks. And this is basically the way that we're going to think about this. I got a question in chat. Is RAG only for Q&A or can it be used effectively for summarization? You can definitely use RAG to augment summarization pipelines. It is not just for Q&A, though, of course, that is the cleanest way to demonstrate a lot of its power. So how do we actually make this whole child chunks, parent chunks, associate, all that? Well, basically, we're going to first just define our documents to be our parent documents. Then we're going to implement a recursive character text splitter. And that's going to be our child splitter. So that's what's going to turn our parent documents into our child documents. We're going to, again, create a quadrant vector store. This time we're going to do kind of the verbose way to do this. So we're going to create a quadrant client. We're going to add a collection. We're going to define the vectors that we need. So this is going to be the size of the text embedding three model from OpenAI. And then we're going to define our distance metric. Then we're going to create that vector store with quadrant. We're going to point it to full documents. And we're going to point it to full documents and we're going to point it to uh you know the correct embedding model and the correct client easy peasy then we're going to create a uh parent document retriever we're going to have our vector store as our parent document vector store our doc store as store this is in memory store and our child splitter as child splitter easy peasy thanks laying chain and then we add our documents. Now, as we add our documents into this retriever, what's going to happen? It's going to get a chunk. It's going to turn that into a bunch of small chunks and associate them through metadata. So now when we actually call this, under the hood, the content is going to be, we're going to get the full chunks back, but we're going to search for the smaller chunks, and that's it. And again, I mean, it looks fine. It looks like it's not too much, you know, so, okay. So far, so good. We're liking that. I assume that, a question from chat, I assume that in production, you'd want to store parent docs and something other than memory on disk vector. Yeah, that's correct. We can start however we want. Probably just leaving a bunch of the memory is less than ideal though. Okay, so that's parent document retriever. Makes sense. Seems cool. We search for small, we get big. The idea right here is that what we want is we want to search little pieces of information because they're likely to contain single ideas, right, which is likely to be best captured by the embedding process. But we know that, like, even if a sentence is very related to our query, likely the sentences or the structure or the text around it is going to be useful context for that small piece of information. And that's the idea of parent document retriever. For self-query, it's a lot more straightforward. We have all of this amazing metadata, right? We have movie title, review date, review title, review URL, author, rate. We have all this stuff and we're just not using it, right? Like we're just not using it at all. And so instead of not using it, which seems just not using it at all. Instead of not using it, which seems maybe not great, we should use it. That's what self-query does. It helps to understand the structure for metadata and then make filtered queries based on that metadata. Whenever you see this description, you want to make sure that you write a natural language description of that because it might be used by the LLM, right? And we are going to define the types so we know what kind of filtering we can do. The kind of filtering we can do is different if it's an integer, then if it's a string, then if it's a date, and so on, etc. We also want a default description. So this is the actual page content, right? So if you remember, our page content is just the review. So we're going to define that as the review. And then, of course, we're going to use GBT35 Turbo. We're going to pass in the vector store we already created. This is the same vector store that we used in our in our actual original naive implementation, right? Because all that metadata is already in there. We don't need a different vector store. We just need a different layer on top of it. We're going to pass in that document content description and then all of our metadata field info, which is here. And then when we ask questions, you know, did people generally like John Wick? Okay, we get kind of the same thing, right? you know, did people generally like John Wick? Okay, we get kind of the same thing, right? We get, yes, they generally like John Wick. But if we ask a question like, do any reviews have a rating of 10? If so, can I have the URLs of those reviews? We get a much more, you know, correct response because we're actually able to filter based on movies that are reviews that have a rating of 10 since we have rating as one of our metadata fields, right? So we get a much better answer here. Just by looking at it, you can tell it's a better answer. And then of course, you know, what happened to John Wick? I mean, this is not really meant to help this perform better. And so it doesn't really, at least by looking at it. So that's our self-query, basically smart filtering. You love to see it. Time-weighted vector store, this one's pretty straightforward as well. We want to penalize pieces of context that aren't often used. And so what we're gonna do is we're going to set this up the same way we did before. This is just creating a new quadrant client. So we've already seen this. You know, this is just the verbose way to do it. We're going to basically scale our semantic similarity scores based on this little formula here. The idea being that the higher we put our decay rate, the more aggressively we're going to penalize our old data, right? So the basic idea here is that if the data is very new, its score is going to be higher. If the data is very old, its score will be lower. And its score is just a combination of these two things, the semantic similarity, sorry about that, the semantic similarity, and this little formula. Now you can set a aggressive or a non-aggressive decay rate, right? If you set the decay rate to be close to one, that means that it's going to very quickly return to just kind of base semantic similarity. If you set the decay rate very close to zero, it's going to very quickly return to just kind of base semantic similarity. If you set the decay rate very close to zero, it's going to take a very long time for it to taper off over the course of the document's lifetime. And so that's it. The way we set this up is, again, very straightforward. We're going to use a time-weighted vector store retriever. We're going to pass under decay rate as a fairly aggressive 0.6. And then we're going to set this k equal to 2, so we retrieve two relevant documents for this, and then we just have to add our documents, and we get a little confirmation that these documents have been added. Sounds great. Now, when we use our time-weighted vector, our retrieval chain, right, so our time-weighted retriever here. You can see, did people generally like John Wick? Yes. People generally like John Wick 4 based on the reviews provided. Well, that's interesting, right? Because we didn't ask about John Wick 4, but because all of that John Wick 4 data is weighted more highly, right? We get that John Wick 4 data. Do any reviews have a rating of 10? If so, can I have the URLs? I'm sorry, but there are no reviews with a rating of 10 in the provided context because there were no reviews of John Wick 4 that were rated a 10. And again, what happened in John Wick? In John Wick 4, there is a lot of action, intense fight scenes, seems to impress some viewers with high energy of fast-paced sequences. Again, the idea is because we waited for higher, it's retrieving John Wick 4 information. And the last one that we have is contextual compression. Contextual compression is, again, the words sound intimidating, but it's really quite straightforward. We retrieve a lot of documents that are very likely related to our query vector, and then we compress those documents into a smaller set of more related documents. More here meaning that they're more related, they have higher relatedness. And that's it, right? You can do this in a number of different ways. So you can actually shrink the context itself. So the text only extracts the relevant text out of each of the contexts. Or you can shrink the number of contexts. So we're going to be using this re-ranking method, which is working on that number of contexts access. And basically, we're going to make a basic retriever, just like we had in our naive example. And then we're going to add this compressor, aka our re-ranker for this example. And the way that looks is straightforward. Again, we define this compressor, and then we define this compression retriever. and then we can we define this compression retriever Bob's your uncle there you go again when we pass information in we ask questions we get good responses you know we get basically what you would hope to get and what we're going to do is post in the YouTube comments a notebook that describes all of the all of the performance as defined by ragas um but uh with that i've we've gone through a lot of content so i'm gonna pass you back to greg uh don't forget to like uh comment subscribe ring the bell notification i know it's kind of a little bit uh a little bit funny but it does does really help because we go live every Wednesday, and we love to see you out here. So with that, I'll pass you back to Greg. Yes, awesome. Thank you so much. Wiz, in case you missed that last piece, that last little fine line, I just want to double-click in on this re-ranking idea. This is maybe the easiest thing to pick up and use straightaway as you start going after naive and into the next sort of advanced piece. And to understand really exactly what re-ranking is doing, we can just recall this semantic rag piece we discussed last week, where the hypothesis was that we could use embeddings of individual sentence to make more meaningful chunks. Well, when you look at Cohere's RERANK algorithm, the idea is that you can optimize your retrieval to improve generations through this quote semantic boost, which is fundamentally sort of giving us a more meaningful list of chunks retrieved. Semantic meaning is at the center of all things NLP, and it's no different here. Finally, it's important to note as a takeaway and a quick trailhead that we can use embedding models directly for re-ranking. Now that we've got this screen share dialed in, I can show you directly on the massive text embedchmark Leaderboard, you'll see there's a re-ranking benchmark list. So I encourage you guys to check this out if this is something that you're potentially interested in looking more into. And finally, we have made our way through quite a lot of content and advanced methods today. We see that there is a fine line, not just between contextual compression and re-ranking, but there's also a quite fine line between the stages of chunking and retrieval that meet at the vector store. Keep this fine line in mind as you're building and you're designing how your data is chunked, what metadata you're gonna leverage to do retrieval. And given your specific questions, given your specific things you want to be able to answer or summarize, you're gonna potentially wanna choose different metadata, chunking strategies, retrievers. And again, we'll drop the quantitative analysis we did. That was just too much for today in the YouTube comments. If you want us to go deeper on any of this stuff, we'd love to hear from you. But otherwise, we've got a few minutes to answer questions that we'll go ahead and get started with from the Slido. Whiz, that was a lot today, man. That was a lot. Okay, maybe we overshot the mark a little bit, but we've got some questions. Paolo asks, hey, nice hat. Thanks, Paolo. Does chunking work for tables in documents? Mute it. And we can't hear you, Chris. Oh, sorry. Yes, I think it can. It kind of depends on what you mean. I would say loosely, though, we don't really want to chunk tables. We would like to keep them together, right? Because they describe a kind of single way to think about something, right? So if you had a very massive table that you wanted to chunk, you would kind of want to retain that structure. So I'm not sure that we would want to chunk tables, but you certainly can. Yeah, why not? Okay. Okay. Manny asks, what's up Manny? When returning results, is there a command for the prompt template that can make the LLM return citations it has with the probability of accuracy in percentage? I guess I'm not quite sure what accuracy is in this case. So kind of. What we can do is we can say things like, so in the case of this notebook, we're returning two things. We're returning the response and the contexts. So basically, that's the citations that it used. That context is what it used to give you the actual original answer. So that's why we set up our chain in that way. When it comes to this probability of accuracy in percent, what we can do is we can see whatever measure we use, we can forward that score, right? So like if we're talking about cosine similarity, we can forward that score. The meaningfulness of that, or a way to directly port it to like percentage accuracy based, that part's a little bit less clear, but it's absolutely true that we can forward the score from our retrieval pipeline, even though it might not mean much in a vacuum. Okay. Manny has a couple either or here. Any vector store specifically required to achieve these retrieval methods or can we do an independent vector store? You do have to use a vector store that supports things like metadata, metadata filtering. Absolutely. Yes. So Quadrant, Chroma, kind of all of the normal ones, even Face, as long as you set it up correctly with an index. So you can make a vector sort of powered by face, but it is not going to work out of the box with every single vector store you can imagine. Quadrant, shout out to quadrant. Learn something new every day. Yeah. All right. Got it. And then can we do both of the, all of these methods in both Langchain and Lama index today? Yeah. Okay. All right. Cool. Cool. Then Richard asks, which are the best rag techniques when we have an app that will add new documents all the time and which for very long documents? Yeah. So when we're adding a lot of new documents, you could see something like the time weighted vector store being useful, right? The idea is that you could add documents to it, and those would be the most recent versions of the documents. So if you're adding, say, blogs to your vector store, right, you could weight the new ones higher. And obviously, that's going to depend on how people interact with them, but that kind of thing is going to work very well. Otherwise, it's up to you about, you know, how that kind of thing is going to work very well. Otherwise, it's up to you about, you know, how that information fits in. You can do, you can add it with specific metadatas and you self-query, you know, just depends on what you need to do with new documents. You know, is it that new ones are more important? Is it that new ones should be categorized based on existing categories? You know, you get, you get the idea. And for long, uh, documents, it's, they're, they're all going to be great. Uh, obviously we can retrieve less of them depending on our context window. Um, but, uh, you know, long documents, something like contextual compression is probably going to be very useful for you as you're going to be able to squeeze out just the valuable juice from each of those, uh, those long chunks or documents. Okay. And then, you know, I're going to be able to squeeze out just the valuable juice from each of those long chunks or documents. Okay. And then I'm going to ask one of these, we got a bunch of straight classic questions, fine tuning, Langchamber's Lama index, et cetera, et cetera. Tell us more about why you chose Quadrant. What are the technical reasons here people want to know it's just so good uh it's very efficient it's very good big scale uh there you go that's that's like i mean quadrant will will will be good for you from when you have one uh you know daily active user to when you have 10 million yeah you know also the most recent company to raise money so that says anything to you it says something to me um go quadrant all right uh if you have more questions please uh throw them on to the youtube comments after the session we will get to them thank you so much everybody uh wiz thanks for walking us through that heck of a sesh today all right time to close out and thank you so much for walking us through that heck of a session today. All right, time to close out. And thank you so much for joining us today. If you enjoyed the session, then you might really enjoy continuing to join us every week. We're back on Wednesday all the time. You can join us same time, same place. We'd love to see you both now on LinkedIn, as well as on YouTube. And if you also want to go a little bit deeper, maybe check out joining our Discord channel. We almost have a thousand people in our Discord and it's pretty popping most of the time. We've got a lot of great projects people are working on. If you're looking for one, it might be a cool place to make some connections. And then if you want to start really taking your game to the next level and you want to start for free, we recommend checking out our open source LLM Ops course. It's free on GitHub and on YouTube. You can get all the boilerplate code and concepts you need straight through just a few videos. You can do it in really a day or two and get that really base of LLM ops. If you're ready to truly accelerate and take it to the next level, figure out all these details, Langchain Lama Index, what's fine tuning? What's the deal? How do I make these decisions? Maybe check out and consider applying to our AI engineering bootcamp. We just kicked off cohort two and cohort three is coming up in May. We'd love to have you if you're interested in getting more one-on-one time with us, with peers that have graduated in the past, getting access to hiring partners and learning the core concepts and code of AI engineering today, then it might be a good option for you. And that's a wrap for today, everybody. Any feedback that you have, we'd love to hear it. I know we had a couple of snags today and we'll continue to improve on our end as we keep building, shipping and sharing. We hope that you will do the same to build, ship and share something awesome this week and tell not just everybody in your network, but maybe even everybody in the AI Makerspace community. So until next time, keep getting after it, keep building, shipping and sharing, and we'll do the same. See you soon, everybody. Bye, guys.
Advanced Retrieval Methods for RAG
In this event, we will break down the retrieval algorithms that AI Engineering practitioners should know and have at hand within their toolbox. Algorithms known to provide greater precision and results at the retrieval step of RAG include the Parent Document Retriever, Self Query Retriever, Contextual Compression, and Time-Weighted Vector Store.

RSVP: https://lu.ma/retrieval4rag

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/3eFnpQg7xrgcnb3TgMGQL6

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/jKdAm5kLb4fRMa8UA