 Hi, this is Lance from LangChain, and this is the third video in our series Rag from Scratch, building up a lot of the motivations for RAG from the very basic components. So we're going to be talking about retrieval today. In the last two short videos, I outlined indexing and gave kind of an overview of this flow, indexing and gave kind of an overview of this flow, which starts with indexing our documents, retrieval of documents relevant to our question, and then generation of answers based on the retrieved documents. And so we saw that the indexing process basically makes documents easy to retrieve. And it goes through a flow that basically looks like you take our documents, you split them in some way into these smaller chunks that can be easily embedded. Those embeddings are then numerical representations of those documents that are easily searchable and they're stored in an index. When given a question that's also embedded, the index performs a similarity search and returns splits that are relevant to the question. Now if we dig a little bit more under the hood, we can think about it like this. If we take a document and embed it, let's imagine that embedding just had three dimensions. So each document is projected into some point in this 3D space. Now the point is that the location in space is determined by the semantic meaning or content in that document. So to follow that then, documents in similar locations in space contain similar semantic information. And this very simple idea is really the cornerstone for a lot of search and retrieval methods that you'll see with modern vector stores so in particular we take our documents we embed them into this in this case a toy 3d space we take our question do the same we can then do a search like a local neighborhood search you can think about in this 3d space around our question to say hey hey, what documents are nearby? And these nearby neighbors are then retrieved because they have similar semantics relative to our question. And that's really what's going on here. So again, we took our documents, we split them, we embed them, and now they exist in this high dimensional space. We've taken our question, embedded it, projected in that same space. And we just do a search around the question for nearby documents and grab ones that are close. And we can pick some number. We can say we want one or two or three or N documents close to my question in this embedding space. And there's a lot of really interesting methods that implement this very effectively. I link one here. And there's a lot of really interesting methods that implement this very effectively. I link one here. And we have a lot of really nice integrations to play with this general idea. So many different embedding models, many different indexes, lots of document loaders, and lots of splitters that can be kind of recombined to test different ways of doing this kind of indexing retrieval. recombined to test different ways of doing this kind of indexing retrieval. So now I'll show a bit of a code walkthrough. So here we defined, we kind of had walked through this previously. This is our notebook. We've installed a few packages. We set a few environment variables using Langsmith. And we showed this previously. This is just an overview showing how to run RAG, like kind of end to end. In the last short talk, we went through indexing. And what I'm going to do very simply is I'm just going to reload our documents. So now I have our documents. I'm going to re-split them. And we saw before how we can build our index. Now here, let's actually do the same thing, but in the slides we actually showed kind of that notion of search in that 3D space. And a nice parameter to think about in building your retriever is k. So k tells you the number of nearby neighbors to fetch when you do that retrieval process. We talked about, you know, in that 3D space, do I want one nearby neighbor or two or three? So here we can specify K equals one, for example. Now we're building our index, so we're taking every split, embedding it, storing it. Now what's nice is I ask a question, what is task decomposition? This is related to the blog post. And I'm going to run get relevant documents. So I run that. And now how many documents do I get back? I get one as expected based upon k equals one. So this retrieved document should be related to my question. Now I can go to Langsmith. And we can open it up and we can look at our retriever. And we can see here is our question. Here's the one document we got back. And okay, so that makes sense this document pertains to task decomposition in particular and it kind of lays out a number of different approaches that can be used to do that this all kind of makes sense and this shows kind of in practice how you can implement this this name this kind of KNN or K-nearest neighbor search really easily just using a few lines of code. And next, we're gonna talk about generation. Thanks.
RAG From Scratch: Part 3 (Retrieval)
This is the third video in our series on RAG. The aim of this series is to build up an understanding of RAG from scratch, starting with the basics of indexing, retrieval, and generation. This video focuses on retrieval, covering the process of document search using an index. 

Code: 
https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_1_to_4.ipynb

Slides:
https://docs.google.com/presentation/d/124I8jlBRCbb0LAUhdmDwbn4nREqxSxZU1RF_eTGXUGc/edit?usp=sharing