 Hi, this is Lance from Langchain. This is the second video in our series Rag from Scratch focused on indexing. So in the past video, you saw the main kind of overall components of rag pipelines, indexing, retrieval, and generation. And here we're going to kind of deep dive on indexing and give just a quick overview of it. So the first aspect of indexing is we have some external documents that we actually want to load and put into what we're trying to call a retriever and the goal of this retriever is simply given an input question I want to fish out documents that are related to my question in some way. Now the way to establish that relationship or relevance or similarity is typically done using some kind of numerical representation of documents. And the reason is that it's very easy to compare vectors, for example, of numbers relative to, you know, just free-form text. And so a lot of approaches have been developed over the years to take text documents and compress them down into a numerical representation that then can be very easily searched. Now, there's a few ways to do that. where you take a document, you look at the frequency of words, and you build what they call sparse vectors, such that the vector locations are a large vocabulary of possible words. Each value represents the number of occurrences of that particular word. And it's sparse because there's, of course, many zeros. It's a very large vocabulary relative to what's present in the document. And there's very good search methods over this type of numerical representation. Now, a bit more recently, embedding methods that are machine learned, so you take a document and you build a compressed, fixed-length representation of that document have been developed with correspondingly very strong search methods over embeddings. So the intuition here is that we take documents and we typically split them because embedding models actually have limited context windows. So you know on the order of maybe 512 tokens up to 8000 tokens or beyond, but they're not infinitely large. So documents are split and each document is compressed into a vector and that vector captures a semantic meaning of the document itself. The vectors are indexed, questions can be embedded in exactly the same way, and then a numerical kind of comparison in some form, you know, using very different types of methods, can be performed on these vectors to fish out relevant documents relative to my question. And let's just do a quick code walkthrough on some of these points. So I have my notebook here. I've installed here. Now I've set a few API keys for Langsmith, which are very useful for tracing, which we'll see shortly. Previously, I walked through this kind of quick start that just showed overall how to lay out these RAG pipelines. And here what I'll do is I'll deep dive a little bit more on indexing, and I'm going to take a question and a document. And first, I'm just going to compute the number of tokens in, for example, the question. And this is interesting because embedding models and LLMs more generally operate on tokens. And so it's kind of nice to understand how large the documents are that I'm trying to feed in. In this case, it's obviously a very small, in this case, question. Now, I'm going to specify open eye embeddings. I specify an embedding model here. and I just say embed, embed query. I can pass my question, my document. And what you can see here is that runs, and this is mapped to now a vector of length 1536. And that fixed length vector representation will be computed for both documents, and really for any documents. So you're always kind of computing this fixed length vector that encodes the semantics of the text that you've passed. Now I can do things like cosine similarity to compare them. And as we'll see here, I can load some documents. This is just like we saw previously. I can split them, and I can index them here, just like we did before. But we can see under the hood, really what we're doing is we're taking each split, we're embedding it using OpenAI embeddings into this vector representation, and that's stored with a link to the raw document itself in our vector store. And next, we'll see how to actually do retrieval using this vector store.
RAG From Scratch: Part 2 (Indexing)
This is the second video in our series on RAG. The aim of this series is to build up an understanding of RAG from scratch, starting with the basics of indexing, retrieval, and generation. This video focuses on indexing, covering the process of document loading, splitting, and embedding. 

Code:
https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_1_to_4.ipynb

Slides:
https://docs.google.com/presentation/d/1MhsCqZs7wTX6P19TFnA9qRSlxH3u-1-0gWkhBiDG9lQ/edit?usp=sharing