 Hi, this is Lance from Langchain. I'm going to talk about indexing and multi-refermentation indexing in particular for the 12th part of our Rag from Scratch series here. So we previously talked about a few different major areas. We talked about query translation, which takes a question and translates it in some way to optimize for retrieval. We talked about routing, which is the process of taking a question, routing it to the right data source, be it a vector store, GraphDB, SQLDB. We talked about query construction. We dug into basically query construction for vector stores, but of course there's also text to SQL, text to Cypher. So now we're going to talk about indexing a bit. In particular, we're going to talk about indexing techniques for vector stores. And I want to highlight one particular method today called multi-representation indexing. So the high-level idea here is derived a bit from a paper called Proposition Indexing, which kind of makes a simple observation. You can think about decoupling raw documents and the units you use for retrieval. So in the typical case, you take a document, you split it up in some way to index it, and then you embed the split directly. This paper talks about actually taking a document, splitting it in some way, but then using an LLM to produce what they call a proposition, which you can think of as like kind of a distillation of that split. So it's kind of using an LLM to modify that split in some way to distill it or make it like a CRISPR, like summary, so to speak, that's better optimized for retrieval. So that's kind of one highlight, one piece of intuition. So we've actually taken that idea and we've kind of built on it a bit in kind of a really nice way that I think is very well suited, actually, for long context LLMs. So the idea is pretty simple. You take a document and you actually distill it or create a proposition like they show in the prior paper. I kind of typically think of this as just produce a summary of the document and you embed that summary. So that summary is meant to be optimized for retrieval. So it might contain a bunch of keywords from the document or like the big ideas, such that when you embed the summary, you embed a question, you do search, you basically can find that document based upon this highly optimized summary for retrieval. So that's kind of represented here in your VectorStore. But here's the catch. You independently store the raw document in a doc store. And when you basically retrieve the summary in the VectorStore, you return the full document for the LLM to perform generation, and this is a nice trick because at generation time, now with long-conduct LLM, for example, the LLM can handle that entire document. You don't need to worry about splitting it or anything. You just simply use the summary to create a really nice representation for fishing out that full doc. Use that full doc in generation. There might be a lot of reasons you want to do that. You want to make sure the LM has the full context to actually answer the question. So that's the big idea. It's a nice trick. And let's walk through some code here. So we have a notebook all set up, just like before. We've done some pip installs. Set some API keys here for langsmith um kind of here's a diagram now let me show an example let's just load two different blog posts uh one's about agents one is about uh you know human data quality um and what we're going to do is let's create a summary of each of those so this is kind of the first step of that process where we're going from like the raw documents to summaries. Let's just have a look and make sure those ran. So okay, cool. So the first doc discusses building autonomous agents. The second doc contains the importance of high quality human data and training. Okay, so that's pretty nice. We have our summaries. Now we're going to go through a process that's pretty simple. First, we define a vector store that's going to index those summaries. Now we're going to define what we call our document storage. You're going to store the full documents. So this multi-vector retriever kind of just pulls those two things together. We basically add our doc store. We add this byte store as basically the full document store. The vector store is our vector store. And now this ID is what we're going to use to reference between the chunks or the summaries and the full documents. That's really it. So now for every document, we'll define a new doc ID. And then we're basically going to take our summary documents. And we're going to extract for each of our summaries we're going to get associated doc id so there we go um so let's go ahead and do that so we have our summary docs which we add to the vector store we have our full documents uh our doc ids and the full raw documents which are added to our doc store. And then let's just do a query, a vector store, like a similarity search on our vector store. So memory and agents, and we can see, okay, so we can extract, you know, from the summaries, we can get, for example, the summary that pertains to agents. That's a good thing. Now let's go ahead and run a query, get relevant documents on our retriever, which basically combines the summaries, which we use for retrieval, then the doc store, which we use to get the full doc back. So we're gonna apply our query. We're gonna basically run this. And here's the key point. We've gotten back the entire article. And we can actually, if you want to look at the whole thing, we can just go ahead and do this. Here we go. So this is the entire article that we get back from that search. So it's a pretty nice trick. Again, we query with just memory and agents. And we can kind of go back to our diagram here. We create for memory and agents. It starts our summaries. It found. We create for memory and agents. It starts our summaries. It found the summary related to memory and agents. It uses that doc ID to reference between the vector store and the doc store. It fishes out the right full doc, returns us the full document, in this case, the full web page. That's really it. Simple idea, nice way to go from basically like nice, simple proposition style or summary style indexing to full document retrieval, which is very useful, especially with long context LLMs. Thank you.
RAG from scratch: Part 12 (Multi-Representation Indexing)
Our RAG From Scratch video series walks through impt RAG concepts in short / focused videos w/ code. This is the 12th video in our series and focuses on some useful tricks for indexing full documents.

Problem: Many RAG approaches focus on splitting documents into chunks and returning some number upon retrieval for the LLM. But chunk size and chunk number can be brittle parameters that many user find difficult to set; both can significantly affect results if they do not contain all context to answer a question.

Idea: Proposition indexing (@tomchen0 et al) is a nice paper that uses an LLM to produce document summaries ("propositions") that are optimized for retrieval. We've built on this with two retrievers: (1) multi-vector retriever embeds summaries, but returns full documents to the LLM. (2) parent-doc retriever embeds chunks but returns full documents to the LLM. Idea is to get best of both worlds: use smaller / concise representations (summaries or chunks) to retrieve, but link them to full documents / context for generation.

The approach is very general, and can be applied to tables or images: in both cases, index a summary but return the raw table or image for reasoning. This gets around challenges w/ directly embedding tables or images (multi-modal embeddings), using a summary as a representation for text-based similarity search.

Code:
https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_12_to_14.ipynb

References:
1/ Proposition indexing:  https://arxiv.org/pdf/2312.06648.pdf
2/ Multi-vector:
https://python.langchain.com/docs/modules/data_connection/retrievers/multi_vector
3/ Parent-document:
https://python.langchain.com/docs/modules/data_connection/retrievers/parent_document_retriever
4/ Blog applying this to tables:
https://blog.langchain.dev/semi-structured-multi-modal-rag/
5/ Blog applying this to images w/ eval:
https://blog.langchain.dev/multi-modal-rag-template/