 Hi, this is Lance from Langchain. This is the 14th part of our Rag from Scratch series. I'm going to be talking about an approach called Cold Bear. So, we've talked about a few different approaches for indexing. And just as kind of a refresher, indexing falls kind of right down here in our flow. We started initially with query translation, taking a question, translating it in some way to optimize retrieval. We talked about routing it to a particular database. We then talked about query construction, so going from natural language to the DSL or domain-specific language for any of the databases that you want to work with. Those are metadata filters for vector stores or cipher for graph DB or SQL for relational DB. So that's kind of flow we talked about today. We talked about some indexing approaches like multi-representation indexing. We gave a small shout out to Greg Cameron in his years on chunking. We talked about hierarchical indexing. I want to include one advanced embedding approach. So we talked a lot about embeddings are obviously very central to semantic similarity search and retrieval. So one of the interesting points that's been brought up is that embedding models, of course, take a document, you can see here on the top, and embed it, basically compress it to a vector. So it's kind of a compression process. You're representing all the semantics of that document in a single vector. You're doing the same to your question, you're doing a similarity search between the question embedding and the document embedding in order to perform retrieval. You're typically taking the, you know, K most similar document embedding is given a question, and that's really how you're typically taking the you know k most similar um document embedding is given a question and that's really how you're doing it now a lot of people said well hey the compressing a full document with all this nuance to a single vector seems a little bit um overly restrictive right and this is a fair question to ask um there's been some interesting approaches to try to address that. And one is this approach method called Colbert. So the intuition is actually pretty straightforward. There's a bunch of good articles I linked down here. This is my little cartoon to explain it, which I think is hopefully kind of helpful. But here's the main idea. Instead of just taking a document and compressing it down to a single vector, basically a single what we might call an embedding vector, we take the document, weing it down to a single vector, basically a single what we might call embedding vector. We take the document, we break it up into tokens. So tokens are just like units of content. It depends on the token areas you use. We talked about this earlier. So you basically tokenize it. And you produce basically an embedding or a vector for every token. And there's some kind of positional weighting that occurs when you do this process. So obviously you look at the implementation to understand the details, but the intuition is that you're producing some kind of representation for every token. Okay? And you're doing the same thing for your questions. You're taking your question, you're breaking it into tokens, and you have some representation or vector per token and then what you're doing is for every token in the question you're computing the similarity across all the tokens in the document and you're finding the max you're taking the max you're storing that and you're doing that process for all the tokens in the question. So again, token two, you compare it to every token in the document, compute the max. And then the final score is in this case the sum of the max similarities between every question token and any document token. So it's an interesting approach. It reports very strong performance. Latency is definitely a question. So kind of production readiness is something you should look into. But it's an approach that's worth mentioning here because it's pretty interesting. And let's walk through the code. So there's actually a really nice library called Rakatouille, which makes it very easy to play with Colbert. So you just pip install it here. I've already done that. And we can use one of their pre-trained models to mediate this process. So I'm basically following their documentation. This is kind of what they recommended. So I'm running this now. Hopefully, this runs somewhat quickly. I'm not sure. I previously have loaded this model, so hopefully it won't take too long. And, yeah, you can see it's pretty quick. I'm on a Mac M2 with 32 gigs. So just as like a context in terms of my system. This is from their documentation. We're just grabbing a Wikipedia page. This is getting a full document on Miyazaki. So that's cool. We're going to grab that. Now this is just from their docs. This is basically how we create an index. So we provide the, you know, some index name, the collection, the dot max document length. And yeah, you should look at their documentation for these flags. These are just the default. So I'm going to create my index. So I get some logging here. So it's working under the hood. And by the way, I actually have their documentation open, so you can kind of follow along. Let's see, yeah, right about here. So you can kind of follow this indexing process, it's great and index, so you load a trained model. This can be either your own pre-trained model or one of ours from the hub. And this is kind of the process we're doing right now. Create index is just a few lines of code, and this is exactly what we're doing. So this is my documents, and this is the indexing step that we just kind of walked through. And it looks like it's done. So you get a bunch of logging here. That's fine. Now let's actually see if this works. So we're going to run drag search. that's fine. Now let's actually see if this works. So we're gonna run drag search, what animation studio did Miyazaki found, set our k parameter, and we get some results. Okay so it's running and cool we get some documents out. So you know it seems to work. Now what's nice is you can run this within linechain as a linechain retriever. So that basically wraps this as a linechain retriever and then you can use it freely as a Retriever within Langchain. It works with all the other different LLMs and all the other components like re-rankers and so forth that we talked through. So you can use this directly as a Retriever. Let's try this out. And boom, nice and fast. And we get our documents. Again, this is a super simple test example. You should run this maybe on more complex cases. But it's a pretty easy spin-up. up it's really interesting alternative indexing approach um using again like we talked through um a very different algorithm for computing doc similarity that may work better i think an interesting regime to consider this would be longer documents so if you want like longer um yeah if you basically want kind of long context embedding i think you should look into for example the max token limits for this approach because it partitions the document into into each token um i would be curious to dig into kind of what the overall context limits are for this approach of colbert but it's really interesting to consider and it reports very strong performance so again i encourage you to play with it and this is just kind of an intro to how to get set up and to start experimenting with it really quickly thanks
RAG From Scratch: Part 14 (ColBERT)
Our RAG From Scratch video series walks through impt RAG concepts in short / focused videos w/ code. This is the 14th video in our series and focuses on indexing with ColBERT for fine-grained similarity search.

Problem: Embedding models compress text into fixed-length (vector) representations that capture the semantic content of the document. This compression is very useful for efficient search / retrieval, but puts a heavy burden on that single vector representation to capture all the semantic nuance / detail of the doc. In some cases, irrelevant / redundant content can dilute the semantic usefulness of the embedding.

Idea: ColBERT (@lateinteraction & @matei_zaharia) is a neat approach to address this with higher granularity embeddings: (1) produce a contextually influenced embedding for each token in the document and query. (2) score similarity between each query token and all document tokens. (3) take the max. (4) do this for all query tokens. (5) take the sum of the max scores (in step 3) for all query tokens to get the similarity score. 
This results in a much more granular token-wise similarity assessment between document and query, and has shown strong performance. 

Code:
https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_12_to_14.ipynb

References:
1/ Paper:
https://arxiv.org/abs/2004.12832

2/ Nice review from @DataStax: 
https://hackernoon.com/how-colbert-helps-developers-overcome-the-limits-of-rag

3/ Nice post from @simonw:
https://til.simonwillison.net/llms/colbert-ragatouille

4/ColBERT repo:
https://github.com/stanford-futuredata/ColBERT

5/ RAGatouille to support RAG w/ ColBERT:
https://github.com/bclavie/RAGatouille