 Hi, this is Lance from Langchain. This is the fifth video focused on query translation in our rag-from-scratch series. We're going to be talking about a technique called hide. So again query translation sits kind of at the front of the overall rag flow and the objective is to take an input question and translate it in some way that improves retrieval. Now, HIDE is an interesting approach that takes advantage of a very simple idea. The basic rag flow takes a question and embeds it, takes a document and embeds it, and looks for similarity between an embedded document and embedded question. But questions and documents are very different text objects. So documents can be like very large chunks taken from dense publications or other sources. Whereas questions are short, kind of terse, potentially ill worded from users. And the intuition behind Hide is take questions and map them into document space using a hypothetical document or by generating a hypothetical document that's the basic intuition and the idea kind of shown here visually is that in principle for certain cases a hypothetical document is closer to a desired document you actually want to retrieve in this you know high dimensional embedding space than the sparse raw input question itself. So again, it's just kind of means of translating raw questions into these hypothetical documents that are then better suited for retrieval. So let's actually do a code walkthrough to see how this works. And it's actually pretty easy to implement, which is really nice. So first, we're just starting with a prompt, and we're using the same notebook that we used for prior videos. We have a blog post on agents already indexed. So what we're going to do is define a prompt to generate a hypothetical document. In this case, we'll say write a paper passage to answer a given question. So let's just run this and see what happens. Again, we're taking our prompt piping it to to open ai check gpd and then using string output parser and so here's a hypothetical document section related to our question okay and this is derived of course mel i'm just kind of embedded uh kind of world knowledge which is you know a sane place to generate hypothetical documents now let's now take that hypothetical document and basically we're going to pipe that into a retriever so this means we're going to fetch documents from our index related to this hypothetical document that's been embedded and you can see we get a few question a few retrieved chunks that are related to this hypothetical document. That's all we've done. And then let's take the final step where we take those retrieved documents here, which we defined, and our question. And we're going to pipe that into this rag prompt. And then we're going to run our kind of rag chain right here, which you've seen before, and we get our answer. So that's really it. We can go to Langsmith and we can actually look at what happened. So here, for example, this was our final rag prompt. Answer the following question based on this context, and here is the retrieved documents that we passed in. So that part's kind of straightforward. We can also look at, okay, this is our retrieval. Okay, no, this is actually what we generated a hypothetical document here. Okay, so this is our hypothetical document. So we've run chat open AI, we generated this passage was our hypothetical document. And then we've run retrieval here. So this is basically showing hypothetical document generation followed by retrieval. So again, here was our passage we passed in. And then here's our retrieved documents from the retriever, which are related to the passage content. So again, in this particular index case, it's possible that the input question was sufficient to retrieve these documents. In fact, given prior examples, I know that some of these same documents are indeed retrieved just from the raw question, but in other contexts, that may not be the case. So folks have reported nice performance using HIDE for certain domains. And the really convenient thing is that you can take this document generation prompt, you can tune this arbitrarily for your domain of interest. So it's absolutely worth experimenting with. It's a neat approach that can overcome some of the challenges with retrieval. Thanks very much.
RAG from scratch: Part 9 (Query Translation -- HyDE)
HyDE (Hypothetical Document Embeddings) is an approach to improve retrieval that generates hypothetical documents that could be used to answer the user input question. These documents, drawn from the LLMs knowledge, are embedded and used to retrieve documents from an index. The idea is that hypothetical documents may be better aligned with the indexes documents than the raw user question.

Slides:
https://docs.google.com/presentation/d/10MmB_QEiS4m00xdyu-92muY-8jC3CdaMpMXbXjzQXsM/edit?usp=sharing

Code:
https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_5_to_9.ipynb

Reference:
https://arxiv.org/pdf/2212.10496.pdf