RAG Retriever: hf vs legacy vs exact vs compressed
In the eval_rag.py file under examples, I notice that the arguments for index_name is “hf” or “legacy”. How are these different from “exact” vs “compressed”?
2020-11-19T23:11:02Z
Hi! The last question was answered here :RAG Retriever : Exact vs. Compressed Index? - #4 by lhoestqRAG Retriever : Exact vs. Compressed Index?exact vs compressed refers to the quantization used for the FAISS index. The compressed one uses an IVF index with product quantization and requires significantly less RAM than the exact one. To reproduce the RAG papers result you will need the exact one though.Note that I will update this week the parameters of both index so that the exact one uses the same as RAG’s paper, and also to have an optimized compressed one.
2020-11-19T23:12:55Z
Hi Jung! I did see that post. Thats what created the confusion for me. The examples uses “hf” vs “legacy” but here we are mentioning “exact” vs “compressed”. If this could be clarified that would be great.
2020-11-19T23:28:29Z
Hi@sashank06, sorry I misunderstood the question.As far as I understand from thesource codeclass LegacyIndexrefered to original index use in the RAG/DPR papers, whileclass HFIndexBaseallows us to use custom datasets (there’sdatasetargument).Therefore, according to above quote, I understand “exact” and “compressed” are subtypes of “legacy”. I may misunderstood, and would be great if@lhoestqcan help clarify here
2020-11-23T16:29:02Z
Hi ! that’s a mistake in the eval_rag.py parameters choices. As specified in the rag configuration (seedocumentation), one can choose between ‘legacy’, ‘exact’ and ‘compressed’. The legacy index is the original index used for RAG/DPR while the other two use thedatasetslibrary indexing implementation.link to the PR that fixes the eval_rag.py parameter description:https://github.com/huggingface/transformers/pull/8730
2021-10-05T20:48:02Z
Hi Quentin!We saw your post saying exact index has to be used for the replication of the paperRAG Retriever : Exact vs. Compressed Index?However, HuggingFace documentation says that legacy index replicates the paper’s resultsRAG — transformers 4.11.2 documentationAlso if Compressed index uses the same wiki dump but FAISS index is loaded in the RAM, why can this not be used to replicate the paper’s results.While using legacy to load the pretrained RAG retriever, we faced “MemoryError: std::bad_alloc”.Kindly help us resolve the confusion as we are trying to load the RAG Retriever’s wiki dump for text based question answering.Also the wiki dump seems to be huge (140G) and we do not have space for that much storage right now.Is there any text based wiki dump that helps us to replicate the paper’s result while being smaller than 140G?Looking forward to your guidance and help. Great thanks for your time!Regards
2021-10-06T09:23:02Z
We saw your post saying exact index has to be used for the replication of the paperRAG Retriever : Exact vs. Compressed Index?However, HuggingFace documentation says that legacy index replicates the paper’s resultsRAG — transformers 4.11.2 documentationThe compressed index has lower retrieval performance than the exact one. That’s why you need to use either the exact or the legacy one to replicate RAG’s performance.While using legacy to load the pretrained RAG retriever, we faced “MemoryError: std::bad_alloc”.The legacy index is 35GB and it needs to fit in RAM. Make sure you have enough RAM when using it.Is there any text based wiki dump that helps us to replicate the paper’s result while being smaller than 140G?You can use the wiki dump from the legacy index:I think this one takes less disk space because the embeddings are stored quantized in the FAISS index (whereas the other indexes store the plain embeddings and it takes around 70GB to download and 70GB to convert to an Arrow dataset file). So you would have around 10GB + the faiss index (35GB).
2021-10-07T23:19:12Z
Great thanks for your response, that’s extremely helpful! May I ask the RAM requirement for ‘exact’ and ‘compressed’ one also? Much appreciated.
2021-10-21T08:45:43Z
It takes 35GB for the exact one and 3GB for the compressed one