Using RAG with local documents
Hi,I have a requirement that model should search for relevant documents to answer the query and I foundRAGfrom Facebook AI which perfectly fits my usecase. I also foundthispost in which HuggingFace explains RAG and came to know that HF implemented RAG which is awesome!My doubt is whether I could extend this functionality so that the model should do retrieval from local documents rather than from HF’s wikipedia corpus.Are there any notebooks to refer to?
2021-04-06T21:19:23Z
hey@saichandra, one possibility would be to useHaystack’simplementation of RAG (which is based on HF transformers), e.g. see here:https://haystack.deepset.ai/docs/latest/tutorial7mdone advantage of using Haystack is that they provide a nice API for FAISS (and other document stores) so you can store the embeddings locally with just a few lines of code.i’ve had mixed results from using RAG with the Natural Questions checkpoints (the answers are often gibberish). if you’re doing QA on a specialised corpus, you might be better off using the classic Retriever-Reader architecture or fine-tuning RAG on your domain
2021-04-21T13:00:39Z
You can easily do this by encoding your local KL. Please follow theuse_own_knowledge_dataset.py
2021-04-21T13:50:56Z
to add to what@shamanezsaid, i also recently learnt that you can add faiss indices directly in thedatasetslibrary:Adding a FAISS or Elastic Search index to a Dataset — datasets 1.6.0 documentation