RAG custom dataset
I just saw that Facebook AI released ablog post about RAG( Retrieval Augmented Generation: Streamlining the creation of intelligent natural language processing models) and that it is already incorporated in theHuggingFace API.I looked quickly, and I couldn’t see how to use a custom dataset with it.It seems like it will only pull down indexed datasets from HuggingFace’s AWS storage. I’m wondering if anyone can show me how toCreate an indexed dataset. I’m assuming this is just a big collection of embeddings that have been made by running documents through a model and taking the output embedding. I’m wondering which model(s) can be used, how many dimensions the embeddings are expected to be, and how to format all of these vectors.Use that custom dataset with HF Rag models.
2020-09-30T12:31:20Z
Indeed, it’s actually very simple to do with datasets and somehow explained on this page:https://huggingface.co/docs/datasets/faiss_and_ea.htmlWe will add an example script on this.
2024-07-23T11:30:17Z
today its a dead page