Create your own search engine
Please read the topic categorydescriptionto understand what this is all aboutCreate your own search engineIn Chapter 5 of theCourse, you learned how to use FAISS to find documents that are most semantically similar to a given query. The goal of this project is to extend this idea to build a retrieval and reranking system, where the retriever returns possibly relevant results, while the reranker evaluates the how relevant these hits are to the query.An example of the architecture might looks as follows (taken from thesentence-transformerslibrary):image945×279 8.53 KBModel(s)Thesentence-transformersmodels on the Hubare great for the reranking task.DatasetsWikipedia is usually a good corpus to test retrieval systems on and you can find a dump in various languages here:wikipediaChallengesImplementing the full retriever-reranking architecture might be a challenge, so a simpler place to start is with a single long document. You can then chunk that document into paragraphs and compute the relevancy scores across each paragraphDesired project outcomesCreate a Streamlit or Gradio app onSpacesthat allows a user to enter a search query about a document (or a whole corpus of documents), and returns the top 5 most relevant paragraphs.Don’t forget to push all your models and datasets to the Hub so others can build on them!Additional resourceshttps://www.sbert.net/examples/applications/retrieve_rerank/README.html#retrieve-re-rankDiscord channelTo chat and organise with other people interested in this project, head over to ourDiscordand:Follow the instructions on the#join-coursechannelJoin theneural-search-enginechannelJust make sure you comment here to indicate that you’ll be contributing to this project
2021-11-15T11:52:11Z
Hi , I will be working on this cool project during the event.
2021-11-15T13:47:01Z
Hey@abhibisht89, cool to hear that you’re tackling this project! I’ve created a Discord channel (see topic description) in case you and others want to use it
2021-11-15T17:44:37Z
Hi, I would like to work on this project during the event
2021-11-15T18:06:46Z
Hi, I would like to work on this project, It’s very interesting
2021-11-15T18:57:42Z
Hi I would like to be part of this project!
2021-11-16T12:56:42Z
Hi I will be working on this project
2021-11-16T13:10:39Z
Hi@kzuriit seems like this project already has 4 team members, so just double check on Discord if that’s the case.If it is, you can either do the project by yourself using your own compute (we reserve the Amazon SageMaker compute for teams), or pick / propose another project in#course:course-event
2021-11-17T16:37:01Z
Hi@lewtun. Yes I have joined discord channel already. Thought I might as well update here that I am working on this project.
2021-11-23T17:33:11Z
Hi@algomuffin@kzuri@abhibisht89very nice web app!However, how did you deal with the wikipedia dataset and transform it into a list of passages?When loading wikipedia dataset in the text field there a many undesirable text as titles, bibliography, external links…How did you filter them out?Thanks!
2021-11-24T05:59:38Z
Hi,Thanks. We use the “simplewiki-2020-11-01.jsonl.gz” dataset due to memory constraints. It is way smaller than the complete wikipedia dataset on huggingface.This is the structure of the dataset:{‘id’: ‘9824’, ‘title’: ‘Aileen Wuornos’, ‘paragraphs’: [‘Aileen Carol Wuornos Pralle (born Aileen Carol Pittman; February 29, 1956\xa0– October 9, 2002) was an American serial killer. She was born in Rochester, Michigan. She confessed to killing six men in Florida and was executed in Florida State Prison by lethal injection for the murders. Wuornos said that the men she killed had raped her or tried to rape her while she was working as a prostitute.’, ‘Wuornos was diagnosed with antisocial personality disorder and borderline personality disorder.’, ‘The movie, “Monster” is about her life. Two documentaries were made about her.’, ‘Wuornos was born Aileen Carol Pittman in Rochester, Michigan. She never met her father. Wuornos was adopted by her grandparents. When she was 13 she became pregnant. She started working as a prostitute when she was 14.’]}this dataset is much cleaner i will say. we use the “paragraphs” and create the embedding out of it.Not much cleaning is needed in this dataset as things are pretty clean.Hope this will help