
# Dataset Card for Open-Domain Question Answering Wikipedia Corpora

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
- [Dataset Structure](#dataset-structure)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Source Data](#source-data)

## Dataset Description

### Dataset Summary

The Wikipedia corpus variants provided can serve as knowledge sources for question-answering systems based on a retriever–reader pipeline. These corpus variants and their corresponding experiments are described further in the paper entitled: 
> Pre-Processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering.


## Dataset Structure

### Data Fields

The dataset consists of passages that have been segmented from Wikipedia articles.
For each passage, the following fields are provided  
- ```docid```: The passage id in the format of (X#Y) where passages from the same article share the same X, but Y denotes the segment id within the article  
- ```title```: The title of the article from where the passage comes
- ```text```: The text content of the passage
  
### Data Splits

There are 6 corpus variants in total
- ```wiki-text-100w-karpukhin```: The original DPR Wikipedia corpus with non-overlapping passages, each 100 words long, from Karpukhin et al.,
> Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih. [Dense Passage Retrieval for Open-Domain Question Answering](https://www.aclweb.org/anthology/2020.emnlp-main.550/). _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, 2020.
- ```wiki-text-100w-tamber```: Our replication of the above corpus
- ```wiki-text-6-3-tamber```:  A corpus similar to above i.e. without tables, infoboxes, and lists. Segmentation is done differently, with a passage size of 6 sentences and a stride of 3 sentences. Note, this means that passages are overlapped.
- ```wiki-text-8-4-tamber```:  Like wiki-text-6-3, but with a passage size of 8 sentences and a stride of 4 sentences.
- ```wiki-all-6-3-tamber```: A corpus with tables, infoboxes, and lists included with a passage size of 6 sentences and a stride of 3 sentences.
- ```wiki-all-8-4-tamber```:  Like wiki-all-6-3, but with a passage size of 8 sentences and a stride of 4 sentences.

## Dataset Creation
### Source Data

#### Initial Data Collection and Normalization

We start with downloading the full December 20, 2018 Wikipedia XML dump: ```enwiki-20181220-pages-articles.xml``` from the Internet Archive: https://archive.org/details/enwiki-20181220. This is then Pre-processed by WikiExtractor: https://github.com/attardi/wikiextractor (making sure to modify the code to include lists as desired and replacing any tables with the string "TABLETOREPLACE") and DrQA: https://github.com/facebookresearch/DrQA/tree/main/scripts/retriever (again making sure to modify the code to not remove lists as desired).

We then apply the [pre-processing script]((https://github.com/castorini/pyserini/blob/master/docs/experiments-wiki-corpora.md)) we make available in [Pyserini](https://github.com/castorini/pyserini) to generate the different corpus variants.

task_categories:question-answering task_categories:text-retrieval task_ids:open-domain-qa annotations_creators:no-annotation multilinguality:monolingual language:en license:cc-by-sa-3.0 region:us
{"annotations_creators": ["no-annotation"], "language_creators": [], "language": ["en"], "license": ["cc-by-sa-3.0"], "multilinguality": ["monolingual"], "size_categories": [], "source_datasets": [], "task_categories": ["question-answering", "text-retrieval"], "task_ids": ["open-domain-qa"], "pretty_name": "Open-Domain Question Answering Wikipedia Corpora", "tags": []}
TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_ids-open-domain-qa #annotations_creators-no-annotation #multilinguality-monolingual #language-English #license-cc-by-sa-3.0 #region-us 


# Dataset Card for Open-Domain Question Answering Wikipedia Corpora

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data

## Dataset Description

### Dataset Summary

The Wikipedia corpus variants provided can serve as knowledge sources for question-answering systems based on a retriever–reader pipeline. These corpus variants and their corresponding experiments are described further in the paper entitled: 
> Pre-Processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering.


## Dataset Structure

### Data Fields

The dataset consists of passages that have been segmented from Wikipedia articles.
For each passage, the following fields are provided  
- : The passage id in the format of (X#Y) where passages from the same article share the same X, but Y denotes the segment id within the article  
- : The title of the article from where the passage comes
- : The text content of the passage
  
### Data Splits

There are 6 corpus variants in total
- : The original DPR Wikipedia corpus with non-overlapping passages, each 100 words long, from Karpukhin et al.,
> Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, 2020.
- : Our replication of the above corpus
- :  A corpus similar to above i.e. without tables, infoboxes, and lists. Segmentation is done differently, with a passage size of 6 sentences and a stride of 3 sentences. Note, this means that passages are overlapped.
- :  Like wiki-text-6-3, but with a passage size of 8 sentences and a stride of 4 sentences.
- : A corpus with tables, infoboxes, and lists included with a passage size of 6 sentences and a stride of 3 sentences.
- :  Like wiki-all-6-3, but with a passage size of 8 sentences and a stride of 4 sentences.

## Dataset Creation
### Source Data

#### Initial Data Collection and Normalization

We start with downloading the full December 20, 2018 Wikipedia XML dump:  from the Internet Archive: URL This is then Pre-processed by WikiExtractor: URL (making sure to modify the code to include lists as desired and replacing any tables with the string "TABLETOREPLACE") and DrQA: URL (again making sure to modify the code to not remove lists as desired).

We then apply the pre-processing script) we make available in Pyserini to generate the different corpus variants.

# Dataset Card for Open-Domain Question Answering Wikipedia Corpora ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data ## Dataset Description ### Dataset Summary

The Wikipedia corpus variants provided can serve as knowledge sources for question-answering systems based on a retriever–reader pipeline. These corpus variants and their corresponding experiments are described further in the paper entitled: 
> Pre-Processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering. ## Dataset Structure ### Data Fields

The dataset consists of passages that have been segmented from Wikipedia articles.
For each passage, the following fields are provided  
- : The passage id in the format of (X#Y) where passages from the same article share the same X, but Y denotes the segment id within the article  
- : The title of the article from where the passage comes
- : The text content of the passage ### Data Splits

There are 6 corpus variants in total
- : The original DPR Wikipedia corpus with non-overlapping passages, each 100 words long, from Karpukhin et al.,
> Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, 2020.
- : Our replication of the above corpus
- :  A corpus similar to above i.e. without tables, infoboxes, and lists. Segmentation is done differently, with a passage size of 6 sentences and a stride of 3 sentences. Note, this means that passages are overlapped.
- :  Like wiki-text-6-3, but with a passage size of 8 sentences and a stride of 4 sentences.
- : A corpus with tables, infoboxes, and lists included with a passage size of 6 sentences and a stride of 3 sentences.
- :  Like wiki-all-6-3, but with a passage size of 8 sentences and a stride of 4 sentences. ## Dataset Creation ### Source Data #### Initial Data Collection and Normalization

We start with downloading the full December 20, 2018 Wikipedia XML dump:  from the Internet Archive: URL This is then Pre-processed by WikiExtractor: URL (making sure to modify the code to include lists as desired and replacing any tables with the string "TABLETOREPLACE") and DrQA: URL (again making sure to modify the code to not remove lists as desired).

We then apply the pre-processing script) we make available in Pyserini to generate the different corpus variants.
TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_ids-open-domain-qa #annotations_creators-no-annotation #multilinguality-monolingual #language-English #license-cc-by-sa-3.0 #region-us 
 # Dataset Card for Open-Domain Question Answering Wikipedia Corpora ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data ## Dataset Description ### Dataset Summary

The Wikipedia corpus variants provided can serve as knowledge sources for question-answering systems based on a retriever–reader pipeline. These corpus variants and their corresponding experiments are described further in the paper entitled: 
> Pre-Processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering. ## Dataset Structure ### Data Fields

The dataset consists of passages that have been segmented from Wikipedia articles.
For each passage, the following fields are provided  
- : The passage id in the format of (X#Y) where passages from the same article share the same X, but Y denotes the segment id within the article  
- : The title of the article from where the passage comes
- : The text content of the passage ### Data Splits

There are 6 corpus variants in total
- : The original DPR Wikipedia corpus with non-overlapping passages, each 100 words long, from Karpukhin et al.,
> Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, 2020.
- : Our replication of the above corpus
- :  A corpus similar to above i.e. without tables, infoboxes, and lists. Segmentation is done differently, with a passage size of 6 sentences and a stride of 3 sentences. Note, this means that passages are overlapped.
- :  Like wiki-text-6-3, but with a passage size of 8 sentences and a stride of 4 sentences.
- : A corpus with tables, infoboxes, and lists included with a passage size of 6 sentences and a stride of 3 sentences.
- :  Like wiki-all-6-3, but with a passage size of 8 sentences and a stride of 4 sentences. ## Dataset Creation ### Source Data #### Initial Data Collection and Normalization

We start with downloading the full December 20, 2018 Wikipedia XML dump:  from the Internet Archive: URL This is then Pre-processed by WikiExtractor: URL (making sure to modify the code to include lists as desired and replacing any tables with the string "TABLETOREPLACE") and DrQA: URL (again making sure to modify the code to not remove lists as desired).

We then apply the pre-processing script) we make available in Pyserini to generate the different corpus variants.
passage: TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_ids-open-domain-qa #annotations_creators-no-annotation #multilinguality-monolingual #language-English #license-cc-by-sa-3.0 #region-us 
# Dataset Card for Open-Domain Question Answering Wikipedia Corpora## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data## Dataset Description### Dataset Summary

The Wikipedia corpus variants provided can serve as knowledge sources for question-answering systems based on a retriever–reader pipeline. These corpus variants and their corresponding experiments are described further in the paper entitled: 
> Pre-Processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering.## Dataset Structure### Data Fields

The dataset consists of passages that have been segmented from Wikipedia articles.
For each passage, the following fields are provided  
- : The passage id in the format of (X#Y) where passages from the same article share the same X, but Y denotes the segment id within the article  
- : The title of the article from where the passage comes
- : The text content of the passage