
# Dataset Card for MAUPQA Dataset

## Dataset Description

- **Paper:** [MAUPQA: Massive Automatically-created Polish Question Answering Dataset](https://arxiv.org/abs/2305.05486), [SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://arxiv.org/abs/2309.08469)
- **Point of Contact:** [Piotr Rybak](mailto:piotr.cezary.rybak@gmail.com)


### Dataset Summary

MAUPQA is a collection of 14 datasets for Polish document retrieval. Most of the datasets are either machine-generated or machine-translated from English. Across all datasets, it consists of over 1M questions, 1M positive, and 7M hard-negative question-passage pairs.

### Supported Tasks and Leaderboards

- `document-retrieval`: The dataset can be used to train a model for document retrieval. Success on this task is typically measured by [top-k retrieval accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html) or [NDCG](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html).

### Languages

The text is in Polish, as spoken by the [Internet users](https://github.com/facebookresearch/cc_net), [Polish Wikipedia](https://pl.wikipedia.org/) editors, or is an output of generative or translation models. The BCP-47 code for Polish is pl-PL.

## Dataset Structure

### Data Instances

The dataset consists of over 8 million question-passage pairs. For each instance, there is a `question`, a passage (`passage_title`, `passage_text`), and a boolean indicator if the passage is `relevant` for the given question (i.e. does it contain the answers).

For a small subset of `question` there is also a list of possible `answers` formulated in a natural language, in a way a Polish
speaker would answer the questions.

```
{
  'question_id': 1,
  'question': 'Na którym kontynencie leży państwo Gujana, panie Krzysztofie?',
  'answers': "['W Ameryce Południowej']",
  'passage_title': 'Gujana (ujednoznacznienie)',
  'passage_text': 'Gujana (region) – region Ameryki Południowej Gujana – państwo w Ameryce Południowej Gujana Brytyjska – dawna kolonia brytyjska; obecnie państwo Gujana Gujana Francuska – departament zamorski Francji; dawniej kolonia francuska Gujana Holenderska – dawna kolonia holenderska; obecnie państwo Surinam',
  'relevant': True,
  'passage_source': 'crawling',
  'subset': '1z10'
}
```

### Data Fields

Question-passage pairs:

- `question_id`: an integer id of the question
- `question`: a string containing the question
- `passage_title`: a string containing the title of the Wikipedia article
- `passage_text`: a string containing the passage text as extracted by the human annotator
- `relevant`: a boolean flag representing whether a passage is relevant to the question (i.e. does it contain the answers)
- `annotated_by`: a string containing the name of the annotator who verified the relevance of the pair
- `answers`: a string containing a list of possible short answers to the question
- `passage_source`: a string containing the method of obtaining the passage. One of the following:
  - `manual-annotation`: the question-passage pair was manually annotated
  - `crawling`: the question-passage pairs were created by taking advantage of the specific structure of crawled website
  - `dataset-translation`: the dataset was created by machine-translating the English dataset
  - `generative-model`: the question was created by the generative model based on the given passage
  - `bm25-negatives`: the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is not relevant
  - `bm25-positives`: the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is relevant
- `subset`: a string containing the name of the dataset

### Data Splits

MAUPQA is a collection of 14 datasets and most of them are weakly labeled. Therefore, the intended use of MAUPQA is for training only. As such, all examples belong to a single `train` split. We recommend using the [PolQA](https://huggingface.co/datasets/ipipan/polqa) dataset for evaluation.

Basic statistics of all 14 datasets:

| dataset           | # questions | # answers | # positive passages | # negative passages |
|-------------------|------------:|----------:|--------------------:|--------------------:|
| 1z10              |      22,835 |    21,415 |              22,014 |             139,471 |
| czy-wiesz-v2      |      29,078 |         - |              29,078 |             143,306 |
| gpt3-cc           |      10,146 |    10,146 |              10,177 |              89,203 |
| gpt3.5-cc         |      29,591 |    29,583 |              29,720 |             251,959 |
| gpt3.5-wiki       |      29,674 |    29,636 |              29,748 |             115,564 |
| mkqa              |       4,036 |     4,036 |               3,968 |              19,814 |
| mqa               |     172,768 |         - |             178,131 |           1,249,659 |
| msmarco           |     389,987 |         - |             416,763 |           3,006,996 |
| multilingual-NLI  |     100,752 |    64,900 |              68,096 |             743,857 |
| nq                |     135,781 |         - |             139,976 |             797,436 |
| poleval2021-pairs |       1,977 |         - |               2,088 |              17,608 |
| poquad            |      56,588 |    46,157 |              46,187 |             299,865 |
| templates         |      15,993 |    14,504 |              15,993 |              45,228 |
| wiki-def          |      18,093 |    18,092 |              18,093 |              84,956 |
| Total             |   1,017,299 |   238,469 |           1,010,032 |           7,004,922 |


## Dataset Creation

### Curation Rationale

Open-domain question answering systems rely heavily on annotated datasets to train neural document retrievers. However, manually annotating such datasets is both difficult and time-consuming. To overcome these difficulties, we experimented with several methods for automatically collecting weakly labeled datasets. As a result, MAUPQA enables the development of robust document retrieval systems for Polish.

### Source Data

#### Initial Data Collection and Normalization

Below, we briefly describe each dataset. For a detailed description please refer to the [paper](https://arxiv.org/abs/2305.05486).

* `1z10`: We transcribe 333 recordings of the [Jeden z Dziesięciu](https://pl.wikipedia.org/wiki/Jeden_z_dziesi%C4%99ciu) TV show using the Whisper model and extract the question-answer pairs using GPT-3.5 model. We use the BM25 retriever and the GPT-3.5-based cross-encoder to match questions with Wikipedia passages.
* `czy-wiesz-v2`:  We first crawl all questions from the [Did you know?](https://pl.wikipedia.org/wiki/Wikiprojekt:Czy_wiesz/archiwum) section on Polish Wikipedia together with a link to the relevant Wikipedia article. Then, we use the [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to choose the most relevant passage.
* `gpt3-cc`: We sample random passages from [CCNet](https://github.com/facebookresearch/cc_net) corpus and prompt GPT-3 to generate a relevant question. 
* `gpt3.5-cc`: We sample random passages from [CCNet](https://github.com/facebookresearch/cc_net) corpus and prompt GPT-3.5 to generate a relevant question. 
* `gpt3.5-wiki`: We sample random passages from Polish Wikipedia and prompt GPT-3.5 to generate a relevant question. 
* `mkqa`: We clean the Polish subset of the [MKQA](https://huggingface.co/datasets/mkqa) dataset by removing questions without answers, requiring long answers (*Why?* and *How?* questions), and ambiguous ones ("Who is the *current* president?*). We use the BM25 retriever and the [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to choose the most relevant passage.
* `mqa`: We clean the Polish subset of the [MQA](https://huggingface.co/datasets/clips/mqa) dataset by removing artificially created questions like "What is the best hotel in *{city}*?" for hundreds of different *cities*. To clean the dataset, we cluster lexically similar questions/passages and remove clusters with over 5 questions.
* `msmarco`: We translate the [MS MARCO](https://huggingface.co/datasets/ms_marco) dataset into Polish using the machine translation model.
* `multilingual-NLI`: We extract question-answer pairs from the Polish subset of the [multilingual-NLI](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7) dataset. We create questions using the following template: "Czy *{premise}*?" (Eng. "Does *{premise}*?") and use hypotheses as passages. We consider `entailment` and `contradiction` labels as relevant and `neutral` as negative.
* `nq`: We translate the [NQ](https://huggingface.co/datasets/natural_questions) dataset into Polish using the machine translation model.
* `poleval2021-pairs`: We take [allegro/polish-question-passage-pairs](https://huggingface.co/datasets/allegro/polish-question-passage-pairs) without any changes.
* `poquad`: We extract question-passages pairs from the training split of the [PoQuAD](https://huggingface.co/datasets/clarin-pl/poquad) dataset.
* `templates`: We take advantage of the Wikipedia structure to generate questions using predefined templates. For example, list pages group together similar entities (e.g. "Writers born in Poland") which allow generating questions like "Where was *{writer name}* born?". In total, we use 33 templates to generate questions. We use the [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to choose the most relevant passage from the linked article.
* `wiki-def`: We use [Wiktionary](https://www.wiktionary.org/) to generate questions based on word definitions. We use definitions that have links to Wikipedia articles to create the question-passage pairs. For example, the definition of "Monday" is "the first day of the week". Based on it, we generate the question "What is the name of *the first day of the week*?".

Additionally, we extend each dataset by sampling the hard negative passages using a BM25 retriever and score using a [multilingual cross-encoder](https://huggingface.co/unicamp-dl/mMiniLM-L6-v2-mmarco-v2) to ensure that passages are not relevant.

#### Who are the source language producers?

The text is in Polish, as spoken by the [Internet users](https://github.com/facebookresearch/cc_net), [Polish Wikipedia](https://pl.wikipedia.org/) editors, or is an output of generative or translation models.

### Annotations

#### Annotation process

The MAUPQA dataset doesn't provide additional annotation except for the annotation present in the source datasets.

#### Who are the annotators?

Please refer to the description of the source datasets.

### Personal and Sensitive Information

The dataset should not contain any personal or sensitive information. However, we use the [CCNet](https://github.com/facebookresearch/cc_net) dataset as a source of passages that we didn't manually inspect for personal and sensitive information.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset was created to promote the research in the open-domain question answering for Polish and allow developing question answering systems.

### Discussion of Biases

The machine-translated datasets might not represent the natural language as used by native Polish speakers. Similarly, the questions generated by the generative models might not be representative or correct.

Most of the question-passage pairs are created automatically using the BM25 retriever and as such it is biased to lexically similar pairs.

### Other Known Limitations

The MAUPQA dataset is mostly automatically generated and can therefore contain a high proportion of noise and incorrectly labeled question-passage pairs.

## Additional Information

### Dataset Curators

The MAUPQA dataset was collected by Piotr Rybak and Maciej Ogrodniczuk from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/) but the source datasets were created by many more researchers. Please refer to the original dataset descriptions for the full authorship.

This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19.

### Licensing Information

CC BY-SA 4.0

### Citation Information

```
@inproceedings{rybak-2023-maupqa,
    title = "{MAUPQA}: Massive Automatically-created {P}olish Question Answering Dataset",
    author = "Rybak, Piotr",
    booktitle = "Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.bsnlp-1.2",
    pages = "11--16",
    abstract = "Recently, open-domain question answering systems have begun to rely heavily on annotated datasets to train neural passage retrievers. However, manually annotating such datasets is both difficult and time-consuming, which limits their availability for less popular languages. In this work, we experiment with several methods for automatically collecting weakly labeled datasets and show how they affect the performance of the neural passage retrieval models. As a result of our work, we publish the MAUPQA dataset, consisting of nearly 400,000 question-passage pairs for Polish, as well as the HerBERT-QA neural retriever.",
}
```

```
@misc{rybak2023silverretriever,
    title={SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering}, 
    author={Piotr Rybak and Maciej Ogrodniczuk},
    year={2023},
    eprint={2309.08469},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```
task_categories:question-answering task_categories:text-retrieval task_ids:open-domain-qa task_ids:document-retrieval annotations_creators:found annotations_creators:machine-generated size_categories:1M<n<10M language:pl license:cc-by-sa-4.0 arxiv:2305.05486 arxiv:2309.08469 region:us
{"annotations_creators": ["found", "machine-generated"], "language": ["pl"], "license": "cc-by-sa-4.0", "size_categories": ["1M<n<10M"], "task_categories": ["question-answering", "text-retrieval"], "task_ids": ["open-domain-qa", "document-retrieval"], "pretty_name": "MAUPQA"}
TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_ids-open-domain-qa #task_ids-document-retrieval #annotations_creators-found #annotations_creators-machine-generated #size_categories-1M<n<10M #language-Polish #license-cc-by-sa-4.0 #arxiv-2305.05486 #arxiv-2309.08469 #region-us 

Dataset Card for MAUPQA Dataset
===============================


Dataset Description
-------------------


* Paper: MAUPQA: Massive Automatically-created Polish Question Answering Dataset, SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering
* Point of Contact: Piotr Rybak


### Dataset Summary


MAUPQA is a collection of 14 datasets for Polish document retrieval. Most of the datasets are either machine-generated or machine-translated from English. Across all datasets, it consists of over 1M questions, 1M positive, and 7M hard-negative question-passage pairs.


### Supported Tasks and Leaderboards


* 'document-retrieval': The dataset can be used to train a model for document retrieval. Success on this task is typically measured by top-k retrieval accuracy or NDCG.


### Languages


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models. The BCP-47 code for Polish is pl-PL.


Dataset Structure
-----------------


### Data Instances


The dataset consists of over 8 million question-passage pairs. For each instance, there is a 'question', a passage ('passage\_title', 'passage\_text'), and a boolean indicator if the passage is 'relevant' for the given question (i.e. does it contain the answers).


For a small subset of 'question' there is also a list of possible 'answers' formulated in a natural language, in a way a Polish
speaker would answer the questions.


### Data Fields


Question-passage pairs:


* 'question\_id': an integer id of the question
* 'question': a string containing the question
* 'passage\_title': a string containing the title of the Wikipedia article
* 'passage\_text': a string containing the passage text as extracted by the human annotator
* 'relevant': a boolean flag representing whether a passage is relevant to the question (i.e. does it contain the answers)
* 'annotated\_by': a string containing the name of the annotator who verified the relevance of the pair
* 'answers': a string containing a list of possible short answers to the question
* 'passage\_source': a string containing the method of obtaining the passage. One of the following:
	+ 'manual-annotation': the question-passage pair was manually annotated
	+ 'crawling': the question-passage pairs were created by taking advantage of the specific structure of crawled website
	+ 'dataset-translation': the dataset was created by machine-translating the English dataset
	+ 'generative-model': the question was created by the generative model based on the given passage
	+ 'bm25-negatives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is not relevant
	+ 'bm25-positives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is relevant
* 'subset': a string containing the name of the dataset


### Data Splits


MAUPQA is a collection of 14 datasets and most of them are weakly labeled. Therefore, the intended use of MAUPQA is for training only. As such, all examples belong to a single 'train' split. We recommend using the PolQA dataset for evaluation.


Basic statistics of all 14 datasets:



Dataset Creation
----------------


### Curation Rationale


Open-domain question answering systems rely heavily on annotated datasets to train neural document retrievers. However, manually annotating such datasets is both difficult and time-consuming. To overcome these difficulties, we experimented with several methods for automatically collecting weakly labeled datasets. As a result, MAUPQA enables the development of robust document retrieval systems for Polish.


### Source Data


#### Initial Data Collection and Normalization


Below, we briefly describe each dataset. For a detailed description please refer to the paper.


* '1z10': We transcribe 333 recordings of the Jeden z Dziesięciu TV show using the Whisper model and extract the question-answer pairs using GPT-3.5 model. We use the BM25 retriever and the GPT-3.5-based cross-encoder to match questions with Wikipedia passages.
* 'czy-wiesz-v2': We first crawl all questions from the Did you know? section on Polish Wikipedia together with a link to the relevant Wikipedia article. Then, we use the multilingual cross-encoder to choose the most relevant passage.
* 'gpt3-cc': We sample random passages from CCNet corpus and prompt GPT-3 to generate a relevant question.
* 'gpt3.5-cc': We sample random passages from CCNet corpus and prompt GPT-3.5 to generate a relevant question.
* 'gpt3.5-wiki': We sample random passages from Polish Wikipedia and prompt GPT-3.5 to generate a relevant question.
* 'mkqa': We clean the Polish subset of the MKQA dataset by removing questions without answers, requiring long answers (*Why?* and *How?* questions), and ambiguous ones ("Who is the *current* president?\*). We use the BM25 retriever and the multilingual cross-encoder to choose the most relevant passage.
* 'mqa': We clean the Polish subset of the MQA dataset by removing artificially created questions like "What is the best hotel in *{city}*?" for hundreds of different *cities*. To clean the dataset, we cluster lexically similar questions/passages and remove clusters with over 5 questions.
* 'msmarco': We translate the MS MARCO dataset into Polish using the machine translation model.
* 'multilingual-NLI': We extract question-answer pairs from the Polish subset of the multilingual-NLI dataset. We create questions using the following template: "Czy *{premise}*?" (Eng. "Does *{premise}*?") and use hypotheses as passages. We consider 'entailment' and 'contradiction' labels as relevant and 'neutral' as negative.
* 'nq': We translate the NQ dataset into Polish using the machine translation model.
* 'poleval2021-pairs': We take allegro/polish-question-passage-pairs without any changes.
* 'poquad': We extract question-passages pairs from the training split of the PoQuAD dataset.
* 'templates': We take advantage of the Wikipedia structure to generate questions using predefined templates. For example, list pages group together similar entities (e.g. "Writers born in Poland") which allow generating questions like "Where was *{writer name}* born?". In total, we use 33 templates to generate questions. We use the multilingual cross-encoder to choose the most relevant passage from the linked article.
* 'wiki-def': We use Wiktionary to generate questions based on word definitions. We use definitions that have links to Wikipedia articles to create the question-passage pairs. For example, the definition of "Monday" is "the first day of the week". Based on it, we generate the question "What is the name of *the first day of the week*?".


Additionally, we extend each dataset by sampling the hard negative passages using a BM25 retriever and score using a multilingual cross-encoder to ensure that passages are not relevant.


#### Who are the source language producers?


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models.


### Annotations


#### Annotation process


The MAUPQA dataset doesn't provide additional annotation except for the annotation present in the source datasets.


#### Who are the annotators?


Please refer to the description of the source datasets.


### Personal and Sensitive Information


The dataset should not contain any personal or sensitive information. However, we use the CCNet dataset as a source of passages that we didn't manually inspect for personal and sensitive information.


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


This dataset was created to promote the research in the open-domain question answering for Polish and allow developing question answering systems.


### Discussion of Biases


The machine-translated datasets might not represent the natural language as used by native Polish speakers. Similarly, the questions generated by the generative models might not be representative or correct.


Most of the question-passage pairs are created automatically using the BM25 retriever and as such it is biased to lexically similar pairs.


### Other Known Limitations


The MAUPQA dataset is mostly automatically generated and can therefore contain a high proportion of noise and incorrectly labeled question-passage pairs.


Additional Information
----------------------


### Dataset Curators


The MAUPQA dataset was collected by Piotr Rybak and Maciej Ogrodniczuk from the Institute of Computer Science, Polish Academy of Sciences but the source datasets were created by many more researchers. Please refer to the original dataset descriptions for the full authorship.


This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19.


### Licensing Information


CC BY-SA 4.0



### Dataset Summary


MAUPQA is a collection of 14 datasets for Polish document retrieval. Most of the datasets are either machine-generated or machine-translated from English. Across all datasets, it consists of over 1M questions, 1M positive, and 7M hard-negative question-passage pairs. ### Supported Tasks and Leaderboards


* 'document-retrieval': The dataset can be used to train a model for document retrieval. Success on this task is typically measured by top-k retrieval accuracy or NDCG. ### Languages


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models. The BCP-47 code for Polish is pl-PL.


Dataset Structure
----------------- ### Data Instances


The dataset consists of over 8 million question-passage pairs. For each instance, there is a 'question', a passage ('passage\_title', 'passage\_text'), and a boolean indicator if the passage is 'relevant' for the given question (i.e. does it contain the answers).


For a small subset of 'question' there is also a list of possible 'answers' formulated in a natural language, in a way a Polish
speaker would answer the questions. ### Data Fields


Question-passage pairs:


* 'question\_id': an integer id of the question
* 'question': a string containing the question
* 'passage\_title': a string containing the title of the Wikipedia article
* 'passage\_text': a string containing the passage text as extracted by the human annotator
* 'relevant': a boolean flag representing whether a passage is relevant to the question (i.e. does it contain the answers)
* 'annotated\_by': a string containing the name of the annotator who verified the relevance of the pair
* 'answers': a string containing a list of possible short answers to the question
* 'passage\_source': a string containing the method of obtaining the passage. One of the following:
	+ 'manual-annotation': the question-passage pair was manually annotated
	+ 'crawling': the question-passage pairs were created by taking advantage of the specific structure of crawled website
	+ 'dataset-translation': the dataset was created by machine-translating the English dataset
	+ 'generative-model': the question was created by the generative model based on the given passage
	+ 'bm25-negatives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is not relevant
	+ 'bm25-positives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is relevant
* 'subset': a string containing the name of the dataset ### Data Splits


MAUPQA is a collection of 14 datasets and most of them are weakly labeled. Therefore, the intended use of MAUPQA is for training only. As such, all examples belong to a single 'train' split. We recommend using the PolQA dataset for evaluation.


Basic statistics of all 14 datasets:



Dataset Creation
---------------- ### Curation Rationale


Open-domain question answering systems rely heavily on annotated datasets to train neural document retrievers. However, manually annotating such datasets is both difficult and time-consuming. To overcome these difficulties, we experimented with several methods for automatically collecting weakly labeled datasets. As a result, MAUPQA enables the development of robust document retrieval systems for Polish. ### Source Data #### Initial Data Collection and Normalization


Below, we briefly describe each dataset. For a detailed description please refer to the paper.


* '1z10': We transcribe 333 recordings of the Jeden z Dziesięciu TV show using the Whisper model and extract the question-answer pairs using GPT-3.5 model. We use the BM25 retriever and the GPT-3.5-based cross-encoder to match questions with Wikipedia passages.
* 'czy-wiesz-v2': We first crawl all questions from the Did you know? section on Polish Wikipedia together with a link to the relevant Wikipedia article. Then, we use the multilingual cross-encoder to choose the most relevant passage.
* 'gpt3-cc': We sample random passages from CCNet corpus and prompt GPT-3 to generate a relevant question.
* 'gpt3.5-cc': We sample random passages from CCNet corpus and prompt GPT-3.5 to generate a relevant question.
* 'gpt3.5-wiki': We sample random passages from Polish Wikipedia and prompt GPT-3.5 to generate a relevant question.
* 'mkqa': We clean the Polish subset of the MKQA dataset by removing questions without answers, requiring long answers (*Why?* and *How?* questions), and ambiguous ones ("Who is the *current* president?\*). We use the BM25 retriever and the multilingual cross-encoder to choose the most relevant passage.
* 'mqa': We clean the Polish subset of the MQA dataset by removing artificially created questions like "What is the best hotel in *{city}*?" for hundreds of different *cities*. To clean the dataset, we cluster lexically similar questions/passages and remove clusters with over 5 questions.
* 'msmarco': We translate the MS MARCO dataset into Polish using the machine translation model.
* 'multilingual-NLI': We extract question-answer pairs from the Polish subset of the multilingual-NLI dataset. We create questions using the following template: "Czy *{premise}*?" (Eng. "Does *{premise}*?") and use hypotheses as passages. We consider 'entailment' and 'contradiction' labels as relevant and 'neutral' as negative.
* 'nq': We translate the NQ dataset into Polish using the machine translation model.
* 'poleval2021-pairs': We take allegro/polish-question-passage-pairs without any changes.
* 'poquad': We extract question-passages pairs from the training split of the PoQuAD dataset.
* 'templates': We take advantage of the Wikipedia structure to generate questions using predefined templates. For example, list pages group together similar entities (e.g. "Writers born in Poland") which allow generating questions like "Where was *{writer name}* born?". In total, we use 33 templates to generate questions. We use the multilingual cross-encoder to choose the most relevant passage from the linked article.
* 'wiki-def': We use Wiktionary to generate questions based on word definitions. We use definitions that have links to Wikipedia articles to create the question-passage pairs. For example, the definition of "Monday" is "the first day of the week". Based on it, we generate the question "What is the name of *the first day of the week*?".


Additionally, we extend each dataset by sampling the hard negative passages using a BM25 retriever and score using a multilingual cross-encoder to ensure that passages are not relevant. #### Who are the source language producers?


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models. ### Annotations #### Annotation process


The MAUPQA dataset doesn't provide additional annotation except for the annotation present in the source datasets. #### Who are the annotators?


Please refer to the description of the source datasets. ### Personal and Sensitive Information


The dataset should not contain any personal or sensitive information. However, we use the CCNet dataset as a source of passages that we didn't manually inspect for personal and sensitive information.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


This dataset was created to promote the research in the open-domain question answering for Polish and allow developing question answering systems. ### Discussion of Biases


The machine-translated datasets might not represent the natural language as used by native Polish speakers. Similarly, the questions generated by the generative models might not be representative or correct.


Most of the question-passage pairs are created automatically using the BM25 retriever and as such it is biased to lexically similar pairs. ### Other Known Limitations


The MAUPQA dataset is mostly automatically generated and can therefore contain a high proportion of noise and incorrectly labeled question-passage pairs.


Additional Information
---------------------- ### Dataset Curators


The MAUPQA dataset was collected by Piotr Rybak and Maciej Ogrodniczuk from the Institute of Computer Science, Polish Academy of Sciences but the source datasets were created by many more researchers. Please refer to the original dataset descriptions for the full authorship.


This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19. ### Licensing Information


CC BY-SA 4.0
TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_ids-open-domain-qa #task_ids-document-retrieval #annotations_creators-found #annotations_creators-machine-generated #size_categories-1M<n<10M #language-Polish #license-cc-by-sa-4.0 #arxiv-2305.05486 #arxiv-2309.08469 #region-us 
 ### Dataset Summary


MAUPQA is a collection of 14 datasets for Polish document retrieval. Most of the datasets are either machine-generated or machine-translated from English. Across all datasets, it consists of over 1M questions, 1M positive, and 7M hard-negative question-passage pairs. ### Supported Tasks and Leaderboards


* 'document-retrieval': The dataset can be used to train a model for document retrieval. Success on this task is typically measured by top-k retrieval accuracy or NDCG. ### Languages


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models. The BCP-47 code for Polish is pl-PL.


Dataset Structure
----------------- ### Data Instances


The dataset consists of over 8 million question-passage pairs. For each instance, there is a 'question', a passage ('passage\_title', 'passage\_text'), and a boolean indicator if the passage is 'relevant' for the given question (i.e. does it contain the answers).


For a small subset of 'question' there is also a list of possible 'answers' formulated in a natural language, in a way a Polish
speaker would answer the questions. ### Data Fields


Question-passage pairs:


* 'question\_id': an integer id of the question
* 'question': a string containing the question
* 'passage\_title': a string containing the title of the Wikipedia article
* 'passage\_text': a string containing the passage text as extracted by the human annotator
* 'relevant': a boolean flag representing whether a passage is relevant to the question (i.e. does it contain the answers)
* 'annotated\_by': a string containing the name of the annotator who verified the relevance of the pair
* 'answers': a string containing a list of possible short answers to the question
* 'passage\_source': a string containing the method of obtaining the passage. One of the following:
	+ 'manual-annotation': the question-passage pair was manually annotated
	+ 'crawling': the question-passage pairs were created by taking advantage of the specific structure of crawled website
	+ 'dataset-translation': the dataset was created by machine-translating the English dataset
	+ 'generative-model': the question was created by the generative model based on the given passage
	+ 'bm25-negatives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is not relevant
	+ 'bm25-positives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is relevant
* 'subset': a string containing the name of the dataset ### Data Splits


MAUPQA is a collection of 14 datasets and most of them are weakly labeled. Therefore, the intended use of MAUPQA is for training only. As such, all examples belong to a single 'train' split. We recommend using the PolQA dataset for evaluation.


Basic statistics of all 14 datasets:



Dataset Creation
---------------- ### Curation Rationale


Open-domain question answering systems rely heavily on annotated datasets to train neural document retrievers. However, manually annotating such datasets is both difficult and time-consuming. To overcome these difficulties, we experimented with several methods for automatically collecting weakly labeled datasets. As a result, MAUPQA enables the development of robust document retrieval systems for Polish. ### Source Data #### Initial Data Collection and Normalization


Below, we briefly describe each dataset. For a detailed description please refer to the paper.


* '1z10': We transcribe 333 recordings of the Jeden z Dziesięciu TV show using the Whisper model and extract the question-answer pairs using GPT-3.5 model. We use the BM25 retriever and the GPT-3.5-based cross-encoder to match questions with Wikipedia passages.
* 'czy-wiesz-v2': We first crawl all questions from the Did you know? section on Polish Wikipedia together with a link to the relevant Wikipedia article. Then, we use the multilingual cross-encoder to choose the most relevant passage.
* 'gpt3-cc': We sample random passages from CCNet corpus and prompt GPT-3 to generate a relevant question.
* 'gpt3.5-cc': We sample random passages from CCNet corpus and prompt GPT-3.5 to generate a relevant question.
* 'gpt3.5-wiki': We sample random passages from Polish Wikipedia and prompt GPT-3.5 to generate a relevant question.
* 'mkqa': We clean the Polish subset of the MKQA dataset by removing questions without answers, requiring long answers (*Why?* and *How?* questions), and ambiguous ones ("Who is the *current* president?\*). We use the BM25 retriever and the multilingual cross-encoder to choose the most relevant passage.
* 'mqa': We clean the Polish subset of the MQA dataset by removing artificially created questions like "What is the best hotel in *{city}*?" for hundreds of different *cities*. To clean the dataset, we cluster lexically similar questions/passages and remove clusters with over 5 questions.
* 'msmarco': We translate the MS MARCO dataset into Polish using the machine translation model.
* 'multilingual-NLI': We extract question-answer pairs from the Polish subset of the multilingual-NLI dataset. We create questions using the following template: "Czy *{premise}*?" (Eng. "Does *{premise}*?") and use hypotheses as passages. We consider 'entailment' and 'contradiction' labels as relevant and 'neutral' as negative.
* 'nq': We translate the NQ dataset into Polish using the machine translation model.
* 'poleval2021-pairs': We take allegro/polish-question-passage-pairs without any changes.
* 'poquad': We extract question-passages pairs from the training split of the PoQuAD dataset.
* 'templates': We take advantage of the Wikipedia structure to generate questions using predefined templates. For example, list pages group together similar entities (e.g. "Writers born in Poland") which allow generating questions like "Where was *{writer name}* born?". In total, we use 33 templates to generate questions. We use the multilingual cross-encoder to choose the most relevant passage from the linked article.
* 'wiki-def': We use Wiktionary to generate questions based on word definitions. We use definitions that have links to Wikipedia articles to create the question-passage pairs. For example, the definition of "Monday" is "the first day of the week". Based on it, we generate the question "What is the name of *the first day of the week*?".


Additionally, we extend each dataset by sampling the hard negative passages using a BM25 retriever and score using a multilingual cross-encoder to ensure that passages are not relevant. #### Who are the source language producers?


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models. ### Annotations #### Annotation process


The MAUPQA dataset doesn't provide additional annotation except for the annotation present in the source datasets. #### Who are the annotators?


Please refer to the description of the source datasets. ### Personal and Sensitive Information


The dataset should not contain any personal or sensitive information. However, we use the CCNet dataset as a source of passages that we didn't manually inspect for personal and sensitive information.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


This dataset was created to promote the research in the open-domain question answering for Polish and allow developing question answering systems. ### Discussion of Biases


The machine-translated datasets might not represent the natural language as used by native Polish speakers. Similarly, the questions generated by the generative models might not be representative or correct.


Most of the question-passage pairs are created automatically using the BM25 retriever and as such it is biased to lexically similar pairs. ### Other Known Limitations


The MAUPQA dataset is mostly automatically generated and can therefore contain a high proportion of noise and incorrectly labeled question-passage pairs.


Additional Information
---------------------- ### Dataset Curators


The MAUPQA dataset was collected by Piotr Rybak and Maciej Ogrodniczuk from the Institute of Computer Science, Polish Academy of Sciences but the source datasets were created by many more researchers. Please refer to the original dataset descriptions for the full authorship.


This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19. ### Licensing Information


CC BY-SA 4.0
passage: TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_ids-open-domain-qa #task_ids-document-retrieval #annotations_creators-found #annotations_creators-machine-generated #size_categories-1M<n<10M #language-Polish #license-cc-by-sa-4.0 #arxiv-2305.05486 #arxiv-2309.08469 #region-us 
### Dataset Summary


MAUPQA is a collection of 14 datasets for Polish document retrieval. Most of the datasets are either machine-generated or machine-translated from English. Across all datasets, it consists of over 1M questions, 1M positive, and 7M hard-negative question-passage pairs.### Supported Tasks and Leaderboards


* 'document-retrieval': The dataset can be used to train a model for document retrieval. Success on this task is typically measured by top-k retrieval accuracy or NDCG.### Languages


The text is in Polish, as spoken by the Internet users, Polish Wikipedia editors, or is an output of generative or translation models. The BCP-47 code for Polish is pl-PL.


Dataset Structure
-----------------### Data Instances


The dataset consists of over 8 million question-passage pairs. For each instance, there is a 'question', a passage ('passage\_title', 'passage\_text'), and a boolean indicator if the passage is 'relevant' for the given question (i.e. does it contain the answers).


For a small subset of 'question' there is also a list of possible 'answers' formulated in a natural language, in a way a Polish
speaker would answer the questions. passage: ### Data Fields


Question-passage pairs:


* 'question\_id': an integer id of the question
* 'question': a string containing the question
* 'passage\_title': a string containing the title of the Wikipedia article
* 'passage\_text': a string containing the passage text as extracted by the human annotator
* 'relevant': a boolean flag representing whether a passage is relevant to the question (i.e. does it contain the answers)
* 'annotated\_by': a string containing the name of the annotator who verified the relevance of the pair
* 'answers': a string containing a list of possible short answers to the question
* 'passage\_source': a string containing the method of obtaining the passage. One of the following:
	+ 'manual-annotation': the question-passage pair was manually annotated
	+ 'crawling': the question-passage pairs were created by taking advantage of the specific structure of crawled website
	+ 'dataset-translation': the dataset was created by machine-translating the English dataset
	+ 'generative-model': the question was created by the generative model based on the given passage
	+ 'bm25-negatives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is not relevant
	+ 'bm25-positives': the passage was found by the BM25 retriever and scored using a multilingual cross-encoder to ensure it is relevant
* 'subset': a string containing the name of the dataset### Data Splits


MAUPQA is a collection of 14 datasets and most of them are weakly labeled. Therefore, the intended use of MAUPQA is for training only. As such, all examples belong to a single 'train' split. We recommend using the PolQA dataset for evaluation.


Basic statistics of all 14 datasets:



Dataset Creation
----------------### Curation Rationale


Open-domain question answering systems rely heavily on annotated datasets to train neural document retrievers. However, manually annotating such datasets is both difficult and time-consuming. To overcome these difficulties, we experimented with several methods for automatically collecting weakly labeled datasets. As a result, MAUPQA enables the development of robust document retrieval systems for Polish.### Source Data