
# Dataset Card for "Wikiomnia"

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
- [Dataset Structure](#dataset-structure)
- [Dataset Creation](#dataset-creation)
- [Additional Information](#additional-information)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)
 
## Dataset Description

- **Repository:** [https://github.com/RussianNLP](https://github.com/RussianNLP)
- **Paper:** [WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia](https://arxiv.org/abs/2204.08009)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Dataset Summary
We present the WikiOmnia dataset, a new publicly available set of QA-pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generative pipeline. The dataset includes every available article from Wikipedia for the Russian language. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

WikiOmnia consists of 2 parts:
1. the voluminous, automatically generated part: 15,9 million triplets consisting of the original article summary, a corresponding generated question and a generated answer;
2. the filtered part: the subsample of 3,5 million triplets, fully verified with automatic means

Wikiomnia adheres to a standard SQuAD format problem, resulting in triplets "text paragraph - question based on paragraph - answer from the paragraph", see the following example:

**Original Wikipedia paragraph**: Коити Масимо (яп. Масимо Ко:ити) — известный режиссёр аниме и основатель японской анимационной студии Bee Train. С
момента основания студии он руководит производством почти всех её картин, а также время от времени принимает участие в работе над анимацией и музыкой.

**English translation**: Koichi Mashimo is a famous anime director and the founder of the Japanese animation studio Bee Train. Since the creation of the studio, he directed almost all studio’s works, and he
also sometimes participates in art and sound tasks. 

**Generated question (ruT5)**: Кто является основателем японской анимационной студии Bee Train?

**Generated answer (ruT5)**: Коити Масимо 

**English QA translation**: Who is the founder of the Japanese animation studio Bee Train? Koichi Mashimo 

## Dataset Creation

Models used for dataset generation:
 - [ruT5](https://huggingface.co/sberbank-ai/ruT5-large) large fine-tuned on SberQuaD
 - [ruGPT-3](https://huggingface.co/sberbank-ai/rugpt3xl) XL fine-tuned on SberQuaD
 - [ruBERT](http://docs.deeppavlov.ai/en/master/features/models/squad.html) DeepPavlov tuned for QA tasks

Source: Wikipedia version March 2021

Special tokens: <[TEXT]>, <[QUESTION]>, <[ANSWER]>

The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-
large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

![](https://raw.githubusercontent.com/RussianNLP/WikiOmnia/main/wikomnia_pipeline.png)

## Additional Information

### Licensing Information

[Apache 2.0 license](https://github.com/RussianNLP/WikiOmnia/blob/main/LICENSE)

### Citation Information

```
@inproceedings{pisarevskaya-shavrina-2022-wikiomnia,
    title = "{W}iki{O}mnia: filtration and evaluation of the generated {QA} corpus on the whole {R}ussian {W}ikipedia",
    author = "Pisarevskaya, Dina  and
      Shavrina, Tatiana",
    booktitle = "Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.gem-1.10",
    pages = "125--135",
    abstract = "The General QA field has been developing the methodology referencing the Stanford Question answering dataset (SQuAD) as the significant benchmark. Compiling factual questions datasets requires manual annotations, limiting the training data{'}s potential size. We present the WikiOmnia dataset, a new publicly available set of QA pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generation and filtration pipeline. To ensure high quality of generated QA pairs, diverse manual and automated evaluation techniques were applied. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).",
}

```
### Contributions

Thanks to [@Deenochka](https://github.com/deenochka), [@TatianaShavrina](https://github.com/TatianaShavrina)
task_categories:question-answering size_categories:1M<n<10M language:ru license:apache-2.0 wikipedia wikiomnia squad QA arxiv:2204.08009 region:us
{"language": ["ru"], "license": "apache-2.0", "size_categories": ["1M<n<10M"], "task_categories": ["question-answering"], "pretty_name": "WikiOmnia", "dataset_info": [{"config_name": "wikiomnia_ruT5_raw", "features": [{"name": "title", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "summary", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "answer", "dtype": "string"}, {"name": "batch_id", "dtype": "string"}], "splits": [{"name": "dev", "num_bytes": 600356136, "num_examples": 266295}, {"name": "test", "num_bytes": 572651444, "num_examples": 267751}], "download_size": 1204094848, "dataset_size": 1173007580}, {"config_name": "wikiomnia_ruT5_filtered", "features": [{"name": "title", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "summary", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "answer", "dtype": "string"}, {"name": "batch_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4157093224, "num_examples": 2088027}], "download_size": 4278635364, "dataset_size": 4157093224}, {"config_name": "wikiomnia_ruGPT3_filtered", "features": [{"name": "title", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "summary", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "answer", "dtype": "string"}, {"name": "batch_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 338607635, "num_examples": 173314}], "download_size": 348694031, "dataset_size": 338607635}, {"config_name": "wikiomnia_ruGPT3_raw", "features": [{"name": "title", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "summary", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "answer", "dtype": "string"}, {"name": "batch_id", "dtype": "string"}], "splits": [{"name": "train_batch1", "num_bytes": 553204785, "num_examples": 260808}, {"name": "train_batch2", "num_bytes": 542823205, "num_examples": 263599}, {"name": "train_batch3", "num_bytes": 582321994, "num_examples": 269736}, {"name": "train_batch4", "num_bytes": 543315355, "num_examples": 265948}, {"name": "train_batch5", "num_bytes": 513288049, "num_examples": 268466}, {"name": "train_batch6", "num_bytes": 943556173, "num_examples": 512147}, {"name": "train_batch7", "num_bytes": 929464509, "num_examples": 508149}, {"name": "train_batch8", "num_bytes": 915128725, "num_examples": 507559}, {"name": "train_batch9", "num_bytes": 926443048, "num_examples": 504292}, {"name": "train_batch10", "num_bytes": 834958539, "num_examples": 463812}, {"name": "train_batch11", "num_bytes": 509866027, "num_examples": 287770}, {"name": "train_batch12", "num_bytes": 478843738, "num_examples": 271410}, {"name": "train_batch13", "num_bytes": 757068702, "num_examples": 385730}, {"name": "train_batch14", "num_bytes": 575937629, "num_examples": 304110}, {"name": "train_batch15", "num_bytes": 517092031, "num_examples": 277507}, {"name": "train_batch16", "num_bytes": 759363156, "num_examples": 402203}, {"name": "train_batch17", "num_bytes": 860544388, "num_examples": 466572}, {"name": "train_batch18", "num_bytes": 935985528, "num_examples": 518348}, {"name": "train_batch19", "num_bytes": 936782197, "num_examples": 514307}, {"name": "train_batch20", "num_bytes": 874299949, "num_examples": 487238}], "download_size": 14939875008, "dataset_size": 14490287727}, {"config_name": "wikiomnia_ruT5_raw_train", "features": [{"name": "title", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "summary", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "answer", "dtype": "string"}, {"name": "batch_id", "dtype": "string"}], "splits": [{"name": "train_batch3", "num_bytes": 612693602, "num_examples": 271391}, {"name": "train_batch4", "num_bytes": 570286147, "num_examples": 265947}, {"name": "train_batch5", "num_bytes": 552502041, "num_examples": 274650}, {"name": "train_batch6", "num_bytes": 1017066184, "num_examples": 525224}, {"name": "train_batch7", "num_bytes": 972351430, "num_examples": 509615}, {"name": "train_batch8", "num_bytes": 973314180, "num_examples": 516828}, {"name": "train_batch9", "num_bytes": 981651841, "num_examples": 512709}, {"name": "train_batch10", "num_bytes": 880664685, "num_examples": 469512}, {"name": "train_batch11", "num_bytes": 543971388, "num_examples": 294631}, {"name": "train_batch12", "num_bytes": 503939060, "num_examples": 273526}, {"name": "train_batch13", "num_bytes": 794421530, "num_examples": 392021}, {"name": "train_batch14", "num_bytes": 610815879, "num_examples": 311452}, {"name": "train_batch15", "num_bytes": 540225492, "num_examples": 278677}, {"name": "train_batch16", "num_bytes": 804003566, "num_examples": 411192}, {"name": "train_batch17", "num_bytes": 903347135, "num_examples": 469871}, {"name": "train_batch18", "num_bytes": 995239085, "num_examples": 528301}, {"name": "train_batch19", "num_bytes": 1003402360, "num_examples": 522264}, {"name": "train_batch20", "num_bytes": 948137237, "num_examples": 499866}], "download_size": 14634332336, "dataset_size": 14208032842}], "tags": ["wikipedia", "wikiomnia", "squad", "QA"]}
TAGS
#task_categories-question-answering #size_categories-1M<n<10M #language-Russian #license-apache-2.0 #wikipedia #wikiomnia #squad #QA #arxiv-2204.08009 #region-us 


# Dataset Card for "Wikiomnia"

## Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
- Dataset Creation
- Additional Information
  - Licensing Information
  - Citation Information
  - Contributions
 
## Dataset Description

- Repository: URL
- Paper: WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia
- Point of Contact: 

### Dataset Summary
We present the WikiOmnia dataset, a new publicly available set of QA-pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generative pipeline. The dataset includes every available article from Wikipedia for the Russian language. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

WikiOmnia consists of 2 parts:
1. the voluminous, automatically generated part: 15,9 million triplets consisting of the original article summary, a corresponding generated question and a generated answer;
2. the filtered part: the subsample of 3,5 million triplets, fully verified with automatic means

Wikiomnia adheres to a standard SQuAD format problem, resulting in triplets "text paragraph - question based on paragraph - answer from the paragraph", see the following example:

Original Wikipedia paragraph: Коити Масимо (яп. Масимо Ко:ити) — известный режиссёр аниме и основатель японской анимационной студии Bee Train. С
момента основания студии он руководит производством почти всех её картин, а также время от времени принимает участие в работе над анимацией и музыкой.

English translation: Koichi Mashimo is a famous anime director and the founder of the Japanese animation studio Bee Train. Since the creation of the studio, he directed almost all studio’s works, and he
also sometimes participates in art and sound tasks. 

Generated question (ruT5): Кто является основателем японской анимационной студии Bee Train?

Generated answer (ruT5): Коити Масимо 

English QA translation: Who is the founder of the Japanese animation studio Bee Train? Koichi Mashimo 

## Dataset Creation

Models used for dataset generation:
 - ruT5 large fine-tuned on SberQuaD
 - ruGPT-3 XL fine-tuned on SberQuaD
 - ruBERT DeepPavlov tuned for QA tasks

Source: Wikipedia version March 2021

Special tokens: <[TEXT]>, <[QUESTION]>, <[ANSWER]>

The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-
large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

![](URL

## Additional Information

### Licensing Information

Apache 2.0 license




### Contributions

Thanks to @Deenochka, @TatianaShavrina
# Dataset Card for "Wikiomnia" ## Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
- Dataset Creation
- Additional Information
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Repository: URL
- Paper: WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia
- Point of Contact: ### Dataset Summary
We present the WikiOmnia dataset, a new publicly available set of QA-pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generative pipeline. The dataset includes every available article from Wikipedia for the Russian language. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

WikiOmnia consists of 2 parts:
1. the voluminous, automatically generated part: 15,9 million triplets consisting of the original article summary, a corresponding generated question and a generated answer;
2. the filtered part: the subsample of 3,5 million triplets, fully verified with automatic means

Wikiomnia adheres to a standard SQuAD format problem, resulting in triplets "text paragraph - question based on paragraph - answer from the paragraph", see the following example:

Original Wikipedia paragraph: Коити Масимо (яп. Масимо Ко:ити) — известный режиссёр аниме и основатель японской анимационной студии Bee Train. С
момента основания студии он руководит производством почти всех её картин, а также время от времени принимает участие в работе над анимацией и музыкой.

English translation: Koichi Mashimo is a famous anime director and the founder of the Japanese animation studio Bee Train. Since the creation of the studio, he directed almost all studio’s works, and he
also sometimes participates in art and sound tasks. 

Generated question (ruT5): Кто является основателем японской анимационной студии Bee Train?

Generated answer (ruT5): Коити Масимо 

English QA translation: Who is the founder of the Japanese animation studio Bee Train? Koichi Mashimo ## Dataset Creation

Models used for dataset generation:
 - ruT5 large fine-tuned on SberQuaD
 - ruGPT-3 XL fine-tuned on SberQuaD
 - ruBERT DeepPavlov tuned for QA tasks

Source: Wikipedia version March 2021

Special tokens: <[TEXT]>, <[QUESTION]>, <[ANSWER]>

The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-
large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

![](URL ## Additional Information ### Licensing Information

Apache 2.0 license ### Contributions

Thanks to @Deenochka, @TatianaShavrina
TAGS
#task_categories-question-answering #size_categories-1M<n<10M #language-Russian #license-apache-2.0 #wikipedia #wikiomnia #squad #QA #arxiv-2204.08009 #region-us 
 # Dataset Card for "Wikiomnia" ## Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
- Dataset Creation
- Additional Information
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Repository: URL
- Paper: WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia
- Point of Contact: ### Dataset Summary
We present the WikiOmnia dataset, a new publicly available set of QA-pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generative pipeline. The dataset includes every available article from Wikipedia for the Russian language. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

WikiOmnia consists of 2 parts:
1. the voluminous, automatically generated part: 15,9 million triplets consisting of the original article summary, a corresponding generated question and a generated answer;
2. the filtered part: the subsample of 3,5 million triplets, fully verified with automatic means

Wikiomnia adheres to a standard SQuAD format problem, resulting in triplets "text paragraph - question based on paragraph - answer from the paragraph", see the following example:

Original Wikipedia paragraph: Коити Масимо (яп. Масимо Ко:ити) — известный режиссёр аниме и основатель японской анимационной студии Bee Train. С
момента основания студии он руководит производством почти всех её картин, а также время от времени принимает участие в работе над анимацией и музыкой.

English translation: Koichi Mashimo is a famous anime director and the founder of the Japanese animation studio Bee Train. Since the creation of the studio, he directed almost all studio’s works, and he
also sometimes participates in art and sound tasks. 

Generated question (ruT5): Кто является основателем японской анимационной студии Bee Train?

Generated answer (ruT5): Коити Масимо 

English QA translation: Who is the founder of the Japanese animation studio Bee Train? Koichi Mashimo ## Dataset Creation

Models used for dataset generation:
 - ruT5 large fine-tuned on SberQuaD
 - ruGPT-3 XL fine-tuned on SberQuaD
 - ruBERT DeepPavlov tuned for QA tasks

Source: Wikipedia version March 2021

Special tokens: <[TEXT]>, <[QUESTION]>, <[ANSWER]>

The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-
large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

![](URL ## Additional Information ### Licensing Information

Apache 2.0 license ### Contributions

Thanks to @Deenochka, @TatianaShavrina
passage: TAGS
#task_categories-question-answering #size_categories-1M<n<10M #language-Russian #license-apache-2.0 #wikipedia #wikiomnia #squad #QA #arxiv-2204.08009 #region-us 
# Dataset Card for "Wikiomnia"## Table of Contents
- Dataset Description
  - Dataset Summary
- Dataset Structure
- Dataset Creation
- Additional Information
  - Licensing Information
  - Citation Information
  - Contributions## Dataset Description

- Repository: URL
- Paper: WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia
- Point of Contact: