
# Dataset Card for TaPaCo Corpus

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages](https://zenodo.org/record/3707949#.X9Dh0cYza3I)
- **Paper:** [TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages](https://www.aclweb.org/anthology/2020.lrec-1.848.pdf)
- **Data:** https://doi.org/10.5281/zenodo.3707949
- **Point of Contact:** [Yves Scherrer](https://blogs.helsinki.fi/yvesscherrer/)

### Dataset Summary
A freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. 
Tatoeba is a crowdsourcing project mainly geared towards language learners. Its aim is to provide example sentences 
and translations for particular linguistic constructions and words. The paraphrase corpus is created by populating a 
graph with Tatoeba sentences and equivalence links between sentences “meaning the same thing”. This graph is then 
traversed to extract sets of paraphrases. Several language-independent filters and pruning steps are applied to 
remove uninteresting sentences. A manual evaluation performed on three languages shows that between half and three 
quarters of inferred paraphrases are correct and that most remaining ones are either correct but trivial, 
or near-paraphrases that neutralize a morphological distinction. The corpus contains a total of 1.9 million 
sentences, with 200 – 250 000 sentences per language. It covers a range of languages for which, to our knowledge,
no other paraphrase dataset exists.

### Supported Tasks and Leaderboards
Paraphrase detection and generation have become popular tasks in NLP
and are increasingly integrated into a wide variety of common downstream tasks such as machine translation
, information retrieval, question answering, and semantic parsing. Most of the existing datasets
 cover only a single language – in most cases English – or a small number of languages. Furthermore, some paraphrase
  datasets focus on lexical and phrasal rather than sentential paraphrases, while others are created (semi
  -)automatically using machine translation.

The number of sentences per language ranges from 200 to 250 000, which makes the dataset
more suitable for fine-tuning and evaluation purposes than
for training. It is well-suited for multi-reference evaluation
of paraphrase generation models, as there is generally not a
single correct way of paraphrasing a given input sentence.

### Languages

The dataset contains paraphrases in Afrikaans, Arabic, Azerbaijani, Belarusian, Berber languages, Bulgarian, Bengali
, Breton, Catalan; Valencian, Chavacano, Mandarin, Czech, Danish, German, Greek, Modern (1453-), English, Esperanto
, Spanish; Castilian, Estonian, Basque, Finnish, French, Galician, Gronings, Hebrew, Hindi, Croatian, Hungarian
, Armenian, Interlingua (International Auxiliary Language Association), Indonesian, Interlingue; Occidental, Ido
, Icelandic, Italian, Japanese, Lojban, Kabyle, Korean, Cornish, Latin, Lingua Franca Nova\t, Lithuanian, Macedonian
, Marathi, Bokmål, Norwegian; Norwegian Bokmål, Low German; Low Saxon; German, Low; Saxon, Low, Dutch; Flemish, ]Old
 Russian, Turkish, Ottoman (1500-1928), Iranian Persian, Polish, Portuguese, Rundi, Romanian; Moldavian; Moldovan, 
 Russian, Slovenian, Serbian, Swedish, Turkmen, Tagalog, Klingon; tlhIngan-Hol, Toki Pona, Turkish, Tatar, 
 Uighur; Uyghur, Ukrainian, Urdu, Vietnamese, Volapük, Waray, Wu Chinese and Yue Chinese

## Dataset Structure

### Data Instances
Each data instance corresponds to a paraphrase, e.g.:
```
{ 
    'paraphrase_set_id': '1483',  
    'sentence_id': '5778896',
    'paraphrase': 'Ɣremt adlis-a.', 
    'lists': ['7546'], 
    'tags': [''],
    'language': 'ber'
}
```

### Data Fields
Each dialogue instance has the following fields:
- `paraphrase_set_id`:  a running number that groups together all sentences that are considered paraphrases of each
 other
- `sentence_id`: OPUS sentence id
- `paraphrase`: Sentential paraphrase in a given language for a given paraphrase_set_id
- `lists`: Contributors can add sentences to list in order to specify the original source of the data
- `tags`: Indicates morphological or phonological properties of the sentence when available
- `language`: Language identifier, one of the 73 languages that belong to this dataset.

### Data Splits

The dataset is having a single `train` split, contains a total of 1.9 million sentences, with 200 – 250 000
 sentences per language

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Creative Commons Attribution 2.0 Generic

### Citation Information

```
@dataset{scherrer_yves_2020_3707949,
  author       = {Scherrer, Yves},
  title        = {{TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages}},
  month        = mar,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0},
  doi          = {10.5281/zenodo.3707949},
  url          = {https://doi.org/10.5281/zenodo.3707949}
}
```

### Contributions

Thanks to [@pacman100](https://github.com/pacman100) for adding this dataset.
task_categories:text2text-generation task_categories:translation task_categories:text-classification task_ids:semantic-similarity-classification annotations_creators:machine-generated language_creators:crowdsourced multilinguality:multilingual size_categories:100K<n<1M size_categories:10K<n<100K size_categories:1K<n<10K size_categories:1M<n<10M size_categories:n<1K source_datasets:extended|other-tatoeba language:af language:ar language:az language:be language:ber language:bg language:bn language:br language:ca language:cbk language:cmn language:cs language:da language:de language:el language:en language:eo language:es language:et language:eu language:fi language:fr language:gl language:gos language:he language:hi language:hr language:hu language:hy language:ia language:id language:ie language:io language:is language:it language:ja language:jbo language:kab language:ko language:kw language:la language:lfn language:lt language:mk language:mr language:nb language:nds language:nl language:orv language:ota language:pes language:pl language:pt language:rn language:ro language:ru language:sl language:sr language:sv language:tk language:tl language:tlh language:tok language:tr language:tt language:ug language:uk language:ur language:vi language:vo language:war language:wuu language:yue license:cc-by-2.0 paraphrase-generation region:us
{"annotations_creators": ["machine-generated"], "language_creators": ["crowdsourced"], "language": ["af", "ar", "az", "be", "ber", "bg", "bn", "br", "ca", "cbk", "cmn", "cs", "da", "de", "el", "en", "eo", "es", "et", "eu", "fi", "fr", "gl", "gos", "he", "hi", "hr", "hu", "hy", "ia", "id", "ie", "io", "is", "it", "ja", "jbo", "kab", "ko", "kw", "la", "lfn", "lt", "mk", "mr", "nb", "nds", "nl", "orv", "ota", "pes", "pl", "pt", "rn", "ro", "ru", "sl", "sr", "sv", "tk", "tl", "tlh", "tok", "tr", "tt", "ug", "uk", "ur", "vi", "vo", "war", "wuu", "yue"], "license": ["cc-by-2.0"], "multilinguality": ["multilingual"], "size_categories": ["100K<n<1M", "10K<n<100K", "1K<n<10K", "1M<n<10M", "n<1K"], "source_datasets": ["extended|other-tatoeba"], "task_categories": ["text2text-generation", "translation", "text-classification"], "task_ids": ["semantic-similarity-classification"], "paperswithcode_id": "tapaco", "pretty_name": "TaPaCo Corpus", "config_names": ["af", "all_languages", "ar", "az", "be", "ber", "bg", "bn", "br", "ca", "cbk", "cmn", "cs", "da", "de", "el", "en", "eo", "es", "et", "eu", "fi", "fr", "gl", "gos", "he", "hi", "hr", "hu", "hy", "ia", "id", "ie", "io", "is", "it", "ja", "jbo", "kab", "ko", "kw", "la", "lfn", "lt", "mk", "mr", "nb", "nds", "nl", "orv", "ota", "pes", "pl", "pt", "rn", "ro", "ru", "sl", "sr", "sv", "tk", "tl", "tlh", "tok", "tr", "tt", "ug", "uk", "ur", "vi", "vo", "war", "wuu", "yue"], "tags": ["paraphrase-generation"], "dataset_info": [{"config_name": "all_languages", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 162802556, "num_examples": 1926192}], "download_size": 32213126, "dataset_size": 162802556}, {"config_name": "af", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21219, "num_examples": 307}], "download_size": 32213126, "dataset_size": 21219}, {"config_name": "ar", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 546200, "num_examples": 6446}], "download_size": 32213126, "dataset_size": 546200}, {"config_name": "az", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 44461, "num_examples": 624}], "download_size": 32213126, "dataset_size": 44461}, {"config_name": "be", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 140376, "num_examples": 1512}], "download_size": 32213126, "dataset_size": 140376}, {"config_name": "ber", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5118620, "num_examples": 67484}], "download_size": 32213126, "dataset_size": 5118620}, {"config_name": "bg", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 590535, "num_examples": 6324}], "download_size": 32213126, "dataset_size": 590535}, {"config_name": "bn", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 146654, "num_examples": 1440}], "download_size": 32213126, "dataset_size": 146654}, {"config_name": "br", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 177919, "num_examples": 2536}], "download_size": 32213126, "dataset_size": 177919}, {"config_name": "ca", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 39404, "num_examples": 518}], "download_size": 32213126, "dataset_size": 39404}, {"config_name": "cbk", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19404, "num_examples": 262}], "download_size": 32213126, "dataset_size": 19404}, {"config_name": "cmn", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 964514, "num_examples": 12549}], "download_size": 32213126, "dataset_size": 964514}, {"config_name": "cs", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 482292, "num_examples": 6659}], "download_size": 32213126, "dataset_size": 482292}, {"config_name": "da", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 848886, "num_examples": 11220}], "download_size": 32213126, "dataset_size": 848886}, {"config_name": "de", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10593377, "num_examples": 125091}], "download_size": 32213126, "dataset_size": 10593377}, {"config_name": "el", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 926054, "num_examples": 10072}], "download_size": 32213126, "dataset_size": 926054}, {"config_name": "en", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15070349, "num_examples": 158053}], "download_size": 32213126, "dataset_size": 15070349}, {"config_name": "eo", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16810965, "num_examples": 207105}], "download_size": 32213126, "dataset_size": 16810965}, {"config_name": "es", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6851135, "num_examples": 85064}], "download_size": 32213126, "dataset_size": 6851135}, {"config_name": "et", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17127, "num_examples": 241}], "download_size": 32213126, "dataset_size": 17127}, {"config_name": "eu", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 42702, "num_examples": 573}], "download_size": 32213126, "dataset_size": 42702}, {"config_name": "fi", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2520167, "num_examples": 31753}], "download_size": 32213126, "dataset_size": 2520167}, {"config_name": "fr", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9481426, "num_examples": 116733}], "download_size": 32213126, "dataset_size": 9481426}, {"config_name": "gl", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 26551, "num_examples": 351}], "download_size": 32213126, "dataset_size": 26551}, {"config_name": "gos", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18442, "num_examples": 279}], "download_size": 32213126, "dataset_size": 18442}, {"config_name": "he", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6024345, "num_examples": 68350}], "download_size": 32213126, "dataset_size": 6024345}, {"config_name": "hi", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 209382, "num_examples": 1913}], "download_size": 32213126, "dataset_size": 209382}, {"config_name": "hr", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 36638, "num_examples": 505}], "download_size": 32213126, "dataset_size": 36638}, {"config_name": "hu", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5289610, "num_examples": 67964}], "download_size": 32213126, "dataset_size": 5289610}, {"config_name": "hy", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 49230, "num_examples": 603}], "download_size": 32213126, "dataset_size": 49230}, {"config_name": "ia", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 194035, "num_examples": 2548}], "download_size": 32213126, "dataset_size": 194035}, {"config_name": "id", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 124568, "num_examples": 1602}], "download_size": 32213126, "dataset_size": 124568}, {"config_name": "ie", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 31956, "num_examples": 488}], "download_size": 32213126, "dataset_size": 31956}, {"config_name": "io", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33892, "num_examples": 480}], "download_size": 32213126, "dataset_size": 33892}, {"config_name": "is", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 132062, "num_examples": 1641}], "download_size": 32213126, "dataset_size": 132062}, {"config_name": "it", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15073750, "num_examples": 198919}], "download_size": 32213126, "dataset_size": 15073750}, {"config_name": "ja", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4314423, "num_examples": 44267}], "download_size": 32213126, "dataset_size": 4314423}, {"config_name": "jbo", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 201564, "num_examples": 2704}], "download_size": 32213126, "dataset_size": 201564}, {"config_name": "kab", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1211051, "num_examples": 15944}], "download_size": 32213126, "dataset_size": 1211051}, {"config_name": "ko", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 40458, "num_examples": 503}], "download_size": 32213126, "dataset_size": 40458}, {"config_name": "kw", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 88577, "num_examples": 1328}], "download_size": 32213126, "dataset_size": 88577}, {"config_name": "la", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 485749, "num_examples": 6889}], "download_size": 32213126, "dataset_size": 485749}, {"config_name": "lfn", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 203383, "num_examples": 2313}], "download_size": 32213126, "dataset_size": 203383}, {"config_name": "lt", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 599166, "num_examples": 8042}], "download_size": 32213126, "dataset_size": 599166}, {"config_name": "mk", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1240185, "num_examples": 14678}], "download_size": 32213126, "dataset_size": 1240185}, {"config_name": "mr", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1838921, "num_examples": 16413}], "download_size": 32213126, "dataset_size": 1838921}, {"config_name": "nb", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 85371, "num_examples": 1094}], "download_size": 32213126, "dataset_size": 85371}, {"config_name": "nds", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 195021, "num_examples": 2633}], "download_size": 32213126, "dataset_size": 195021}, {"config_name": "nl", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1790975, "num_examples": 23561}], "download_size": 32213126, "dataset_size": 1790975}, {"config_name": "orv", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 40484, "num_examples": 471}], "download_size": 32213126, "dataset_size": 40484}, {"config_name": "ota", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 44996, "num_examples": 486}], "download_size": 32213126, "dataset_size": 44996}, {"config_name": "pes", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 433406, "num_examples": 4285}], "download_size": 32213126, "dataset_size": 433406}, {"config_name": "pl", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1722188, "num_examples": 22391}], "download_size": 32213126, "dataset_size": 1722188}, {"config_name": "pt", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6141178, "num_examples": 78430}], "download_size": 32213126, "dataset_size": 6141178}, {"config_name": "rn", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 47387, "num_examples": 648}], "download_size": 32213126, "dataset_size": 47387}, {"config_name": "ro", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 162955, "num_examples": 2092}], "download_size": 32213126, "dataset_size": 162955}, {"config_name": "ru", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24540667, "num_examples": 251263}], "download_size": 32213126, "dataset_size": 24540667}, {"config_name": "sl", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 49610, "num_examples": 706}], "download_size": 32213126, "dataset_size": 49610}, {"config_name": "sr", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 667308, "num_examples": 8175}], "download_size": 32213126, "dataset_size": 667308}, {"config_name": "sv", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 509884, "num_examples": 7005}], "download_size": 32213126, "dataset_size": 509884}, {"config_name": "tk", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 95047, "num_examples": 1165}], "download_size": 32213126, "dataset_size": 95047}, {"config_name": "tl", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 76059, "num_examples": 1017}], "download_size": 32213126, "dataset_size": 76059}, {"config_name": "tlh", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 185309, "num_examples": 2804}], "download_size": 32213126, "dataset_size": 185309}, {"config_name": "toki", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 310864, "num_examples": 3738}], "download_size": 32213126, "dataset_size": 310864}, {"config_name": "tr", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11271158, "num_examples": 142088}], "download_size": 32213126, "dataset_size": 11271158}, {"config_name": "tt", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 277269, "num_examples": 2398}], "download_size": 32213126, "dataset_size": 277269}, {"config_name": "ug", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 118474, "num_examples": 1183}], "download_size": 32213126, "dataset_size": 118474}, {"config_name": "uk", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4885677, "num_examples": 54431}], "download_size": 32213126, "dataset_size": 4885677}, {"config_name": "ur", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24075, "num_examples": 252}], "download_size": 32213126, "dataset_size": 24075}, {"config_name": "vi", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 84773, "num_examples": 962}], "download_size": 32213126, "dataset_size": 84773}, {"config_name": "vo", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 22164, "num_examples": 328}], "download_size": 32213126, "dataset_size": 22164}, {"config_name": "war", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 25759, "num_examples": 327}], "download_size": 32213126, "dataset_size": 25759}, {"config_name": "wuu", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 31640, "num_examples": 408}], "download_size": 32213126, "dataset_size": 31640}, {"config_name": "yue", "features": [{"name": "paraphrase_set_id", "dtype": "string"}, {"name": "sentence_id", "dtype": "string"}, {"name": "paraphrase", "dtype": "string"}, {"name": "lists", "sequence": "string"}, {"name": "tags", "sequence": "string"}, {"name": "language", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 42766, "num_examples": 561}], "download_size": 32213126, "dataset_size": 42766}]}
TAGS
#task_categories-text2text-generation #task_categories-translation #task_categories-text-classification #task_ids-semantic-similarity-classification #annotations_creators-machine-generated #language_creators-crowdsourced #multilinguality-multilingual #size_categories-100K<n<1M #size_categories-10K<n<100K #size_categories-1K<n<10K #size_categories-1M<n<10M #size_categories-n<1K #source_datasets-extended|other-tatoeba #language-Afrikaans #language-Arabic #language-Azerbaijani #language-Belarusian #language-ber #language-Bulgarian #language-Bengali #language-Breton #language-Catalan #language-Chavacano #language-Mandarin Chinese #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Finnish #language-French #language-Galician #language-Gronings #language-Hebrew #language-Hindi #language-Croatian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Lojban #language-Kabyle #language-Korean #language-Cornish #language-Latin #language-Lingua Franca Nova #language-Lithuanian #language-Macedonian #language-Marathi #language-Norwegian Bokmål #language-Low German #language-Dutch #language-Old Russian #language-Ottoman Turkish (1500-1928) #language-Iranian Persian #language-Polish #language-Portuguese #language-Rundi #language-Romanian #language-Russian #language-Slovenian #language-Serbian #language-Swedish #language-Turkmen #language-Tagalog #language-Klingon #language-Toki Pona #language-Turkish #language-Tatar #language-Uighur #language-Ukrainian #language-Urdu #language-Vietnamese #language-Volapük #language-Waray (Philippines) #language-Wu Chinese #language-Yue Chinese #license-cc-by-2.0 #paraphrase-generation #region-us 


# Dataset Card for TaPaCo Corpus

## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages
- Paper: TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages
- Data: URL
- Point of Contact: Yves Scherrer

### Dataset Summary
A freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. 
Tatoeba is a crowdsourcing project mainly geared towards language learners. Its aim is to provide example sentences 
and translations for particular linguistic constructions and words. The paraphrase corpus is created by populating a 
graph with Tatoeba sentences and equivalence links between sentences “meaning the same thing”. This graph is then 
traversed to extract sets of paraphrases. Several language-independent filters and pruning steps are applied to 
remove uninteresting sentences. A manual evaluation performed on three languages shows that between half and three 
quarters of inferred paraphrases are correct and that most remaining ones are either correct but trivial, 
or near-paraphrases that neutralize a morphological distinction. The corpus contains a total of 1.9 million 
sentences, with 200 – 250 000 sentences per language. It covers a range of languages for which, to our knowledge,
no other paraphrase dataset exists.

### Supported Tasks and Leaderboards
Paraphrase detection and generation have become popular tasks in NLP
and are increasingly integrated into a wide variety of common downstream tasks such as machine translation
, information retrieval, question answering, and semantic parsing. Most of the existing datasets
 cover only a single language – in most cases English – or a small number of languages. Furthermore, some paraphrase
  datasets focus on lexical and phrasal rather than sentential paraphrases, while others are created (semi
  -)automatically using machine translation.

The number of sentences per language ranges from 200 to 250 000, which makes the dataset
more suitable for fine-tuning and evaluation purposes than
for training. It is well-suited for multi-reference evaluation
of paraphrase generation models, as there is generally not a
single correct way of paraphrasing a given input sentence.

### Languages

The dataset contains paraphrases in Afrikaans, Arabic, Azerbaijani, Belarusian, Berber languages, Bulgarian, Bengali
, Breton, Catalan; Valencian, Chavacano, Mandarin, Czech, Danish, German, Greek, Modern (1453-), English, Esperanto
, Spanish; Castilian, Estonian, Basque, Finnish, French, Galician, Gronings, Hebrew, Hindi, Croatian, Hungarian
, Armenian, Interlingua (International Auxiliary Language Association), Indonesian, Interlingue; Occidental, Ido
, Icelandic, Italian, Japanese, Lojban, Kabyle, Korean, Cornish, Latin, Lingua Franca Nova\t, Lithuanian, Macedonian
, Marathi, Bokmål, Norwegian; Norwegian Bokmål, Low German; Low Saxon; German, Low; Saxon, Low, Dutch; Flemish, ]Old
 Russian, Turkish, Ottoman (1500-1928), Iranian Persian, Polish, Portuguese, Rundi, Romanian; Moldavian; Moldovan, 
 Russian, Slovenian, Serbian, Swedish, Turkmen, Tagalog, Klingon; tlhIngan-Hol, Toki Pona, Turkish, Tatar, 
 Uighur; Uyghur, Ukrainian, Urdu, Vietnamese, Volapük, Waray, Wu Chinese and Yue Chinese

## Dataset Structure

### Data Instances
Each data instance corresponds to a paraphrase, e.g.:


### Data Fields
Each dialogue instance has the following fields:
- 'paraphrase_set_id':  a running number that groups together all sentences that are considered paraphrases of each
 other
- 'sentence_id': OPUS sentence id
- 'paraphrase': Sentential paraphrase in a given language for a given paraphrase_set_id
- 'lists': Contributors can add sentences to list in order to specify the original source of the data
- 'tags': Indicates morphological or phonological properties of the sentence when available
- 'language': Language identifier, one of the 73 languages that belong to this dataset.

### Data Splits

The dataset is having a single 'train' split, contains a total of 1.9 million sentences, with 200 – 250 000
 sentences per language

## Dataset Creation

### Curation Rationale



### Source Data

#### Initial Data Collection and Normalization



#### Who are the source language producers?



### Annotations

#### Annotation process



#### Who are the annotators?



### Personal and Sensitive Information



## Considerations for Using the Data

### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



## Additional Information

### Dataset Curators



### Licensing Information

Creative Commons Attribution 2.0 Generic





### Contributions

Thanks to @pacman100 for adding this dataset.
# Dataset Card for TaPaCo Corpus ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages
- Paper: TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages
- Data: URL
- Point of Contact: Yves Scherrer ### Dataset Summary
A freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. 
Tatoeba is a crowdsourcing project mainly geared towards language learners. Its aim is to provide example sentences 
and translations for particular linguistic constructions and words. The paraphrase corpus is created by populating a 
graph with Tatoeba sentences and equivalence links between sentences “meaning the same thing”. This graph is then 
traversed to extract sets of paraphrases. Several language-independent filters and pruning steps are applied to 
remove uninteresting sentences. A manual evaluation performed on three languages shows that between half and three 
quarters of inferred paraphrases are correct and that most remaining ones are either correct but trivial, 
or near-paraphrases that neutralize a morphological distinction. The corpus contains a total of 1.9 million 
sentences, with 200 – 250 000 sentences per language. It covers a range of languages for which, to our knowledge,
no other paraphrase dataset exists. ### Supported Tasks and Leaderboards
Paraphrase detection and generation have become popular tasks in NLP
and are increasingly integrated into a wide variety of common downstream tasks such as machine translation
, information retrieval, question answering, and semantic parsing. Most of the existing datasets
 cover only a single language – in most cases English – or a small number of languages. Furthermore, some paraphrase
  datasets focus on lexical and phrasal rather than sentential paraphrases, while others are created (semi
  -)automatically using machine translation.

The number of sentences per language ranges from 200 to 250 000, which makes the dataset
more suitable for fine-tuning and evaluation purposes than
for training. It is well-suited for multi-reference evaluation
of paraphrase generation models, as there is generally not a
single correct way of paraphrasing a given input sentence. ### Languages

The dataset contains paraphrases in Afrikaans, Arabic, Azerbaijani, Belarusian, Berber languages, Bulgarian, Bengali
, Breton, Catalan; Valencian, Chavacano, Mandarin, Czech, Danish, German, Greek, Modern (1453-), English, Esperanto
, Spanish; Castilian, Estonian, Basque, Finnish, French, Galician, Gronings, Hebrew, Hindi, Croatian, Hungarian
, Armenian, Interlingua (International Auxiliary Language Association), Indonesian, Interlingue; Occidental, Ido
, Icelandic, Italian, Japanese, Lojban, Kabyle, Korean, Cornish, Latin, Lingua Franca Nova\t, Lithuanian, Macedonian
, Marathi, Bokmål, Norwegian; Norwegian Bokmål, Low German; Low Saxon; German, Low; Saxon, Low, Dutch; Flemish, ]Old
 Russian, Turkish, Ottoman (1500-1928), Iranian Persian, Polish, Portuguese, Rundi, Romanian; Moldavian; Moldovan, 
 Russian, Slovenian, Serbian, Swedish, Turkmen, Tagalog, Klingon; tlhIngan-Hol, Toki Pona, Turkish, Tatar, 
 Uighur; Uyghur, Ukrainian, Urdu, Vietnamese, Volapük, Waray, Wu Chinese and Yue Chinese ## Dataset Structure ### Data Instances
Each data instance corresponds to a paraphrase, e.g.: ### Data Fields
Each dialogue instance has the following fields:
- 'paraphrase_set_id':  a running number that groups together all sentences that are considered paraphrases of each
 other
- 'sentence_id': OPUS sentence id
- 'paraphrase': Sentential paraphrase in a given language for a given paraphrase_set_id
- 'lists': Contributors can add sentences to list in order to specify the original source of the data
- 'tags': Indicates morphological or phonological properties of the sentence when available
- 'language': Language identifier, one of the 73 languages that belong to this dataset. ### Data Splits

The dataset is having a single 'train' split, contains a total of 1.9 million sentences, with 200 – 250 000
 sentences per language ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Creative Commons Attribution 2.0 Generic ### Contributions

Thanks to @pacman100 for adding this dataset.
TAGS
#task_categories-text2text-generation #task_categories-translation #task_categories-text-classification #task_ids-semantic-similarity-classification #annotations_creators-machine-generated #language_creators-crowdsourced #multilinguality-multilingual #size_categories-100K<n<1M #size_categories-10K<n<100K #size_categories-1K<n<10K #size_categories-1M<n<10M #size_categories-n<1K #source_datasets-extended|other-tatoeba #language-Afrikaans #language-Arabic #language-Azerbaijani #language-Belarusian #language-ber #language-Bulgarian #language-Bengali #language-Breton #language-Catalan #language-Chavacano #language-Mandarin Chinese #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Finnish #language-French #language-Galician #language-Gronings #language-Hebrew #language-Hindi #language-Croatian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Lojban #language-Kabyle #language-Korean #language-Cornish #language-Latin #language-Lingua Franca Nova #language-Lithuanian #language-Macedonian #language-Marathi #language-Norwegian Bokmål #language-Low German #language-Dutch #language-Old Russian #language-Ottoman Turkish (1500-1928) #language-Iranian Persian #language-Polish #language-Portuguese #language-Rundi #language-Romanian #language-Russian #language-Slovenian #language-Serbian #language-Swedish #language-Turkmen #language-Tagalog #language-Klingon #language-Toki Pona #language-Turkish #language-Tatar #language-Uighur #language-Ukrainian #language-Urdu #language-Vietnamese #language-Volapük #language-Waray (Philippines) #language-Wu Chinese #language-Yue Chinese #license-cc-by-2.0 #paraphrase-generation #region-us 
 # Dataset Card for TaPaCo Corpus ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages
- Paper: TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages
- Data: URL
- Point of Contact: Yves Scherrer ### Dataset Summary
A freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. 
Tatoeba is a crowdsourcing project mainly geared towards language learners. Its aim is to provide example sentences 
and translations for particular linguistic constructions and words. The paraphrase corpus is created by populating a 
graph with Tatoeba sentences and equivalence links between sentences “meaning the same thing”. This graph is then 
traversed to extract sets of paraphrases. Several language-independent filters and pruning steps are applied to 
remove uninteresting sentences. A manual evaluation performed on three languages shows that between half and three 
quarters of inferred paraphrases are correct and that most remaining ones are either correct but trivial, 
or near-paraphrases that neutralize a morphological distinction. The corpus contains a total of 1.9 million 
sentences, with 200 – 250 000 sentences per language. It covers a range of languages for which, to our knowledge,
no other paraphrase dataset exists. ### Supported Tasks and Leaderboards
Paraphrase detection and generation have become popular tasks in NLP
and are increasingly integrated into a wide variety of common downstream tasks such as machine translation
, information retrieval, question answering, and semantic parsing. Most of the existing datasets
 cover only a single language – in most cases English – or a small number of languages. Furthermore, some paraphrase
  datasets focus on lexical and phrasal rather than sentential paraphrases, while others are created (semi
  -)automatically using machine translation.

The number of sentences per language ranges from 200 to 250 000, which makes the dataset
more suitable for fine-tuning and evaluation purposes than
for training. It is well-suited for multi-reference evaluation
of paraphrase generation models, as there is generally not a
single correct way of paraphrasing a given input sentence. ### Languages

The dataset contains paraphrases in Afrikaans, Arabic, Azerbaijani, Belarusian, Berber languages, Bulgarian, Bengali
, Breton, Catalan; Valencian, Chavacano, Mandarin, Czech, Danish, German, Greek, Modern (1453-), English, Esperanto
, Spanish; Castilian, Estonian, Basque, Finnish, French, Galician, Gronings, Hebrew, Hindi, Croatian, Hungarian
, Armenian, Interlingua (International Auxiliary Language Association), Indonesian, Interlingue; Occidental, Ido
, Icelandic, Italian, Japanese, Lojban, Kabyle, Korean, Cornish, Latin, Lingua Franca Nova\t, Lithuanian, Macedonian
, Marathi, Bokmål, Norwegian; Norwegian Bokmål, Low German; Low Saxon; German, Low; Saxon, Low, Dutch; Flemish, ]Old
 Russian, Turkish, Ottoman (1500-1928), Iranian Persian, Polish, Portuguese, Rundi, Romanian; Moldavian; Moldovan, 
 Russian, Slovenian, Serbian, Swedish, Turkmen, Tagalog, Klingon; tlhIngan-Hol, Toki Pona, Turkish, Tatar, 
 Uighur; Uyghur, Ukrainian, Urdu, Vietnamese, Volapük, Waray, Wu Chinese and Yue Chinese ## Dataset Structure ### Data Instances
Each data instance corresponds to a paraphrase, e.g.: ### Data Fields
Each dialogue instance has the following fields:
- 'paraphrase_set_id':  a running number that groups together all sentences that are considered paraphrases of each
 other
- 'sentence_id': OPUS sentence id
- 'paraphrase': Sentential paraphrase in a given language for a given paraphrase_set_id
- 'lists': Contributors can add sentences to list in order to specify the original source of the data
- 'tags': Indicates morphological or phonological properties of the sentence when available
- 'language': Language identifier, one of the 73 languages that belong to this dataset. ### Data Splits

The dataset is having a single 'train' split, contains a total of 1.9 million sentences, with 200 – 250 000
 sentences per language ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Creative Commons Attribution 2.0 Generic ### Contributions

Thanks to @pacman100 for adding this dataset.