
# Dataset Card for Flores 200

## Table of Contents

- [Dataset Card for Flores 200](#dataset-card-for-flores-200)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
    - [Dataset Creation](#dataset-creation)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)

## Dataset Description

- **Home:** [Flores](https://github.com/facebookresearch/flores)
- **Repository:** [Github](https://github.com/facebookresearch/flores)

### Dataset Summary

FLORES is a benchmark dataset for machine translation between English and low-resource languages.

>The creation of FLORES-200 doubles the existing language coverage of FLORES-101. 
Given the nature of the new languages, which have less standardization and require 
more specialized professional translations, the verification process became more complex. 
This required modifications to the translation workflow. FLORES-200 has several languages 
which were not translated from English. Specifically, several languages were translated 
from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also 
includes two script alternatives for four languages. FLORES-200 consists of translations 
from 842 distinct web articles, totaling 3001 sentences. These sentences are divided 
into three splits: dev, devtest, and test (hidden). On average, sentences are approximately 
21 words long.

**Disclaimer**: *The Flores-200 dataset is hosted by the Facebook and licensed under the [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/).
### Supported Tasks and Leaderboards
#### Multilingual Machine Translation
Refer to the [Dynabench leaderboard](https://dynabench.org/flores/Flores%20MT%20Evaluation%20(FULL)) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on [Large-Scale Multilingual Machine Translation](http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html). Flores 200 is an extention of this.

### Languages
The dataset contains parallel sentences for 200 languages, as mentioned in the original [Github](https://github.com/facebookresearch/flores/blob/master/README.md) page for the project. Languages are identified with the ISO 639-3 code (e.g. `eng`, `fra`, `rus`) plus an additional code describing the script (e.g., "eng_Latn", "ukr_Cyrl"). See [the webpage for code descriptions](https://github.com/facebookresearch/flores/blob/main/flores200/README.md).

Use the configuration `all` to access the full set of parallel sentences for all the available languages in a single command. 

Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng_Latn-ukr_Cyrl" will provide sentences in the format below).

## Dataset Structure
### Data Instances
A sample from the `dev` split for the Ukrainian language (`ukr_Cyrl` config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.
```python
{
	'id': 1,
	'sentence': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.',
	'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet',
	'domain': 'wikinews',
	'topic': 'health',
	'has_image': 0,
	'has_hyperlink': 0
}
```
When using a hyphenated pairing or using the `all` function, data will be presented as follows:

```python
{
    'id': 1, 
    'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet', 
    'domain': 'wikinews', 
    'topic': 'health', 
    'has_image': 0, 
    'has_hyperlink': 0, 
    'sentence_eng_Latn': 'On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.', 
    'sentence_ukr_Cyrl': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.'
}
```


The text is provided as-in the original dataset, without further preprocessing or tokenization.
### Data Fields
- `id`: Row number for the data entry, starting at 1.
- `sentence`: The full sentence in the specific language (may have _lang for pairings)
- `URL`: The URL for the English article from which the sentence was extracted.
- `domain`: The domain of the sentence.
- `topic`: The topic of the sentence.
- `has_image`: Whether the  original article contains an image.
- `has_hyperlink`: Whether the  sentence contains a hyperlink.
### Data Splits
|            config| `dev`| `devtest`|
|-----------------:|-----:|---------:|
|all configurations|   997|     1012:|
### Dataset Creation
Please refer to the original article [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) for additional information on dataset creation.
## Additional Information
### Dataset Curators
See paper for details.
### Licensing Information
Licensed with Creative Commons Attribution Share Alike 4.0. License available [here](https://creativecommons.org/licenses/by-sa/4.0/).
### Citation Information
Please cite the authors if you use these corpora in your work:
```bibtex
@article{nllb2022,
  author    = {NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi,  Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang},
  title     = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  year      = {2022}
}
```

Please also cite prior work that this dataset builds on:

```bibtex
@inproceedings{,
  title={The FLORES-101  Evaluation Benchmark for Low-Resource and Multilingual Machine Translation},
  author={Goyal, Naman and Gao, Cynthia and Chaudhary, Vishrav and Chen, Peng-Jen and Wenzek, Guillaume and Ju, Da and Krishnan, Sanjana and Ranzato, Marc'Aurelio and Guzm\'{a}n, Francisco and Fan, Angela},
  year={2021}
}
```

```bibtex
@inproceedings{,
  title={Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English},
  author={Guzm\'{a}n, Francisco and Chen, Peng-Jen and Ott, Myle and Pino, Juan and Lample, Guillaume and Koehn, Philipp and Chaudhary, Vishrav and Ranzato, Marc'Aurelio},
  journal={arXiv preprint arXiv:1902.01382},
  year={2019}
}
```
task_categories:text2text-generation task_categories:translation annotations_creators:found language_creators:expert-generated multilinguality:multilingual multilinguality:translation size_categories:unknown source_datasets:extended|flores language:ace language:acm language:acq language:aeb language:af language:ajp language:ak language:als language:am language:apc language:ar language:ars language:ary language:arz language:as language:ast language:awa language:ayr language:azb language:azj language:ba language:bm language:ban language:be language:bem language:bn language:bho language:bjn language:bo language:bs language:bug language:bg language:ca language:ceb language:cs language:cjk language:ckb language:crh language:cy language:da language:de language:dik language:dyu language:dz language:el language:en language:eo language:et language:eu language:ee language:fo language:fj language:fi language:fon language:fr language:fur language:fuv language:gaz language:gd language:ga language:gl language:gn language:gu language:ht language:ha language:he language:hi language:hne language:hr language:hu language:hy language:ig language:ilo language:id language:is language:it language:jv language:ja language:kab language:kac language:kam language:kn language:ks language:ka language:kk language:kbp language:kea language:khk language:km language:ki language:rw language:ky language:kmb language:kmr language:knc language:kg language:ko language:lo language:lij language:li language:ln language:lt language:lmo language:ltg language:lb language:lua language:lg language:luo language:lus language:lvs language:mag language:mai language:ml language:mar language:min language:mk language:mt language:mni language:mos language:mi language:my language:nl language:nn language:nb language:npi language:nso language:nus language:ny language:oc language:ory language:pag language:pa language:pap language:pbt language:pes language:plt language:pl language:pt language:prs language:quy language:ro language:rn language:ru language:sg language:sa language:sat language:scn language:shn language:si language:sk language:sl language:sm language:sn language:sd language:so language:st language:es language:sc language:sr language:ss language:su language:sv language:swh language:szl language:ta language:taq language:tt language:te language:tg language:tl language:th language:ti language:tpi language:tn language:ts language:tk language:tum language:tr language:tw language:tzm language:ug language:uk language:umb language:ur language:uzn language:vec language:vi language:war language:wo language:xh language:ydd language:yo language:yue language:zh language:zsm language:zu license:cc-by-sa-4.0 conditional-text-generation arxiv:2207.04672 region:us
{"annotations_creators": ["found"], "language_creators": ["expert-generated"], "language": ["ace", "acm", "acq", "aeb", "af", "ajp", "ak", "als", "am", "apc", "ar", "ars", "ary", "arz", "as", "ast", "awa", "ayr", "azb", "azj", "ba", "bm", "ban", "be", "bem", "bn", "bho", "bjn", "bo", "bs", "bug", "bg", "ca", "ceb", "cs", "cjk", "ckb", "crh", "cy", "da", "de", "dik", "dyu", "dz", "el", "en", "eo", "et", "eu", "ee", "fo", "fj", "fi", "fon", "fr", "fur", "fuv", "gaz", "gd", "ga", "gl", "gn", "gu", "ht", "ha", "he", "hi", "hne", "hr", "hu", "hy", "ig", "ilo", "id", "is", "it", "jv", "ja", "kab", "kac", "kam", "kn", "ks", "ka", "kk", "kbp", "kea", "khk", "km", "ki", "rw", "ky", "kmb", "kmr", "knc", "kg", "ko", "lo", "lij", "li", "ln", "lt", "lmo", "ltg", "lb", "lua", "lg", "luo", "lus", "lvs", "mag", "mai", "ml", "mar", "min", "mk", "mt", "mni", "mos", "mi", "my", "nl", "nn", "nb", "npi", "nso", "nus", "ny", "oc", "ory", "pag", "pa", "pap", "pbt", "pes", "plt", "pl", "pt", "prs", "quy", "ro", "rn", "ru", "sg", "sa", "sat", "scn", "shn", "si", "sk", "sl", "sm", "sn", "sd", "so", "st", "es", "sc", "sr", "ss", "su", "sv", "swh", "szl", "ta", "taq", "tt", "te", "tg", "tl", "th", "ti", "tpi", "tn", "ts", "tk", "tum", "tr", "tw", "tzm", "ug", "uk", "umb", "ur", "uzn", "vec", "vi", "war", "wo", "xh", "ydd", "yo", "yue", "zh", "zsm", "zu"], "license": ["cc-by-sa-4.0"], "multilinguality": ["multilingual", "translation"], "size_categories": ["unknown"], "source_datasets": ["extended|flores"], "task_categories": ["text2text-generation", "translation"], "task_ids": [], "paperswithcode_id": "flores", "pretty_name": "flores200", "language_details": "ace_Arab, ace_Latn, acm_Arab, acq_Arab, aeb_Arab, afr_Latn, ajp_Arab, aka_Latn, amh_Ethi, apc_Arab, arb_Arab, ars_Arab, ary_Arab, arz_Arab, asm_Beng, ast_Latn, awa_Deva, ayr_Latn, azb_Arab, azj_Latn, bak_Cyrl, bam_Latn, ban_Latn,bel_Cyrl, bem_Latn, ben_Beng, bho_Deva, bjn_Arab, bjn_Latn, bod_Tibt, bos_Latn, bug_Latn, bul_Cyrl, cat_Latn, ceb_Latn, ces_Latn, cjk_Latn, ckb_Arab, crh_Latn, cym_Latn, dan_Latn, deu_Latn, dik_Latn, dyu_Latn, dzo_Tibt, ell_Grek, eng_Latn, epo_Latn, est_Latn, eus_Latn, ewe_Latn, fao_Latn, pes_Arab, fij_Latn, fin_Latn, fon_Latn, fra_Latn, fur_Latn, fuv_Latn, gla_Latn, gle_Latn, glg_Latn, grn_Latn, guj_Gujr, hat_Latn, hau_Latn, heb_Hebr, hin_Deva, hne_Deva, hrv_Latn, hun_Latn, hye_Armn, ibo_Latn, ilo_Latn, ind_Latn, isl_Latn, ita_Latn, jav_Latn, jpn_Jpan, kab_Latn, kac_Latn, kam_Latn, kan_Knda, kas_Arab, kas_Deva, kat_Geor, knc_Arab, knc_Latn, kaz_Cyrl, kbp_Latn, kea_Latn, khm_Khmr, kik_Latn, kin_Latn, kir_Cyrl, kmb_Latn, kon_Latn, kor_Hang, kmr_Latn, lao_Laoo, lvs_Latn, lij_Latn, lim_Latn, lin_Latn, lit_Latn, lmo_Latn, ltg_Latn, ltz_Latn, lua_Latn, lug_Latn, luo_Latn, lus_Latn, mag_Deva, mai_Deva, mal_Mlym, mar_Deva, min_Latn, mkd_Cyrl, plt_Latn, mlt_Latn, mni_Beng, khk_Cyrl, mos_Latn, mri_Latn, zsm_Latn, mya_Mymr, nld_Latn, nno_Latn, nob_Latn, npi_Deva, nso_Latn, nus_Latn, nya_Latn, oci_Latn, gaz_Latn, ory_Orya, pag_Latn, pan_Guru, pap_Latn, pol_Latn, por_Latn, prs_Arab, pbt_Arab, quy_Latn, ron_Latn, run_Latn, rus_Cyrl, sag_Latn, san_Deva, sat_Beng, scn_Latn, shn_Mymr, sin_Sinh, slk_Latn, slv_Latn, smo_Latn, sna_Latn, snd_Arab, som_Latn, sot_Latn, spa_Latn, als_Latn, srd_Latn, srp_Cyrl, ssw_Latn, sun_Latn, swe_Latn, swh_Latn, szl_Latn, tam_Taml, tat_Cyrl, tel_Telu, tgk_Cyrl, tgl_Latn, tha_Thai, tir_Ethi, taq_Latn, taq_Tfng, tpi_Latn, tsn_Latn, tso_Latn, tuk_Latn, tum_Latn, tur_Latn, twi_Latn, tzm_Tfng, uig_Arab, ukr_Cyrl, umb_Latn, urd_Arab, uzn_Latn, vec_Latn, vie_Latn, war_Latn, wol_Latn, xho_Latn, ydd_Hebr, yor_Latn, yue_Hant, zho_Hans, zho_Hant, zul_Latn", "tags": ["conditional-text-generation"]}
TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-found #language_creators-expert-generated #multilinguality-multilingual #multilinguality-translation #size_categories-unknown #source_datasets-extended|flores #language-Achinese #language-Mesopotamian Arabic #language-Ta'izzi-Adeni Arabic #language-Tunisian Arabic #language-Afrikaans #language-South Levantine Arabic #language-Akan #language-Tosk Albanian #language-Amharic #language-Levantine Arabic #language-Arabic #language-Najdi Arabic #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Awadhi #language-Central Aymara #language-South Azerbaijani #language-North Azerbaijani #language-Bashkir #language-Bambara #language-Balinese #language-Belarusian #language-Bemba (Zambia) #language-Bengali #language-Bhojpuri #language-Banjar #language-Tibetan #language-Bosnian #language-Buginese #language-Bulgarian #language-Catalan #language-Cebuano #language-Czech #language-Chokwe #language-Central Kurdish #language-Crimean Tatar #language-Welsh #language-Danish #language-German #language-Southwestern Dinka #language-Dyula #language-Dzongkha #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Estonian #language-Basque #language-Ewe #language-Faroese #language-Fijian #language-Finnish #language-Fon #language-French #language-Friulian #language-Nigerian Fulfulde #language-West Central Oromo #language-Scottish Gaelic #language-Irish #language-Galician #language-Guarani #language-Gujarati #language-Haitian #language-Hausa #language-Hebrew #language-Hindi #language-Chhattisgarhi #language-Croatian #language-Hungarian #language-Armenian #language-Igbo #language-Iloko #language-Indonesian #language-Icelandic #language-Italian #language-Javanese #language-Japanese #language-Kabyle #language-Kachin #language-Kamba (Kenya) #language-Kannada #language-Kashmiri #language-Georgian #language-Kazakh #language-Kabiyè #language-Kabuverdianu #language-Halh Mongolian #language-Khmer #language-Kikuyu #language-Kinyarwanda #language-Kirghiz #language-Kimbundu #language-Northern Kurdish #language-Central Kanuri #language-Kongo #language-Korean #language-Lao #language-Ligurian #language-Limburgan #language-Lingala #language-Lithuanian #language-Lombard #language-Latgalian #language-Luxembourgish #language-Luba-Lulua #language-Ganda #language-Luo (Kenya and Tanzania) #language-Lushai #language-Standard Latvian #language-Magahi #language-Maithili #language-Malayalam #language-Marathi #language-Minangkabau #language-Macedonian #language-Maltese #language-Manipuri #language-Mossi #language-Maori #language-Burmese #language-Dutch #language-Norwegian Nynorsk #language-Norwegian Bokmål #language-Nepali (individual language) #language-Pedi #language-Nuer #language-Nyanja #language-Occitan (post 1500) #language-Odia #language-Pangasinan #language-Panjabi #language-Papiamento #language-Southern Pashto #language-Iranian Persian #language-Plateau Malagasy #language-Polish #language-Portuguese #language-Dari #language-Ayacucho Quechua #language-Romanian #language-Rundi #language-Russian #language-Sango #language-Sanskrit #language-Santali #language-Sicilian #language-Shan #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Sindhi #language-Somali #language-Southern Sotho #language-Spanish #language-Sardinian #language-Serbian #language-Swati #language-Sundanese #language-Swedish #language-Swahili (individual language) #language-Silesian #language-Tamil #language-Tamasheq #language-Tatar #language-Telugu #language-Tajik #language-Tagalog #language-Thai #language-Tigrinya #language-Tok Pisin #language-Tswana #language-Tsonga #language-Turkmen #language-Tumbuka #language-Turkish #language-Twi #language-Central Atlas Tamazight #language-Uighur #language-Ukrainian #language-Umbundu #language-Urdu #language-Northern Uzbek #language-Venetian #language-Vietnamese #language-Waray (Philippines) #language-Wolof #language-Xhosa #language-Eastern Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #language-Standard Malay #language-Zulu #license-cc-by-sa-4.0 #conditional-text-generation #arxiv-2207.04672 #region-us 

Dataset Card for Flores 200
===========================


Table of Contents
-----------------


* Dataset Card for Flores 200
	+ Table of Contents
	+ Dataset Description
		- Dataset Summary
		- Supported Tasks and Leaderboards
		- Languages
	+ Dataset Structure
		- Data Instances
		- Data Fields
		- Data Splits
		- Dataset Creation
	+ Additional Information
		- Dataset Curators
		- Licensing Information
		- Citation Information


Dataset Description
-------------------


* Home: Flores
* Repository: Github


### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES-200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores-200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.


### Supported Tasks and Leaderboards


#### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this.


### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.


Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.


Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
-----------------


### Data Instances


A sample from the 'dev' split for the Ukrainian language ('ukr\_Cyrl' config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.


When using a hyphenated pairing or using the 'all' function, data will be presented as follows:


The text is provided as-in the original dataset, without further preprocessing or tokenization.


### Data Fields


* 'id': Row number for the data entry, starting at 1.
* 'sentence': The full sentence in the specific language (may have \_lang for pairings)
* 'URL': The URL for the English article from which the sentence was extracted.
* 'domain': The domain of the sentence.
* 'topic': The topic of the sentence.
* 'has\_image': Whether the original article contains an image.
* 'has\_hyperlink': Whether the sentence contains a hyperlink.


### Data Splits



### Dataset Creation


Please refer to the original article No Language Left Behind: Scaling Human-Centered Machine Translation for additional information on dataset creation.


Additional Information
----------------------


### Dataset Curators


See paper for details.


### Licensing Information


Licensed with Creative Commons Attribution Share Alike 4.0. License available here.


Please cite the authors if you use these corpora in your work:


Please also cite prior work that this dataset builds on:



### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES-200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores-200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. ### Supported Tasks and Leaderboards #### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this. ### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.


Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.


Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
----------------- ### Data Instances


A sample from the 'dev' split for the Ukrainian language ('ukr\_Cyrl' config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.


When using a hyphenated pairing or using the 'all' function, data will be presented as follows:


The text is provided as-in the original dataset, without further preprocessing or tokenization. ### Data Fields


* 'id': Row number for the data entry, starting at 1.
* 'sentence': The full sentence in the specific language (may have \_lang for pairings)
* 'URL': The URL for the English article from which the sentence was extracted.
* 'domain': The domain of the sentence.
* 'topic': The topic of the sentence.
* 'has\_image': Whether the original article contains an image.
* 'has\_hyperlink': Whether the sentence contains a hyperlink. ### Data Splits ### Dataset Creation


Please refer to the original article No Language Left Behind: Scaling Human-Centered Machine Translation for additional information on dataset creation.


Additional Information
---------------------- ### Dataset Curators


See paper for details. ### Licensing Information


Licensed with Creative Commons Attribution Share Alike 4.0. License available here.


Please cite the authors if you use these corpora in your work:


Please also cite prior work that this dataset builds on:
TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-found #language_creators-expert-generated #multilinguality-multilingual #multilinguality-translation #size_categories-unknown #source_datasets-extended|flores #language-Achinese #language-Mesopotamian Arabic #language-Ta'izzi-Adeni Arabic #language-Tunisian Arabic #language-Afrikaans #language-South Levantine Arabic #language-Akan #language-Tosk Albanian #language-Amharic #language-Levantine Arabic #language-Arabic #language-Najdi Arabic #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Awadhi #language-Central Aymara #language-South Azerbaijani #language-North Azerbaijani #language-Bashkir #language-Bambara #language-Balinese #language-Belarusian #language-Bemba (Zambia) #language-Bengali #language-Bhojpuri #language-Banjar #language-Tibetan #language-Bosnian #language-Buginese #language-Bulgarian #language-Catalan #language-Cebuano #language-Czech #language-Chokwe #language-Central Kurdish #language-Crimean Tatar #language-Welsh #language-Danish #language-German #language-Southwestern Dinka #language-Dyula #language-Dzongkha #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Estonian #language-Basque #language-Ewe #language-Faroese #language-Fijian #language-Finnish #language-Fon #language-French #language-Friulian #language-Nigerian Fulfulde #language-West Central Oromo #language-Scottish Gaelic #language-Irish #language-Galician #language-Guarani #language-Gujarati #language-Haitian #language-Hausa #language-Hebrew #language-Hindi #language-Chhattisgarhi #language-Croatian #language-Hungarian #language-Armenian #language-Igbo #language-Iloko #language-Indonesian #language-Icelandic #language-Italian #language-Javanese #language-Japanese #language-Kabyle #language-Kachin #language-Kamba (Kenya) #language-Kannada #language-Kashmiri #language-Georgian #language-Kazakh #language-Kabiyè #language-Kabuverdianu #language-Halh Mongolian #language-Khmer #language-Kikuyu #language-Kinyarwanda #language-Kirghiz #language-Kimbundu #language-Northern Kurdish #language-Central Kanuri #language-Kongo #language-Korean #language-Lao #language-Ligurian #language-Limburgan #language-Lingala #language-Lithuanian #language-Lombard #language-Latgalian #language-Luxembourgish #language-Luba-Lulua #language-Ganda #language-Luo (Kenya and Tanzania) #language-Lushai #language-Standard Latvian #language-Magahi #language-Maithili #language-Malayalam #language-Marathi #language-Minangkabau #language-Macedonian #language-Maltese #language-Manipuri #language-Mossi #language-Maori #language-Burmese #language-Dutch #language-Norwegian Nynorsk #language-Norwegian Bokmål #language-Nepali (individual language) #language-Pedi #language-Nuer #language-Nyanja #language-Occitan (post 1500) #language-Odia #language-Pangasinan #language-Panjabi #language-Papiamento #language-Southern Pashto #language-Iranian Persian #language-Plateau Malagasy #language-Polish #language-Portuguese #language-Dari #language-Ayacucho Quechua #language-Romanian #language-Rundi #language-Russian #language-Sango #language-Sanskrit #language-Santali #language-Sicilian #language-Shan #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Sindhi #language-Somali #language-Southern Sotho #language-Spanish #language-Sardinian #language-Serbian #language-Swati #language-Sundanese #language-Swedish #language-Swahili (individual language) #language-Silesian #language-Tamil #language-Tamasheq #language-Tatar #language-Telugu #language-Tajik #language-Tagalog #language-Thai #language-Tigrinya #language-Tok Pisin #language-Tswana #language-Tsonga #language-Turkmen #language-Tumbuka #language-Turkish #language-Twi #language-Central Atlas Tamazight #language-Uighur #language-Ukrainian #language-Umbundu #language-Urdu #language-Northern Uzbek #language-Venetian #language-Vietnamese #language-Waray (Philippines) #language-Wolof #language-Xhosa #language-Eastern Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #language-Standard Malay #language-Zulu #license-cc-by-sa-4.0 #conditional-text-generation #arxiv-2207.04672 #region-us 
 ### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES-200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores-200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. ### Supported Tasks and Leaderboards #### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this. ### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.


Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.


Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
----------------- ### Data Instances


A sample from the 'dev' split for the Ukrainian language ('ukr\_Cyrl' config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.


When using a hyphenated pairing or using the 'all' function, data will be presented as follows:


The text is provided as-in the original dataset, without further preprocessing or tokenization. ### Data Fields


* 'id': Row number for the data entry, starting at 1.
* 'sentence': The full sentence in the specific language (may have \_lang for pairings)
* 'URL': The URL for the English article from which the sentence was extracted.
* 'domain': The domain of the sentence.
* 'topic': The topic of the sentence.
* 'has\_image': Whether the original article contains an image.
* 'has\_hyperlink': Whether the sentence contains a hyperlink. ### Data Splits ### Dataset Creation


Please refer to the original article No Language Left Behind: Scaling Human-Centered Machine Translation for additional information on dataset creation.


Additional Information
---------------------- ### Dataset Curators


See paper for details. ### Licensing Information


Licensed with Creative Commons Attribution Share Alike 4.0. License available here.


Please cite the authors if you use these corpora in your work:


Please also cite prior work that this dataset builds on:
passage:  passage: TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-found #language_creators-expert-generated #multilinguality-multilingual #multilinguality-translation #size_categories-unknown #source_datasets-extended|flores #language-Achinese #language-Mesopotamian Arabic #language-Ta'izzi-Adeni Arabic #language-Tunisian Arabic #language-Afrikaans #language-South Levantine Arabic #language-Akan #language-Tosk Albanian #language-Amharic #language-Levantine Arabic #language-Arabic #language-Najdi Arabic #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Awadhi #language-Central Aymara #language-South Azerbaijani #language-North Azerbaijani #language-Bashkir #language-Bambara #language-Balinese #language-Belarusian #language-Bemba (Zambia) #language-Bengali #language-Bhojpuri #language-Banjar #language-Tibetan #language-Bosnian #language-Buginese #language-Bulgarian #language-Catalan #language-Cebuano #language-Czech #language-Chokwe #language-Central Kurdish #language-Crimean Tatar #language-Welsh #language-Danish #language-German #language-Southwestern Dinka #language-Dyula #language-Dzongkha #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Estonian #language-Basque #language-Ewe #language-Faroese #language-Fijian #language-Finnish #language-Fon #language-French #language-Friulian #language-Nigerian Fulfulde #language-West Central Oromo #language-Scottish Gaelic #language-Irish #language-Galician #language-Guarani #language-Gujarati #language-Haitian #language-Hausa #language-Hebrew #language-Hindi #language-Chhattisgarhi #language-Croatian #language-Hungarian #language-Armenian #language-Igbo #language-Iloko #language-Indonesian #language-Icelandic #language-Italian #language-Javanese #language-Japanese #language-Kabyle #language-Kachin #language-Kamba (Kenya) #language-Kannada #language-Kashmiri #language-Georgian #language-Kazakh #language-Kabiyè #language-Kabuverdianu #language-Halh Mongolian #language-Khmer #language-Kikuyu #language-Kinyarwanda #language-Kirghiz #language-Kimbundu #language-Northern Kurdish #language-Central Kanuri #language-Kongo #language-Korean #language-Lao #language-Ligurian #language-Limburgan #language-Lingala #language-Lithuanian #language-Lombard #language-Latgalian #language-Luxembourgish #language-Luba-Lulua #language-Ganda #language-Luo (Kenya and Tanzania) #language-Lushai #language-Standard Latvian #language-Magahi #language-Maithili #language-Malayalam #language-Marathi #language-Minangkabau #language-Macedonian #language-Maltese #language-Manipuri #language-Mossi #language-Maori #language-Burmese #language-Dutch #language-Norwegian Nynorsk #language-Norwegian Bokmål #language-Nepali (individual language) #language-Pedi #language-Nuer #language-Nyanja #language-Occitan (post 1500) #language-Odia #language-Pangasinan #language-Panjabi #language-Papiamento #language-Southern Pashto #language-Iranian Persian #language-Plateau Malagasy #language-Polish #language-Portuguese #language-Dari #language-Ayacucho Quechua #language-Romanian #language-Rundi #language-Russian #language-Sango #language-Sanskrit #language-Santali #language-Sicilian #language-Shan #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Sindhi #language-Somali #language-Southern Sotho #language-Spanish #language-Sardinian #language-Serbian #language-Swati #language-Sundanese #language-Swedish #language-Swahili (individual language) #language-Silesian #language-Tamil #language-Tamasheq #language-Tatar #language-Telugu #language-Tajik #language-Tagalog #language-Thai #language-Tigrinya #language-Tok Pisin #language-Tswana #language-Tsonga #language-Turkmen #language-Tumbuka #language-Turkish #language-Twi #language-Central Atlas Tamazight #language-Uighur #language-Ukrainian #language-Umbundu #language-Urdu #language-Northern Uzbek #language-Venetian #language-Vietnamese #language-Waray (Philippines) #language-Wolof #language-Xhosa #language-Eastern Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #language-Standard Malay #language-Zulu #license-cc-by-sa-4.0 #conditional-text-generation #arxiv-2207.04672 #region-us 
### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES-200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores-200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.### Supported Tasks and Leaderboards#### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this.### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.


Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.


Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
-----------------