
# Dataset Card for Flores200

## Table of Contents

- [Dataset Card for Flores200](#dataset-card-for-flores200)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
    - [Dataset Creation](#dataset-creation)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)

## Dataset Description

- **Home:** [Flores](https://github.com/facebookresearch/flores)
- **Repository:** [Github](https://github.com/facebookresearch/flores)

### Dataset Summary

FLORES is a benchmark dataset for machine translation between English and low-resource languages.

>The creation of FLORES200 doubles the existing language coverage of FLORES-101. 
Given the nature of the new languages, which have less standardization and require 
more specialized professional translations, the verification process became more complex. 
This required modifications to the translation workflow. FLORES-200 has several languages 
which were not translated from English. Specifically, several languages were translated 
from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also 
includes two script alternatives for four languages. FLORES-200 consists of translations 
from 842 distinct web articles, totaling 3001 sentences. These sentences are divided 
into three splits: dev, devtest, and test (hidden). On average, sentences are approximately 
21 words long.

**Disclaimer**: *The Flores200 dataset is hosted by the Facebook and licensed under the [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/).
### Supported Tasks and Leaderboards
#### Multilingual Machine Translation
Refer to the [Dynabench leaderboard](https://dynabench.org/flores/Flores%20MT%20Evaluation%20(FULL)) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on [Large-Scale Multilingual Machine Translation](http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html). Flores 200 is an extention of this.
### Languages
The dataset contains parallel sentences for 200 languages, as mentioned in the original [Github](https://github.com/facebookresearch/flores/blob/master/README.md) page for the project. Languages are identified with the ISO 639-3 code (e.g. `eng`, `fra`, `rus`) plus an additional code describing the script (e.g., "eng_Latn", "ukr_Cyrl"). See [the webpage for code descriptions](https://github.com/facebookresearch/flores/blob/main/flores200/README.md).
Use the configuration `all` to access the full set of parallel sentences for all the available languages in a single command. 
Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng_Latn-ukr_Cyrl" will provide sentences in the format below).
## Dataset Structure
### Data Instances
A sample from the `dev` split for the Ukrainian language (`ukr_Cyrl` config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.
```python
{
	'id': 1,
	'sentence': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.',
	'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet',
	'domain': 'wikinews',
	'topic': 'health',
	'has_image': 0,
	'has_hyperlink': 0
}
```
When using a hyphenated pairing or using the `all` function, data will be presented as follows:
```python
{
    'id': 1, 
    'URL': 'https://en.wikinews.org/wiki/Scientists_say_new_medical_diagnostic_chip_can_sort_cells_anywhere_with_an_inkjet', 
    'domain': 'wikinews', 
    'topic': 'health', 
    'has_image': 0, 
    'has_hyperlink': 0, 
    'sentence_eng_Latn': 'On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.', 
    'sentence_ukr_Cyrl': 'У понеділок, науковці зі Школи медицини Стенфордського університету оголосили про винайдення нового діагностичного інструменту, що може сортувати клітини за їх видами: це малесенький друкований чіп, який можна виготовити за допомогою стандартних променевих принтерів десь по одному центу США за штуку.'
}
```
The text is provided as-in the original dataset, without further preprocessing or tokenization.
### Data Fields
- `id`: Row number for the data entry, starting at 1.
- `sentence`: The full sentence in the specific language (may have _lang for pairings)
- `URL`: The URL for the English article from which the sentence was extracted.
- `domain`: The domain of the sentence.
- `topic`: The topic of the sentence.
- `has_image`: Whether the  original article contains an image.
- `has_hyperlink`: Whether the  sentence contains a hyperlink.
### Data Splits
|            config| `dev`| `devtest`|
|-----------------:|-----:|---------:|
|all configurations|   997|     1012:|
### Dataset Creation
Please refer to the original article [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) for additional information on dataset creation.
## Additional Information
### Dataset Curators
See paper for details.
### Licensing Information
Licensed with Creative Commons Attribution Share Alike 4.0. License available [here](https://creativecommons.org/licenses/by-sa/4.0/).
### Citation Information
Please cite the authors if you use these corpora in your work:
```bibtex
@article{nllb2022,
  author    = {NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi,  Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang},
  title     = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  year      = {2022}
}
```

task_categories:text2text-generation task_categories:translation annotations_creators:found language_creators:expert-generated multilinguality:multilingual multilinguality:translation size_categories:unknown source_datasets:extended|flores language:ace language:acm language:acq language:aeb language:afr language:ajp language:aka language:als language:amh language:apc language:arb language:ars language:ary language:arz language:asm language:ast language:awa language:ayr language:azb language:azj language:bak language:bam language:ban language:bel language:bem language:ben language:bho language:bjn language:bod language:bos language:bug language:bul language:cat language:ceb language:ces language:cjk language:ckb language:crh language:cym language:dan language:deu language:dik language:dyu language:dzo language:ell language:eng language:epo language:est language:eus language:ewe language:fao language:fij language:fin language:fon language:fra language:fur language:fuv language:gaz language:gla language:gle language:glg language:grn language:guj language:hat language:hau language:heb language:hin language:hne language:hrv language:hun language:hye language:ibo language:ilo language:ind language:isl language:ita language:jav language:jpn language:kab language:kac language:kam language:kan language:kas language:kat language:kaz language:kbp language:kea language:khk language:khm language:kik language:kin language:kir language:kmb language:kmr language:knc language:kon language:kor language:lao language:lij language:lim language:lin language:lit language:lmo language:ltg language:ltz language:lua language:lug language:luo language:lus language:lvs language:mag language:mai language:mal language:mar language:min language:mkd language:mlt language:mni language:mos language:mri language:mya language:nld language:nno language:nob language:npi language:nso language:nus language:nya language:oci language:ory language:pag language:pan language:pap language:pbt language:pes language:plt language:pol language:por language:prs language:quy language:ron language:run language:rus language:sag language:san language:sat language:scn language:shn language:sin language:slk language:slv language:smo language:sna language:snd language:som language:sot language:spa language:srd language:srp language:ssw language:sun language:swe language:swh language:szl language:tam language:taq language:tat language:tel language:tgk language:tgl language:tha language:tir language:tpi language:tsn language:tso language:tuk language:tum language:tur language:twi language:tzm language:uig language:ukr language:umb language:urd language:uzn language:vec language:vie language:war language:wol language:xho language:ydd language:yor language:yue language:zho language:zsm language:zul license:cc-by-sa-4.0 conditional-text-generation arxiv:2207.04672 region:us
{"annotations_creators": ["found"], "language_creators": ["expert-generated"], "language": ["ace", "acm", "acq", "aeb", "afr", "ajp", "aka", "als", "amh", "apc", "arb", "ars", "ary", "arz", "asm", "ast", "awa", "ayr", "azb", "azj", "bak", "bam", "ban", "bel", "bem", "ben", "bho", "bjn", "bod", "bos", "bug", "bul", "cat", "ceb", "ces", "cjk", "ckb", "crh", "cym", "dan", "deu", "dik", "dyu", "dzo", "ell", "eng", "epo", "est", "eus", "ewe", "fao", "fij", "fin", "fon", "fra", "fur", "fuv", "gaz", "gla", "gle", "glg", "grn", "guj", "hat", "hau", "heb", "hin", "hne", "hrv", "hun", "hye", "ibo", "ilo", "ind", "isl", "ita", "jav", "jpn", "kab", "kac", "kam", "kan", "kas", "kat", "kaz", "kbp", "kea", "khk", "khm", "kik", "kin", "kir", "kmb", "kmr", "knc", "kon", "kor", "lao", "lij", "lim", "lin", "lit", "lmo", "ltg", "ltz", "lua", "lug", "luo", "lus", "lvs", "mag", "mai", "mal", "mar", "min", "mkd", "mlt", "mni", "mos", "mri", "mya", "nld", "nno", "nob", "npi", "nso", "nus", "nya", "oci", "ory", "pag", "pan", "pap", "pbt", "pes", "plt", "pol", "por", "prs", "quy", "ron", "run", "rus", "sag", "san", "sat", "scn", "shn", "sin", "slk", "slv", "smo", "sna", "snd", "som", "sot", "spa", "srd", "srp", "ssw", "sun", "swe", "swh", "szl", "tam", "taq", "tat", "tel", "tgk", "tgl", "tha", "tir", "tpi", "tsn", "tso", "tuk", "tum", "tur", "twi", "tzm", "uig", "ukr", "umb", "urd", "uzn", "vec", "vie", "war", "wol", "xho", "ydd", "yor", "yue", "zho", "zsm", "zul"], "license": ["cc-by-sa-4.0"], "multilinguality": ["multilingual", "translation"], "size_categories": ["unknown"], "source_datasets": ["extended|flores"], "task_categories": ["text2text-generation", "translation"], "task_ids": [], "paperswithcode_id": "flores", "pretty_name": "flores200", "tags": ["conditional-text-generation"]}
TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-found #language_creators-expert-generated #multilinguality-multilingual #multilinguality-translation #size_categories-unknown #source_datasets-extended|flores #language-Achinese #language-Mesopotamian Arabic #language-Ta'izzi-Adeni Arabic #language-Tunisian Arabic #language-Afrikaans #language-South Levantine Arabic #language-Akan #language-Tosk Albanian #language-Amharic #language-Levantine Arabic #language-Standard Arabic #language-Najdi Arabic #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Awadhi #language-Central Aymara #language-South Azerbaijani #language-North Azerbaijani #language-Bashkir #language-Bambara #language-Balinese #language-Belarusian #language-Bemba (Zambia) #language-Bengali #language-Bhojpuri #language-Banjar #language-Tibetan #language-Bosnian #language-Buginese #language-Bulgarian #language-Catalan #language-Cebuano #language-Czech #language-Chokwe #language-Central Kurdish #language-Crimean Tatar #language-Welsh #language-Danish #language-German #language-Southwestern Dinka #language-Dyula #language-Dzongkha #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Estonian #language-Basque #language-Ewe #language-Faroese #language-Fijian #language-Finnish #language-Fon #language-French #language-Friulian #language-Nigerian Fulfulde #language-West Central Oromo #language-Scottish Gaelic #language-Irish #language-Galician #language-Guarani #language-Gujarati #language-Haitian #language-Hausa #language-Hebrew #language-Hindi #language-Chhattisgarhi #language-Croatian #language-Hungarian #language-Armenian #language-Igbo #language-Iloko #language-Indonesian #language-Icelandic #language-Italian #language-Javanese #language-Japanese #language-Kabyle #language-Kachin #language-Kamba (Kenya) #language-Kannada #language-Kashmiri #language-Georgian #language-Kazakh #language-Kabiyè #language-Kabuverdianu #language-Halh Mongolian #language-Khmer #language-Kikuyu #language-Kinyarwanda #language-Kirghiz #language-Kimbundu #language-Northern Kurdish #language-Central Kanuri #language-Kongo #language-Korean #language-Lao #language-Ligurian #language-Limburgan #language-Lingala #language-Lithuanian #language-Lombard #language-Latgalian #language-Luxembourgish #language-Luba-Lulua #language-Ganda #language-Luo (Kenya and Tanzania) #language-Lushai #language-Standard Latvian #language-Magahi #language-Maithili #language-Malayalam #language-Marathi #language-Minangkabau #language-Macedonian #language-Maltese #language-Manipuri #language-Mossi #language-Maori #language-Burmese #language-Dutch #language-Norwegian Nynorsk #language-Norwegian Bokmål #language-Nepali (individual language) #language-Pedi #language-Nuer #language-Nyanja #language-Occitan (post 1500) #language-Odia #language-Pangasinan #language-Panjabi #language-Papiamento #language-Southern Pashto #language-Iranian Persian #language-Plateau Malagasy #language-Polish #language-Portuguese #language-Dari #language-Ayacucho Quechua #language-Romanian #language-Rundi #language-Russian #language-Sango #language-Sanskrit #language-Santali #language-Sicilian #language-Shan #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Sindhi #language-Somali #language-Southern Sotho #language-Spanish #language-Sardinian #language-Serbian #language-Swati #language-Sundanese #language-Swedish #language-Swahili (individual language) #language-Silesian #language-Tamil #language-Tamasheq #language-Tatar #language-Telugu #language-Tajik #language-Tagalog #language-Thai #language-Tigrinya #language-Tok Pisin #language-Tswana #language-Tsonga #language-Turkmen #language-Tumbuka #language-Turkish #language-Twi #language-Central Atlas Tamazight #language-Uighur #language-Ukrainian #language-Umbundu #language-Urdu #language-Northern Uzbek #language-Venetian #language-Vietnamese #language-Waray (Philippines) #language-Wolof #language-Xhosa #language-Eastern Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #language-Standard Malay #language-Zulu #license-cc-by-sa-4.0 #conditional-text-generation #arxiv-2207.04672 #region-us 

Dataset Card for Flores200
==========================


Table of Contents
-----------------


* Dataset Card for Flores200
	+ Table of Contents
	+ Dataset Description
		- Dataset Summary
		- Supported Tasks and Leaderboards
		- Languages
	+ Dataset Structure
		- Data Instances
		- Data Fields
		- Data Splits
		- Dataset Creation
	+ Additional Information
		- Dataset Curators
		- Licensing Information
		- Citation Information


Dataset Description
-------------------


* Home: Flores
* Repository: Github


### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.


### Supported Tasks and Leaderboards


#### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this.


### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.
Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.
Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
-----------------


### Data Instances


A sample from the 'dev' split for the Ukrainian language ('ukr\_Cyrl' config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.


When using a hyphenated pairing or using the 'all' function, data will be presented as follows:


The text is provided as-in the original dataset, without further preprocessing or tokenization.


### Data Fields


* 'id': Row number for the data entry, starting at 1.
* 'sentence': The full sentence in the specific language (may have \_lang for pairings)
* 'URL': The URL for the English article from which the sentence was extracted.
* 'domain': The domain of the sentence.
* 'topic': The topic of the sentence.
* 'has\_image': Whether the original article contains an image.
* 'has\_hyperlink': Whether the sentence contains a hyperlink.


### Data Splits



### Dataset Creation


Please refer to the original article No Language Left Behind: Scaling Human-Centered Machine Translation for additional information on dataset creation.


Additional Information
----------------------


### Dataset Curators


See paper for details.


### Licensing Information


Licensed with Creative Commons Attribution Share Alike 4.0. License available here.


Please cite the authors if you use these corpora in your work:



### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. ### Supported Tasks and Leaderboards #### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this. ### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.
Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.
Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
----------------- ### Data Instances


A sample from the 'dev' split for the Ukrainian language ('ukr\_Cyrl' config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.


When using a hyphenated pairing or using the 'all' function, data will be presented as follows:


The text is provided as-in the original dataset, without further preprocessing or tokenization. ### Data Fields


* 'id': Row number for the data entry, starting at 1.
* 'sentence': The full sentence in the specific language (may have \_lang for pairings)
* 'URL': The URL for the English article from which the sentence was extracted.
* 'domain': The domain of the sentence.
* 'topic': The topic of the sentence.
* 'has\_image': Whether the original article contains an image.
* 'has\_hyperlink': Whether the sentence contains a hyperlink. ### Data Splits ### Dataset Creation


Please refer to the original article No Language Left Behind: Scaling Human-Centered Machine Translation for additional information on dataset creation.


Additional Information
---------------------- ### Dataset Curators


See paper for details. ### Licensing Information


Licensed with Creative Commons Attribution Share Alike 4.0. License available here.


Please cite the authors if you use these corpora in your work:
TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-found #language_creators-expert-generated #multilinguality-multilingual #multilinguality-translation #size_categories-unknown #source_datasets-extended|flores #language-Achinese #language-Mesopotamian Arabic #language-Ta'izzi-Adeni Arabic #language-Tunisian Arabic #language-Afrikaans #language-South Levantine Arabic #language-Akan #language-Tosk Albanian #language-Amharic #language-Levantine Arabic #language-Standard Arabic #language-Najdi Arabic #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Awadhi #language-Central Aymara #language-South Azerbaijani #language-North Azerbaijani #language-Bashkir #language-Bambara #language-Balinese #language-Belarusian #language-Bemba (Zambia) #language-Bengali #language-Bhojpuri #language-Banjar #language-Tibetan #language-Bosnian #language-Buginese #language-Bulgarian #language-Catalan #language-Cebuano #language-Czech #language-Chokwe #language-Central Kurdish #language-Crimean Tatar #language-Welsh #language-Danish #language-German #language-Southwestern Dinka #language-Dyula #language-Dzongkha #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Estonian #language-Basque #language-Ewe #language-Faroese #language-Fijian #language-Finnish #language-Fon #language-French #language-Friulian #language-Nigerian Fulfulde #language-West Central Oromo #language-Scottish Gaelic #language-Irish #language-Galician #language-Guarani #language-Gujarati #language-Haitian #language-Hausa #language-Hebrew #language-Hindi #language-Chhattisgarhi #language-Croatian #language-Hungarian #language-Armenian #language-Igbo #language-Iloko #language-Indonesian #language-Icelandic #language-Italian #language-Javanese #language-Japanese #language-Kabyle #language-Kachin #language-Kamba (Kenya) #language-Kannada #language-Kashmiri #language-Georgian #language-Kazakh #language-Kabiyè #language-Kabuverdianu #language-Halh Mongolian #language-Khmer #language-Kikuyu #language-Kinyarwanda #language-Kirghiz #language-Kimbundu #language-Northern Kurdish #language-Central Kanuri #language-Kongo #language-Korean #language-Lao #language-Ligurian #language-Limburgan #language-Lingala #language-Lithuanian #language-Lombard #language-Latgalian #language-Luxembourgish #language-Luba-Lulua #language-Ganda #language-Luo (Kenya and Tanzania) #language-Lushai #language-Standard Latvian #language-Magahi #language-Maithili #language-Malayalam #language-Marathi #language-Minangkabau #language-Macedonian #language-Maltese #language-Manipuri #language-Mossi #language-Maori #language-Burmese #language-Dutch #language-Norwegian Nynorsk #language-Norwegian Bokmål #language-Nepali (individual language) #language-Pedi #language-Nuer #language-Nyanja #language-Occitan (post 1500) #language-Odia #language-Pangasinan #language-Panjabi #language-Papiamento #language-Southern Pashto #language-Iranian Persian #language-Plateau Malagasy #language-Polish #language-Portuguese #language-Dari #language-Ayacucho Quechua #language-Romanian #language-Rundi #language-Russian #language-Sango #language-Sanskrit #language-Santali #language-Sicilian #language-Shan #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Sindhi #language-Somali #language-Southern Sotho #language-Spanish #language-Sardinian #language-Serbian #language-Swati #language-Sundanese #language-Swedish #language-Swahili (individual language) #language-Silesian #language-Tamil #language-Tamasheq #language-Tatar #language-Telugu #language-Tajik #language-Tagalog #language-Thai #language-Tigrinya #language-Tok Pisin #language-Tswana #language-Tsonga #language-Turkmen #language-Tumbuka #language-Turkish #language-Twi #language-Central Atlas Tamazight #language-Uighur #language-Ukrainian #language-Umbundu #language-Urdu #language-Northern Uzbek #language-Venetian #language-Vietnamese #language-Waray (Philippines) #language-Wolof #language-Xhosa #language-Eastern Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #language-Standard Malay #language-Zulu #license-cc-by-sa-4.0 #conditional-text-generation #arxiv-2207.04672 #region-us 
 ### Dataset Summary


FLORES is a benchmark dataset for machine translation between English and low-resource languages.



> 
> The creation of FLORES200 doubles the existing language coverage of FLORES-101.
> Given the nature of the new languages, which have less standardization and require
> more specialized professional translations, the verification process became more complex.
> This required modifications to the translation workflow. FLORES-200 has several languages
> which were not translated from English. Specifically, several languages were translated
> from Spanish, French, Russian and Modern Standard Arabic. Moreover, FLORES-200 also
> includes two script alternatives for four languages. FLORES-200 consists of translations
> from 842 distinct web articles, totaling 3001 sentences. These sentences are divided
> into three splits: dev, devtest, and test (hidden). On average, sentences are approximately
> 21 words long.
> 
> 
> 


Disclaimer: \*The Flores200 dataset is hosted by the Facebook and licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. ### Supported Tasks and Leaderboards #### Multilingual Machine Translation


Refer to the Dynabench leaderboard) for additional details on model evaluation on FLORES-101 in the context of the WMT2021 shared task on Large-Scale Multilingual Machine Translation. Flores 200 is an extention of this. ### Languages


The dataset contains parallel sentences for 200 languages, as mentioned in the original Github page for the project. Languages are identified with the ISO 639-3 code (e.g. 'eng', 'fra', 'rus') plus an additional code describing the script (e.g., "eng\_Latn", "ukr\_Cyrl"). See the webpage for code descriptions.
Use the configuration 'all' to access the full set of parallel sentences for all the available languages in a single command.
Use a hyphenated pairing to get two langauges in one datapoint (e.g., "eng\_Latn-ukr\_Cyrl" will provide sentences in the format below).


Dataset Structure
----------------- ### Data Instances


A sample from the 'dev' split for the Ukrainian language ('ukr\_Cyrl' config) is provided below. All configurations have the same structure, and all sentences are aligned across configurations and splits.


When using a hyphenated pairing or using the 'all' function, data will be presented as follows:


The text is provided as-in the original dataset, without further preprocessing or tokenization. ### Data Fields


* 'id': Row number for the data entry, starting at 1.
* 'sentence': The full sentence in the specific language (may have \_lang for pairings)
* 'URL': The URL for the English article from which the sentence was extracted.
* 'domain': The domain of the sentence.
* 'topic': The topic of the sentence.
* 'has\_image': Whether the original article contains an image.
* 'has\_hyperlink': Whether the sentence contains a hyperlink. ### Data Splits ### Dataset Creation


Please refer to the original article No Language Left Behind: Scaling Human-Centered Machine Translation for additional information on dataset creation.


Additional Information
---------------------- ### Dataset Curators


See paper for details. ### Licensing Information


Licensed with Creative Commons Attribution Share Alike 4.0. License available here.


Please cite the authors if you use these corpora in your work: