# Dataset Card for allenai/wmt22_african

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
  - [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** https://www.statmt.org/wmt22/large-scale-multilingual-translation-task.html
- **Repository:** [Needs More Information]
- **Paper:** [Needs More Information]
- **Leaderboard:** [Needs More Information]
- **Point of Contact:** [Needs More Information]

### Dataset Summary

This dataset was created based on [metadata](https://github.com/facebookresearch/LASER/tree/main/data/wmt22_african) for mined bitext released by Meta AI.  It contains bitext for 248 pairs for the African languages that are part of the [2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages](https://www.statmt.org/wmt22/large-scale-multilingual-translation-task.html).

#### How to use the data

There are two ways to access the data:

* Via the Hugging Face Python datasets library 

```
from datasets import load_dataset
dataset = load_dataset("allenai/wmt22_african")
```

* Clone the git repo
```
git lfs install
git clone https://huggingface.co/datasets/allenai/wmt22_african
```


### Supported Tasks and Leaderboards

This dataset is one of resources allowed under the Constrained Track for the [2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages](https://www.statmt.org/wmt22/large-scale-multilingual-translation-task.html).

### Languages

#### Focus languages

| Language | Code |
| -------- | ---- |
| Afrikaans | afr |
| Amharic | amh |
| Chichewa | nya |
| Nigerian Fulfulde | fuv |
| Hausa | hau |
| Igbo | ibo |
| Kamba | kam |
| Kinyarwanda | kin |
| Lingala | lin |
| Luganda | lug |
| Luo | luo |
| Northern Sotho | nso |
| Oroma | orm |
| Shona | sna |
| Somali | som |
| Swahili | swh |
| Swati | ssw |
| Tswana | tsn | 
| Umbundu | umb |
| Wolof | wol |
| Xhosa | xho |
| Xitsonga | tso |
| Yoruba | yor |
| Zulu | zul |

Colonial linguae francae: English - eng, French - fra

## Dataset Structure

The dataset contains gzipped tab delimited text files for each direction.  Each text file contains lines with parallel sentences.

### Data Instances

The dataset contains 248 language pairs.

Sentence counts for each pair can be found [here](https://huggingface.co/datasets/allenai/wmt22_african/blob/main/sentence_counts.txt).

### Data Fields

Every instance for a language pair contains the following fields: 'translation' (containing sentence pairs), 'laser_score', 'source_sentence_lid', 'target_sentence_lid', where 'lid' is language classification probability.

Example:
```
{
'translation': 
    {
        'afr': 'In Mei 2007, in ooreenstemming met die spesifikasies van die Java Gemeenskapproses, het Sun Java tegnologie geherlisensieer onder die GNU General Public License.', 
        'eng': 'As of May 2007, in compliance with the specifications of the Java Community Process, Sun relicensed most of its Java technologies under the GNU General Public License.'
    }, 
'laser_score': 1.0717015266418457, 
'source_sentence_lid': 0.9996600151062012, 
'target_sentence_lid': 0.9972000122070312
}
```
### Data Splits

The data is not split into train, dev, and test.

## Dataset Creation

### Curation Rationale

Parallel sentences from monolingual data in Common Crawl and ParaCrawl were identified via [Language-Agnostic Sentence Representation (LASER)](https://github.com/facebookresearch/LASER) encoders.

### Source Data

#### Initial Data Collection and Normalization

Monolingual data was obtained from Common Crawl and ParaCrawl.

#### Who are the source language producers?

Contributors to web text in Common Crawl and ParaCrawl.

### Annotations

#### Annotation process

The data was not human annotated. The metadata used to create the dataset can be found here: https://github.com/facebookresearch/LASER/tree/main/data/wmt22_african

#### Who are the annotators?

The data was not human annotated.  Parallel text from Common Crawl and Para Crawl monolingual data were identified automatically via [LASER](https://github.com/facebookresearch/LASER) encoders. 

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

This dataset provides data for training machine learning systems for many languages that have low resources available for NLP.

### Discussion of Biases

Biases in the data have not been studied.

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

[Needs More Information]

### Licensing Information

The dataset is released under the terms of [ODC-BY](https://opendatacommons.org/licenses/by/1-0/). By using this, you are also bound by the Internet Archive [Terms of Use](https://archive.org/about/terms.php) in respect of the content contained in the dataset.

### Citation Information

NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv, 2022.

### Contributions

We thank the AllenNLP team at AI2 for hosting and releasing this data, including [Akshita Bhagia](https://akshitab.github.io/) (for engineering efforts to create the huggingface dataset), and [Jesse Dodge](https://jessedodge.github.io/) (for organizing the connection).

region:us
{}
TAGS
#region-us 

Dataset Card for allenai/wmt22\_african
=======================================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information


Dataset Description
-------------------


* Homepage: URL
* Repository:
* Paper:
* Leaderboard:
* Point of Contact:


### Dataset Summary


This dataset was created based on metadata for mined bitext released by Meta AI. It contains bitext for 248 pairs for the African languages that are part of the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages.


#### How to use the data


There are two ways to access the data:


* Via the Hugging Face Python datasets library
* Clone the git repo


### Supported Tasks and Leaderboards


This dataset is one of resources allowed under the Constrained Track for the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages.


### Languages


#### Focus languages



Colonial linguae francae: English - eng, French - fra


Dataset Structure
-----------------


The dataset contains gzipped tab delimited text files for each direction. Each text file contains lines with parallel sentences.


### Data Instances


The dataset contains 248 language pairs.


Sentence counts for each pair can be found here.


### Data Fields


Every instance for a language pair contains the following fields: 'translation' (containing sentence pairs), 'laser\_score', 'source\_sentence\_lid', 'target\_sentence\_lid', where 'lid' is language classification probability.


Example:


### Data Splits


The data is not split into train, dev, and test.


Dataset Creation
----------------


### Curation Rationale


Parallel sentences from monolingual data in Common Crawl and ParaCrawl were identified via Language-Agnostic Sentence Representation (LASER) encoders.


### Source Data


#### Initial Data Collection and Normalization


Monolingual data was obtained from Common Crawl and ParaCrawl.


#### Who are the source language producers?


Contributors to web text in Common Crawl and ParaCrawl.


### Annotations


#### Annotation process


The data was not human annotated. The metadata used to create the dataset can be found here: URL


#### Who are the annotators?


The data was not human annotated. Parallel text from Common Crawl and Para Crawl monolingual data were identified automatically via LASER encoders.


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


This dataset provides data for training machine learning systems for many languages that have low resources available for NLP.


### Discussion of Biases


Biases in the data have not been studied.


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


The dataset is released under the terms of ODC-BY. By using this, you are also bound by the Internet Archive Terms of Use in respect of the content contained in the dataset.


NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv, 2022.


### Contributions


We thank the AllenNLP team at AI2 for hosting and releasing this data, including Akshita Bhagia (for engineering efforts to create the huggingface dataset), and Jesse Dodge (for organizing the connection).



### Dataset Summary


This dataset was created based on metadata for mined bitext released by Meta AI. It contains bitext for 248 pairs for the African languages that are part of the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages. #### How to use the data


There are two ways to access the data:


* Via the Hugging Face Python datasets library
* Clone the git repo ### Supported Tasks and Leaderboards


This dataset is one of resources allowed under the Constrained Track for the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages. ### Languages #### Focus languages



Colonial linguae francae: English - eng, French - fra


Dataset Structure
-----------------


The dataset contains gzipped tab delimited text files for each direction. Each text file contains lines with parallel sentences. ### Data Instances


The dataset contains 248 language pairs.


Sentence counts for each pair can be found here. ### Data Fields


Every instance for a language pair contains the following fields: 'translation' (containing sentence pairs), 'laser\_score', 'source\_sentence\_lid', 'target\_sentence\_lid', where 'lid' is language classification probability.


Example: ### Data Splits


The data is not split into train, dev, and test.


Dataset Creation
---------------- ### Curation Rationale


Parallel sentences from monolingual data in Common Crawl and ParaCrawl were identified via Language-Agnostic Sentence Representation (LASER) encoders. ### Source Data #### Initial Data Collection and Normalization


Monolingual data was obtained from Common Crawl and ParaCrawl. #### Who are the source language producers?


Contributors to web text in Common Crawl and ParaCrawl. ### Annotations #### Annotation process


The data was not human annotated. The metadata used to create the dataset can be found here: URL #### Who are the annotators?


The data was not human annotated. Parallel text from Common Crawl and Para Crawl monolingual data were identified automatically via LASER encoders. ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


This dataset provides data for training machine learning systems for many languages that have low resources available for NLP. ### Discussion of Biases


Biases in the data have not been studied. ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


The dataset is released under the terms of ODC-BY. By using this, you are also bound by the Internet Archive Terms of Use in respect of the content contained in the dataset.


NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv, 2022. ### Contributions


We thank the AllenNLP team at AI2 for hosting and releasing this data, including Akshita Bhagia (for engineering efforts to create the huggingface dataset), and Jesse Dodge (for organizing the connection).
TAGS
#region-us 
 ### Dataset Summary


This dataset was created based on metadata for mined bitext released by Meta AI. It contains bitext for 248 pairs for the African languages that are part of the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages. #### How to use the data


There are two ways to access the data:


* Via the Hugging Face Python datasets library
* Clone the git repo ### Supported Tasks and Leaderboards


This dataset is one of resources allowed under the Constrained Track for the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages. ### Languages #### Focus languages



Colonial linguae francae: English - eng, French - fra


Dataset Structure
-----------------


The dataset contains gzipped tab delimited text files for each direction. Each text file contains lines with parallel sentences. ### Data Instances


The dataset contains 248 language pairs.


Sentence counts for each pair can be found here. ### Data Fields


Every instance for a language pair contains the following fields: 'translation' (containing sentence pairs), 'laser\_score', 'source\_sentence\_lid', 'target\_sentence\_lid', where 'lid' is language classification probability.


Example: ### Data Splits


The data is not split into train, dev, and test.


Dataset Creation
---------------- ### Curation Rationale


Parallel sentences from monolingual data in Common Crawl and ParaCrawl were identified via Language-Agnostic Sentence Representation (LASER) encoders. ### Source Data #### Initial Data Collection and Normalization


Monolingual data was obtained from Common Crawl and ParaCrawl. #### Who are the source language producers?


Contributors to web text in Common Crawl and ParaCrawl. ### Annotations #### Annotation process


The data was not human annotated. The metadata used to create the dataset can be found here: URL #### Who are the annotators?


The data was not human annotated. Parallel text from Common Crawl and Para Crawl monolingual data were identified automatically via LASER encoders. ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


This dataset provides data for training machine learning systems for many languages that have low resources available for NLP. ### Discussion of Biases


Biases in the data have not been studied. ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


The dataset is released under the terms of ODC-BY. By using this, you are also bound by the Internet Archive Terms of Use in respect of the content contained in the dataset.


NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv, 2022. ### Contributions


We thank the AllenNLP team at AI2 for hosting and releasing this data, including Akshita Bhagia (for engineering efforts to create the huggingface dataset), and Jesse Dodge (for organizing the connection).