
# Dataset Card for mC4

## Table of Contents

- [Dataset Card for mC4](#dataset-card-for-mc4)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
      - [Who are the source language producers?](#who-are-the-source-language-producers)
    - [Annotations](#annotations)
      - [Annotation process](#annotation-process)
      - [Who are the annotators?](#who-are-the-annotators)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://huggingface.co/datasets/allenai/c4
- **Paper:** https://arxiv.org/abs/1910.10683

### Dataset Summary

A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".

This is the version prepared by AllenAI, hosted at this address: https://huggingface.co/datasets/allenai/c4

108 languages are available and are reported in the table below.

Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script.

| language code   | language name        |
|:----------------|:---------------------|
| af              | Afrikaans            |
| am              | Amharic              |
| ar              | Arabic               |
| az              | Azerbaijani          |
| be              | Belarusian           |
| bg              | Bulgarian            |
| bg-Latn         | Bulgarian (Latin)    |
| bn              | Bangla               |
| ca              | Catalan              |
| ceb             | Cebuano              |
| co              | Corsican             |
| cs              | Czech                |
| cy              | Welsh                |
| da              | Danish               |
| de              | German               |
| el              | Greek                |
| el-Latn         | Greek (Latin)        |
| en              | English              |
| eo              | Esperanto            |
| es              | Spanish              |
| et              | Estonian             |
| eu              | Basque               |
| fa              | Persian              |
| fi              | Finnish              |
| fil             | Filipino             |
| fr              | French               |
| fy              | Western Frisian      |
| ga              | Irish                |
| gd              | Scottish Gaelic      |
| gl              | Galician             |
| gu              | Gujarati             |
| ha              | Hausa                |
| haw             | Hawaiian             |
| hi              | Hindi                |
| hi-Latn         | Hindi (Latin script) |
| hmn             | Hmong, Mong          |
| ht              | Haitian              |
| hu              | Hungarian            |
| hy              | Armenian             |
| id              | Indonesian           |
| ig              | Igbo                 |
| is              | Icelandic            |
| it              | Italian              |
| iw              | former Hebrew        |
| ja              | Japanese             |
| ja-Latn         | Japanese (Latin)     |
| jv              | Javanese             |
| ka              | Georgian             |
| kk              | Kazakh               |
| km              | Khmer                |
| kn              | Kannada              |
| ko              | Korean               |
| ku              | Kurdish              |
| ky              | Kyrgyz               |
| la              | Latin                |
| lb              | Luxembourgish        |
| lo              | Lao                  |
| lt              | Lithuanian           |
| lv              | Latvian              |
| mg              | Malagasy             |
| mi              | Maori                |
| mk              | Macedonian           |
| ml              | Malayalam            |
| mn              | Mongolian            |
| mr              | Marathi              |
| ms              | Malay                |
| mt              | Maltese              |
| my              | Burmese              |
| ne              | Nepali               |
| nl              | Dutch                |
| no              | Norwegian            |
| ny              | Nyanja               |
| pa              | Punjabi              |
| pl              | Polish               |
| ps              | Pashto               |
| pt              | Portuguese           |
| ro              | Romanian             |
| ru              | Russian              |
| ru-Latn         | Russian (Latin)      |
| sd              | Sindhi               |
| si              | Sinhala              |
| sk              | Slovak               |
| sl              | Slovenian            |
| sm              | Samoan           |
| sn              | Shona                |
| so              | Somali               |
| sq              | Albanian             |
| sr              | Serbian              |
| st              | Southern Sotho       |
| su              | Sundanese            |
| sv              | Swedish              |
| sw              | Swahili              |
| ta              | Tamil                |
| te              | Telugu               |
| tg              | Tajik                |
| th              | Thai                 |
| tr              | Turkish              |
| uk              | Ukrainian            |
| und             | Unknown language     |
| ur              | Urdu                 |
| uz              | Uzbek                |
| vi              | Vietnamese           |
| xh              | Xhosa                |
| yi              | Yiddish              |
| yo              | Yoruba               |
| zh              | Chinese              |
| zh-Latn         | Chinese (Latin)      |
| zu              | Zulu                 |

You can load the mC4 subset of any language like this:

```python
from datasets import load_dataset

en_mc4 = load_dataset("mc4", "en")
```

And if you can even specify a list of languages:

```python
from datasets import load_dataset

mc4_subset_with_five_languages = load_dataset("mc4", languages=["en", "fr", "es", "de", "zh"])
```

### Supported Tasks and Leaderboards

mC4 is mainly intended to pretrain language models and word representations.

### Languages

The dataset supports 108 languages.

## Dataset Structure

### Data Instances

An example form the `en` config is:

```
{'timestamp': '2018-06-24T01:32:39Z',
 'text': 'Farm Resources in Plumas County\nShow Beginning Farmer Organizations & Professionals (304)\nThere are 304 resources serving Plumas County in the following categories:\nMap of Beginning Farmer Organizations & Professionals serving Plumas County\nVictoria Fisher - Office Manager - Loyalton, CA\nAmy Lynn Rasband - UCCE Plumas-Sierra Administrative Assistant II - Quincy , CA\nShow Farm Income Opportunities Organizations & Professionals (353)\nThere are 353 resources serving Plumas County in the following categories:\nFarm Ranch And Forest Retailers (18)\nMap of Farm Income Opportunities Organizations & Professionals serving Plumas County\nWarner Valley Wildlife Area - Plumas County\nShow Farm Resources Organizations & Professionals (297)\nThere are 297 resources serving Plumas County in the following categories:\nMap of Farm Resources Organizations & Professionals serving Plumas County\nThere are 57 resources serving Plumas County in the following categories:\nMap of Organic Certification Organizations & Professionals serving Plumas County',
 'url': 'http://www.californialandcan.org/Plumas/Farm-Resources/'}
```

### Data Fields

The data have several fields:

- `url`: url of the source as a string
- `text`: text content as a string
- `timestamp`: timestamp as a string

### Data Splits

To build mC4, the authors used [CLD3](https://github.com/google/cld3) to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table:

| config   | train   | validation   |
|:---------|:--------|:-------------|
| af       | ?       | ?            |
| am       | ?       | ?            |
| ar       | ?       | ?            |
| az       | ?       | ?            |
| be       | ?       | ?            |
| bg       | ?       | ?            |
| bg-Latn  | ?       | ?            |
| bn       | ?       | ?            |
| ca       | ?       | ?            |
| ceb      | ?       | ?            |
| co       | ?       | ?            |
| cs       | ?       | ?            |
| cy       | ?       | ?            |
| da       | ?       | ?            |
| de       | ?       | ?            |
| el       | ?       | ?            |
| el-Latn  | ?       | ?            |
| en       | ?       | ?            |
| eo       | ?       | ?            |
| es       | ?       | ?            |
| et       | ?       | ?            |
| eu       | ?       | ?            |
| fa       | ?       | ?            |
| fi       | ?       | ?            |
| fil      | ?       | ?            |
| fr       | ?       | ?            |
| fy       | ?       | ?            |
| ga       | ?       | ?            |
| gd       | ?       | ?            |
| gl       | ?       | ?            |
| gu       | ?       | ?            |
| ha       | ?       | ?            |
| haw      | ?       | ?            |
| hi       | ?       | ?            |
| hi-Latn  | ?       | ?            |
| hmn      | ?       | ?            |
| ht       | ?       | ?            |
| hu       | ?       | ?            |
| hy       | ?       | ?            |
| id       | ?       | ?            |
| ig       | ?       | ?            |
| is       | ?       | ?            |
| it       | ?       | ?            |
| iw       | ?       | ?            |
| ja       | ?       | ?            |
| ja-Latn  | ?       | ?            |
| jv       | ?       | ?            |
| ka       | ?       | ?            |
| kk       | ?       | ?            |
| km       | ?       | ?            |
| kn       | ?       | ?            |
| ko       | ?       | ?            |
| ku       | ?       | ?            |
| ky       | ?       | ?            |
| la       | ?       | ?            |
| lb       | ?       | ?            |
| lo       | ?       | ?            |
| lt       | ?       | ?            |
| lv       | ?       | ?            |
| mg       | ?       | ?            |
| mi       | ?       | ?            |
| mk       | ?       | ?            |
| ml       | ?       | ?            |
| mn       | ?       | ?            |
| mr       | ?       | ?            |
| ms       | ?       | ?            |
| mt       | ?       | ?            |
| my       | ?       | ?            |
| ne       | ?       | ?            |
| nl       | ?       | ?            |
| no       | ?       | ?            |
| ny       | ?       | ?            |
| pa       | ?       | ?            |
| pl       | ?       | ?            |
| ps       | ?       | ?            |
| pt       | ?       | ?            |
| ro       | ?       | ?            |
| ru       | ?       | ?            |
| ru-Latn  | ?       | ?            |
| sd       | ?       | ?            |
| si       | ?       | ?            |
| sk       | ?       | ?            |
| sl       | ?       | ?            |
| sm       | ?       | ?            |
| sn       | ?       | ?            |
| so       | ?       | ?            |
| sq       | ?       | ?            |
| sr       | ?       | ?            |
| st       | ?       | ?            |
| su       | ?       | ?            |
| sv       | ?       | ?            |
| sw       | ?       | ?            |
| ta       | ?       | ?            |
| te       | ?       | ?            |
| tg       | ?       | ?            |
| th       | ?       | ?            |
| tr       | ?       | ?            |
| uk       | ?       | ?            |
| und      | ?       | ?            |
| ur       | ?       | ?            |
| uz       | ?       | ?            |
| vi       | ?       | ?            |
| xh       | ?       | ?            |
| yi       | ?       | ?            |
| yo       | ?       | ?            |
| zh       | ?       | ?            |
| zh-Latn  | ?       | ?            |
| zu       | ?       | ?            |

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.

### Citation Information

```
@article{2019t5,
    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
    journal = {arXiv e-prints},
    year = {2019},
    archivePrefix = {arXiv},
    eprint = {1910.10683},
}
```

### Contributions

Thanks to [@dirkgr](https://github.com/dirkgr) and [@lhoestq](https://github.com/lhoestq) for adding this dataset.

task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:found multilinguality:multilingual size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M size_categories:1M<n<10M size_categories:10M<n<100M size_categories:100M<n<1B size_categories:1B<n<10B source_datasets:original language:af language:am language:ar language:az language:be language:bg language:bn language:ca language:ceb language:co language:cs language:cy language:da language:de language:el language:en language:eo language:es language:et language:eu language:fa language:fi language:fil language:fr language:fy language:ga language:gd language:gl language:gu language:ha language:haw language:he language:hi language:hmn language:ht language:hu language:hy language:id language:ig language:is language:it language:iw language:ja language:jv language:ka language:kk language:km language:kn language:ko language:ku language:ky language:la language:lb language:lo language:lt language:lv language:mg language:mi language:mk language:ml language:mn language:mr language:ms language:mt language:my language:ne language:nl language:no language:ny language:pa language:pl language:ps language:pt language:ro language:ru language:sd language:si language:sk language:sl language:sm language:sn language:so language:sq language:sr language:st language:su language:sv language:sw language:ta language:te language:tg language:th language:tr language:uk language:und language:ur language:uz language:vi language:xh language:yi language:yo language:zh language:zu license:odc-by arxiv:1910.10683 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["af", "am", "ar", "az", "be", "bg", "bn", "ca", "ceb", "co", "cs", "cy", "da", "de", "el", "en", "eo", "es", "et", "eu", "fa", "fi", "fil", "fr", "fy", "ga", "gd", "gl", "gu", "ha", "haw", "he", "hi", "hmn", "ht", "hu", "hy", "id", "ig", "is", "it", "iw", "ja", "jv", "ka", "kk", "km", "kn", "ko", "ku", "ky", "la", "lb", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", "my", "ne", "nl", "no", "ny", "pa", "pl", "ps", "pt", "ro", "ru", "sd", "si", "sk", "sl", "sm", "sn", "so", "sq", "sr", "st", "su", "sv", "sw", "ta", "te", "tg", "th", "tr", "uk", "und", "ur", "uz", "vi", "xh", "yi", "yo", "zh", "zu"], "license": ["odc-by"], "multilinguality": ["multilingual"], "size_categories": ["n<1K", "1K<n<10K", "10K<n<100K", "100K<n<1M", "1M<n<10M", "10M<n<100M", "100M<n<1B", "1B<n<10B"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "paperswithcode_id": "mc4", "pretty_name": "mC4", "language_bcp47": ["bg-Latn", "el-Latn", "hi-Latn", "ja-Latn", "ru-Latn", "zh-Latn"]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #size_categories-10M<n<100M #size_categories-100M<n<1B #size_categories-1B<n<10B #source_datasets-original #language-Afrikaans #language-Amharic #language-Arabic #language-Azerbaijani #language-Belarusian #language-Bulgarian #language-Bengali #language-Catalan #language-Cebuano #language-Corsican #language-Czech #language-Welsh #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-Filipino #language-French #language-Western Frisian #language-Irish #language-Scottish Gaelic #language-Galician #language-Gujarati #language-Hausa #language-Hawaiian #language-Hebrew #language-Hindi #language-Hmong #language-Haitian #language-Hungarian #language-Armenian #language-Indonesian #language-Igbo #language-Icelandic #language-Italian #language-iw #language-Japanese #language-Javanese #language-Georgian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Kurdish #language-Kirghiz #language-Latin #language-Luxembourgish #language-Lao #language-Lithuanian #language-Latvian #language-Malagasy #language-Maori #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Malay (macrolanguage) #language-Maltese #language-Burmese #language-Nepali (macrolanguage) #language-Dutch #language-Norwegian #language-Nyanja #language-Panjabi #language-Polish #language-Pushto #language-Portuguese #language-Romanian #language-Russian #language-Sindhi #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Southern Sotho #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Tamil #language-Telugu #language-Tajik #language-Thai #language-Turkish #language-Ukrainian #language-Undetermined #language-Urdu #language-Uzbek #language-Vietnamese #language-Xhosa #language-Yiddish #language-Yoruba #language-Chinese #language-Zulu #license-odc-by #arxiv-1910.10683 #region-us 

Dataset Card for mC4
====================


Table of Contents
-----------------


* Dataset Card for mC4
	+ Table of Contents
	+ Dataset Description
		- Dataset Summary
		- Supported Tasks and Leaderboards
		- Languages
	+ Dataset Structure
		- Data Instances
		- Data Fields
		- Data Splits
	+ Dataset Creation
		- Curation Rationale
		- Source Data
			* Initial Data Collection and Normalization
			* Who are the source language producers?
		- Annotations
			* Annotation process
			* Who are the annotators?
		- Personal and Sensitive Information
	+ Considerations for Using the Data
		- Social Impact of Dataset
		- Discussion of Biases
		- Other Known Limitations
	+ Additional Information
		- Dataset Curators
		- Licensing Information
		- Citation Information
		- Contributions


Dataset Description
-------------------


* Homepage: URL
* Paper: URL


### Dataset Summary


A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "URL".


This is the version prepared by AllenAI, hosted at this address: URL


108 languages are available and are reported in the table below.


Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script.



You can load the mC4 subset of any language like this:


And if you can even specify a list of languages:


### Supported Tasks and Leaderboards


mC4 is mainly intended to pretrain language models and word representations.


### Languages


The dataset supports 108 languages.


Dataset Structure
-----------------


### Data Instances


An example form the 'en' config is:


### Data Fields


The data have several fields:


* 'url': url of the source as a string
* 'text': text content as a string
* 'timestamp': timestamp as a string


### Data Splits


To build mC4, the authors used CLD3 to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table:



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.


### Contributions


Thanks to @dirkgr and @lhoestq for adding this dataset.



### Dataset Summary


A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "URL".


This is the version prepared by AllenAI, hosted at this address: URL


108 languages are available and are reported in the table below.


Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script.



You can load the mC4 subset of any language like this:


And if you can even specify a list of languages: ### Supported Tasks and Leaderboards


mC4 is mainly intended to pretrain language models and word representations. ### Languages


The dataset supports 108 languages.


Dataset Structure
----------------- ### Data Instances


An example form the 'en' config is: ### Data Fields


The data have several fields:


* 'url': url of the source as a string
* 'text': text content as a string
* 'timestamp': timestamp as a string ### Data Splits


To build mC4, the authors used CLD3 to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table:



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset. ### Contributions


Thanks to @dirkgr and @lhoestq for adding this dataset.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #size_categories-10M<n<100M #size_categories-100M<n<1B #size_categories-1B<n<10B #source_datasets-original #language-Afrikaans #language-Amharic #language-Arabic #language-Azerbaijani #language-Belarusian #language-Bulgarian #language-Bengali #language-Catalan #language-Cebuano #language-Corsican #language-Czech #language-Welsh #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-Filipino #language-French #language-Western Frisian #language-Irish #language-Scottish Gaelic #language-Galician #language-Gujarati #language-Hausa #language-Hawaiian #language-Hebrew #language-Hindi #language-Hmong #language-Haitian #language-Hungarian #language-Armenian #language-Indonesian #language-Igbo #language-Icelandic #language-Italian #language-iw #language-Japanese #language-Javanese #language-Georgian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Kurdish #language-Kirghiz #language-Latin #language-Luxembourgish #language-Lao #language-Lithuanian #language-Latvian #language-Malagasy #language-Maori #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Malay (macrolanguage) #language-Maltese #language-Burmese #language-Nepali (macrolanguage) #language-Dutch #language-Norwegian #language-Nyanja #language-Panjabi #language-Polish #language-Pushto #language-Portuguese #language-Romanian #language-Russian #language-Sindhi #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Southern Sotho #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Tamil #language-Telugu #language-Tajik #language-Thai #language-Turkish #language-Ukrainian #language-Undetermined #language-Urdu #language-Uzbek #language-Vietnamese #language-Xhosa #language-Yiddish #language-Yoruba #language-Chinese #language-Zulu #license-odc-by #arxiv-1910.10683 #region-us 
 ### Dataset Summary


A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "URL".


This is the version prepared by AllenAI, hosted at this address: URL


108 languages are available and are reported in the table below.


Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script.



You can load the mC4 subset of any language like this:


And if you can even specify a list of languages: ### Supported Tasks and Leaderboards


mC4 is mainly intended to pretrain language models and word representations. ### Languages


The dataset supports 108 languages.


Dataset Structure
----------------- ### Data Instances


An example form the 'en' config is: ### Data Fields


The data have several fields:


* 'url': url of the source as a string
* 'text': text content as a string
* 'timestamp': timestamp as a string ### Data Splits


To build mC4, the authors used CLD3 to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table:



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset. ### Contributions


Thanks to @dirkgr and @lhoestq for adding this dataset.
passage: 