
# Dataset Card for "oscar"

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [https://oscar-corpus.com](https://oscar-corpus.com)
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Dataset Summary

OSCAR or **O**pen **S**uper-large **C**rawled [**A**LMAnaCH](https://team.inria.fr/almanach/) co**R**pus is a huge multilingual corpus obtained by language classification and filtering of the [Common Crawl](https://commoncrawl.org/) corpus using the [goclassy](https://github.com/pjox/goclassy) architecture. Data is distributed by language in both original and deduplicated form.

The version here is the original OSCAR 2019 release: https://oscar-project.org/post/oscar-2019/

For more recent versions, visit the [oscar-corpus](https://huggingface.co/oscar-corpus) organization on the Hub:
- OSCAR 22.01 (released in January 2022): [oscar-corpus/OSCAR-2201](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201)
- OSCAR 21.09 (released in September 2021): [oscar-corpus/OSCAR-2109](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109)

### Supported Tasks and Leaderboards

OSCAR is mainly inteded to pretrain language models and word represantations.

### Languages

All the data is distributed by language, both the original and the deduplicated versions of the data are available. 166 different languages are available. The table in subsection [Data Splits Sample Size](#data-splits-sample-size) provides the language code for each subcorpus as well as the number of words (space separated tokens), lines and sizes for both the original and the deduplicated versions of OSCAR.

## Dataset Structure

We show detailed information for all the configurations of the dataset.

### Data Instances


<details>
  <summary>Click to expand the Data/size information for each language (deduplicated)</summary>

#### unshuffled_deduplicated_af

- **Size of downloaded dataset files:** 65.99 MB
- **Size of the generated dataset:** 172.30 MB
- **Total amount of disk used:** 238.29 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "aanlyn markte as gevolg van ons voortgesette 'n begrip opsie handel sakeplan pdf terwyl ons steeds die gereelde ons binêre opsies handel"
}
```

#### unshuffled_deduplicated_als

- **Size of downloaded dataset files:** 1.26 MB
- **Size of the generated dataset:** 2.96 MB
- **Total amount of disk used:** 4.22 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"De Nazionalpark hät e Flächi vo 170,3 km² und isch dodemit s grösti Naturschutzgebiet vo de Schwiz. Er ligt uf em Gebiet vo de ..."
}
```

#### unshuffled_deduplicated_am

- **Size of downloaded dataset files:** 61.35 MB
- **Size of the generated dataset:** 216.15 MB
- **Total amount of disk used:** 277.50 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"አየር መንገዱ ከአዲስ አበባ ወደ ሮም ጣሊያን በማምራት ላይ በነበረበት ጊዜ ረዳት አብራሪው የጉዞውን አቅጣጫ በመቀየር ጄኔቭ አውሮፓላን ማረፊያ በማሳረፍ እጁን ለፖሊስ ሰጥቷል።\\nየኢትዮጵያ መንግስት የ..."
}
```

#### unshuffled_deduplicated_an

- **Size of downloaded dataset files:** 0.14 MB
- **Size of the generated dataset:** 0.85 MB
- **Total amount of disk used:** 0.99 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"واااااااأسفاه الأمم تفتخر ب 0 أمي ووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووو..."
}
```

#### unshuffled_deduplicated_ar

- **Size of downloaded dataset files:** 9.67 GB
- **Size of the generated dataset:** 33.57 GB
- **Total amount of disk used:** 43.23 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"مرحبا بك عزيز الزائر نتمنى لك أوقاتاً سعيدة معنا وأن نزداد شرفا بخدمتك ولا تنسى التسجيل معنا لتستفيد بكل جديد\\nأهلا وسهلا بك زا..."
}
```

#### unshuffled_deduplicated_arz

- **Size of downloaded dataset files:** 10.02 MB
- **Size of the generated dataset:** 35.91 MB
- **Total amount of disk used:** 45.94 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"بنى عجل : قبيلة من عجل بن لجيم بن صعب بن على بن بكر بن وائل انتقل اغلبهم الى البصرة فى العراق و اصفهان و خراسان فى ايران و اذرب..."
}
```

#### unshuffled_deduplicated_as

- **Size of downloaded dataset files:** 15.51 MB
- **Size of the generated dataset:** 74.07 MB
- **Total amount of disk used:** 89.58 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"আমি, এই সংগঠনৰ সদস্য সকলে একেলগ হৈ অসমকে ধৰি ভাৰতৰ উত্তৰ পূৰ্বাঞ্চলৰ অমূল্য কলা-সাংস্কৃতিক সম্পদৰাজি বৃহত্তৰ অষ্ট্ৰেলিয়াৰ সন্মু..."
}
```

#### unshuffled_deduplicated_ast

- **Size of downloaded dataset files:** 0.86 MB
- **Size of the generated dataset:** 2.17 MB
- **Total amount of disk used:** 3.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"The Killers llanzaron el so álbum debú, Hot Fuss, en xunu de 2004 nel Reinu Xuníu, al traviés de la discográfica Lizard King, y..."
}
```

#### unshuffled_deduplicated_av

- **Size of downloaded dataset files:** 0.07 MB
- **Size of the generated dataset:** 0.34 MB
- **Total amount of disk used:** 0.41 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Жинда малъараб ва божизе бегьулеб рагІудаса кьуризе бегьуларо гьев. Гьес насихІат гьабизе кколелъул бацІцІадаб диналъул рахъалъ..."
}
```

#### unshuffled_deduplicated_az

- **Size of downloaded dataset files:** 521.74 MB
- **Size of the generated dataset:** 1.53 GB
- **Total amount of disk used:** 2.05 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"AZTV-Artıq 7 ildir ki, Abşeron rayonu dotasiya almadan bütün xərclərini yerli daxilolmalar hesabına maliyyələşdirir.\\nDünən, 10..."
}
```

#### unshuffled_deduplicated_azb

- **Size of downloaded dataset files:** 5.19 MB
- **Size of the generated dataset:** 20.08 MB
- **Total amount of disk used:** 25.27 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"لعلی ١٣-جو عصرده یاشاییب یاراتمیش گؤرکملی آذربایجان شاعرلریندندیر. ١٢٢٤-جی ایلده تبریزده آنادان اولموشدور، گنج یاشلاریندا تیجار..."
}
```

#### unshuffled_deduplicated_ba

- **Size of downloaded dataset files:** 25.98 MB
- **Size of the generated dataset:** 93.84 MB
- **Total amount of disk used:** 119.82 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Күҙәтеү ҡуласаһы моделен хәҙер Мифтахетдин Аҡмулла исемендәге Башҡорт дәүләт педагогия университетында ла эшләргә мөмкин\\t\\nКүҙ..."
}
```

#### unshuffled_deduplicated_bar

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "                                                                                                                                          vo"
}
```

#### unshuffled_deduplicated_bcl

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"& ÿ ó / í 0 - ø û ù ö ú ð ï ú \\u0014 ù þ ô ö í ÷ ò \\u0014 ÷ í ù û ö í \\u0001 û ñ ç þ \\u0001 ð \\u0007 þ ò ñ ñ ò ô \\u0017 û ö ô ÷..."
}
```

#### unshuffled_deduplicated_be

- **Size of downloaded dataset files:** 306.70 MB
- **Size of the generated dataset:** 1.08 GB
- **Total amount of disk used:** 1.39 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Брэсцкія ўлады не дазволілі прафсаюзу РЭП правесці пікетаванне ў парку Воінаў-інтэрнацыяналістаў 30 мая 2018 года.\\nСітуацыю пр..."
}
```

#### unshuffled_deduplicated_bg

- **Size of downloaded dataset files:** 3.85 GB
- **Size of the generated dataset:** 14.45 GB
- **Total amount of disk used:** 18.30 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ЖАЛБОПОДАТЕЛЯТ директор на Дирекция „ Обжалване и данъчно-осигурителна практика“- Бургас, редовно призован, се представлява от ..."
}
```

#### unshuffled_deduplicated_bh

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.04 MB
- **Total amount of disk used:** 0.04 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"सुकमा जिला भारत के छत्तीसगढ़ राज्य में एगो जिला बाटे। एकर मुख्यालय सुकमा शहर बाटे। एकर कुल रकबा 5636 वर्ग कि॰मी॰ बाटे।\"..."
}
```

#### unshuffled_deduplicated_bn

- **Size of downloaded dataset files:** 1.26 GB
- **Size of the generated dataset:** 6.24 GB
- **Total amount of disk used:** 7.50 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ভড়ং সর্বস্ব বাংলা আর্ট অ্যান্ড কালচারের হিসাব গুলিয়ে দেওয়ার ম্যাজিকের নাম ব্রাত্য রাইসু November 23, 2017\\nTagged with ডায়োজিনি..."
}
```

#### unshuffled_deduplicated_bo

- **Size of downloaded dataset files:** 22.37 MB
- **Size of the generated dataset:** 144.65 MB
- **Total amount of disk used:** 167.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"བོད་མི་འདི་དག་ནི་རང་རྒྱུད་སྒོ་རུ་ཕུད་དེ་གཞན་རྒྱུད་པང་དུ་ཉར་ནས་གསོ་སྐྱོང་བྱེད་དགོས་ཟེར་བ་དང་གཅིག་མཚུངས་རེད།\\nཚན་རིག་ནི་དང་ཐོག་རང..."
}
```

#### unshuffled_deduplicated_bpy

- **Size of downloaded dataset files:** 0.19 MB
- **Size of the generated dataset:** 1.78 MB
- **Total amount of disk used:** 1.97 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"পৌরসভা এহার আয়তন (লয়াহান) ২,৭৩০,.৬৩ বর্গ কিলোমিটার। পৌরসভা এহার মাপাহানর অক্ষাংশ বারো দ্রাঘিমাংশ ইলতাই 18.63° S 48.18° W ।[১]..."
}
```

#### unshuffled_deduplicated_br

- **Size of downloaded dataset files:** 6.47 MB
- **Size of the generated dataset:** 17.00 MB
- **Total amount of disk used:** 23.47 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Ar mank Magalhães(Daveoù a vank) a zo ur spesad evned, Spheniscus magellanicus an anv skiantel anezhañ.\\nGallout a reer implijo..."
}
```

#### unshuffled_deduplicated_bs

- **Size of downloaded dataset files:** 0.04 MB
- **Size of the generated dataset:** 0.15 MB
- **Total amount of disk used:** 0.18 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ž šř é ú šř šř ě šř ž é č ě ž ů ě ď éé ýš ě ě Ž č š ý ě ď é ýš ě ď ě éé ýš ě č ž ě š ý ď ě ýš é ú č ž č š ý ď ý ž é éě ď é č ýš..."
}
```

#### unshuffled_deduplicated_bxr

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"2002 оной хабар буряад хэлэ бэшэгэй һалбари Үндэһэтэнэй хүмүүнлиг ухаанай дээдэ һургуули болгогдожо өөршэлэгдөө.\\nХарин мүнөө б..."
}
```

#### unshuffled_deduplicated_ca

- **Size of downloaded dataset files:** 1.73 GB
- **Size of the generated dataset:** 4.57 GB
- **Total amount of disk used:** 6.30 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Daniel Vendrell, conegut com Vandrell, ha sigut un dels il•lustradors contemporanis més influents, representant a la nova onada..."
}
```

#### unshuffled_deduplicated_cbk

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano..."
}
```

#### unshuffled_deduplicated_ce

- **Size of downloaded dataset files:** 1.87 MB
- **Size of the generated dataset:** 7.04 MB
- **Total amount of disk used:** 8.90 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Шаьш анархисташ ду бохучу жигархойн дIахьедарехь дуьйцу, оьрсийн ницкъаллийн структурийн а, федералан каналан а Iалашонаш \\\"мар..."
}
```

#### unshuffled_deduplicated_ceb

- **Size of downloaded dataset files:** 7.12 MB
- **Size of the generated dataset:** 24.83 MB
- **Total amount of disk used:** 31.95 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Si Isko walay pupamilok nga nagtan-aw sa unahan, natugaw. “Naunsa ka gud diha Isko nga layo man kaayo ang imong panan-aw?” ni I..."
}
```

#### unshuffled_deduplicated_ckb

- **Size of downloaded dataset files:** 60.32 MB
- **Size of the generated dataset:** 237.72 MB
- **Total amount of disk used:** 298.05 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"رسی رۆژ - ساڵێک دوای بومەلەرزەی کرماشان میوانی بەرنامە : کاک سیاوەش حەیاتی چالاکی مەدەنی -قەسری شیرین\\nپارچە موزیک 30 / 10 / 20..."
}
```

#### unshuffled_deduplicated_cs

- **Size of downloaded dataset files:** 10.49 GB
- **Size of the generated dataset:** 25.71 GB
- **Total amount of disk used:** 36.20 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Akce anarchistů proti připravovanému novému služební řádu a nízkým mzdám 1903 – Historie českého anarchismu (1880 – 1939)\\nRost..."
}
```

#### unshuffled_deduplicated_cv

- **Size of downloaded dataset files:** 7.47 MB
- **Size of the generated dataset:** 27.49 MB
- **Total amount of disk used:** 34.95 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Шыранӑ чухне ӑнсӑртран латин кирилл саспаллисем вырӑнне латин саспаллисене ҫырсан, сайт эсир ҫырнине юсама тӑрӑшӗ.\\nКу сайтра ч..."
}
```

#### unshuffled_deduplicated_cy

- **Size of downloaded dataset files:** 53.63 MB
- **Size of the generated dataset:** 141.22 MB
- **Total amount of disk used:** 194.86 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Mae capeli Cymreig yr Andes ym Mhatagonia wedi cyhoeddi na fydd gwasanaethau yno weddill y mis, oherwydd yr eira trwm sydd wedi..."
}
```

#### unshuffled_deduplicated_da

- **Size of downloaded dataset files:** 3.82 GB
- **Size of the generated dataset:** 10.24 GB
- **Total amount of disk used:** 14.06 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Den 2.-5. februar 2016 løb det tredje kursus i uddannelsen af 4kommunesamarbejdets Local Impact Coaches, af stablen i Gentofte ..."
}
```

#### unshuffled_deduplicated_de

- **Size of downloaded dataset files:** 60.80 GB
- **Size of the generated dataset:** 156.30 GB
- **Total amount of disk used:** 217.10 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Auf dieser Seite gibt es mind. ein YouTube Video. Cookies für diese Website wurden abgelehnt. Dadurch können keine YouTube Vide..."
}
```

#### unshuffled_deduplicated_diq

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Zıwanê Slawki, zıwano merdumanê Slawano. Zıwanê Slawki yew lızgeyê Zıwananê Hind u Ewropao. Keyeyê Zıwananê Slawki beno hirê letey:"
}
```

#### unshuffled_deduplicated_dsb

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Pśiklaskaju južo pśed pśedstajenim... 1500 źiśi njamóžo wěcej docakaś, měsćańska hala w Chóśebuzu - wupśedana."
}
```

#### unshuffled_deduplicated_dv

- **Size of downloaded dataset files:** 16.84 MB
- **Size of the generated dataset:** 82.19 MB
- **Total amount of disk used:** 99.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ބ. އަތޮޅުގައި ހުޅުވަން ތައްޔާރުވަމުން އަންނަ ވައްކަރު ރިސޯޓުގައި ވަޒީފާ އަދާކުރަން ޝައުގުވެރިވާ ފަރާތްތަކަށް ކުރިމަތިލުމުގެ ފުރ..."
}
```

#### unshuffled_deduplicated_el

- **Size of downloaded dataset files:** 7.91 GB
- **Size of the generated dataset:** 28.74 GB
- **Total amount of disk used:** 36.65 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Νεκρός εντοπίστηκε μέσα στο σπίτι του στην οδό Ηρώδου Αττικού στον αριθμό 7 ο επικεφαλής του προξενικού τμήματος της Ρωσικής πρ..."
}
```

#### unshuffled_deduplicated_eml

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"A séguit dal prucès ad rubutiśasiòṅ di abitànt dal pòpul ad Mikenes, Angoras 'l è finî dènt'r a 'n robot cun la tèsta dna rana ..."
}
```

#### unshuffled_deduplicated_en

- **Size of downloaded dataset files:** 496.50 GB
- **Size of the generated dataset:** 1299.75 GB
- **Total amount of disk used:** 1796.24 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Mtendere Village was inspired by the vision of Chief Napoleon Dzombe, which he shared with John Blanchard during his first visi..."
}
```

#### unshuffled_deduplicated_eo

- **Size of downloaded dataset files:** 92.86 MB
- **Size of the generated dataset:** 240.12 MB
- **Total amount of disk used:** 332.99 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Ĉu ... preĝi | mediti | ricevi instigojn || kanti | muziki || informiĝi | legi | studi || prepari Diservon\\nTemas pri kolekto d..."
}
```

#### unshuffled_deduplicated_es

- **Size of downloaded dataset files:** 60.46 GB
- **Size of the generated dataset:** 160.86 GB
- **Total amount of disk used:** 221.32 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Como se librará de la celulitis en el gimnasio La piel superflua en las manos después del adelgazamiento, Los bailes fáciles pa..."
}
```

#### unshuffled_deduplicated_et

- **Size of downloaded dataset files:** 966.79 MB
- **Size of the generated dataset:** 2.45 GB
- **Total amount of disk used:** 3.41 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"MTÜ AB Video järgib oma tegevuses kodanikuühenduste eetilise tegevuse üldtunnustatud põhimõtteid, mis on lühidalt kokkuvõetud 7..."
}
```

#### unshuffled_deduplicated_eu

- **Size of downloaded dataset files:** 134.68 MB
- **Size of the generated dataset:** 363.93 MB
- **Total amount of disk used:** 498.61 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Gure jarduerek eraikuntzarekin, elkarbizitzarekin, hirigintzarekin eta ekologiarekin dute harremana, baita ideia eta konponbideak irudikatu eta garatzearekin ere, eraikuntza sektorea hobetuz, pertsonen erosotasuna eta bizi-kalitatea hobetzeko."
}
```

#### unshuffled_deduplicated_fa

- **Size of downloaded dataset files:** 10.46 GB
- **Size of the generated dataset:** 40.06 GB
- **Total amount of disk used:** 50.52 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"قـــــــــــــــــرار بود با هم کنـــــــــــــار بیایم نه اینکه از کنــــــــــــار هم رد بشیم...!!!\\nاگر روزی دلت لبریز غم بو..."
}
```

#### unshuffled_deduplicated_fi

- **Size of downloaded dataset files:** 5.38 GB
- **Size of the generated dataset:** 13.99 GB
- **Total amount of disk used:** 19.37 GB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Kiitos Deelle kaikesta - 1,5 viikkoa kulunut, kun Dee ei ole enää ollut omani. Reilu viikko sitten sunnuntaina vein Deen uuteen kotiinsa. Itselläni on ollut niin ristiriitaiset t..."
}
```

#### unshuffled_deduplicated_fr

- **Size of downloaded dataset files:** 55.46 GB
- **Size of the generated dataset:** 148.28 GB
- **Total amount of disk used:** 203.75 GB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Média de débat d'idées, de culture et de littérature. Récits, décryptages, analyses, portraits et critiques autour de la vie des idées. Magazine engagé, ouvert aux autres et au monde.. Bring up to date in french"
}
```

#### unshuffled_deduplicated_frr

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Hiragana’ Practice’Sheet’1’(A -O)’ ’ Name:’________ __________________________’Section:’_______________ _’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ..."
}
```

#### unshuffled_deduplicated_fy

- **Size of downloaded dataset files:** 10.27 MB
- **Size of the generated dataset:** 26.73 MB
- **Total amount of disk used:** 37.00 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Nim in sêfte ride op Holmsjön, yn ien fan 'e lytse marren yn de omkriten, of nim se op avontueren lykas nonresidential. lâns Indalsälven wetter. Holm Sportklubb hawwe kano 's te huur, yn gearwurking mei de Baltyske Power konferinsje."
}
```

#### unshuffled_deduplicated_ga

- **Size of downloaded dataset files:** 22.22 MB
- **Size of the generated dataset:** 63.86 MB
- **Total amount of disk used:** 86.08 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Is fóram é seo chun plé a dhéanamh ar an leabhar atá roghnaithe do mhí na Samhna 2013 amháin. Ní féidir ach le baill chláraithe..."
}
```

#### unshuffled_deduplicated_gd

- **Size of downloaded dataset files:** 0.42 MB
- **Size of the generated dataset:** 1.36 MB
- **Total amount of disk used:** 1.78 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Zhou Yujun, a 'phàrtaidh Rùnaire Comataidh Sgìre Yanfeng ann Hengyang bhaile agus a Sgìre pàrtaidh agus an riaghaltas a' bhuidheann-riochdachaidh a 'tighinn a chèilidh air ar companaidh air Apr. 14, 2017."
}
```

#### unshuffled_deduplicated_gl

- **Size of downloaded dataset files:** 155.85 MB
- **Size of the generated dataset:** 408.34 MB
- **Total amount of disk used:** 564.19 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"O persoal de Inditex da provincia de Pontevedra segue a reclamar iguais condicións laborais no conxunto do país - CIG: Confeder..."
}
```

#### unshuffled_deduplicated_gn

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"º ÑÆÚÓ À Ã Ð É Æ ¾ ÄÂ Î À ¼ Æ É ÄÛ = Ü Ý\\\"Þ ßà á â ã ä å æçè ã é ê â å àë ì æê íî é á ë ï í çì àð í Ü à ñ ê é ò ä ì\"..."
}
```

#### unshuffled_deduplicated_gom

- **Size of downloaded dataset files:** 0.38 MB
- **Size of the generated dataset:** 1.87 MB
- **Total amount of disk used:** 2.24 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"दुष्ट शीळ हें कौरवांचें । रामें सविस्तर देखूनि साचें । बोलिले वचनें जें दुर्वाचे । करी तयांचें अनुस्मरण ॥२२०॥\"..."
}
```

#### unshuffled_deduplicated_gu

- **Size of downloaded dataset files:** 162.97 MB
- **Size of the generated dataset:** 759.34 MB
- **Total amount of disk used:** 922.32 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"અધિક માસ ચાલે છે. સમગ્ર ભારતમાં અને તેમાંય ખાસ કરીને પવિત્ર કે ધાર્મિક કહેવાય છે તેવા સ્થાનક પર કથાનો દોર ચાલે છે. ઉનાળાની કાળઝ..."
}
```

#### unshuffled_deduplicated_he

- **Size of downloaded dataset files:** 3.04 GB
- **Size of the generated dataset:** 10.47 GB
- **Total amount of disk used:** 13.51 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"זקוקים לרשתות נגד יתושים? מחפשים רשת מתאימה לחלון צר וקטן? רשתות נגד יתושים אקורדיון של חברת קליר-מש הן הפתרון.\\nרשתות לחלונות ..."
}
```

#### unshuffled_deduplicated_hi

- **Size of downloaded dataset files:** 2.01 GB
- **Size of the generated dataset:** 9.57 GB
- **Total amount of disk used:** 11.58 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"'आइटम गर्ल' बनकर हिट हुई थीं राखी सावंत, आज करीना-कटरीना तक फॉलो कर रही हैं ट्रेंड नक्‍सलियों का दम निकालेगा बाइक ग्रेनेड लॉन्च..."
}
```

#### unshuffled_deduplicated_hr

- **Size of downloaded dataset files:** 46.74 MB
- **Size of the generated dataset:** 121.50 MB
- **Total amount of disk used:** 168.23 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"U raspravi je sudjelovao i HSS-ov saborski zastupnik rekavši kako poljoprivrednici ne osjete mjere o kojima ministar govori jer..."
}
```

#### unshuffled_deduplicated_hsb

- **Size of downloaded dataset files:** 0.72 MB
- **Size of the generated dataset:** 1.89 MB
- **Total amount of disk used:** 2.61 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Budyšin (SN/BŠe). Elektronikarjo mějachu lětsa cyle hinaši zazběh do swojeho wukubłanja. Wokrjesne rjemjeslnistwo bě mjenujcy w..."
}
```

#### unshuffled_deduplicated_ht

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan..."
}
```

#### unshuffled_deduplicated_hu

- **Size of downloaded dataset files:** 7.37 GB
- **Size of the generated dataset:** 19.09 GB
- **Total amount of disk used:** 26.46 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"monster - Amatőr, házi szex videók és kezdő csjaok pornó filmjei. - Free amateur, home made sex videos and online porn movies. ..."
}
```

#### unshuffled_deduplicated_hy

- **Size of downloaded dataset files:** 393.62 MB
- **Size of the generated dataset:** 1.56 GB
- **Total amount of disk used:** 1.96 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Արցախի Հանրապետության հռչակման 26-րդ տարեդարձի կապակցությամբ Շուշիի Արվեստի կենտրոնում կազմակերպվել է մոսկվաբնակ նկարիչներ՝ հայ..."
}
```

#### unshuffled_deduplicated_ia

- **Size of downloaded dataset files:** 0.05 MB
- **Size of the generated dataset:** 0.38 MB
- **Total amount of disk used:** 0.43 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha h..."
}
```

#### unshuffled_deduplicated_id

- **Size of downloaded dataset files:** 6.00 GB
- **Size of the generated dataset:** 17.05 GB
- **Total amount of disk used:** 23.05 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Perihal dari itu, kalau kunci hal yang demikian hilang, pemilik wajib melapor ke bengkel sah untuk dibuatkan kunci baru dengan ..."
}
```

#### unshuffled_deduplicated_ie

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Plastic Yo Yo Metal Yo Yos Wooden Yo Yo Keychain Yo Yo Translucent Yo Yo Light Up Yo Yo Globe Yo Yo Stress Reliever Yo Yo Jellyfish Yo Yo Sports Ball Yo Yo Sound Yo Yo Miniature Yo Yo Promotional Yo Yo Novelty Yo Yo Video Game Yo Yo ECO Recycled Yo Yo"
}
```

#### unshuffled_deduplicated_ilo

- **Size of downloaded dataset files:** 0.23 MB
- **Size of the generated dataset:** 0.68 MB
- **Total amount of disk used:** 0.91 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Segun ken ni Ping-ay, ti yellow corn ti maysa kadagiti nadakamat a liberalized agricultural commodity iti daytoy a free trade k..."
}
```

#### unshuffled_deduplicated_io

- **Size of downloaded dataset files:** 0.04 MB
- **Size of the generated dataset:** 0.14 MB
- **Total amount of disk used:** 0.19 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Chekia esas parlamentala republiko. La chefo di stato esas la prezidanto. Til 2013 lu elektesis dal parlamento. Pos ta yaro, ol..."
}
```

#### unshuffled_deduplicated_is

- **Size of downloaded dataset files:** 332.87 MB
- **Size of the generated dataset:** 894.28 MB
- **Total amount of disk used:** 1.23 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Eyjar.net - upplýsinga- og fréttamiðill um Vestmannaeyjar - Fréttir - Nái núverandi stefna stjórnvalda fram að ganga mun það va..."
}
```

#### unshuffled_deduplicated_it

- **Size of downloaded dataset files:** 27.93 GB
- **Size of the generated dataset:** 74.09 GB
- **Total amount of disk used:** 102.03 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Jaundice - causes, treatment & pathology massaggio a osteochondrosis dellindizio di una controindicazione\\nTrattamento su un co..."
}
```

#### unshuffled_deduplicated_ja

- **Size of downloaded dataset files:** 40.80 GB
- **Size of the generated dataset:** 113.63 GB
- **Total amount of disk used:** 154.44 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"神社などへ一緒に同行して、様々な角度のショットで家族写真やお子様の写真を撮影致します！お好みに合わせて様々な写真を取ることができますので、その場でカメラマンへのリクエストも可能です！お子様の晴れ姿を、緊張していない自然な笑顔で残しませんか？\\n※七五三の..."
}
```

#### unshuffled_deduplicated_jbo

- **Size of downloaded dataset files:** 0.20 MB
- **Size of the generated dataset:** 0.70 MB
- **Total amount of disk used:** 0.91 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "ni'o 23 la cimast. cu 23moi djedi fi'o masti la cimast. noi ke'a cu cimoi masti .i 22 la cimast. cu purlamdei .ije 24 la cimast. cu bavlamdei"
}
```

#### unshuffled_deduplicated_jv

- **Size of downloaded dataset files:** 0.21 MB
- **Size of the generated dataset:** 0.62 MB
- **Total amount of disk used:** 0.82 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"José Mourinho (diwaca: [ʒuˈzɛ moˈɾiɲu]; lair ing Setubal, Portugal, 26 Januari 1963; umur 55 taun) iku salah siji pelatih bal k..."
}
```

#### unshuffled_deduplicated_ka

- **Size of downloaded dataset files:** 377.23 MB
- **Size of the generated dataset:** 1.99 GB
- **Total amount of disk used:** 2.36 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"წამიყვანე შენთან ერთად (ქართულად) / Возьми меня с собой (картулад) / (რუსული სერიალები ქართულად) (რუსების პორნო ონლაინში) (ruse..."
}
```

#### unshuffled_deduplicated_kk

- **Size of downloaded dataset files:** 389.12 MB
- **Size of the generated dataset:** 1.59 GB
- **Total amount of disk used:** 1.97 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Түлкібас ауданында «Латын негізді әліпби мен емле ережесі туралы насихат» жобасының тобы семинар өткізді\\nЕлорданың «Қазақстан»..."
}
```

#### unshuffled_deduplicated_km

- **Size of downloaded dataset files:** 114.48 MB
- **Size of the generated dataset:** 610.61 MB
- **Total amount of disk used:** 725.09 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ខ្សឹបដាក់ត្រចៀក៖ លោក សួស សុផានិត នាយផ្នែករដ្ឋបាលព្រៃឈើ ស្រុកភ្នំក្រវាញ់ ដែលទើបឡើងកាន់តំណែងថ្មី បើកដៃឲ្យឈ្នួញ ប្រព្រឹត្តបទល្មើស ..."
}
```

#### unshuffled_deduplicated_kn

- **Size of downloaded dataset files:** 215.52 MB
- **Size of the generated dataset:** 1.08 GB
- **Total amount of disk used:** 1.30 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ರಾಷ್ಟ್ರಪತಿ ಪ್ರಣಬ್ ಮುಖರ್ಜಿಯಿಂದ ಪದ್ಮ ಪ್ರಶಸ್ತಿ ಪ್ರದಾನ | President Pranab Mukherjee Confers Padma Awards | Photo Gallery on Kannada..."
}
```

#### unshuffled_deduplicated_ko

- **Size of downloaded dataset files:** 4.46 GB
- **Size of the generated dataset:** 12.00 GB
- **Total amount of disk used:** 16.47 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"CIA 프로젝트에서는 데이터베이스로 들어오는 요청을 중간에 수집(Sniffing)하고 수집한 데이터를 분석(Parsing)하여 그로 인한 결과를 판단하여 알릴 수 있는 시스템(Push Service)이 필요하다. 그리고 연구를 ..."
}
```

#### unshuffled_deduplicated_krc

- **Size of downloaded dataset files:** 0.62 MB
- **Size of the generated dataset:** 2.41 MB
- **Total amount of disk used:** 3.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Шамханланы, Бийлени къаршысына ябушуп, Батыр уланларыбызны къоллары булан «ортакъ ожакъ» къургъанбыз. Шо иш уллу зараллы иш бол..."
}
```

#### unshuffled_deduplicated_ku

- **Size of downloaded dataset files:** 23.34 MB
- **Size of the generated dataset:** 63.09 MB
- **Total amount of disk used:** 86.43 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Me di 114 bernameyên xwe yên berê da perçeyên ji berhemên zanyarî yên kurdzanên mezin bi wergera kurdî da ...\\nMe di 114 bernam..."
}
```

#### unshuffled_deduplicated_kv

- **Size of downloaded dataset files:** 0.33 MB
- **Size of the generated dataset:** 1.21 MB
- **Total amount of disk used:** 1.54 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Коми кытшыслӧн ыджытжык тор вӧр увтын куйлӧ, сійӧн и фаунасӧ татӧн аркмӧтӧны вӧрын олісь подаэз. Ассямаӧн лоӧ сія, мый кытшас с..."
}
```

#### unshuffled_deduplicated_kw

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼Pray without ceasing🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏..."
}
```

#### unshuffled_deduplicated_ky

- **Size of downloaded dataset files:** 106.22 MB
- **Size of the generated dataset:** 408.40 MB
- **Total amount of disk used:** 514.61 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Turmush: Бишкек шаардык кеңешинин кезексиз отурумунда мэрге ишенбөөчүлүк көрсөтүү маселеси каралат, - депутат Т.Сагынов\\nБишкек..."
}
```

#### unshuffled_deduplicated_la

- **Size of downloaded dataset files:** 3.42 MB
- **Size of the generated dataset:** 9.79 MB
- **Total amount of disk used:** 13.22 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Hæ sunt generationes Noë: Noë vir justus atque perfectus fuit in generationibus suis; cum Deo ambulavit.\\nEcce ego adducam aqua..."
}
```

#### unshuffled_deduplicated_lb

- **Size of downloaded dataset files:** 8.30 MB
- **Size of the generated dataset:** 21.42 MB
- **Total amount of disk used:** 29.72 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Während dem Gaardefestival \\\"Ambiance Jardins\\\" vum 15. bis de 17. Mee huet den SNJ nees zesumme mam Groupe Animateur en Inform..."
}
```

#### unshuffled_deduplicated_lez

- **Size of downloaded dataset files:** 0.77 MB
- **Size of the generated dataset:** 3.08 MB
- **Total amount of disk used:** 3.84 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Ахцегь хуьр, виридалай ч1ехи лезги хуьрерикая я. Ам Урусатдин виридалай къиблепатавай хуьрерикай я. Ин хуьр...\"..."
}
```

#### unshuffled_deduplicated_li

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.03 MB
- **Total amount of disk used:** 0.04 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"'t Good Goedenraad aan de Ezerbaek besjteit oet 'n kesjtièl mèt gesjlote haof en 'n park van 26 hectare. Hie in sjtoon väól beu..."
}
```

#### unshuffled_deduplicated_lmo

- **Size of downloaded dataset files:** 0.10 MB
- **Size of the generated dataset:** 0.46 MB
- **Total amount of disk used:** 0.57 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Serét (en tortonés: Sregh; en piemontés: Srèj) l'è 'n cümü italià, de la regiù del Piemónt, en Pruvìncia de Alessandria. El g'h..."
}
```

#### unshuffled_deduplicated_lo

- **Size of downloaded dataset files:** 23.63 MB
- **Size of the generated dataset:** 119.29 MB
- **Total amount of disk used:** 142.92 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"ຜູ້ພິພາກສາ ປະຈຳເຂດ ສຫລ ທ່ານນຶ່ງ ຕັດສິນວ່າ ໂຄງການເກັບກຳຂໍ້ມູນ ທາງໂທລະສັບ ຂອງອົງການ ຄວາມໝັ້ນຄົງແຫ່ງຊາດ ແມ່ນຖືກຕ້ອງ ຕາມກົດໝາຍ.\\nກະ..."
}
```

#### unshuffled_deduplicated_lrc

- **Size of downloaded dataset files:** 0.02 MB
- **Size of the generated dataset:** 0.06 MB
- **Total amount of disk used:** 0.08 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"آرلینگتون یئ گئل د شأریا ڤولاتچە ڤیرجینیا و یئ گئل د شأریا ڤولات ڤولاتچە یا یأکاگئرئتە ئمریکاە. ئی شأر دویومی کألوٙن شأر د راسا..."
}
```

#### unshuffled_deduplicated_lt

- **Size of downloaded dataset files:** 1.65 GB
- **Size of the generated dataset:** 4.20 GB
- **Total amount of disk used:** 5.86 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Čir vir vir pavasaris! Čia čia čia… dalinamės labai simpatiška video pamokėle, kurią pristato ab888art galerija.\\nBe galo papra..."
}
```

#### unshuffled_deduplicated_lv

- **Size of downloaded dataset files:** 710.45 MB
- **Size of the generated dataset:** 1.91 GB
- **Total amount of disk used:** 2.62 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Dekoratīvi sliekšņi MITSUBISHI OUTLANDER 2007, izgatavoti no ovālas formas, pulētas nerūsējošā tērauda caurules...\\ndažādas tūn..."
}
```

#### unshuffled_deduplicated_mai

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"१ · २ · ३ · ४ · ५ · ६ · ७ · ८ · ९ · १० · ११ · १२ · १३ · १४ · १५ · १६ · १७ · १८ · १९ · २० · २१ · २२ · २३ · २४ · २५ · २६ · २७ · २..."
}
```

#### unshuffled_deduplicated_mg

- **Size of downloaded dataset files:** 4.30 MB
- **Size of the generated dataset:** 13.59 MB
- **Total amount of disk used:** 17.89 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Nanamboatra taratasy apetaka sy soso-kevitra ho an'ny olona te-hanatevin-daharana ity fihetsiketsehana ity i Anocrena.\\nNosorat..."
}
```

#### unshuffled_deduplicated_mhr

- **Size of downloaded dataset files:** 1.63 MB
- **Size of the generated dataset:** 6.26 MB
- **Total amount of disk used:** 7.89 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Акрет жап годым Уганда кундемым Пигмей племена- влак айлен шогеныт. мемнан эран 1 курым гыч Банту племена влакат тиде кундемышк..."
}
```

#### unshuffled_deduplicated_min

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.31 MB
- **Total amount of disk used:** 0.33 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ..."
}
```

#### unshuffled_deduplicated_mk

- **Size of downloaded dataset files:** 303.12 MB
- **Size of the generated dataset:** 1.19 GB
- **Total amount of disk used:** 1.49 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"„Филм плус“ е насловен првиот филмски месечник во Македонија, чиј прв број ќе биде промовиран вечер во „Менада“. Новото македон..."
}
```

#### unshuffled_deduplicated_ml

- **Size of downloaded dataset files:** 496.80 MB
- **Size of the generated dataset:** 2.69 GB
- **Total amount of disk used:** 3.18 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"സ്ത്രീ പ്രവേശനം സര്‍ക്കാര്‍ പൂര്‍ണമായും അംഗീകരിക്കുന്നുവെന്നും ശബരിമലയുടെ സുരക്ഷയില്‍ ഇടപെടുമെന്നും സര്‍ക്കാര്‍ ഹൈക്കോടതിയില്‍\\..."
}
```

#### unshuffled_deduplicated_mn

- **Size of downloaded dataset files:** 219.52 MB
- **Size of the generated dataset:** 883.46 MB
- **Total amount of disk used:** 1.10 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"МУБИС-ын багш мэргэжлийн хөрвөх сургалтыг төгссөн багшид багшлах эрх олгох тухай ~ БМДИ-ийн захирлын тушаал - Багшийн мэргэжил ..."
}
```

#### unshuffled_deduplicated_mr

- **Size of downloaded dataset files:** 299.68 MB
- **Size of the generated dataset:** 1.49 GB
- **Total amount of disk used:** 1.79 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Home / motivational marathi story / उद्योजकता (Entrepreneurship) / यांना हे जमलय, तर आपल्याला का नाही जमणार ?\\nयापैकी कोणाचीही ..."
}
```

#### unshuffled_deduplicated_mrj

- **Size of downloaded dataset files:** 0.29 MB
- **Size of the generated dataset:** 1.10 MB
- **Total amount of disk used:** 1.38 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Лӹпӹвлӓ (латинлӓ Lepidoptera ; алыкмарла лыве-влак) — капшангывлӓ йыхыш пырышы сӱмӓн нӹл шылдыран капшангывлӓ. Цилӓжӹ 180000 тӹ..."
}
```

#### unshuffled_deduplicated_ms

- **Size of downloaded dataset files:** 16.39 MB
- **Size of the generated dataset:** 49.45 MB
- **Total amount of disk used:** 65.85 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Sanad pertama daripada Zuhair bin Harb daripada ‘Affan daripada Hammad daripada Thabit daripada Anas.\\nSanad kedua daripada ‘Ab..."
}
```

#### unshuffled_deduplicated_mt

- **Size of downloaded dataset files:** 5.90 MB
- **Size of the generated dataset:** 17.68 MB
- **Total amount of disk used:** 23.58 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "tibgħat il-kawża lura lill-Qorti Ġenerali għall-annullament jew għat-tnaqqis tal-penalità imposta mill-Kummissjoni bid-deċiżjoni inizjali kif emendata bid-deċiżjoni ta’ rettifika;"
}
```

#### unshuffled_deduplicated_mwl

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Deciplina social i outónoma que angloba atebidades de ouserbaçon, de análeze, de çcriçon, cumparaçon, de sistematizaçon i de sp..."
}
```

#### unshuffled_deduplicated_my

- **Size of downloaded dataset files:** 207.14 MB
- **Size of the generated dataset:** 1.11 GB
- **Total amount of disk used:** 1.32 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ျမ၀တီ - ရန္ကုန္တိုင္းေဒသႀကီး ေျမာက္ဥကၠလာပႏွင္႕ ဗဟန္းၿမိဳ႔နယ္ မေကြးတိုင္း ေဒသႀကီး ပခုကၠဴၿမိဳ႔နယ္တို႔၌ ျမန္မာ႕တပ္မေတာ္အား ေထာက္ခံ..."
}
```

#### unshuffled_deduplicated_myv

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"2018 иень умарьковонь 6-це чистэ сась паро куля! Россиянь культурань Министерствась макссь невтемань конёв (прокатной удостовер..."
}
```

#### unshuffled_deduplicated_mzn

- **Size of downloaded dataset files:** 0.16 MB
- **Size of the generated dataset:** 0.63 MB
- **Total amount of disk used:** 0.79 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"قرآن یا قوران اسلام ِآسمونی کتاب هسته. مسلمونون گانّّه قرآن ره خدا، وحی جه برسنی‌یه، «محمد معجزه» هسته و ثقلین حدیث دله ونه خَو..."
}
```

#### unshuffled_deduplicated_nah

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "In mācuīlpōhualxihuitl VI (inic chicuacē) in mācuīlpōhualli xiuhitl cāhuitl īhuīcpa 501 xihuitl oc 600 xihuitl."
}
```

#### unshuffled_deduplicated_nap

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ò AUDIT í Ç è î ÿ å å 30 ò ÿ ÿ é, õ ñ ì ÿ, ê ã- ò à ì. å â å í ç â à à é ñ è å é ó ó ë. å å å û è å î é è à. à è à AUDIT 1-7 â ..."
}
```

#### unshuffled_deduplicated_nds

- **Size of downloaded dataset files:** 5.27 MB
- **Size of the generated dataset:** 13.48 MB
- **Total amount of disk used:** 18.76 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Dor kann sik vun nu af an de hele plattdüütsche Welt – vun Niebüll bit New York, vun Helgoland bit Honolulu – drapen. Allens, w..."
}
```

#### unshuffled_deduplicated_ne

- **Size of downloaded dataset files:** 240.63 MB
- **Size of the generated dataset:** 1.24 GB
- **Total amount of disk used:** 1.48 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"बर्दिबास नगरपालिकाको तेस्रो नगर परिषदबाट पारित आ.व.२०७३।७४ को संशोधित र २०७४।७५ को प्रस्तावित नीति, कार्यक्रम तथा बजेट\\nअार्थिक..."
}
```

#### unshuffled_deduplicated_new

- **Size of downloaded dataset files:** 0.83 MB
- **Size of the generated dataset:** 4.26 MB
- **Total amount of disk used:** 5.09 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"थ्व शहरयागु अक्षांश ३४.७००१६४ उत्तर व देशान्तर ८६.३७६४६९ पश्चिम खः (34.700164° N 86.376469° W)। थ्व थासे ७२२६७३२ वर्ग मिटर (२.७..."
}
```

#### unshuffled_deduplicated_nl

- **Size of downloaded dataset files:** 15.73 GB
- **Size of the generated dataset:** 41.91 GB
- **Total amount of disk used:** 57.65 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Op vrijdag 31 augustus wordt het nieuwe studiejaar van de masteropleiding architectuur geopend met een dagexcursie naar Venlo.\\..."
}
```

#### unshuffled_deduplicated_nn

- **Size of downloaded dataset files:** 23.58 MB
- **Size of the generated dataset:** 58.32 MB
- **Total amount of disk used:** 81.90 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Planomtale krav til innhald Bakgrunn: Spørsmål frå fleire kommunar om kva ein planomtale/planbeskrivelse bør innehalde Fylkeskommunen og fylkesmannen har i ein del saker reist motsegn på formelt grunnlag"
}
```

#### unshuffled_deduplicated_no

- **Size of downloaded dataset files:** 1.96 GB
- **Size of the generated dataset:** 5.11 GB
- **Total amount of disk used:** 7.07 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Ytterligere aktører i primærhelsetjenesten og andre NHS-virksomheter ble infisert, inkludert legekontor.Læreren vår er så attra..."
}
```

#### unshuffled_deduplicated_oc

- **Size of downloaded dataset files:** 1.34 MB
- **Size of the generated dataset:** 4.00 MB
- **Total amount of disk used:** 5.34 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": ".рф (rf, còdi punycode: .xn--p1ai)[1] es lo nom de domeni en rus per Russia. Foguèt activat lo 12 de mai de 2010. Lo còdi latin es .ru."
}
```

#### unshuffled_deduplicated_or

- **Size of downloaded dataset files:** 38.72 MB
- **Size of the generated dataset:** 197.63 MB
- **Total amount of disk used:** 236.36 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ଭୁବନେଶ୍ୱର, ୨୭/୧– (ଓଡ଼ିଆ ପୁଅ) ସିପିଆଇ ଜାତୀୟ ପରିଷଦର ଆହ୍ୱାନକ୍ରମେ ଗତକାଲି ଜାନୁୟାରୀ ୨୬ ସାଧାରଣତନ୍ତ୍ର ଦିବସକୁ ଦେଶ ବ୍ୟାପୀ ସମ୍ବିଧାନ ସୁରକ୍ଷା ..."
}
```

#### unshuffled_deduplicated_os

- **Size of downloaded dataset files:** 2.83 MB
- **Size of the generated dataset:** 11.00 MB
- **Total amount of disk used:** 13.83 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"1. Лæппу æмæ чызг казрæдзийы зæрдæмæ куы фæцæуынц æмæ, куы сфæнд кæнынц сæ цард баиу кæнын, уæд лæппу бар ракуры чызгæй, цæмæй ..."
}
```

#### unshuffled_deduplicated_pa

- **Size of downloaded dataset files:** 102.39 MB
- **Size of the generated dataset:** 483.04 MB
- **Total amount of disk used:** 585.42 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ਰਜਿ: ਨੰ: PB/JL-138/2018-20 ਜਿਲਦ 63, ਬਾਨੀ ਸੰਪਾਦਕ (ਸਵ:) ਡਾ: ਸਾਧੂ ਸਿੰਘ ਹਮਦਰਦ ਫ਼ੋਨ : 0181-2455961-62-63, 5032400, ਫੈਕਸ : 2455960, 2..."
}
```

#### unshuffled_deduplicated_pam

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Áku pu i Anak ning Aláya at ngeni ipákit kó kékayu ngan nûng makanánu lang susúlat détinang kulit a mágkas. Lauan ya ing tarátu..."
}
```

#### unshuffled_deduplicated_pl

- **Size of downloaded dataset files:** 20.19 GB
- **Size of the generated dataset:** 50.59 GB
- **Total amount of disk used:** 70.78 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"System informatyczny - Załącznik nr 1 do zarządzenia Wójta Gminy Podegrodzie Nr 530/2013 z dnia 27 maja 2013 r\\nSystem informat..."
}
```

#### unshuffled_deduplicated_pms

- **Size of downloaded dataset files:** 0.71 MB
- **Size of the generated dataset:** 2.00 MB
- **Total amount of disk used:** 2.72 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Louvigné-du-Désert a l'é na comun-a fransèisa ant la region aministrativa dla Brëtagna, ant ël dipartiment d'Ille-et-Vilaine. A..."
}
```

#### unshuffled_deduplicated_pnb

- **Size of downloaded dataset files:** 2.58 MB
- **Size of the generated dataset:** 9.44 MB
- **Total amount of disk used:** 12.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"ایہ فائل Wikimedia Commons توں اے تے دوجیاں ویونتاں تے وی ورتی جاےکدی اے۔ گل بات اس دے فائل گل بات صفہ تے تھلے دتی گئی۔\"..."
}
```

#### unshuffled_deduplicated_ps

- **Size of downloaded dataset files:** 71.83 MB
- **Size of the generated dataset:** 254.79 MB
- **Total amount of disk used:** 326.61 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Many people usually use the time period ‘business to business (B2B) advertising,’ however most of them do not know precisely wh..."
}
```

#### unshuffled_deduplicated_pt

- **Size of downloaded dataset files:** 26.00 GB
- **Size of the generated dataset:** 68.37 GB
- **Total amount of disk used:** 94.37 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Você pode estar lendo este texto no sofá, levantar pra pegar uma breja na geladeira, dar uma cagada e sentar novamente, sem int..."
}
```

#### unshuffled_deduplicated_qu

- **Size of downloaded dataset files:** 0.02 MB
- **Size of the generated dataset:** 0.07 MB
- **Total amount of disk used:** 0.09 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Warayu wichay (kastilla simipi: Ascensión de Guarayos) nisqaqa Buliwya mama llaqtapi, Santa Krus suyupi, huk llaqtam, Warayu pruwinsyap uma llaqtanmi."
}
```

#### unshuffled_deduplicated_rm

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"practicists agrars / practicistas agraras AFP pon far ina furmaziun da basa scursanida per cuntanscher in attestat federal da q..."
}
```

#### unshuffled_deduplicated_ro

- **Size of downloaded dataset files:** 4.48 GB
- **Size of the generated dataset:** 11.66 GB
- **Total amount of disk used:** 16.14 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"“În viață, oportunitatea nu este totul. Cine atrage Lumina, cineva bun în umbră. Timpul ne creează.” maestru\\nLyn.Evans: Ce mar..."
}
```

#### unshuffled_deduplicated_ru

- **Size of downloaded dataset files:** 166.68 GB
- **Size of the generated dataset:** 611.70 GB
- **Total amount of disk used:** 778.38 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Доступ к данному профилю для публичного просмотра закрыт администрацией сайта - профиль находится на модерации.\\nРазработчикам ..."
}
```

#### unshuffled_deduplicated_sa

- **Size of downloaded dataset files:** 7.27 MB
- **Size of the generated dataset:** 38.33 MB
- **Total amount of disk used:** 45.60 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"अनिरुद्धनगरे क्रीडिता रामलीला सम्‍प्रति समाप्‍ता अस्ति । तस्‍य कानिचन् चित्राणि पूर्वमेव प्रकाशितानि सन्ति । द्वौ चलचित्रौ अपि ..."
}
```

#### unshuffled_deduplicated_sah

- **Size of downloaded dataset files:** 7.01 MB
- **Size of the generated dataset:** 27.46 MB
- **Total amount of disk used:** 34.49 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████..."
}
```

#### unshuffled_deduplicated_scn

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "La gilusìa è nu sintimentu dulurusu ca nasci d'un disideriu di pussessu sclusivu ntê cunfrunti dâ pirsuna amata e dû timuri, dû suspettu o dâ cirtizza dâ sò nfidiltati."
}
```

#### unshuffled_deduplicated_sd

- **Size of downloaded dataset files:** 74.17 MB
- **Size of the generated dataset:** 275.48 MB
- **Total amount of disk used:** 349.66 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"هر ڪو ڄاڻي ٿو ته جڏهن توهان هڪ وڏي خريد ڪرڻ چاهيون ٿا, توهان پڄي ضروري حڪم ۾ ان جي ڪم ڪرڻ جي هٿ ۾ لاڳاپو ڪيو آهي. جي شيء آهي ته..."
}
```

#### unshuffled_deduplicated_sh

- **Size of downloaded dataset files:** 1.45 MB
- **Size of the generated dataset:** 6.44 MB
- **Total amount of disk used:** 7.87 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Opština Gornja Radgona se nalazi u sjeveroistočnoj Sloveniji i graniči s susjednom Austriji duž rijeke Mure. Sa tridesetim nase..."
}
```

#### unshuffled_deduplicated_si

- **Size of downloaded dataset files:** 175.62 MB
- **Size of the generated dataset:** 842.57 MB
- **Total amount of disk used:** 1.02 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"ලාංකීය සිතිවිලි සිංහල බ්ලොග් කියවනය කොත්තු සින්ඩිය ලංකා Blogger හත්මාළුව ලංකා බ්ලොග් කියවනය මාතලන්ගේ සින්ඩිය මොබයිල්lk\\nඅවකාශය ..."
}
```

#### unshuffled_deduplicated_sk

- **Size of downloaded dataset files:** 1.96 GB
- **Size of the generated dataset:** 4.80 GB
- **Total amount of disk used:** 6.76 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Aktivity | Agentúra podporovaného zamestnávania | vzdelávanie pre klientov, vzdelávanie pre odborníkov, kurzy\\nŠpecializované k..."
}
```

#### unshuffled_deduplicated_sl

- **Size of downloaded dataset files:** 523.22 MB
- **Size of the generated dataset:** 1.32 GB
- **Total amount of disk used:** 1.85 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Če Creatures, ki je želel, da pridejo na čas, predvsem je povedlo – razlikuje od ljubosumja začel grizenja kolen (ali zadnjica)..."
}
```

#### unshuffled_deduplicated_so

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"тттттттттттттттттттттттттттттттт тттттттттттттттттттттттттттттттт тттттттттттттттттттттттттттттттт ттттттттттттттттуууууууууууу..."
}
```

#### unshuffled_deduplicated_sq

- **Size of downloaded dataset files:** 445.36 MB
- **Size of the generated dataset:** 1.21 GB
- **Total amount of disk used:** 1.66 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Çfarë do të më pëlqente tek një femër ose çfarë do të më shndërronte në një shpërthim drite? – Albert Vataj\\nTë gjithëve një zo..."
}
```

#### unshuffled_deduplicated_sr

- **Size of downloaded dataset files:** 665.03 MB
- **Size of the generated dataset:** 2.36 GB
- **Total amount of disk used:** 3.03 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Корисни савети за сваки дан. На сајту су разне категорије, као што су љепота, мода, кување и поправка властитим рукама.\\nШколск..."
}
```

#### unshuffled_deduplicated_su

- **Size of downloaded dataset files:** 0.05 MB
- **Size of the generated dataset:** 0.16 MB
- **Total amount of disk used:** 0.21 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Kartu krédit nyaéta \"duit plastik\" anu dikaluarkeun ku bank pikeun alat pambayaran di tempat-tempat nu tangtu samisal jiga di hotél, réstoran, tempat rékréasi jeung sajabana.[1]"
}
```

#### unshuffled_deduplicated_sv

- **Size of downloaded dataset files:** 10.19 GB
- **Size of the generated dataset:** 26.33 GB
- **Total amount of disk used:** 36.51 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"1783 är ett viktigt årtal i den nya tidens historia. Det året slöts en fred i Paris och därmed blev de 13 brittiska kolonierna ..."
}
```

#### unshuffled_deduplicated_sw

- **Size of downloaded dataset files:** 2.95 MB
- **Size of the generated dataset:** 8.98 MB
- **Total amount of disk used:** 11.92 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Miripuko hiyo inakuja mwanzoni mwa Wiki Takatifu kuelekea Pasaka na ikiwa ni wiki chache tu kabla ya Papa Francis kuanza ziara yake katika nchi hiyo yenye idadi kubwa kabisa ya watu katika ulimwengu wa nchi za Kiarabu."
}
```

#### unshuffled_deduplicated_ta

- **Size of downloaded dataset files:** 971.12 MB
- **Size of the generated dataset:** 5.48 GB
- **Total amount of disk used:** 6.45 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"பொழுது சாய்ந்து வெகு நேரமாகிவிட்டது. கூலி வேலைக்குப் போயிருந்த 'சித்தாள் ' பெண்கள் எல்லோரும் வீடு திரும்பி விட்டார்கள். இன்னும்..."
}
```

#### unshuffled_deduplicated_te

- **Size of downloaded dataset files:** 342.43 MB
- **Size of the generated dataset:** 1.70 GB
- **Total amount of disk used:** 2.04 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"హర్యానాలో టోల్ దగ్గర సిబ్బంది.. స్థానిక ప్రజలు కొట్టుకున్నారు. కర్నాల్ అనే గ్రామానికి సమీపంలో టోల్ గేట్ ఉంది. అయితే సాధారణంగా స..."
}
```

#### unshuffled_deduplicated_tg

- **Size of downloaded dataset files:** 62.90 MB
- **Size of the generated dataset:** 261.68 MB
- **Total amount of disk used:** 324.60 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Ҳумайро гуфтааст, мухолифи низом аст, низоме, ки дар Тоҷикистон вуҷуд дорад. Ба ин маънӣ, худро мухолифи давлату ҳукумати Тоҷик..."
}
```

#### unshuffled_deduplicated_th

- **Size of downloaded dataset files:** 3.54 GB
- **Size of the generated dataset:** 17.11 GB
- **Total amount of disk used:** 20.65 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ฟันที่แลดูขาวสะอาดไม่มีเศษอาหารติดอยู่ เหงือกสีชมพู ไม่เจ็บ หรือมีเลือดออกเวลาแปรงฟันหรือขัดฟัน ไม่มีปัญหาเรื่องกลิ่นปาก ทำให้ก..."
}
```

#### unshuffled_deduplicated_tk

- **Size of downloaded dataset files:** 2.22 MB
- **Size of the generated dataset:** 7.12 MB
- **Total amount of disk used:** 9.34 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Türkmenistanyň Prezidenti agyr atletika boýunça dünýä çempionatyna taýýarlyk işleriniň barşy bilen tanyşdy\\nHalallykdan kemal t..."
}
```

#### unshuffled_deduplicated_tl

- **Size of downloaded dataset files:** 151.34 MB
- **Size of the generated dataset:** 431.69 MB
- **Total amount of disk used:** 583.04 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"“Gusto ko manawagan sa mga Unit Head ng Chanel 2 Salve. Kasi napapansin ko iyon mga alaga ko ang taping halos once a week lang,..."
}
```

#### unshuffled_deduplicated_tr

- **Size of downloaded dataset files:** 10.39 GB
- **Size of the generated dataset:** 28.47 GB
- **Total amount of disk used:** 38.86 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Son yıllarda görülen ay tutulmalarına göre daha etkili olacağı söylenen Kanlı veya Kırmızı Ay Tutulmasına saatler kaldı. Bu akş..."
}
```

#### unshuffled_deduplicated_tt

- **Size of downloaded dataset files:** 85.89 MB
- **Size of the generated dataset:** 321.37 MB
- **Total amount of disk used:** 407.26 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"\\\"Иремнең вафатына 40 көн узгач, Алмаз да безнең өйгә кереп үлде\\\". Арчада 35 яшьлек ир өстенә кондызлар ега башлаган агач төшк..."
}
```

#### unshuffled_deduplicated_tyv

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Экии, хүндүлуг аалчылар болгаш тыва дылдың деткикчилери! Тыва дылдың болгаш чогаалдың ховар бир башкызынга, Менги Ооржакка, ажы..."
}
```

#### unshuffled_deduplicated_ug

- **Size of downloaded dataset files:** 20.53 MB
- **Size of the generated dataset:** 86.44 MB
- **Total amount of disk used:** 106.97 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"زاڭ-ءتۇزىم | عىلىم-تەحنيكا | ءتىل-ادەبيەت | تۇرمىس | دەنە تاربيە | ساياحات-ورتا | سۋرەتتى حابار | سىر سۇحبات | ارناۋلى تاقىرىپ ..."
}
```

#### unshuffled_deduplicated_uk

- **Size of downloaded dataset files:** 8.04 GB
- **Size of the generated dataset:** 29.86 GB
- **Total amount of disk used:** 37.90 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Про надання роз'яснення (щодо форми письмового зобов'язання громадян про зворотне ввезення/вивезення товарів), Державна митна с..."
}
```

#### unshuffled_deduplicated_ur

- **Size of downloaded dataset files:** 483.59 MB
- **Size of the generated dataset:** 1.82 GB
- **Total amount of disk used:** 2.31 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"آئیے اہم اسلامی کتب کو یونیکوڈ میں انٹرنیٹ پر پیش کرنے کے لئے مل جل کر آن لائن ٹائپنگ کریں۔ محدث ٹائپنگ پراجیکٹ کے ذریعے آپ روز..."
}
```

#### unshuffled_deduplicated_uz

- **Size of downloaded dataset files:** 4.30 MB
- **Size of the generated dataset:** 12.00 MB
- **Total amount of disk used:** 16.29 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Qurama tog'lari tizmasining Toshkentdan 154 km uzoqlikdagi Toshkent-Ush yo'li yeqasidaxushmanzara tabiat qo'ynida joylashgan maydoni 30 ga.\nBolalarni sog'lomlashtirish oromgohi Bo'stonliq tumani Oqtosh muntaqasining soy-salqin gushasida joylashgan."
}
```

#### unshuffled_deduplicated_vec

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Par ogni pónto, ła derivada ła xe ła pendensa de ła reta tangente a ła curva de ła funsion f. Ła reta de cołor róso l'è senpre ..."
}
```

#### unshuffled_deduplicated_vi

- **Size of downloaded dataset files:** 10.71 GB
- **Size of the generated dataset:** 33.60 GB
- **Total amount of disk used:** 44.31 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Canh chua cá bông lau không chỉ là món ăn giải nhiệt, thanh mát ngày hè mà còn là món siêu bổ dưỡng, rất tốt cho người gầy ốm. ..."
}
```

#### unshuffled_deduplicated_vo

- **Size of downloaded dataset files:** 0.30 MB
- **Size of the generated dataset:** 2.10 MB
- **Total amount of disk used:** 2.40 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Sarniguet binon zif in ziläk: Hautes-Pyrénées, in topäd: Midi-Pyrénées, in Fransän. Sarniguet topon videtü 43°19’ 7’’ N e lunetü 0°5’ 19’’ L."
}
```

#### unshuffled_deduplicated_wa

- **Size of downloaded dataset files:** 0.08 MB
- **Size of the generated dataset:** 0.22 MB
- **Total amount of disk used:** 0.29 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Cisse pådje ci n' est co k' on djermon, dj' ô bén k' el pådje est djusse sibåtcheye, eyet co trop tene; et s' divreut ele ecråxhî ene miete."
}
```

#### unshuffled_deduplicated_war

- **Size of downloaded dataset files:** 0.55 MB
- **Size of the generated dataset:** 2.36 MB
- **Total amount of disk used:** 2.90 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "An Honce amo in usa ka baryo ngan munisipalidad ha distrito han Rožňava ha rehiyon han Košice ha nasod han Slovakia.\nAn Rumegies amo in usa ka komyun ha departamento han Nord ngan ha rehiyon han Nord-Pas-de-Calais ha nasod han Fransya."
}
```

#### unshuffled_deduplicated_wuu

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.03 MB
- **Total amount of disk used:** 0.04 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"伊春元旦天气 伊春腊八天气 伊春春节天气 伊春情人节天气 伊春元宵节天气 伊春愚人节天气 伊春清明节天气 伊春劳动节天气 伊春母亲节天气 伊春端午节天气 伊春七夕节天气 伊春教师节天气 伊春中秋节天气 伊春国庆节天气 伊春重阳节天气 伊春万圣节天气 伊春..."
}
```

#### unshuffled_deduplicated_xal

- **Size of downloaded dataset files:** 0.03 MB
- **Size of the generated dataset:** 0.12 MB
- **Total amount of disk used:** 0.15 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Арнгудин Орн гисн Европд бәәдг һазр. 2007 җилин тooһaр эн орн нутгт 3,600,523 әмтн бәәдг билә. Арнгудин Орнин хотл балһсна нерн..."
}
```

#### unshuffled_deduplicated_xmf

- **Size of downloaded dataset files:** 0.94 MB
- **Size of the generated dataset:** 4.63 MB
- **Total amount of disk used:** 5.58 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"მოჩამილი ტექსტი წჷმორინელი რე Creative Commons Attribution-ShareAlike ლიცენზიათ; შილებე გეძინელი პირობეფიშ არსებუა. კილიშკილიშა..."
}
```

#### unshuffled_deduplicated_yi

- **Size of downloaded dataset files:** 22.20 MB
- **Size of the generated dataset:** 88.29 MB
- **Total amount of disk used:** 110.49 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ממשותדיק - חבֿרה, איך אַרבעט איצט אױף אַ זשורנאַל. טאָמער איר האָט עפּעס צוצוגעבן זאָלט איר שיקן מיר אַן אָנזאָג. ס'װעט הײסן \\\"..."
}
```

#### unshuffled_deduplicated_yo

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.03 MB
- **Total amount of disk used:** 0.04 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Copyright © 2018 BBC. BBC kò mọ̀ nípa àwọn ohun tí ó wà ní àwọn ojú òpó tí ó wà ní ìta. Ọwọ́ tí a fi mú ìbáṣepọ̀ ti ìta.\"..."
}
```

#### unshuffled_deduplicated_yue

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 你還不爆 我累了 投降輸一半可以嗎\"..."
}
```

#### unshuffled_deduplicated_zh

- **Size of downloaded dataset files:** 99.98 GB
- **Size of the generated dataset:** 267.88 GB
- **Total amount of disk used:** 367.86 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"中国铝灰网 中国有色金属矿产网 中国黄莲网 中国水轮发电机网 中国抽油泵网 中国数控雕刻机网 中国不锈钢抛光网 中国磨具加工网 中国压铸铝网 中国耐水腻子网 中国手机摄像头网 中国粗粮网 中国车门锁网 中国钛粉网 中国轮圈网\\n天天中奖彩票图 天天中彩票..."
}
```

</details>


<details>
  <summary>Click to expand the Data/size information for each language (original)</summary>

#### unshuffled_original_af

- **Size of downloaded dataset files:** 85.79 MB
- **Size of the generated dataset:** 254.08 MB
- **Total amount of disk used:** 339.87 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "aanlyn markte as gevolg van ons voortgesette 'n begrip opsie handel sakeplan pdf terwyl ons steeds die gereelde ons binêre opsies handel"
}
```

#### unshuffled_original_als

- **Size of downloaded dataset files:** 1.49 MB
- **Size of the generated dataset:** 5.30 MB
- **Total amount of disk used:** 6.78 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"De Nazionalpark hät e Flächi vo 170,3 km² und isch dodemit s grösti Naturschutzgebiet vo de Schwiz. Er ligt uf em Gebiet vo de ..."
}
```

#### unshuffled_original_am

- **Size of downloaded dataset files:** 102.79 MB
- **Size of the generated dataset:** 378.06 MB
- **Total amount of disk used:** 480.85 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"አየር መንገዱ ከአዲስ አበባ ወደ ሮም ጣሊያን በማምራት ላይ በነበረበት ጊዜ ረዳት አብራሪው የጉዞውን አቅጣጫ በመቀየር ጄኔቭ አውሮፓላን ማረፊያ በማሳረፍ እጁን ለፖሊስ ሰጥቷል።\\nየኢትዮጵያ መንግስት የ..."
}
```

#### unshuffled_original_an

- **Size of downloaded dataset files:** 0.15 MB
- **Size of the generated dataset:** 1.33 MB
- **Total amount of disk used:** 1.48 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"واااااااأسفاه الأمم تفتخر ب 0 أمي ووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووووو..."
}
```

#### unshuffled_original_ar

- **Size of downloaded dataset files:** 22.23 GB
- **Size of the generated dataset:** 87.94 GB
- **Total amount of disk used:** 110.17 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"مرحبا بك عزيز الزائر نتمنى لك أوقاتاً سعيدة معنا وأن نزداد شرفا بخدمتك ولا تنسى التسجيل معنا لتستفيد بكل جديد\\nأهلا وسهلا بك زا..."
}
```

#### unshuffled_original_arz

- **Size of downloaded dataset files:** 15.90 MB
- **Size of the generated dataset:** 70.13 MB
- **Total amount of disk used:** 86.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"بنى عجل : قبيلة من عجل بن لجيم بن صعب بن على بن بكر بن وائل انتقل اغلبهم الى البصرة فى العراق و اصفهان و خراسان فى ايران و اذرب..."
}
```

#### unshuffled_original_as

- **Size of downloaded dataset files:** 21.43 MB
- **Size of the generated dataset:** 117.73 MB
- **Total amount of disk used:** 139.17 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"আমি, এই সংগঠনৰ সদস্য সকলে একেলগ হৈ অসমকে ধৰি ভাৰতৰ উত্তৰ পূৰ্বাঞ্চলৰ অমূল্য কলা-সাংস্কৃতিক সম্পদৰাজি বৃহত্তৰ অষ্ট্ৰেলিয়াৰ সন্মু..."
}
```

#### unshuffled_original_ast

- **Size of downloaded dataset files:** 0.92 MB
- **Size of the generated dataset:** 2.54 MB
- **Total amount of disk used:** 3.46 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"The Killers llanzaron el so álbum debú, Hot Fuss, en xunu de 2004 nel Reinu Xuníu, al traviés de la discográfica Lizard King, y..."
}
```

#### unshuffled_original_av

- **Size of downloaded dataset files:** 0.08 MB
- **Size of the generated dataset:** 0.42 MB
- **Total amount of disk used:** 0.50 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Жинда малъараб ва божизе бегьулеб рагІудаса кьуризе бегьуларо гьев. Гьес насихІат гьабизе кколелъул бацІцІадаб диналъул рахъалъ..."
}
```

#### unshuffled_original_az

- **Size of downloaded dataset files:** 927.76 MB
- **Size of the generated dataset:** 2.96 GB
- **Total amount of disk used:** 3.89 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"AZTV-Artıq 7 ildir ki, Abşeron rayonu dotasiya almadan bütün xərclərini yerli daxilolmalar hesabına maliyyələşdirir.\\nDünən, 10..."
}
```

#### unshuffled_original_azb

- **Size of downloaded dataset files:** 6.64 MB
- **Size of the generated dataset:** 28.47 MB
- **Total amount of disk used:** 35.11 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"لعلی ١٣-جو عصرده یاشاییب یاراتمیش گؤرکملی آذربایجان شاعرلریندندیر. ١٢٢٤-جی ایلده تبریزده آنادان اولموشدور، گنج یاشلاریندا تیجار..."
}
```

#### unshuffled_original_ba

- **Size of downloaded dataset files:** 33.22 MB
- **Size of the generated dataset:** 133.70 MB
- **Total amount of disk used:** 166.92 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Күҙәтеү ҡуласаһы моделен хәҙер Мифтахетдин Аҡмулла исемендәге Башҡорт дәүләт педагогия университетында ла эшләргә мөмкин\\t\\nКүҙ..."
}
```

#### unshuffled_original_bar

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "                                                                                                                                          vo"
}
```

#### unshuffled_original_bcl

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"& ÿ ó / í 0 - ø û ù ö ú ð ï ú \\u0014 ù þ ô ö í ÷ ò \\u0014 ÷ í ù û ö í \\u0001 û ñ ç þ \\u0001 ð \\u0007 þ ò ñ ñ ò ô \\u0017 û ö ô ÷..."
}
```

#### unshuffled_original_be

- **Size of downloaded dataset files:** 498.29 MB
- **Size of the generated dataset:** 1.88 GB
- **Total amount of disk used:** 2.38 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Брэсцкія ўлады не дазволілі прафсаюзу РЭП правесці пікетаванне ў парку Воінаў-інтэрнацыяналістаў 30 мая 2018 года.\\nСітуацыю пр..."
}
```

#### unshuffled_original_bg

- **Size of downloaded dataset files:** 8.34 GB
- **Size of the generated dataset:** 33.75 GB
- **Total amount of disk used:** 42.09 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ЖАЛБОПОДАТЕЛЯТ директор на Дирекция „ Обжалване и данъчно-осигурителна практика“- Бургас, редовно призован, се представлява от ..."
}
```

#### unshuffled_original_bh

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.12 MB
- **Total amount of disk used:** 0.13 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"सुकमा जिला भारत के छत्तीसगढ़ राज्य में एगो जिला बाटे। एकर मुख्यालय सुकमा शहर बाटे। एकर कुल रकबा 5636 वर्ग कि॰मी॰ बाटे।\"..."
}
```

#### unshuffled_original_bn

- **Size of downloaded dataset files:** 2.14 GB
- **Size of the generated dataset:** 10.77 GB
- **Total amount of disk used:** 12.91 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ভড়ং সর্বস্ব বাংলা আর্ট অ্যান্ড কালচারের হিসাব গুলিয়ে দেওয়ার ম্যাজিকের নাম ব্রাত্য রাইসু November 23, 2017\\nভড়ং সর্বস্ব বাংলা আর..."
}
```

#### unshuffled_original_bo

- **Size of downloaded dataset files:** 28.94 MB
- **Size of the generated dataset:** 195.40 MB
- **Total amount of disk used:** 224.34 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"བོད་མི་འདི་དག་ནི་རང་རྒྱུད་སྒོ་རུ་ཕུད་དེ་གཞན་རྒྱུད་པང་དུ་ཉར་ནས་གསོ་སྐྱོང་བྱེད་དགོས་ཟེར་བ་དང་གཅིག་མཚུངས་རེད།\\nཚན་རིག་ནི་དང་ཐོག་རང..."
}
```

#### unshuffled_original_bpy

- **Size of downloaded dataset files:** 0.34 MB
- **Size of the generated dataset:** 4.35 MB
- **Total amount of disk used:** 4.69 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"পৌরসভা এহার আয়তন (লয়াহান) ২,৭৩০,.৬৩ বর্গ কিলোমিটার। পৌরসভা এহার মাপাহানর অক্ষাংশ বারো দ্রাঘিমাংশ ইলতাই 18.63° S 48.18° W ।[১]..."
}
```

#### unshuffled_original_br

- **Size of downloaded dataset files:** 9.18 MB
- **Size of the generated dataset:** 30.20 MB
- **Total amount of disk used:** 39.38 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Ar mank Magalhães(Daveoù a vank) a zo ur spesad evned, Spheniscus magellanicus an anv skiantel anezhañ.\\nGallout a reer implijo..."
}
```

#### unshuffled_original_bs

- **Size of downloaded dataset files:** 0.05 MB
- **Size of the generated dataset:** 0.48 MB
- **Total amount of disk used:** 0.53 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ž šř é ú šř šř ě šř ž é č ě ž ů ě ď éé ýš ě ě Ž č š ý ě ď é ýš ě ď ě éé ýš ě č ž ě š ý ď ě ýš é ú č ž č š ý ď ý ž é éě ď é č ýš..."
}
```

#### unshuffled_original_bxr

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"2002 оной хабар буряад хэлэ бэшэгэй һалбари Үндэһэтэнэй хүмүүнлиг ухаанай дээдэ һургуули болгогдожо өөршэлэгдөө.\\nХарин мүнөө б..."
}
```

#### unshuffled_original_ca

- **Size of downloaded dataset files:** 3.10 GB
- **Size of the generated dataset:** 8.62 GB
- **Total amount of disk used:** 11.73 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Daniel Vendrell, conegut com Vandrell, ha sigut un dels il•lustradors contemporanis més influents, representant a la nova onada..."
}
```

#### unshuffled_original_cbk

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano yo gano..."
}
```

#### unshuffled_original_ce

- **Size of downloaded dataset files:** 2.09 MB
- **Size of the generated dataset:** 8.73 MB
- **Total amount of disk used:** 10.82 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Шаьш анархисташ ду бохучу жигархойн дIахьедарехь дуьйцу, оьрсийн ницкъаллийн структурийн а, федералан каналан а Iалашонаш \\\"мар..."
}
```

#### unshuffled_original_ceb

- **Size of downloaded dataset files:** 11.07 MB
- **Size of the generated dataset:** 40.97 MB
- **Total amount of disk used:** 52.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Si Isko walay pupamilok nga nagtan-aw sa unahan, natugaw. “Naunsa ka gud diha Isko nga layo man kaayo ang imong panan-aw?” ni I..."
}
```

#### unshuffled_original_ckb

- **Size of downloaded dataset files:** 111.88 MB
- **Size of the generated dataset:** 510.97 MB
- **Total amount of disk used:** 622.85 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"رسی رۆژ - ساڵێک دوای بومەلەرزەی کرماشان میوانی بەرنامە : کاک سیاوەش حەیاتی چالاکی مەدەنی -قەسری شیرین\\nپارچە موزیک 30 / 10 / 20..."
}
```

#### unshuffled_original_cs

- **Size of downloaded dataset files:** 21.72 GB
- **Size of the generated dataset:** 57.08 GB
- **Total amount of disk used:** 78.80 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Akce anarchistů proti připravovanému novému služební řádu a nízkým mzdám 1903 – Historie českého anarchismu (1880 – 1939)\\nRost..."
}
```

#### unshuffled_original_cv

- **Size of downloaded dataset files:** 9.40 MB
- **Size of the generated dataset:** 41.05 MB
- **Total amount of disk used:** 50.45 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Шыранӑ чухне ӑнсӑртран латин кирилл саспаллисем вырӑнне латин саспаллисене ҫырсан, сайт эсир ҫырнине юсама тӑрӑшӗ.\\nКу сайтра ч..."
}
```

#### unshuffled_original_cy

- **Size of downloaded dataset files:** 81.74 MB
- **Size of the generated dataset:** 224.93 MB
- **Total amount of disk used:** 306.67 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Mae capeli Cymreig yr Andes ym Mhatagonia wedi cyhoeddi na fydd gwasanaethau yno weddill y mis, oherwydd yr eira trwm sydd wedi..."
}
```

#### unshuffled_original_da

- **Size of downloaded dataset files:** 6.00 GB
- **Size of the generated dataset:** 16.76 GB
- **Total amount of disk used:** 22.76 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Den 2.-5. februar 2016 løb det tredje kursus i uddannelsen af 4kommunesamarbejdets Local Impact Coaches, af stablen i Gentofte ..."
}
```

#### unshuffled_original_de

- **Size of downloaded dataset files:** 119.51 GB
- **Size of the generated dataset:** 331.22 GB
- **Total amount of disk used:** 450.73 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Auf dieser Seite gibt es mind. ein YouTube Video. Cookies für diese Website wurden abgelehnt. Dadurch können keine YouTube Vide..."
}
```

#### unshuffled_original_diq

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Zıwanê Slawki, zıwano merdumanê Slawano. Zıwanê Slawki yew lızgeyê Zıwananê Hind u Ewropao. Keyeyê Zıwananê Slawki beno hirê letey:"
}
```

#### unshuffled_original_dsb

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Pśiklaskaju južo pśed pśedstajenim... 1500 źiśi njamóžo wěcej docakaś, měsćańska hala w Chóśebuzu - wupśedana."
}
```

#### unshuffled_original_dv

- **Size of downloaded dataset files:** 24.91 MB
- **Size of the generated dataset:** 131.63 MB
- **Total amount of disk used:** 156.54 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ބ. އަތޮޅުގައި ހުޅުވަން ތައްޔާރުވަމުން އަންނަ ވައްކަރު ރިސޯޓުގައި ވަޒީފާ އަދާކުރަން ޝައުގުވެރިވާ ފަރާތްތަކަށް ކުރިމަތިލުމުގެ ފުރ..."
}
```

#### unshuffled_original_el

- **Size of downloaded dataset files:** 17.31 GB
- **Size of the generated dataset:** 66.27 GB
- **Total amount of disk used:** 83.58 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Νεκρός εντοπίστηκε μέσα στο σπίτι του στην οδό Ηρώδου Αττικού στον αριθμό 7 ο επικεφαλής του προξενικού τμήματος της Ρωσικής πρ..."
}
```

#### unshuffled_original_eml

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"A séguit dal prucès ad rubutiśasiòṅ di abitànt dal pòpul ad Mikenes, Angoras 'l è finî dènt'r a 'n robot cun la tèsta dna rana ..."
}
```

#### unshuffled_original_en

- **Size of downloaded dataset files:** 903.83 GB
- **Size of the generated dataset:** 2525.44 GB
- **Total amount of disk used:** 3429.27 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Mtendere Village was inspired by the vision of Chief Napoleon Dzombe, which he shared with John Blanchard during his first visi..."
}
```

#### unshuffled_original_eo

- **Size of downloaded dataset files:** 117.07 MB
- **Size of the generated dataset:** 314.18 MB
- **Total amount of disk used:** 431.27 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Ĉu ... preĝi | mediti | ricevi instigojn || kanti | muziki || informiĝi | legi | studi || prepari Diservon\\nTemas pri kolekto d..."
}
```

#### unshuffled_original_es

- **Size of downloaded dataset files:** 106.04 GB
- **Size of the generated dataset:** 298.49 GB
- **Total amount of disk used:** 404.53 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Como se librará de la celulitis en el gimnasio La piel superflua en las manos después del adelgazamiento, Los bailes fáciles pa..."
}
```

#### unshuffled_original_et

- **Size of downloaded dataset files:** 1.88 GB
- **Size of the generated dataset:** 5.17 GB
- **Total amount of disk used:** 7.06 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"MTÜ AB Video järgib oma tegevuses kodanikuühenduste eetilise tegevuse üldtunnustatud põhimõtteid, mis on lühidalt kokkuvõetud 7..."
}
```

#### unshuffled_original_eu

- **Size of downloaded dataset files:** 248.19 MB
- **Size of the generated dataset:** 894.83 MB
- **Total amount of disk used:** 1.14 GB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Gure jarduerek eraikuntzarekin, elkarbizitzarekin, hirigintzarekin eta ekologiarekin dute harremana, baita ideia eta konponbideak irudikatu eta garatzearekin ere, eraikuntza sektorea hobetuz, pertsonen erosotasuna eta bizi-kalitatea hobetzeko."
}
```

#### unshuffled_original_fa

- **Size of downloaded dataset files:** 20.96 GB
- **Size of the generated dataset:** 84.21 GB
- **Total amount of disk used:** 105.17 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"قـــــــــــــــــرار بود با هم کنـــــــــــــار بیایم نه اینکه از کنــــــــــــار هم رد بشیم...!!!\\nاگر روزی دلت لبریز غم بو..."
}
```

#### unshuffled_original_fi

- **Size of downloaded dataset files:** 9.97 GB
- **Size of the generated dataset:** 28.57 GB
- **Total amount of disk used:** 38.54 GB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Kiitos Deelle kaikesta - 1,5 viikkoa kulunut, kun Dee ei ole enää ollut omani. Reilu viikko sitten sunnuntaina vein Deen uuteen kotiinsa. Itselläni on ollut niin ristiriitaiset t..."
}
```

#### unshuffled_original_fr

- **Size of downloaded dataset files:** 105.32 GB
- **Size of the generated dataset:** 303.19 GB
- **Total amount of disk used:** 408.51 GB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Média de débat d'idées, de culture et de littérature. Récits, décryptages, analyses, portraits et critiques autour de la vie des idées. Magazine engagé, ouvert aux autres et au monde.. Bring up to date in french"
}
```

#### unshuffled_original_frr

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Hiragana’ Practice’Sheet’1’(A -O)’ ’ Name:’________ __________________________’Section:’_______________ _’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ’ ..."
}
```

#### unshuffled_original_fy

- **Size of downloaded dataset files:** 12.40 MB
- **Size of the generated dataset:** 36.24 MB
- **Total amount of disk used:** 48.64 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Nim in sêfte ride op Holmsjön, yn ien fan 'e lytse marren yn de omkriten, of nim se op avontueren lykas nonresidential. lâns Indalsälven wetter. Holm Sportklubb hawwe kano 's te huur, yn gearwurking mei de Baltyske Power konferinsje."
}
```

#### unshuffled_original_ga

- **Size of downloaded dataset files:** 29.27 MB
- **Size of the generated dataset:** 92.37 MB
- **Total amount of disk used:** 121.63 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Is fóram é seo chun plé a dhéanamh ar an leabhar atá roghnaithe do mhí na Samhna 2013 amháin. Ní féidir ach le baill chláraithe..."
}
```

#### unshuffled_original_gd

- **Size of downloaded dataset files:** 0.52 MB
- **Size of the generated dataset:** 2.02 MB
- **Total amount of disk used:** 2.55 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Zhou Yujun, a 'phàrtaidh Rùnaire Comataidh Sgìre Yanfeng ann Hengyang bhaile agus a Sgìre pàrtaidh agus an riaghaltas a' bhuidheann-riochdachaidh a 'tighinn a chèilidh air ar companaidh air Apr. 14, 2017."
}
```

#### unshuffled_original_gl

- **Size of downloaded dataset files:** 235.38 MB
- **Size of the generated dataset:** 656.48 MB
- **Total amount of disk used:** 891.87 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"O persoal de Inditex da provincia de Pontevedra segue a reclamar iguais condicións laborais no conxunto do país - CIG: Confeder..."
}
```

#### unshuffled_original_gn

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.04 MB
- **Total amount of disk used:** 0.05 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"º ÑÆÚÓ À Ã Ð É Æ ¾ ÄÂ Î À ¼ Æ É ÄÛ = Ü Ý\\\"Þ ßà á â ã ä å æçè ã é ê â å àë ì æê íî é á ë ï í çì àð í Ü à ñ ê é ò ä ì\"..."
}
```

#### unshuffled_original_gom

- **Size of downloaded dataset files:** 0.44 MB
- **Size of the generated dataset:** 2.25 MB
- **Total amount of disk used:** 2.71 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"दुष्ट शीळ हें कौरवांचें । रामें सविस्तर देखूनि साचें । बोलिले वचनें जें दुर्वाचे । करी तयांचें अनुस्मरण ॥२२०॥\"..."
}
```

#### unshuffled_original_gu

- **Size of downloaded dataset files:** 232.02 MB
- **Size of the generated dataset:** 1.09 GB
- **Total amount of disk used:** 1.33 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"અધિક માસ ચાલે છે. સમગ્ર ભારતમાં અને તેમાંય ખાસ કરીને પવિત્ર કે ધાર્મિક કહેવાય છે તેવા સ્થાનક પર કથાનો દોર ચાલે છે. ઉનાળાની કાળઝ..."
}
```

#### unshuffled_original_he

- **Size of downloaded dataset files:** 5.66 GB
- **Size of the generated dataset:** 21.11 GB
- **Total amount of disk used:** 26.77 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"זקוקים לרשתות נגד יתושים? מחפשים רשת מתאימה לחלון צר וקטן? רשתות נגד יתושים אקורדיון של חברת קליר-מש הן הפתרון.\\nרשתות לחלונות ..."
}
```

#### unshuffled_original_hi

- **Size of downloaded dataset files:** 3.66 GB
- **Size of the generated dataset:** 17.93 GB
- **Total amount of disk used:** 21.59 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"'आइटम गर्ल' बनकर हिट हुई थीं राखी सावंत, आज करीना-कटरीना तक फॉलो कर रही हैं ट्रेंड नक्‍सलियों का दम निकालेगा बाइक ग्रेनेड लॉन्च..."
}
```

#### unshuffled_original_hr

- **Size of downloaded dataset files:** 79.42 MB
- **Size of the generated dataset:** 243.83 MB
- **Total amount of disk used:** 323.24 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"U raspravi je sudjelovao i HSS-ov saborski zastupnik rekavši kako poljoprivrednici ne osjete mjere o kojima ministar govori jer..."
}
```

#### unshuffled_original_hsb

- **Size of downloaded dataset files:** 1.39 MB
- **Size of the generated dataset:** 4.49 MB
- **Total amount of disk used:** 5.87 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Budyšin (SN/BŠe). Elektronikarjo mějachu lětsa cyle hinaši zazběh do swojeho wukubłanja. Wokrjesne rjemjeslnistwo bě mjenujcy w..."
}
```

#### unshuffled_original_ht

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan..."
}
```

#### unshuffled_original_hu

- **Size of downloaded dataset files:** 15.69 GB
- **Size of the generated dataset:** 43.07 GB
- **Total amount of disk used:** 58.77 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"monster - Amatőr, házi szex videók és kezdő csjaok pornó filmjei. - Free amateur, home made sex videos and online porn movies. ..."
}
```

#### unshuffled_original_hy

- **Size of downloaded dataset files:** 897.36 MB
- **Size of the generated dataset:** 3.94 GB
- **Total amount of disk used:** 4.84 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Արցախի Հանրապետության հռչակման 26-րդ տարեդարձի կապակցությամբ Շուշիի Արվեստի կենտրոնում կազմակերպվել է մոսկվաբնակ նկարիչներ՝ հայ..."
}
```

#### unshuffled_original_ia

- **Size of downloaded dataset files:** 0.08 MB
- **Size of the generated dataset:** 0.69 MB
- **Total amount of disk used:** 0.78 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha h..."
}
```

#### unshuffled_original_id

- **Size of downloaded dataset files:** 10.60 GB
- **Size of the generated dataset:** 32.32 GB
- **Total amount of disk used:** 42.91 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Perihal dari itu, kalau kunci hal yang demikian hilang, pemilik wajib melapor ke bengkel sah untuk dibuatkan kunci baru dengan ..."
}
```

#### unshuffled_original_ie

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Plastic Yo Yo Metal Yo Yos Wooden Yo Yo Keychain Yo Yo Translucent Yo Yo Light Up Yo Yo Globe Yo Yo Stress Reliever Yo Yo Jellyfish Yo Yo Sports Ball Yo Yo Sound Yo Yo Miniature Yo Yo Promotional Yo Yo Novelty Yo Yo Video Game Yo Yo ECO Recycled Yo Yo"
}
```

#### unshuffled_original_ilo

- **Size of downloaded dataset files:** 0.27 MB
- **Size of the generated dataset:** 0.92 MB
- **Total amount of disk used:** 1.20 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Segun ken ni Ping-ay, ti yellow corn ti maysa kadagiti nadakamat a liberalized agricultural commodity iti daytoy a free trade k..."
}
```

#### unshuffled_original_io

- **Size of downloaded dataset files:** 0.04 MB
- **Size of the generated dataset:** 0.16 MB
- **Total amount of disk used:** 0.20 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Chekia esas parlamentala republiko. La chefo di stato esas la prezidanto. Til 2013 lu elektesis dal parlamento. Pos ta yaro, ol..."
}
```

#### unshuffled_original_is

- **Size of downloaded dataset files:** 533.03 MB
- **Size of the generated dataset:** 1.52 GB
- **Total amount of disk used:** 2.06 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Eyjar.net - upplýsinga- og fréttamiðill um Vestmannaeyjar - Fréttir - Nái núverandi stefna stjórnvalda fram að ganga mun það va..."
}
```

#### unshuffled_original_it

- **Size of downloaded dataset files:** 52.16 GB
- **Size of the generated dataset:** 147.38 GB
- **Total amount of disk used:** 199.54 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Jaundice - causes, treatment & pathology massaggio a osteochondrosis dellindizio di una controindicazione\\nTrattamento su un co..."
}
```

#### unshuffled_original_ja

- **Size of downloaded dataset files:** 79.56 GB
- **Size of the generated dataset:** 232.22 GB
- **Total amount of disk used:** 311.78 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"神社などへ一緒に同行して、様々な角度のショットで家族写真やお子様の写真を撮影致します！お好みに合わせて様々な写真を取ることができますので、その場でカメラマンへのリクエストも可能です！お子様の晴れ姿を、緊張していない自然な笑顔で残しませんか？\\n※七五三の..."
}
```

#### unshuffled_original_jbo

- **Size of downloaded dataset files:** 0.21 MB
- **Size of the generated dataset:** 0.77 MB
- **Total amount of disk used:** 0.98 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "ni'o 23 la cimast. cu 23moi djedi fi'o masti la cimast. noi ke'a cu cimoi masti .i 22 la cimast. cu purlamdei .ije 24 la cimast. cu bavlamdei"
}
```

#### unshuffled_original_jv

- **Size of downloaded dataset files:** 0.22 MB
- **Size of the generated dataset:** 0.69 MB
- **Total amount of disk used:** 0.91 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"José Mourinho (diwaca: [ʒuˈzɛ moˈɾiɲu]; lair ing Setubal, Portugal, 26 Januari 1963; umur 55 taun) iku salah siji pelatih bal k..."
}
```

#### unshuffled_original_ka

- **Size of downloaded dataset files:** 680.74 MB
- **Size of the generated dataset:** 3.77 GB
- **Total amount of disk used:** 4.45 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"წამიყვანე შენთან ერთად (ქართულად) / Возьми меня с собой (картулад) / (რუსული სერიალები ქართულად) (რუსების პორნო ონლაინში) (ruse..."
}
```

#### unshuffled_original_kk

- **Size of downloaded dataset files:** 615.06 MB
- **Size of the generated dataset:** 2.83 GB
- **Total amount of disk used:** 3.45 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Түлкібас ауданында «Латын негізді әліпби мен емле ережесі туралы насихат» жобасының тобы семинар өткізді\\nЕлорданың «Қазақстан»..."
}
```

#### unshuffled_original_km

- **Size of downloaded dataset files:** 193.28 MB
- **Size of the generated dataset:** 1.10 GB
- **Total amount of disk used:** 1.30 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ខ្សឹបដាក់ត្រចៀក៖ លោក សួស សុផានិត នាយផ្នែករដ្ឋបាលព្រៃឈើ ស្រុកភ្នំក្រវាញ់ ដែលទើបឡើងកាន់តំណែងថ្មី បើកដៃឲ្យឈ្នួញ ប្រព្រឹត្តបទល្មើស ..."
}
```

#### unshuffled_original_kn

- **Size of downloaded dataset files:** 342.15 MB
- **Size of the generated dataset:** 1.76 GB
- **Total amount of disk used:** 2.11 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ರಾಷ್ಟ್ರಪತಿ ಪ್ರಣಬ್ ಮುಖರ್ಜಿಯಿಂದ ಪದ್ಮ ಪ್ರಶಸ್ತಿ ಪ್ರದಾನ | President Pranab Mukherjee Confers Padma Awards | Photo Gallery on Kannada..."
}
```

#### unshuffled_original_ko

- **Size of downloaded dataset files:** 8.81 GB
- **Size of the generated dataset:** 25.29 GB
- **Total amount of disk used:** 34.10 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"CIA 프로젝트에서는 데이터베이스로 들어오는 요청을 중간에 수집(Sniffing)하고 수집한 데이터를 분석(Parsing)하여 그로 인한 결과를 판단하여 알릴 수 있는 시스템(Push Service)이 필요하다. 그리고 연구를 ..."
}
```

#### unshuffled_original_krc

- **Size of downloaded dataset files:** 0.66 MB
- **Size of the generated dataset:** 2.68 MB
- **Total amount of disk used:** 3.34 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Шамханланы, Бийлени къаршысына ябушуп, Батыр уланларыбызны къоллары булан «ортакъ ожакъ» къургъанбыз. Шо иш уллу зараллы иш бол..."
}
```

#### unshuffled_original_ku

- **Size of downloaded dataset files:** 33.38 MB
- **Size of the generated dataset:** 99.06 MB
- **Total amount of disk used:** 132.44 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Me di 114 bernameyên xwe yên berê da perçeyên ji berhemên zanyarî yên kurdzanên mezin bi wergera kurdî da ...\\nMe di 114 bernam..."
}
```

#### unshuffled_original_kv

- **Size of downloaded dataset files:** 0.40 MB
- **Size of the generated dataset:** 2.38 MB
- **Total amount of disk used:** 2.78 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Коми кытшыслӧн ыджытжык тор вӧр увтын куйлӧ, сійӧн и фаунасӧ татӧн аркмӧтӧны вӧрын олісь подаэз. Ассямаӧн лоӧ сія, мый кытшас с..."
}
```

#### unshuffled_original_kw

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.04 MB
- **Total amount of disk used:** 0.05 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼Pray without ceasing🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼🙏..."
}
```

#### unshuffled_original_ky

- **Size of downloaded dataset files:** 152.64 MB
- **Size of the generated dataset:** 630.79 MB
- **Total amount of disk used:** 783.43 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Turmush: Бишкек шаардык кеңешинин кезексиз отурумунда мэрге ишенбөөчүлүк көрсөтүү маселеси каралат, - депутат Т.Сагынов\\nБишкек..."
}
```

#### unshuffled_original_la

- **Size of downloaded dataset files:** 5.46 MB
- **Size of the generated dataset:** 27.80 MB
- **Total amount of disk used:** 33.26 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Hæ sunt generationes Noë: Noë vir justus atque perfectus fuit in generationibus suis; cum Deo ambulavit.\\nEcce ego adducam aqua..."
}
```

#### unshuffled_original_lb

- **Size of downloaded dataset files:** 10.73 MB
- **Size of the generated dataset:** 30.60 MB
- **Total amount of disk used:** 41.32 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Während dem Gaardefestival \\\"Ambiance Jardins\\\" vum 15. bis de 17. Mee huet den SNJ nees zesumme mam Groupe Animateur en Inform..."
}
```

#### unshuffled_original_lez

- **Size of downloaded dataset files:** 0.83 MB
- **Size of the generated dataset:** 3.38 MB
- **Total amount of disk used:** 4.20 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Ахцегь хуьр, виридалай ч1ехи лезги хуьрерикая я. Ам Урусатдин виридалай къиблепатавай хуьрерикай я. Ин хуьр...\"..."
}
```

#### unshuffled_original_li

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.03 MB
- **Total amount of disk used:** 0.04 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"'t Good Goedenraad aan de Ezerbaek besjteit oet 'n kesjtièl mèt gesjlote haof en 'n park van 26 hectare. Hie in sjtoon väól beu..."
}
```

#### unshuffled_original_lmo

- **Size of downloaded dataset files:** 0.10 MB
- **Size of the generated dataset:** 0.47 MB
- **Total amount of disk used:** 0.58 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Serét (en tortonés: Sregh; en piemontés: Srèj) l'è 'n cümü italià, de la regiù del Piemónt, en Pruvìncia de Alessandria. El g'h..."
}
```

#### unshuffled_original_lo

- **Size of downloaded dataset files:** 33.92 MB
- **Size of the generated dataset:** 182.36 MB
- **Total amount of disk used:** 216.28 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"ຜູ້ພິພາກສາ ປະຈຳເຂດ ສຫລ ທ່ານນຶ່ງ ຕັດສິນວ່າ ໂຄງການເກັບກຳຂໍ້ມູນ ທາງໂທລະສັບ ຂອງອົງການ ຄວາມໝັ້ນຄົງແຫ່ງຊາດ ແມ່ນຖືກຕ້ອງ ຕາມກົດໝາຍ.\\nກະ..."
}
```

#### unshuffled_original_lrc

- **Size of downloaded dataset files:** 0.02 MB
- **Size of the generated dataset:** 0.07 MB
- **Total amount of disk used:** 0.09 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"آرلینگتون یئ گئل د شأریا ڤولاتچە ڤیرجینیا و یئ گئل د شأریا ڤولات ڤولاتچە یا یأکاگئرئتە ئمریکاە. ئی شأر دویومی کألوٙن شأر د راسا..."
}
```

#### unshuffled_original_lt

- **Size of downloaded dataset files:** 3.44 GB
- **Size of the generated dataset:** 9.45 GB
- **Total amount of disk used:** 12.89 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Čir vir vir pavasaris! Čia čia čia… dalinamės labai simpatiška video pamokėle, kurią pristato ab888art galerija.\\nBe galo papra..."
}
```

#### unshuffled_original_lv

- **Size of downloaded dataset files:** 1.49 GB
- **Size of the generated dataset:** 4.27 GB
- **Total amount of disk used:** 5.75 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Dekoratīvi sliekšņi MITSUBISHI OUTLANDER 2007, izgatavoti no ovālas formas, pulētas nerūsējošā tērauda caurules...\\ndažādas tūn..."
}
```

#### unshuffled_original_mai

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.33 MB
- **Total amount of disk used:** 0.34 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"१ · २ · ३ · ४ · ५ · ६ · ७ · ८ · ९ · १० · ११ · १२ · १३ · १४ · १५ · १६ · १७ · १८ · १९ · २० · २१ · २२ · २३ · २४ · २५ · २६ · २७ · २..."
}
```

#### unshuffled_original_mg

- **Size of downloaded dataset files:** 6.22 MB
- **Size of the generated dataset:** 21.79 MB
- **Total amount of disk used:** 28.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Nanamboatra taratasy apetaka sy soso-kevitra ho an'ny olona te-hanatevin-daharana ity fihetsiketsehana ity i Anocrena.\\nNosorat..."
}
```

#### unshuffled_original_mhr

- **Size of downloaded dataset files:** 1.84 MB
- **Size of the generated dataset:** 7.55 MB
- **Total amount of disk used:** 9.38 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Акрет жап годым Уганда кундемым Пигмей племена- влак айлен шогеныт. мемнан эран 1 курым гыч Банту племена влакат тиде кундемышк..."
}
```

#### unshuffled_original_min

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.63 MB
- **Total amount of disk used:** 0.64 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ‏‏‎ ..."
}
```

#### unshuffled_original_mk

- **Size of downloaded dataset files:** 508.24 MB
- **Size of the generated dataset:** 2.20 GB
- **Total amount of disk used:** 2.71 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"„Филм плус“ е насловен првиот филмски месечник во Македонија, чиј прв број ќе биде промовиран вечер во „Менада“. Новото македон..."
}
```

#### unshuffled_original_ml

- **Size of downloaded dataset files:** 938.69 MB
- **Size of the generated dataset:** 5.24 GB
- **Total amount of disk used:** 6.18 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"സ്ത്രീ പ്രവേശനം സര്‍ക്കാര്‍ പൂര്‍ണമായും അംഗീകരിക്കുന്നുവെന്നും ശബരിമലയുടെ സുരക്ഷയില്‍ ഇടപെടുമെന്നും സര്‍ക്കാര്‍ ഹൈക്കോടതിയില്‍\\..."
}
```

#### unshuffled_original_mn

- **Size of downloaded dataset files:** 472.36 MB
- **Size of the generated dataset:** 2.33 GB
- **Total amount of disk used:** 2.81 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Монгол улс, Улаанбаатар хот - 14191 Энхтайваны өргөн чөлөө - 10, Багш хөгжлийн ордон, Багшийн мэргэжил дээшлүүлэх институт\\nБаг..."
}
```

#### unshuffled_original_mr

- **Size of downloaded dataset files:** 525.31 MB
- **Size of the generated dataset:** 2.82 GB
- **Total amount of disk used:** 3.34 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Home / motivational marathi story / उद्योजकता (Entrepreneurship) / यांना हे जमलय, तर आपल्याला का नाही जमणार ?\\nयापैकी कोणाचीही ..."
}
```

#### unshuffled_original_mrj

- **Size of downloaded dataset files:** 0.30 MB
- **Size of the generated dataset:** 1.16 MB
- **Total amount of disk used:** 1.47 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Лӹпӹвлӓ (латинлӓ Lepidoptera ; алыкмарла лыве-влак) — капшангывлӓ йыхыш пырышы сӱмӓн нӹл шылдыран капшангывлӓ. Цилӓжӹ 180000 тӹ..."
}
```

#### unshuffled_original_ms

- **Size of downloaded dataset files:** 28.46 MB
- **Size of the generated dataset:** 122.33 MB
- **Total amount of disk used:** 150.79 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Sanad pertama daripada Zuhair bin Harb daripada ‘Affan daripada Hammad daripada Thabit daripada Anas.\\nSanad kedua daripada ‘Ab..."
}
```

#### unshuffled_original_mt

- **Size of downloaded dataset files:** 7.53 MB
- **Size of the generated dataset:** 24.47 MB
- **Total amount of disk used:** 32.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "tibgħat il-kawża lura lill-Qorti Ġenerali għall-annullament jew għat-tnaqqis tal-penalità imposta mill-Kummissjoni bid-deċiżjoni inizjali kif emendata bid-deċiżjoni ta’ rettifika;"
}
```

#### unshuffled_original_mwl

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Deciplina social i outónoma que angloba atebidades de ouserbaçon, de análeze, de çcriçon, cumparaçon, de sistematizaçon i de sp..."
}
```

#### unshuffled_original_my

- **Size of downloaded dataset files:** 369.85 MB
- **Size of the generated dataset:** 2.02 GB
- **Total amount of disk used:** 2.39 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ျမ၀တီ - ရန္ကုန္တိုင္းေဒသႀကီး ေျမာက္ဥကၠလာပႏွင္႕ ဗဟန္းၿမိဳ႔နယ္ မေကြးတိုင္း ေဒသႀကီး ပခုကၠဴၿမိဳ႔နယ္တို႔၌ ျမန္မာ႕တပ္မေတာ္အား ေထာက္ခံ..."
}
```

#### unshuffled_original_myv

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"2018 иень умарьковонь 6-це чистэ сась паро куля! Россиянь культурань Министерствась макссь невтемань конёв (прокатной удостовер..."
}
```

#### unshuffled_original_mzn

- **Size of downloaded dataset files:** 0.18 MB
- **Size of the generated dataset:** 0.72 MB
- **Total amount of disk used:** 0.90 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"قرآن یا قوران اسلام ِآسمونی کتاب هسته. مسلمونون گانّّه قرآن ره خدا، وحی جه برسنی‌یه، «محمد معجزه» هسته و ثقلین حدیث دله ونه خَو..."
}
```

#### unshuffled_original_nah

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "In mācuīlpōhualxihuitl VI (inic chicuacē) in mācuīlpōhualli xiuhitl cāhuitl īhuīcpa 501 xihuitl oc 600 xihuitl."
}
```

#### unshuffled_original_nap

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.02 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ò AUDIT í Ç è î ÿ å å 30 ò ÿ ÿ é, õ ñ ì ÿ, ê ã- ò à ì. å â å í ç â à à é ñ è å é ó ó ë. å å å û è å î é è à. à è à AUDIT 1-7 â ..."
}
```

#### unshuffled_original_nds

- **Size of downloaded dataset files:** 6.74 MB
- **Size of the generated dataset:** 18.23 MB
- **Total amount of disk used:** 24.99 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Dor kann sik vun nu af an de hele plattdüütsche Welt – vun Niebüll bit New York, vun Helgoland bit Honolulu – drapen. Allens, w..."
}
```

#### unshuffled_original_ne

- **Size of downloaded dataset files:** 355.29 MB
- **Size of the generated dataset:** 1.87 GB
- **Total amount of disk used:** 2.22 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"बर्दिबास नगरपालिकाको तेस्रो नगर परिषदबाट पारित आ.व.२०७३।७४ को संशोधित र २०७४।७५ को प्रस्तावित नीति, कार्यक्रम तथा बजेट\\nअार्थिक..."
}
```

#### unshuffled_original_new

- **Size of downloaded dataset files:** 1.03 MB
- **Size of the generated dataset:** 5.77 MB
- **Total amount of disk used:** 6.79 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"थ्व शहरयागु अक्षांश ३४.७००१६४ उत्तर व देशान्तर ८६.३७६४६९ पश्चिम खः (34.700164° N 86.376469° W)। थ्व थासे ७२२६७३२ वर्ग मिटर (२.७..."
}
```

#### unshuffled_original_nl

- **Size of downloaded dataset files:** 29.35 GB
- **Size of the generated dataset:** 83.23 GB
- **Total amount of disk used:** 112.58 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Op vrijdag 31 augustus wordt het nieuwe studiejaar van de masteropleiding architectuur geopend met een dagexcursie naar Venlo.\\..."
}
```

#### unshuffled_original_nn

- **Size of downloaded dataset files:** 32.86 MB
- **Size of the generated dataset:** 90.84 MB
- **Total amount of disk used:** 123.70 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "Planomtale krav til innhald Bakgrunn: Spørsmål frå fleire kommunar om kva ein planomtale/planbeskrivelse bør innehalde Fylkeskommunen og fylkesmannen har i ein del saker reist motsegn på formelt grunnlag"
}
```

#### unshuffled_original_no

- **Size of downloaded dataset files:** 3.11 GB
- **Size of the generated dataset:** 8.65 GB
- **Total amount of disk used:** 11.76 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Ytterligere aktører i primærhelsetjenesten og andre NHS-virksomheter ble infisert, inkludert legekontor.Læreren vår er så attra..."
}
```

#### unshuffled_original_oc

- **Size of downloaded dataset files:** 1.57 MB
- **Size of the generated dataset:** 6.12 MB
- **Total amount of disk used:** 7.71 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": ".рф (rf, còdi punycode: .xn--p1ai)[1] es lo nom de domeni en rus per Russia. Foguèt activat lo 12 de mai de 2010. Lo còdi latin es .ru."
}
```

#### unshuffled_original_or

- **Size of downloaded dataset files:** 49.84 MB
- **Size of the generated dataset:** 260.15 MB
- **Total amount of disk used:** 309.99 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ଭୁବନେଶ୍ୱର, ୨୭/୧– (ଓଡ଼ିଆ ପୁଅ) ସିପିଆଇ ଜାତୀୟ ପରିଷଦର ଆହ୍ୱାନକ୍ରମେ ଗତକାଲି ଜାନୁୟାରୀ ୨୬ ସାଧାରଣତନ୍ତ୍ର ଦିବସକୁ ଦେଶ ବ୍ୟାପୀ ସମ୍ବିଧାନ ସୁରକ୍ଷା ..."
}
```

#### unshuffled_original_os

- **Size of downloaded dataset files:** 3.09 MB
- **Size of the generated dataset:** 12.90 MB
- **Total amount of disk used:** 15.99 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"1. Лæппу æмæ чызг казрæдзийы зæрдæмæ куы фæцæуынц æмæ, куы сфæнд кæнынц сæ цард баиу кæнын, уæд лæппу бар ракуры чызгæй, цæмæй ..."
}
```

#### unshuffled_original_pa

- **Size of downloaded dataset files:** 164.21 MB
- **Size of the generated dataset:** 801.16 MB
- **Total amount of disk used:** 965.37 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ਰਜਿ: ਨੰ: PB/JL-138/2018-20 ਜਿਲਦ 63, ਬਾਨੀ ਸੰਪਾਦਕ (ਸਵ:) ਡਾ: ਸਾਧੂ ਸਿੰਘ ਹਮਦਰਦ ਫ਼ੋਨ : 0181-2455961-62-63, 5032400, ਫੈਕਸ : 2455960, 2..."
}
```

#### unshuffled_original_pam

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Áku pu i Anak ning Aláya at ngeni ipákit kó kékayu ngan nûng makanánu lang susúlat détinang kulit a mágkas. Lauan ya ing tarátu..."
}
```

#### unshuffled_original_pl

- **Size of downloaded dataset files:** 42.88 GB
- **Size of the generated dataset:** 117.12 GB
- **Total amount of disk used:** 160.01 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"System informatyczny - Załącznik nr 1 do zarządzenia Wójta Gminy Podegrodzie Nr 530/2013 z dnia 27 maja 2013 r\\nSystem informat..."
}
```

#### unshuffled_original_pms

- **Size of downloaded dataset files:** 0.75 MB
- **Size of the generated dataset:** 2.15 MB
- **Total amount of disk used:** 2.92 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Louvigné-du-Désert a l'é na comun-a fransèisa ant la region aministrativa dla Brëtagna, ant ël dipartiment d'Ille-et-Vilaine. A..."
}
```

#### unshuffled_original_pnb

- **Size of downloaded dataset files:** 3.22 MB
- **Size of the generated dataset:** 12.04 MB
- **Total amount of disk used:** 15.26 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"ایہ فائل Wikimedia Commons توں اے تے دوجیاں ویونتاں تے وی ورتی جاےکدی اے۔ گل بات اس دے فائل گل بات صفہ تے تھلے دتی گئی۔\"..."
}
```

#### unshuffled_original_ps

- **Size of downloaded dataset files:** 103.66 MB
- **Size of the generated dataset:** 379.51 MB
- **Total amount of disk used:** 483.17 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Many people usually use the time period ‘business to business (B2B) advertising,’ however most of them do not know precisely wh..."
}
```

#### unshuffled_original_pt

- **Size of downloaded dataset files:** 47.26 GB
- **Size of the generated dataset:** 132.64 GB
- **Total amount of disk used:** 179.89 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Você pode estar lendo este texto no sofá, levantar pra pegar uma breja na geladeira, dar uma cagada e sentar novamente, sem int..."
}
```

#### unshuffled_original_qu

- **Size of downloaded dataset files:** 0.02 MB
- **Size of the generated dataset:** 0.08 MB
- **Total amount of disk used:** 0.10 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Warayu wichay (kastilla simipi: Ascensión de Guarayos) nisqaqa Buliwya mama llaqtapi, Santa Krus suyupi, huk llaqtam, Warayu pruwinsyap uma llaqtanmi."
}
```

#### unshuffled_original_rm

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"practicists agrars / practicistas agraras AFP pon far ina furmaziun da basa scursanida per cuntanscher in attestat federal da q..."
}
```

#### unshuffled_original_ro

- **Size of downloaded dataset files:** 9.53 GB
- **Size of the generated dataset:** 26.87 GB
- **Total amount of disk used:** 36.40 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"“În viață, oportunitatea nu este totul. Cine atrage Lumina, cineva bun în umbră. Timpul ne creează.” maestru\\nLyn.Evans: Ce mar..."
}
```

#### unshuffled_original_ru

- **Size of downloaded dataset files:** 319.76 GB
- **Size of the generated dataset:** 1241.63 GB
- **Total amount of disk used:** 1561.38 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Доступ к данному профилю для публичного просмотра закрыт администрацией сайта - профиль находится на модерации.\\nРазработчикам ..."
}
```

#### unshuffled_original_sa

- **Size of downloaded dataset files:** 17.52 MB
- **Size of the generated dataset:** 97.06 MB
- **Total amount of disk used:** 114.58 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"अनिरुद्धनगरे क्रीडिता रामलीला सम्‍प्रति समाप्‍ता अस्ति । तस्‍य कानिचन् चित्राणि पूर्वमेव प्रकाशितानि सन्ति । द्वौ चलचित्रौ अपि ..."
}
```

#### unshuffled_original_sah

- **Size of downloaded dataset files:** 9.08 MB
- **Size of the generated dataset:** 43.82 MB
- **Total amount of disk used:** 52.90 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████..."
}
```

#### unshuffled_original_scn

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
{
    "id": 0,
    "text": "La gilusìa è nu sintimentu dulurusu ca nasci d'un disideriu di pussessu sclusivu ntê cunfrunti dâ pirsuna amata e dû timuri, dû suspettu o dâ cirtizza dâ sò nfidiltati."
}
```

#### unshuffled_original_sd

- **Size of downloaded dataset files:** 90.62 MB
- **Size of the generated dataset:** 364.25 MB
- **Total amount of disk used:** 454.88 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"هر ڪو ڄاڻي ٿو ته جڏهن توهان هڪ وڏي خريد ڪرڻ چاهيون ٿا, توهان پڄي ضروري حڪم ۾ ان جي ڪم ڪرڻ جي هٿ ۾ لاڳاپو ڪيو آهي. جي شيء آهي ته..."
}
```

#### unshuffled_original_sh

- **Size of downloaded dataset files:** 3.46 MB
- **Size of the generated dataset:** 25.84 MB
- **Total amount of disk used:** 29.30 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Opština Gornja Radgona se nalazi u sjeveroistočnoj Sloveniji i graniči s susjednom Austriji duž rijeke Mure. Sa tridesetim nase..."
}
```

#### unshuffled_original_si

- **Size of downloaded dataset files:** 310.93 MB
- **Size of the generated dataset:** 1.47 GB
- **Total amount of disk used:** 1.78 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"ලාංකීය සිතිවිලි සිංහල බ්ලොග් කියවනය කොත්තු සින්ඩිය ලංකා Blogger හත්මාළුව ලංකා බ්ලොග් කියවනය මාතලන්ගේ සින්ඩිය මොබයිල්lk\\nඅවකාශය ..."
}
```

#### unshuffled_original_sk

- **Size of downloaded dataset files:** 3.71 GB
- **Size of the generated dataset:** 9.81 GB
- **Total amount of disk used:** 13.52 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Aktivity | Agentúra podporovaného zamestnávania | vzdelávanie pre klientov, vzdelávanie pre odborníkov, kurzy\\nŠpecializované k..."
}
```

#### unshuffled_original_sl

- **Size of downloaded dataset files:** 956.20 MB
- **Size of the generated dataset:** 2.68 GB
- **Total amount of disk used:** 3.63 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Če Creatures, ki je želel, da pridejo na čas, predvsem je povedlo – razlikuje od ljubosumja začel grizenja kolen (ali zadnjica)..."
}
```

#### unshuffled_original_so

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.06 MB
- **Total amount of disk used:** 0.06 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"тттттттттттттттттттттттттттттттт тттттттттттттттттттттттттттттттт тттттттттттттттттттттттттттттттт ттттттттттттттттуууууууууууу..."
}
```

#### unshuffled_original_sq

- **Size of downloaded dataset files:** 861.84 MB
- **Size of the generated dataset:** 2.44 GB
- **Total amount of disk used:** 3.30 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Çfarë do të më pëlqente tek një femër ose çfarë do të më shndërronte në një shpërthim drite? – Albert Vataj\\nTë gjithëve një zo..."
}
```

#### unshuffled_original_sr

- **Size of downloaded dataset files:** 1.08 GB
- **Size of the generated dataset:** 4.13 GB
- **Total amount of disk used:** 5.21 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Корисни савети за сваки дан. На сајту су разне категорије, као што су љепота, мода, кување и поправка властитим рукама.\\nШколск..."
}
```

#### unshuffled_original_su

- **Size of downloaded dataset files:** 0.06 MB
- **Size of the generated dataset:** 0.23 MB
- **Total amount of disk used:** 0.28 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Kartu krédit nyaéta \"duit plastik\" anu dikaluarkeun ku bank pikeun alat pambayaran di tempat-tempat nu tangtu samisal jiga di hotél, réstoran, tempat rékréasi jeung sajabana.[1]"
}
```

#### unshuffled_original_sv

- **Size of downloaded dataset files:** 17.18 GB
- **Size of the generated dataset:** 47.00 GB
- **Total amount of disk used:** 64.18 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"1783 är ett viktigt årtal i den nya tidens historia. Det året slöts en fred i Paris och därmed blev de 13 brittiska kolonierna ..."
}
```

#### unshuffled_original_sw

- **Size of downloaded dataset files:** 3.71 MB
- **Size of the generated dataset:** 14.07 MB
- **Total amount of disk used:** 17.78 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Miripuko hiyo inakuja mwanzoni mwa Wiki Takatifu kuelekea Pasaka na ikiwa ni wiki chache tu kabla ya Papa Francis kuanza ziara yake katika nchi hiyo yenye idadi kubwa kabisa ya watu katika ulimwengu wa nchi za Kiarabu."
}
```

#### unshuffled_original_ta

- **Size of downloaded dataset files:** 1.74 GB
- **Size of the generated dataset:** 9.93 GB
- **Total amount of disk used:** 11.67 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"பொழுது சாய்ந்து வெகு நேரமாகிவிட்டது. கூலி வேலைக்குப் போயிருந்த 'சித்தாள் ' பெண்கள் எல்லோரும் வீடு திரும்பி விட்டார்கள். இன்னும்..."
}
```

#### unshuffled_original_te

- **Size of downloaded dataset files:** 522.47 MB
- **Size of the generated dataset:** 2.61 GB
- **Total amount of disk used:** 3.13 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"హర్యానాలో టోల్ దగ్గర సిబ్బంది.. స్థానిక ప్రజలు కొట్టుకున్నారు. కర్నాల్ అనే గ్రామానికి సమీపంలో టోల్ గేట్ ఉంది. అయితే సాధారణంగా స..."
}
```

#### unshuffled_original_tg

- **Size of downloaded dataset files:** 90.97 MB
- **Size of the generated dataset:** 397.43 MB
- **Total amount of disk used:** 488.41 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Ҳумайро гуфтааст, мухолифи низом аст, низоме, ки дар Тоҷикистон вуҷуд дорад. Ба ин маънӣ, худро мухолифи давлату ҳукумати Тоҷик..."
}
```

#### unshuffled_original_th

- **Size of downloaded dataset files:** 7.38 GB
- **Size of the generated dataset:** 38.29 GB
- **Total amount of disk used:** 45.67 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ฟันที่แลดูขาวสะอาดไม่มีเศษอาหารติดอยู่ เหงือกสีชมพู ไม่เจ็บ หรือมีเลือดออกเวลาแปรงฟันหรือขัดฟัน ไม่มีปัญหาเรื่องกลิ่นปาก ทำให้ก..."
}
```

#### unshuffled_original_tk

- **Size of downloaded dataset files:** 2.96 MB
- **Size of the generated dataset:** 10.66 MB
- **Total amount of disk used:** 13.62 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"Türkmenistanyň Prezidenti agyr atletika boýunça dünýä çempionatyna taýýarlyk işleriniň barşy bilen tanyşdy\\nHalallykdan kemal t..."
}
```

#### unshuffled_original_tl

- **Size of downloaded dataset files:** 204.89 MB
- **Size of the generated dataset:** 606.30 MB
- **Total amount of disk used:** 811.19 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"“Gusto ko manawagan sa mga Unit Head ng Chanel 2 Salve. Kasi napapansin ko iyon mga alaga ko ang taping halos once a week lang,..."
}
```

#### unshuffled_original_tr

- **Size of downloaded dataset files:** 21.96 GB
- **Size of the generated dataset:** 63.58 GB
- **Total amount of disk used:** 85.54 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Son yıllarda görülen ay tutulmalarına göre daha etkili olacağı söylenen Kanlı veya Kırmızı Ay Tutulmasına saatler kaldı. Bu akş..."
}
```

#### unshuffled_original_tt

- **Size of downloaded dataset files:** 151.06 MB
- **Size of the generated dataset:** 703.42 MB
- **Total amount of disk used:** 854.47 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"\\\"Иремнең вафатына 40 көн узгач, Алмаз да безнең өйгә кереп үлде\\\". Арчада 35 яшьлек ир өстенә кондызлар ега башлаган агач төшк..."
}
```

#### unshuffled_original_tyv

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.01 MB
- **Total amount of disk used:** 0.01 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Экии, хүндүлуг аалчылар болгаш тыва дылдың деткикчилери! Тыва дылдың болгаш чогаалдың ховар бир башкызынга, Менги Ооржакка, ажы..."
}
```

#### unshuffled_original_ug

- **Size of downloaded dataset files:** 27.92 MB
- **Size of the generated dataset:** 127.42 MB
- **Total amount of disk used:** 155.35 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"زاڭ-ءتۇزىم | عىلىم-تەحنيكا | ءتىل-ادەبيەت | تۇرمىس | دەنە تاربيە | ساياحات-ورتا | سۋرەتتى حابار | سىر سۇحبات | ارناۋلى تاقىرىپ ..."
}
```

#### unshuffled_original_uk

- **Size of downloaded dataset files:** 14.42 GB
- **Size of the generated dataset:** 56.44 GB
- **Total amount of disk used:** 70.86 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Про надання роз'яснення (щодо форми письмового зобов'язання громадян про зворотне ввезення/вивезення товарів), Державна митна с..."
}
```

#### unshuffled_original_ur

- **Size of downloaded dataset files:** 712.61 MB
- **Size of the generated dataset:** 2.80 GB
- **Total amount of disk used:** 3.51 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"آئیے اہم اسلامی کتب کو یونیکوڈ میں انٹرنیٹ پر پیش کرنے کے لئے مل جل کر آن لائن ٹائپنگ کریں۔ محدث ٹائپنگ پراجیکٹ کے ذریعے آپ روز..."
}
```

#### unshuffled_original_uz

- **Size of downloaded dataset files:** 5.78 MB
- **Size of the generated dataset:** 21.46 MB
- **Total amount of disk used:** 27.24 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Qurama tog'lari tizmasining Toshkentdan 154 km uzoqlikdagi Toshkent-Ush yo'li yeqasidaxushmanzara tabiat qo'ynida joylashgan maydoni 30 ga.\nBolalarni sog'lomlashtirish oromgohi Bo'stonliq tumani Oqtosh muntaqasining soy-salqin gushasida joylashgan."
}
```

#### unshuffled_original_vec

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.02 MB
- **Total amount of disk used:** 0.03 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Par ogni pónto, ła derivada ła xe ła pendensa de ła reta tangente a ła curva de ła funsion f. Ła reta de cołor róso l'è senpre ..."
}
```

#### unshuffled_original_vi

- **Size of downloaded dataset files:** 21.50 GB
- **Size of the generated dataset:** 72.23 GB
- **Total amount of disk used:** 93.73 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Canh chua cá bông lau không chỉ là món ăn giải nhiệt, thanh mát ngày hè mà còn là món siêu bổ dưỡng, rất tốt cho người gầy ốm. ..."
}
```

#### unshuffled_original_vo

- **Size of downloaded dataset files:** 0.30 MB
- **Size of the generated dataset:** 2.12 MB
- **Total amount of disk used:** 2.42 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Sarniguet binon zif in ziläk: Hautes-Pyrénées, in topäd: Midi-Pyrénées, in Fransän. Sarniguet topon videtü 43°19’ 7’’ N e lunetü 0°5’ 19’’ L."
}
```

#### unshuffled_original_wa

- **Size of downloaded dataset files:** 0.09 MB
- **Size of the generated dataset:** 0.29 MB
- **Total amount of disk used:** 0.38 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "Cisse pådje ci n' est co k' on djermon, dj' ô bén k' el pådje est djusse sibåtcheye, eyet co trop tene; et s' divreut ele ecråxhî ene miete."
}
```

#### unshuffled_original_war

- **Size of downloaded dataset files:** 0.64 MB
- **Size of the generated dataset:** 2.68 MB
- **Total amount of disk used:** 3.32 MB

An example of 'train' looks as follows.
```
{
    "id": 1,
    "text": "An Honce amo in usa ka baryo ngan munisipalidad ha distrito han Rožňava ha rehiyon han Košice ha nasod han Slovakia.\nAn Rumegies amo in usa ka komyun ha departamento han Nord ngan ha rehiyon han Nord-Pas-de-Calais ha nasod han Fransya."
}
```

#### unshuffled_original_wuu

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.12 MB
- **Total amount of disk used:** 0.13 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"伊春元旦天气 伊春腊八天气 伊春春节天气 伊春情人节天气 伊春元宵节天气 伊春愚人节天气 伊春清明节天气 伊春劳动节天气 伊春母亲节天气 伊春端午节天气 伊春七夕节天气 伊春教师节天气 伊春中秋节天气 伊春国庆节天气 伊春重阳节天气 伊春万圣节天气 伊春..."
}
```

#### unshuffled_original_xal

- **Size of downloaded dataset files:** 0.03 MB
- **Size of the generated dataset:** 0.12 MB
- **Total amount of disk used:** 0.15 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Арнгудин Орн гисн Европд бәәдг һазр. 2007 җилин тooһaр эн орн нутгт 3,600,523 әмтн бәәдг билә. Арнгудин Орнин хотл балһсна нерн..."
}
```

#### unshuffled_original_xmf

- **Size of downloaded dataset files:** 1.05 MB
- **Size of the generated dataset:** 6.12 MB
- **Total amount of disk used:** 7.17 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"მოჩამილი ტექსტი წჷმორინელი რე Creative Commons Attribution-ShareAlike ლიცენზიათ; შილებე გეძინელი პირობეფიშ არსებუა. კილიშკილიშა..."
}
```

#### unshuffled_original_yi

- **Size of downloaded dataset files:** 33.33 MB
- **Size of the generated dataset:** 147.60 MB
- **Total amount of disk used:** 180.94 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"ממשותדיק - חבֿרה, איך אַרבעט איצט אױף אַ זשורנאַל. טאָמער איר האָט עפּעס צוצוגעבן זאָלט איר שיקן מיר אַן אָנזאָג. ס'װעט הײסן \\\"..."
}
```

#### unshuffled_original_yo

- **Size of downloaded dataset files:** 0.01 MB
- **Size of the generated dataset:** 0.06 MB
- **Total amount of disk used:** 0.06 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 0,
    "text": "\"Copyright © 2018 BBC. BBC kò mọ̀ nípa àwọn ohun tí ó wà ní àwọn ojú òpó tí ó wà ní ìta. Ọwọ́ tí a fi mú ìbáṣepọ̀ ti ìta.\"..."
}
```

#### unshuffled_original_yue

- **Size of downloaded dataset files:** 0.00 MB
- **Size of the generated dataset:** 0.00 MB
- **Total amount of disk used:** 0.00 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 我 灌 我 灌 我 灌 灌 灌 你還不爆 我累了 投降輸一半可以嗎\"..."
}
```

#### unshuffled_original_zh

- **Size of downloaded dataset files:** 206.00 GB
- **Size of the generated dataset:** 545.61 GB
- **Total amount of disk used:** 751.61 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "id": 1,
    "text": "\"中国铝灰网 中国有色金属矿产网 中国黄莲网 中国水轮发电机网 中国抽油泵网 中国数控雕刻机网 中国不锈钢抛光网 中国磨具加工网 中国压铸铝网 中国耐水腻子网 中国手机摄像头网 中国粗粮网 中国车门锁网 中国钛粉网 中国轮圈网\\n天天中奖彩票图 天天中彩票..."
}
```

</details>

### Data Fields

The data fields are the same among all configs.

- `id`: a `int64` feature.
- `text`: a `string` feature.

### Data Splits


<details>
  <summary>Click to expand the number of samples per configuration</summary>

| Language          | Language code | Name original           | Train original | Words original  | Size original | Name deduplicated           | Train deduplicated | Words deduplicated | Size deduplicated |
| ----------------- | ------------- | ----------------------- | -------------- | --------------- | ------------- | --------------------------- | ------------------ | ------------------ | ----------------- |
| Afrikaans         | af            | unshuffled_original_af  | 201117         | 43,482,801      | 241M          | unshuffled_deduplicated_af  | 130640             | 29,533,437         | 163M              |
| Albanian          | sq            | unshuffled_original_sq  | 672077         | 374,196,110     | 2.3G          | unshuffled_deduplicated_sq  | 461598             | 186,856,699        | 1.2G              |
| Alemannic         | als           | unshuffled_original_als | 7324           | 841,750         | 5.0M          | unshuffled_deduplicated_als | 4518               | 459,001            | 2.8M              |
| Amharic           | am            | unshuffled_original_am  | 83663          | 28,301,601      | 360M          | unshuffled_deduplicated_am  | 43102              | 16,086,628         | 206M              |
| Arabic            | ar            | unshuffled_original_ar  | 16365602       | 8,117,162,828   | 82G           | unshuffled_deduplicated_ar  | 9006977            | 3,171,221,354      | 32G               |
| Aragonese         | an            | unshuffled_original_an  | 2449           | 52,896          | 1.3M          | unshuffled_deduplicated_an  | 2025               | 45,669             | 801K              |
| Armenian          | hy            | unshuffled_original_hy  | 659430         | 273,919,388     | 3.7G          | unshuffled_deduplicated_hy  | 396093             | 110,196,043        | 1.5G              |
| Assamese          | as            | unshuffled_original_as  | 14985          | 6,956,663       | 113M          | unshuffled_deduplicated_as  | 9212               | 4,366,570          | 71M               |
| Asturian          | ast           | unshuffled_original_ast | 6999           | 381,005         | 2.4M          | unshuffled_deduplicated_ast | 5343               | 325,237            | 2.0M              |
| Avaric            | av            | unshuffled_original_av  | 456            | 24,720          | 409K          | unshuffled_deduplicated_av  | 360                | 19,478             | 324K              |
| Azerbaijani       | az            | unshuffled_original_az  | 912330         | 322,641,710     | 2.8G          | unshuffled_deduplicated_az  | 626796             | 167,742,296        | 1.5G              |
| Bashkir           | ba            | unshuffled_original_ba  | 42551          | 9,796,764       | 128M          | unshuffled_deduplicated_ba  | 27050              | 6,922,589          | 90M               |
| Basque            | eu            | unshuffled_original_eu  | 506883         | 120,456,652     | 848M          | unshuffled_deduplicated_eu  | 256513             | 45,359,710         | 342M              |
| Bavarian          | bar           | unshuffled_original_bar | 4              | 399             | 503           | unshuffled_deduplicated_bar | 4                  | 399                | 503               |
| Belarusian        | be            | unshuffled_original_be  | 586031         | 144,579,630     | 1.8G          | unshuffled_deduplicated_be  | 307405             | 83,499,037         | 1.1G              |
| Bengali           | bn            | unshuffled_original_bn  | 1675515        | 623,575,733     | 11G           | unshuffled_deduplicated_bn  | 1114481            | 363,766,143        | 5.8G              |
| Bihari            | bh            | unshuffled_original_bh  | 336            | 8,848           | 110K          | unshuffled_deduplicated_bh  | 82                 | 2,875              | 34K               |
| Bishnupriya       | bpy           | unshuffled_original_bpy | 6046           | 198,286         | 4.1M          | unshuffled_deduplicated_bpy | 1770               | 96,940             | 1.7M              |
| Bosnian           | bs            | unshuffled_original_bs  | 2143           | 106,448         | 447K          | unshuffled_deduplicated_bs  | 702                | 20,485             | 116K              |
| Breton            | br            | unshuffled_original_br  | 37085          | 5,013,241       | 29M           | unshuffled_deduplicated_br  | 14724              | 2,890,384          | 16M               |
| Bulgarian         | bg            | unshuffled_original_bg  | 5869686        | 2,947,648,106   | 32G           | unshuffled_deduplicated_bg  | 3398679            | 1,268,114,977      | 14G               |
| Burmese           | my            | unshuffled_original_my  | 232329         | 56,111,184      | 1.9G          | unshuffled_deduplicated_my  | 136639             | 30,102,173         | 1.1G              |
| Catalan           | ca            | unshuffled_original_ca  | 4390754        | 1,360,212,450   | 8.0G          | unshuffled_deduplicated_ca  | 2458067            | 729,333,440        | 4.3G              |
| Cebuano           | ceb           | unshuffled_original_ceb | 56248          | 6,603,567       | 39M           | unshuffled_deduplicated_ceb | 26145              | 3,675,024          | 24M               |
| Central Bikol     | bcl           | unshuffled_original_bcl | 1              | 312             | 885           | unshuffled_deduplicated_bcl | 1                  | 312                | 885               |
| Central Khmer     | km            | unshuffled_original_km  | 159363         | 20,690,610      | 1.1G          | unshuffled_deduplicated_km  | 108346             | 10,082,245         | 581M              |
| Central Kurdish   | ckb           | unshuffled_original_ckb | 103639         | 48,478,334      | 487M          | unshuffled_deduplicated_ckb | 68210              | 18,726,721         | 226M              |
| Chavacano         | cbk           | unshuffled_original_cbk | 1              | 130             | 520           | unshuffled_deduplicated_cbk | 1                  | 130                | 520               |
| Chechen           | ce            | unshuffled_original_ce  | 4042           | 711,051         | 8.3M          | unshuffled_deduplicated_ce  | 2984               | 568,146            | 6.7M              |
| Chinese           | zh            | unshuffled_original_zh  | 60137667       | 14,986,424,850  | 508G          | unshuffled_deduplicated_zh  | 41708901           | 6,350,215,113      | 249G              |
| Chuvash           | cv            | unshuffled_original_cv  | 20281          | 3,041,614       | 39M           | unshuffled_deduplicated_cv  | 10130              | 2,054,810          | 26M               |
| Cornish           | kw            | unshuffled_original_kw  | 203            | 8,329           | 44K           | unshuffled_deduplicated_kw  | 68                 | 2,704              | 14K               |
| Croatian          | hr            | unshuffled_original_hr  | 582219         | 34,232,765      | 226M          | unshuffled_deduplicated_hr  | 321484             | 16,727,640         | 110M              |
| Czech             | cs            | unshuffled_original_cs  | 21001388       | 7,715,977,441   | 53G           | unshuffled_deduplicated_cs  | 12308039           | 3,540,997,509      | 24G               |
| Danish            | da            | unshuffled_original_da  | 7664010        | 2,637,463,889   | 16G           | unshuffled_deduplicated_da  | 4771098            | 1,620,091,317      | 9.5G              |
| Dhivehi           | dv            | unshuffled_original_dv  | 21018          | 7,559,472       | 126M          | unshuffled_deduplicated_dv  | 17024              | 4,726,660          | 79M               |
| Dimli             | diq           | unshuffled_original_diq | 1              | 19              | 146           | unshuffled_deduplicated_diq | 1                  | 19                 | 146               |
| Dutch             | nl            | unshuffled_original_nl  | 34682142       | 13,020,136,373  | 78G           | unshuffled_deduplicated_nl  | 20812149           | 6,598,786,137      | 39G               |
| Eastern Mari      | mhr           | unshuffled_original_mhr | 3212           | 565,992         | 7.2M          | unshuffled_deduplicated_mhr | 2515               | 469,297            | 6.0M              |
| Egyptian Arabic   | arz           | unshuffled_original_arz | 158113         | 7,305,151       | 66M           | unshuffled_deduplicated_arz | 79928              | 3,659,419          | 33M               |
| Emilian-Romagnol  | eml           | unshuffled_original_eml | 84             | 6,376           | 25K           | unshuffled_deduplicated_eml | 80                 | 6,121              | 24K               |
| English           | en            | unshuffled_original_en  | 455994980      | 418,187,793,408 | 2.3T          | unshuffled_deduplicated_en  | 304230423          | 215,841,256,971    | 1.2T              |
| Erzya             | myv           | unshuffled_original_myv | 6              | 90              | 1.4K          | unshuffled_deduplicated_myv | 5                  | 78                 | 1.2K              |
| Esperanto         | eo            | unshuffled_original_eo  | 121171         | 48,486,161      | 299M          | unshuffled_deduplicated_eo  | 84752              | 37,324,446         | 228M              |
| Estonian          | et            | unshuffled_original_et  | 2093621        | 643,163,730     | 4.8G          | unshuffled_deduplicated_et  | 1172041            | 309,931,463        | 2.3G              |
| Finnish           | fi            | unshuffled_original_fi  | 8557453        | 3,196,666,419   | 27G           | unshuffled_deduplicated_fi  | 5326443            | 1,597,855,468      | 13G               |
| French            | fr            | unshuffled_original_fr  | 96742378       | 46,896,036,417  | 282G          | unshuffled_deduplicated_fr  | 59448891           | 23,206,776,649     | 138G              |
| Galician          | gl            | unshuffled_original_gl  | 544388         | 102,011,291     | 620M          | unshuffled_deduplicated_gl  | 284320             | 63,600,602         | 384M              |
| Georgian          | ka            | unshuffled_original_ka  | 563916         | 171,950,621     | 3.6G          | unshuffled_deduplicated_ka  | 372158             | 91,569,739         | 1.9G              |
| German            | de            | unshuffled_original_de  | 104913504      | 44,878,908,446  | 308G          | unshuffled_deduplicated_de  | 62398034           | 21,529,164,172     | 145G              |
| Goan Konkani      | gom           | unshuffled_original_gom | 640            | 124,277         | 2.2M          | unshuffled_deduplicated_gom | 484                | 102,306            | 1.8M              |
| Guarani           | gn            | unshuffled_original_gn  | 106            | 7,382           | 36K           | unshuffled_deduplicated_gn  | 68                 | 4,680              | 24K               |
| Gujarati          | gu            | unshuffled_original_gu  | 240691         | 72,045,701      | 1.1G          | unshuffled_deduplicated_gu  | 169834             | 50,023,432         | 722M              |
| Haitian           | ht            | unshuffled_original_ht  | 13             | 1,014           | 3.9K          | unshuffled_deduplicated_ht  | 9                  | 832                | 3.3K              |
| Hebrew            | he            | unshuffled_original_he  | 3808397        | 2,067,753,528   | 20G           | unshuffled_deduplicated_he  | 2375030            | 1,032,018,056      | 9.8G              |
| Hindi             | hi            | unshuffled_original_hi  | 3264660        | 1,372,234,782   | 17G           | unshuffled_deduplicated_hi  | 1909387            | 745,774,934        | 8.9G              |
| Hungarian         | hu            | unshuffled_original_hu  | 11197780       | 5,163,936,345   | 40G           | unshuffled_deduplicated_hu  | 6582908            | 2,339,127,555      | 18G               |
| Icelandic         | is            | unshuffled_original_is  | 625673         | 219,900,094     | 1.5G          | unshuffled_deduplicated_is  | 389515             | 129,818,331        | 846M              |
| Ido               | io            | unshuffled_original_io  | 694            | 25,702          | 147K          | unshuffled_deduplicated_io  | 617                | 22,773             | 130K              |
| Iloko             | ilo           | unshuffled_original_ilo | 2638           | 142,942         | 874K          | unshuffled_deduplicated_ilo | 1578               | 105,564            | 636K              |
| Indonesian        | id            | unshuffled_original_id  | 16236463       | 4,574,692,265   | 30G           | unshuffled_deduplicated_id  | 9948521            | 2,394,957,629      | 16G               |
| Interlingua       | ia            | unshuffled_original_ia  | 1040           | 180,231         | 662K          | unshuffled_deduplicated_ia  | 529                | 100,019            | 360K              |
| Interlingue       | ie            | unshuffled_original_ie  | 101            | 5,352           | 24K           | unshuffled_deduplicated_ie  | 11                 | 602                | 1.6K              |
| Irish             | ga            | unshuffled_original_ga  | 83223          | 14,483,593      | 88M           | unshuffled_deduplicated_ga  | 46493              | 10,017,303         | 60M               |
| Italian           | it            | unshuffled_original_it  | 46981781       | 22,248,707,341  | 137G          | unshuffled_deduplicated_it  | 28522082           | 11,250,012,896     | 69G               |
| Japanese          | ja            | unshuffled_original_ja  | 62721527       | 4,962,979,182   | 216G          | unshuffled_deduplicated_ja  | 39496439           | 1,123,067,063      | 106G              |
| Javanese          | jv            | unshuffled_original_jv  | 1445           | 104,896         | 659K          | unshuffled_deduplicated_jv  | 1163               | 86,654             | 583K              |
| Kalmyk            | xal           | unshuffled_original_xal | 39             | 10,277          | 113K          | unshuffled_deduplicated_xal | 36                 | 10,155             | 112K              |
| Kannada           | kn            | unshuffled_original_kn  | 350363         | 81,186,863      | 1.7G          | unshuffled_deduplicated_kn  | 251064             | 49,343,462         | 1.1G              |
| Karachay-Balkar   | krc           | unshuffled_original_krc | 1581           | 185,436         | 2.6M          | unshuffled_deduplicated_krc | 1377               | 166,496            | 2.3M              |
| Kazakh            | kk            | unshuffled_original_kk  | 524591         | 191,126,469     | 2.7G          | unshuffled_deduplicated_kk  | 338073             | 108,388,743        | 1.5G              |
| Kirghiz           | ky            | unshuffled_original_ky  | 146993         | 44,194,823      | 600M          | unshuffled_deduplicated_ky  | 86561              | 28,982,620         | 388M              |
| Komi              | kv            | unshuffled_original_kv  | 1549           | 201,404         | 2.3M          | unshuffled_deduplicated_kv  | 924                | 95,243             | 1.2M              |
| Korean            | ko            | unshuffled_original_ko  | 7345075        | 2,368,765,142   | 24G           | unshuffled_deduplicated_ko  | 3675420            | 1,120,375,149      | 12G               |
| Kurdish           | ku            | unshuffled_original_ku  | 46535          | 15,561,003      | 94M           | unshuffled_deduplicated_ku  | 29054              | 9,946,440          | 60M               |
| Lao               | lo            | unshuffled_original_lo  | 52910          | 4,133,311       | 174M          | unshuffled_deduplicated_lo  | 32652              | 2,583,342          | 114M              |
| Latin             | la            | unshuffled_original_la  | 94588          | 4,122,201       | 26M           | unshuffled_deduplicated_la  | 18808              | 1,328,038          | 8.3M              |
| Latvian           | lv            | unshuffled_original_lv  | 1593820        | 520,761,977     | 4.0G          | unshuffled_deduplicated_lv  | 843195             | 236,428,905        | 1.8G              |
| Lezghian          | lez           | unshuffled_original_lez | 1485           | 247,646         | 3.3M          | unshuffled_deduplicated_lez | 1381               | 224,871            | 3.0M              |
| Limburgan         | li            | unshuffled_original_li  | 137            | 4,730           | 29K           | unshuffled_deduplicated_li  | 118                | 4,283              | 27K               |
| Lithuanian        | lt            | unshuffled_original_lt  | 2977757        | 1,159,661,742   | 8.8G          | unshuffled_deduplicated_lt  | 1737411            | 516,183,525        | 3.9G              |
| Lojban            | jbo           | unshuffled_original_jbo | 832            | 154,330         | 736K          | unshuffled_deduplicated_jbo | 617                | 141,973            | 678K              |
| Lombard           | lmo           | unshuffled_original_lmo | 1401           | 75,229          | 443K          | unshuffled_deduplicated_lmo | 1374               | 73,665             | 433K              |
| Low German        | nds           | unshuffled_original_nds | 18174          | 2,906,347       | 18M           | unshuffled_deduplicated_nds | 8714               | 2,146,417          | 13M               |
| Lower Sorbian     | dsb           | unshuffled_original_dsb | 65             | 1,787           | 13K           | unshuffled_deduplicated_dsb | 37                 | 966                | 7.1K              |
| Luxembourgish     | lb            | unshuffled_original_lb  | 34807          | 4,403,577       | 29M           | unshuffled_deduplicated_lb  | 21735              | 3,087,650          | 21M               |
| Macedonian        | mk            | unshuffled_original_mk  | 437871         | 189,289,873     | 2.1G          | unshuffled_deduplicated_mk  | 299457             | 102,849,595        | 1.2G              |
| Maithili          | mai           | unshuffled_original_mai | 123            | 69,161          | 317K          | unshuffled_deduplicated_mai | 25                 | 874                | 11K               |
| Malagasy          | mg            | unshuffled_original_mg  | 17957          | 3,068,360       | 21M           | unshuffled_deduplicated_mg  | 13343              | 1,872,044          | 13M               |
| Malay             | ms            | unshuffled_original_ms  | 534016         | 16,696,882      | 111M          | unshuffled_deduplicated_ms  | 183443             | 6,045,753          | 42M               |
| Malayalam         | ml            | unshuffled_original_ml  | 603937         | 189,534,472     | 4.9G          | unshuffled_deduplicated_ml  | 453904             | 95,892,551         | 2.5G              |
| Maltese           | mt            | unshuffled_original_mt  | 26598          | 2,995,654       | 24M           | unshuffled_deduplicated_mt  | 16383              | 2,163,358          | 17M               |
| Marathi           | mr            | unshuffled_original_mr  | 326804         | 162,609,404     | 2.7G          | unshuffled_deduplicated_mr  | 212556             | 82,130,803         | 1.4G              |
| Mazanderani       | mzn           | unshuffled_original_mzn | 1055           | 73,870          | 691K          | unshuffled_deduplicated_mzn | 917                | 64,481             | 602K              |
| Minangkabau       | min           | unshuffled_original_min | 220            | 5,682           | 608K          | unshuffled_deduplicated_min | 166                | 4,825              | 310K              |
| Mingrelian        | xmf           | unshuffled_original_xmf | 3783           | 299,098         | 5.8M          | unshuffled_deduplicated_xmf | 2418               | 228,629            | 4.4M              |
| Mirandese         | mwl           | unshuffled_original_mwl | 8              | 171             | 1.2K          | unshuffled_deduplicated_mwl | 7                  | 152                | 1.1K              |
| Modern Greek      | el            | unshuffled_original_el  | 10425596       | 5,479,180,137   | 62G           | unshuffled_deduplicated_el  | 6521169            | 2,412,419,435      | 27G               |
| Mongolian         | mn            | unshuffled_original_mn  | 395605         | 181,307,167     | 2.2G          | unshuffled_deduplicated_mn  | 197878             | 68,362,013         | 838M              |
| Nahuatl languages | nah           | unshuffled_original_nah | 61             | 1,234           | 12K           | unshuffled_deduplicated_nah | 58                 | 1,193              | 11K               |
| Neapolitan        | nap           | unshuffled_original_nap | 73             | 5,282           | 17K           | unshuffled_deduplicated_nap | 55                 | 4,147              | 13K               |
| Nepali            | ne            | unshuffled_original_ne  | 299938         | 107,448,208     | 1.8G          | unshuffled_deduplicated_ne  | 219334             | 71,628,317         | 1.2G              |
| Newari            | new           | unshuffled_original_new | 4696           | 564,697         | 5.5M          | unshuffled_deduplicated_new | 2126               | 288,995            | 4.1M              |
| Northern Frisian  | frr           | unshuffled_original_frr | 7              | 1,516           | 4.4K          | unshuffled_deduplicated_frr | 7                  | 1,516              | 4.4K              |
| Northern Luri     | lrc           | unshuffled_original_lrc | 88             | 8,022           | 76K           | unshuffled_deduplicated_lrc | 72                 | 6,740              | 63K               |
| Norwegian         | no            | unshuffled_original_no  | 5546211        | 1,344,326,388   | 8.0G          | unshuffled_deduplicated_no  | 3229940            | 804,894,377        | 4.7G              |
| Norwegian Nynorsk | nn            | unshuffled_original_nn  | 185884         | 14,764,980      | 85M           | unshuffled_deduplicated_nn  | 109118             | 9,435,139          | 54M               |
| Occitan           | oc            | unshuffled_original_oc  | 10709          | 750,301         | 5.8M          | unshuffled_deduplicated_oc  | 6485               | 512,678            | 3.7M              |
| Oriya             | or            | unshuffled_original_or  | 59463          | 14,938,567      | 248M          | unshuffled_deduplicated_or  | 44230              | 11,321,740         | 188M              |
| Ossetian          | os            | unshuffled_original_os  | 5213           | 1,031,268       | 13M           | unshuffled_deduplicated_os  | 2559               | 878,765            | 11M               |
| Pampanga          | pam           | unshuffled_original_pam | 3              | 130             | 760           | unshuffled_deduplicated_pam | 1                  | 52                 | 304               |
| Panjabi           | pa            | unshuffled_original_pa  | 127467         | 61,847,806      | 763M          | unshuffled_deduplicated_pa  | 87235              | 37,555,835         | 460M              |
| Persian           | fa            | unshuffled_original_fa  | 13704702       | 9,096,554,121   | 79G           | unshuffled_deduplicated_fa  | 8203495            | 4,363,505,319      | 38G               |
| Piemontese        | pms           | unshuffled_original_pms | 3225           | 362,013         | 2.1M          | unshuffled_deduplicated_pms | 2859               | 337,246            | 1.9M              |
| Polish            | pl            | unshuffled_original_pl  | 35440972       | 15,277,255,137  | 109G          | unshuffled_deduplicated_pl  | 20682611           | 6,708,709,674      | 47G               |
| Portuguese        | pt            | unshuffled_original_pt  | 42114520       | 20,641,903,898  | 124G          | unshuffled_deduplicated_pt  | 26920397           | 10,751,156,918     | 64G               |
| Pushto            | ps            | unshuffled_original_ps  | 98216          | 46,559,441      | 361M          | unshuffled_deduplicated_ps  | 67921              | 31,347,348         | 242M              |
| Quechua           | qu            | unshuffled_original_qu  | 452            | 10,186          | 78K           | unshuffled_deduplicated_qu  | 411                | 8,691              | 67K               |
| Romanian          | ro            | unshuffled_original_ro  | 9387265        | 3,984,317,058   | 25G           | unshuffled_deduplicated_ro  | 5044757            | 1,741,794,069      | 11G               |
| Romansh           | rm            | unshuffled_original_rm  | 41             | 1,093           | 7.4K          | unshuffled_deduplicated_rm  | 34                 | 960                | 6.5K              |
| Russia Buriat     | bxr           | unshuffled_original_bxr | 42             | 963             | 13K           | unshuffled_deduplicated_bxr | 36                 | 809                | 11K               |
| Russian           | ru            | unshuffled_original_ru  | 161836003      | 92,522,407,837  | 1.2T          | unshuffled_deduplicated_ru  | 115954598          | 46,692,691,520     | 568G              |
| Sanskrit          | sa            | unshuffled_original_sa  | 14291          | 4,331,569       | 93M           | unshuffled_deduplicated_sa  | 7121               | 1,713,930          | 37M               |
| Scottish Gaelic   | gd            | unshuffled_original_gd  | 5799           | 310,689         | 1.9M          | unshuffled_deduplicated_gd  | 3883               | 207,110            | 1.3M              |
| Serbian           | sr            | unshuffled_original_sr  | 1013619        | 364,395,411     | 3.9G          | unshuffled_deduplicated_sr  | 645747             | 207,561,168        | 2.2G              |
| Serbo-Croatian    | sh            | unshuffled_original_sh  | 36700          | 5,292,184       | 25M           | unshuffled_deduplicated_sh  | 17610              | 1,040,573          | 5.8M              |
| Sicilian          | scn           | unshuffled_original_scn | 21             | 554             | 3.3K          | unshuffled_deduplicated_scn | 17                 | 468                | 2.8K              |
| Sindhi            | sd            | unshuffled_original_sd  | 44280          | 43,530,158      | 347M          | unshuffled_deduplicated_sd  | 33925              | 33,028,015         | 263M              |
| Sinhala           | si            | unshuffled_original_si  | 203082         | 93,053,465      | 1.4G          | unshuffled_deduplicated_si  | 120684             | 50,864,857         | 802M              |
| Slovak            | sk            | unshuffled_original_sk  | 5492194        | 1,322,247,763   | 9.1G          | unshuffled_deduplicated_sk  | 2820821            | 656,346,179        | 4.5G              |
| Slovenian         | sl            | unshuffled_original_sl  | 1746604        | 387,399,700     | 2.5G          | unshuffled_deduplicated_sl  | 886223             | 193,926,684        | 1.3G              |
| Somali            | so            | unshuffled_original_so  | 156            | 1,202           | 61K           | unshuffled_deduplicated_so  | 42                 | 472                | 16K               |
| South Azerbaijani | azb           | unshuffled_original_azb | 15446          | 2,175,054       | 27M           | unshuffled_deduplicated_azb | 9985               | 1,528,709          | 19M               |
| Spanish           | es            | unshuffled_original_es  | 88199221       | 47,545,122,279  | 278G          | unshuffled_deduplicated_es  | 56326016           | 25,928,290,729     | 149G              |
| Sundanese         | su            | unshuffled_original_su  | 805            | 30,321          | 211K          | unshuffled_deduplicated_su  | 511                | 20,278             | 141K              |
| Swahili           | sw            | unshuffled_original_sw  | 41986          | 2,211,927       | 13M           | unshuffled_deduplicated_sw  | 24803              | 1,376,963          | 8.1M              |
| Swedish           | sv            | unshuffled_original_sv  | 17395625       | 7,155,994,312   | 44G           | unshuffled_deduplicated_sv  | 11014487           | 4,106,120,608      | 25G               |
| Tagalog           | tl            | unshuffled_original_tl  | 458206         | 98,949,299      | 573M          | unshuffled_deduplicated_tl  | 294132             | 70,121,601         | 407M              |
| Tajik             | tg            | unshuffled_original_tg  | 89002          | 31,758,142      | 379M          | unshuffled_deduplicated_tg  | 56259              | 21,029,893         | 249M              |
| Tamil             | ta            | unshuffled_original_ta  | 1263280        | 420,537,132     | 9.3G          | unshuffled_deduplicated_ta  | 833101             | 226,013,330        | 5.1G              |
| Tatar             | tt            | unshuffled_original_tt  | 135923         | 51,034,893      | 670M          | unshuffled_deduplicated_tt  | 82738              | 23,825,695         | 305M              |
| Telugu            | te            | unshuffled_original_te  | 475703         | 123,711,517     | 2.5G          | unshuffled_deduplicated_te  | 312644             | 79,094,167         | 1.6G              |
| Thai              | th            | unshuffled_original_th  | 6064129        | 951,743,087     | 36G           | unshuffled_deduplicated_th  | 3749826            | 368,965,202        | 16G               |
| Tibetan           | bo            | unshuffled_original_bo  | 26795          | 1,483,589       | 187M          | unshuffled_deduplicated_bo  | 15762              | 936,556            | 138M              |
| Turkish           | tr            | unshuffled_original_tr  | 18535253       | 7,577,388,700   | 60G           | unshuffled_deduplicated_tr  | 11596446           | 3,365,734,289      | 27G               |
| Turkmen           | tk            | unshuffled_original_tk  | 6456           | 1,113,869       | 11M           | unshuffled_deduplicated_tk  | 4694               | 752,326            | 6.8M              |
| Tuvinian          | tyv           | unshuffled_original_tyv | 34             | 759             | 12K           | unshuffled_deduplicated_tyv | 24                 | 540                | 7.9K              |
| Uighur            | ug            | unshuffled_original_ug  | 22255          | 8,657,141       | 122M          | unshuffled_deduplicated_ug  | 15503              | 5,852,225          | 83M               |
| Ukrainian         | uk            | unshuffled_original_uk  | 12973467       | 4,204,381,276   | 53G           | unshuffled_deduplicated_uk  | 7782375            | 2,252,380,351      | 28G               |
| Upper Sorbian     | hsb           | unshuffled_original_hsb | 7959           | 545,351         | 4.2M          | unshuffled_deduplicated_hsb | 3084               | 236,867            | 1.8M              |
| Urdu              | ur            | unshuffled_original_ur  | 638596         | 331,817,982     | 2.7G          | unshuffled_deduplicated_ur  | 428674             | 218,030,228        | 1.7G              |
| Uzbek             | uz            | unshuffled_original_uz  | 27537          | 2,450,256       | 21M           | unshuffled_deduplicated_uz  | 15074              | 1,381,644          | 12M               |
| Venetian          | vec           | unshuffled_original_vec | 73             | 3,492           | 18K           | unshuffled_deduplicated_vec | 64                 | 3,199              | 17K               |
| Vietnamese        | vi            | unshuffled_original_vi  | 14898250       | 12,036,845,359  | 68G           | unshuffled_deduplicated_vi  | 9897709            | 5,577,159,843      | 32G               |
| Volapük           | vo            | unshuffled_original_vo  | 3366           | 321,121         | 2.0M          | unshuffled_deduplicated_vo  | 3317               | 318,568            | 2.0M              |
| Walloon           | wa            | unshuffled_original_wa  | 1001           | 50,720          | 273K          | unshuffled_deduplicated_wa  | 677                | 37,543             | 203K              |
| Waray             | war           | unshuffled_original_war | 9760           | 397,315         | 2.5M          | unshuffled_deduplicated_war | 9161               | 336,311            | 2.2M              |
| Welsh             | cy            | unshuffled_original_cy  | 157698         | 37,422,441      | 213M          | unshuffled_deduplicated_cy  | 98225              | 23,574,673         | 133M              |
| Western Frisian   | fy            | unshuffled_original_fy  | 33053          | 5,691,077       | 35M           | unshuffled_deduplicated_fy  | 20661              | 4,223,816          | 26M               |
| Western Mari      | mrj           | unshuffled_original_mrj | 757            | 93,338          | 1.2M          | unshuffled_deduplicated_mrj | 669                | 87,780             | 1.1M              |
| Western Panjabi   | pnb           | unshuffled_original_pnb | 4599           | 1,426,986       | 12M           | unshuffled_deduplicated_pnb | 3463               | 1,111,112          | 9.0M              |
| Wu Chinese        | wuu           | unshuffled_original_wuu | 214            | 11,189          | 109K          | unshuffled_deduplicated_wuu | 64                 | 4,333              | 32K               |
| Yakut             | sah           | unshuffled_original_sah | 22301          | 2,547,623       | 42M           | unshuffled_deduplicated_sah | 8555               | 1,789,174          | 26M               |
| Yiddish           | yi            | unshuffled_original_yi  | 59364          | 13,834,320      | 141M          | unshuffled_deduplicated_yi  | 32919              | 8,212,970          | 84M               |
| Yoruba            | yo            | unshuffled_original_yo  | 214            | 8,906           | 55K           | unshuffled_deduplicated_yo  | 49                 | 3,518              | 27K               |
| Yue Chinese       | yue           | unshuffled_original_yue | 11             | 186             | 3.7K          | unshuffled_deduplicated_yue | 7                  | 128                | 2.2K              |



</details>

## Dataset Creation

### Curation Rationale

OSCAR was constructed new pipeline derived from the [fastText's one](https://github.com/facebookresearch/fastText), called [_goclassy_](https://github.com/pjox/goclassy). Goclassy reuses the [fastText linear classifier](https://fasttext.cc) and the pre-trained fastText model for language recognition, but it completely rewrites and parallelises their pipeline in an asynchronous manner.

The order of operations is more or less the same as in the fastText pre-processing pipeline but instead of clustering multiple operations into a single blocking process, a worker is launched for each operation but bounding the number of possible parallel operations at a given time by the number of available threads instead of the number of CPUs. Goclassy is implemented in the [Go programming language](https://golang.org/) so it lets the [Go runtime](https://golang.org/src/runtime/mprof.go) handle the scheduling of the processes. Thus the goclassy's pipeline one does not have to wait for a whole WET file to download, decompress and classify in order to start downloading and processing the next one, a new file will start downloading and processing as soon as the scheduler is able to allocate a new process.

Filtering and cleaning processes at line level are done before feeding each line to the classifier. Lines shorter than 100 UTF-8 characters and lines containing invalid UTF-8 characters are discarted and are not classified. After all files are proccesed the deduplicated versions are constructed and everything is then splitted in shards and compressed.

### Source Data

#### Initial Data Collection and Normalization

[Common Crawl](https://commoncrawl.org/) is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable. Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metdata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected [nofollow](http://microformats.org/wiki/rel-nofollow) and [robots.txt](https://www.robotstxt.org/) policies.

Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics.

To construct OSCAR the WET files of Common Crawl were used. These contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format and is stored on Amazon Web Services. In the case of OSCAR, the **November 2018** snapshot was used. It surpasses 20TB of uncompressed data and contains more than 50 thousand plain text files where each file consists of the plain text from multiple websites along its metadata header.

#### Who are the source language producers?

The data comes from multiple web pages in a large variety of languages.

### Annotations

The dataset does not contain any additional annotations.

#### Annotation process

N/A

#### Who are the annotators?

N/A

### Personal and Sensitive Information

Being constructed from Common Crawl, Personal and sensitive information might be present. This **must** be considered before training deep learning models with OSCAR, specially in the case of text-generation models.

## Considerations for Using the Data

### Social Impact of Dataset

OSCAR is intended to bring more data to a wide variety of lanuages, the aim of the corpus is to make large amounts of data available to lower resource languages in order to facilitate the pre-training of state-of-the-art language modeling architectures.

### Discussion of Biases

OSCAR is not properly filtered yet and this can be reflected on the models trained with it. Care is advised specially concerning biases of the resulting models.

### Other Known Limitations

The [fastText linear classifier](https://fasttext.cc) is limed both in performance and the variety of languages it can recognize, so the quality of some OSCAR sub-corpora might be lower than expected, specially for the lowest-resource langiuages. Some audits have already been done by [third parties](https://arxiv.org/abs/2010.14571).

## Additional Information

### Dataset Curators

The corpus was put together by [Pedro J. Ortiz](https://pjortiz.eu/), [Benoît Sagot](http://pauillac.inria.fr/~sagot/), and [Laurent Romary](https://cv.archives-ouvertes.fr/laurentromary), during work done at [Inria](https://www.inria.fr/en), particularly at the [ALMAnaCH team](https://team.inria.fr/almanach/).

### Licensing Information

    These data are released under this licensing scheme
    We do not own any of the text from which these data has been extracted.
    We license the actual packaging of these data under the Creative Commons CC0 license ("no rights reserved") http://creativecommons.org/publicdomain/zero/1.0/
    To the extent possible under law, Inria has waived all copyright and related or neighboring rights to OSCAR
    This work is published from: France.

    Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
    * Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
    * Clearly identify the copyrighted work claimed to be infringed.
    * Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

    We will comply to legitimate requests by removing the affected sources from the next release of the corpus.

### Citation Information

```
@inproceedings{ortiz-suarez-etal-2020-monolingual,
    title = "A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages",
    author = "Ortiz Su{'a}rez, Pedro Javier  and
      Romary, Laurent  and
      Sagot, Benoit",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.156",
    pages = "1703--1714",
    abstract = "We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.",
}

@inproceedings{OrtizSuarezSagotRomary2019,
  author    = {Pedro Javier {Ortiz Su{'a}rez} and Benoit Sagot and Laurent Romary},
  title     = {Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures},
  series = {Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019},
  editor    = {Piotr Bański and Adrien Barbaresi and Hanno Biber and Evelyn Breiteneder and Simon Clematide and Marc Kupietz and Harald L{"u}ngen and Caroline Iliadi},
  publisher = {Leibniz-Institut f{"u}r Deutsche Sprache},
  address   = {Mannheim},
  doi       = {10.14618/ids-pub-9021},
  url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},
  pages     = {9 -- 16},
  year      = {2019},
  abstract  = {Common Crawl is a considerably large, heterogeneous multilingual corpus comprised of crawled documents from the internet, surpassing 20TB of data and distributed as a set of more than 50 thousand plain text files where each contains many documents written in a wide variety of languages. Even though each document has a metadata block associated to it, this data lacks any information about the language in which each document is written, making it extremely difficult to use Common Crawl for monolingual applications. We propose a general, highly parallel, multithreaded pipeline to clean and classify Common Crawl by language; we specifically design it so that it runs efficiently on medium to low resource infrastructures where I/O speeds are the main constraint. We develop the pipeline so that it can be easily reapplied to any kind of heterogeneous corpus and so that it can be parameterised to a wide range of infrastructures. We also distribute a 6.3TB version of Common Crawl, filtered, classified by language, shuffled at line level in order to avoid copyright issues, and ready to be used for NLP applications.},
  language  = {en}
}

```

### Contributions

Thanks to [@pjox](https://github.com/pjox) and [@lhoestq](https://github.com/lhoestq) for adding this dataset.
task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:found multilinguality:multilingual size_categories:100K<n<1M size_categories:100M<n<1B size_categories:10K<n<100K size_categories:10M<n<100M size_categories:1K<n<10K size_categories:1M<n<10M size_categories:n<1K source_datasets:original language:af language:als language:am language:an language:ar language:arz language:as language:ast language:av language:az language:azb language:ba language:bar language:bcl language:be language:bg language:bh language:bn language:bo language:bpy language:br language:bs language:bxr language:ca language:cbk language:ce language:ceb language:ckb language:cs language:cv language:cy language:da language:de language:diq language:dsb language:dv language:el language:eml language:en language:eo language:es language:et language:eu language:fa language:fi language:fr language:frr language:fy language:ga language:gd language:gl language:gn language:gom language:gu language:he language:hi language:hr language:hsb language:ht language:hu language:hy language:ia language:id language:ie language:ilo language:io language:is language:it language:ja language:jbo language:jv language:ka language:kk language:km language:kn language:ko language:krc language:ku language:kv language:kw language:ky language:la language:lb language:lez language:li language:lmo language:lo language:lrc language:lt language:lv language:mai language:mg language:mhr language:min language:mk language:ml language:mn language:mr language:mrj language:ms language:mt language:mwl language:my language:myv language:mzn language:nah language:nap language:nds language:ne language:new language:nl language:nn language:no language:oc language:or language:os language:pa language:pam language:pl language:pms language:pnb language:ps language:pt language:qu language:rm language:ro language:ru language:sa language:sah language:scn language:sd language:sh language:si language:sk language:sl language:so language:sq language:sr language:su language:sv language:sw language:ta language:te language:tg language:th language:tk language:tl language:tr language:tt language:tyv language:ug language:uk language:ur language:uz language:vec language:vi language:vo language:wa language:war language:wuu language:xal language:xmf language:yi language:yo language:yue language:zh license:cc0-1.0 arxiv:2010.14571 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["af", "als", "am", "an", "ar", "arz", "as", "ast", "av", "az", "azb", "ba", "bar", "bcl", "be", "bg", "bh", "bn", "bo", "bpy", "br", "bs", "bxr", "ca", "cbk", "ce", "ceb", "ckb", "cs", "cv", "cy", "da", "de", "diq", "dsb", "dv", "el", "eml", "en", "eo", "es", "et", "eu", "fa", "fi", "fr", "frr", "fy", "ga", "gd", "gl", "gn", "gom", "gu", "he", "hi", "hr", "hsb", "ht", "hu", "hy", "ia", "id", "ie", "ilo", "io", "is", "it", "ja", "jbo", "jv", "ka", "kk", "km", "kn", "ko", "krc", "ku", "kv", "kw", "ky", "la", "lb", "lez", "li", "lmo", "lo", "lrc", "lt", "lv", "mai", "mg", "mhr", "min", "mk", "ml", "mn", "mr", "mrj", "ms", "mt", "mwl", "my", "myv", "mzn", "nah", "nap", "nds", "ne", "new", "nl", "nn", "no", "oc", "or", "os", "pa", "pam", "pl", "pms", "pnb", "ps", "pt", "qu", "rm", "ro", "ru", "sa", "sah", "scn", "sd", "sh", "si", "sk", "sl", "so", "sq", "sr", "su", "sv", "sw", "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "tyv", "ug", "uk", "ur", "uz", "vec", "vi", "vo", "wa", "war", "wuu", "xal", "xmf", "yi", "yo", "yue", "zh"], "license": ["cc0-1.0"], "multilinguality": ["multilingual"], "size_categories": ["100K<n<1M", "100M<n<1B", "10K<n<100K", "10M<n<100M", "1K<n<10K", "1M<n<10M", "n<1K"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "paperswithcode_id": "oscar", "pretty_name": "OSCAR", "config_names": ["unshuffled_deduplicated_af", "unshuffled_deduplicated_als", "unshuffled_deduplicated_am", "unshuffled_deduplicated_an", "unshuffled_deduplicated_ar", "unshuffled_deduplicated_arz", "unshuffled_deduplicated_as", "unshuffled_deduplicated_ast", "unshuffled_deduplicated_av", "unshuffled_deduplicated_az", "unshuffled_deduplicated_azb", "unshuffled_deduplicated_ba", "unshuffled_deduplicated_bar", "unshuffled_deduplicated_bcl", "unshuffled_deduplicated_be", "unshuffled_deduplicated_bg", "unshuffled_deduplicated_bh", "unshuffled_deduplicated_bn", "unshuffled_deduplicated_bo", "unshuffled_deduplicated_bpy", "unshuffled_deduplicated_br", "unshuffled_deduplicated_bs", "unshuffled_deduplicated_bxr", "unshuffled_deduplicated_ca", "unshuffled_deduplicated_cbk", "unshuffled_deduplicated_ce", "unshuffled_deduplicated_ceb", "unshuffled_deduplicated_ckb", "unshuffled_deduplicated_cs", "unshuffled_deduplicated_cv", "unshuffled_deduplicated_cy", "unshuffled_deduplicated_da", "unshuffled_deduplicated_de", "unshuffled_deduplicated_diq", "unshuffled_deduplicated_dsb", "unshuffled_deduplicated_dv", "unshuffled_deduplicated_el", "unshuffled_deduplicated_eml", "unshuffled_deduplicated_en", "unshuffled_deduplicated_eo", "unshuffled_deduplicated_es", "unshuffled_deduplicated_et", "unshuffled_deduplicated_eu", "unshuffled_deduplicated_fa", "unshuffled_deduplicated_fi", "unshuffled_deduplicated_fr", "unshuffled_deduplicated_frr", "unshuffled_deduplicated_fy", "unshuffled_deduplicated_ga", "unshuffled_deduplicated_gd", "unshuffled_deduplicated_gl", "unshuffled_deduplicated_gn", "unshuffled_deduplicated_gom", "unshuffled_deduplicated_gu", "unshuffled_deduplicated_he", "unshuffled_deduplicated_hi", "unshuffled_deduplicated_hr", "unshuffled_deduplicated_hsb", "unshuffled_deduplicated_ht", "unshuffled_deduplicated_hu", "unshuffled_deduplicated_hy", "unshuffled_deduplicated_ia", "unshuffled_deduplicated_id", "unshuffled_deduplicated_ie", "unshuffled_deduplicated_ilo", "unshuffled_deduplicated_io", "unshuffled_deduplicated_is", "unshuffled_deduplicated_it", "unshuffled_deduplicated_ja", "unshuffled_deduplicated_jbo", "unshuffled_deduplicated_jv", "unshuffled_deduplicated_ka", "unshuffled_deduplicated_kk", "unshuffled_deduplicated_km", "unshuffled_deduplicated_kn", "unshuffled_deduplicated_ko", "unshuffled_deduplicated_krc", "unshuffled_deduplicated_ku", "unshuffled_deduplicated_kv", "unshuffled_deduplicated_kw", "unshuffled_deduplicated_ky", "unshuffled_deduplicated_la", "unshuffled_deduplicated_lb", "unshuffled_deduplicated_lez", "unshuffled_deduplicated_li", "unshuffled_deduplicated_lmo", "unshuffled_deduplicated_lo", "unshuffled_deduplicated_lrc", "unshuffled_deduplicated_lt", "unshuffled_deduplicated_lv", "unshuffled_deduplicated_mai", "unshuffled_deduplicated_mg", "unshuffled_deduplicated_mhr", "unshuffled_deduplicated_min", "unshuffled_deduplicated_mk", "unshuffled_deduplicated_ml", "unshuffled_deduplicated_mn", "unshuffled_deduplicated_mr", "unshuffled_deduplicated_mrj", "unshuffled_deduplicated_ms", "unshuffled_deduplicated_mt", "unshuffled_deduplicated_mwl", "unshuffled_deduplicated_my", "unshuffled_deduplicated_myv", "unshuffled_deduplicated_mzn", "unshuffled_deduplicated_nah", "unshuffled_deduplicated_nap", "unshuffled_deduplicated_nds", "unshuffled_deduplicated_ne", "unshuffled_deduplicated_new", "unshuffled_deduplicated_nl", "unshuffled_deduplicated_nn", "unshuffled_deduplicated_no", "unshuffled_deduplicated_oc", "unshuffled_deduplicated_or", "unshuffled_deduplicated_os", "unshuffled_deduplicated_pa", "unshuffled_deduplicated_pam", "unshuffled_deduplicated_pl", "unshuffled_deduplicated_pms", "unshuffled_deduplicated_pnb", "unshuffled_deduplicated_ps", "unshuffled_deduplicated_pt", "unshuffled_deduplicated_qu", "unshuffled_deduplicated_rm", "unshuffled_deduplicated_ro", "unshuffled_deduplicated_ru", "unshuffled_deduplicated_sa", "unshuffled_deduplicated_sah", "unshuffled_deduplicated_scn", "unshuffled_deduplicated_sd", "unshuffled_deduplicated_sh", "unshuffled_deduplicated_si", "unshuffled_deduplicated_sk", "unshuffled_deduplicated_sl", "unshuffled_deduplicated_so", "unshuffled_deduplicated_sq", "unshuffled_deduplicated_sr", "unshuffled_deduplicated_su", "unshuffled_deduplicated_sv", "unshuffled_deduplicated_sw", "unshuffled_deduplicated_ta", "unshuffled_deduplicated_te", "unshuffled_deduplicated_tg", "unshuffled_deduplicated_th", "unshuffled_deduplicated_tk", "unshuffled_deduplicated_tl", "unshuffled_deduplicated_tr", "unshuffled_deduplicated_tt", "unshuffled_deduplicated_tyv", "unshuffled_deduplicated_ug", "unshuffled_deduplicated_uk", "unshuffled_deduplicated_ur", "unshuffled_deduplicated_uz", "unshuffled_deduplicated_vec", "unshuffled_deduplicated_vi", "unshuffled_deduplicated_vo", "unshuffled_deduplicated_wa", "unshuffled_deduplicated_war", "unshuffled_deduplicated_wuu", "unshuffled_deduplicated_xal", "unshuffled_deduplicated_xmf", "unshuffled_deduplicated_yi", "unshuffled_deduplicated_yo", "unshuffled_deduplicated_yue", "unshuffled_deduplicated_zh", "unshuffled_original_af", "unshuffled_original_als", "unshuffled_original_am", "unshuffled_original_an", "unshuffled_original_ar", "unshuffled_original_arz", "unshuffled_original_as", "unshuffled_original_ast", "unshuffled_original_av", "unshuffled_original_az", "unshuffled_original_azb", "unshuffled_original_ba", "unshuffled_original_bar", "unshuffled_original_bcl", "unshuffled_original_be", "unshuffled_original_bg", "unshuffled_original_bh", "unshuffled_original_bn", "unshuffled_original_bo", "unshuffled_original_bpy", "unshuffled_original_br", "unshuffled_original_bs", "unshuffled_original_bxr", "unshuffled_original_ca", "unshuffled_original_cbk", "unshuffled_original_ce", "unshuffled_original_ceb", "unshuffled_original_ckb", "unshuffled_original_cs", "unshuffled_original_cv", "unshuffled_original_cy", "unshuffled_original_da", "unshuffled_original_de", "unshuffled_original_diq", "unshuffled_original_dsb", "unshuffled_original_dv", "unshuffled_original_el", "unshuffled_original_eml", "unshuffled_original_en", "unshuffled_original_eo", "unshuffled_original_es", "unshuffled_original_et", "unshuffled_original_eu", "unshuffled_original_fa", "unshuffled_original_fi", "unshuffled_original_fr", "unshuffled_original_frr", "unshuffled_original_fy", "unshuffled_original_ga", "unshuffled_original_gd", "unshuffled_original_gl", "unshuffled_original_gn", "unshuffled_original_gom", "unshuffled_original_gu", "unshuffled_original_he", "unshuffled_original_hi", "unshuffled_original_hr", "unshuffled_original_hsb", "unshuffled_original_ht", "unshuffled_original_hu", "unshuffled_original_hy", "unshuffled_original_ia", "unshuffled_original_id", "unshuffled_original_ie", "unshuffled_original_ilo", "unshuffled_original_io", "unshuffled_original_is", "unshuffled_original_it", "unshuffled_original_ja", "unshuffled_original_jbo", "unshuffled_original_jv", "unshuffled_original_ka", "unshuffled_original_kk", "unshuffled_original_km", "unshuffled_original_kn", "unshuffled_original_ko", "unshuffled_original_krc", "unshuffled_original_ku", "unshuffled_original_kv", "unshuffled_original_kw", "unshuffled_original_ky", "unshuffled_original_la", "unshuffled_original_lb", "unshuffled_original_lez", "unshuffled_original_li", "unshuffled_original_lmo", "unshuffled_original_lo", "unshuffled_original_lrc", "unshuffled_original_lt", "unshuffled_original_lv", "unshuffled_original_mai", "unshuffled_original_mg", "unshuffled_original_mhr", "unshuffled_original_min", "unshuffled_original_mk", "unshuffled_original_ml", "unshuffled_original_mn", "unshuffled_original_mr", "unshuffled_original_mrj", "unshuffled_original_ms", "unshuffled_original_mt", "unshuffled_original_mwl", "unshuffled_original_my", "unshuffled_original_myv", "unshuffled_original_mzn", "unshuffled_original_nah", "unshuffled_original_nap", "unshuffled_original_nds", "unshuffled_original_ne", "unshuffled_original_new", "unshuffled_original_nl", "unshuffled_original_nn", "unshuffled_original_no", "unshuffled_original_oc", "unshuffled_original_or", "unshuffled_original_os", "unshuffled_original_pa", "unshuffled_original_pam", "unshuffled_original_pl", "unshuffled_original_pms", "unshuffled_original_pnb", "unshuffled_original_ps", "unshuffled_original_pt", "unshuffled_original_qu", "unshuffled_original_rm", "unshuffled_original_ro", "unshuffled_original_ru", "unshuffled_original_sa", "unshuffled_original_sah", "unshuffled_original_scn", "unshuffled_original_sd", "unshuffled_original_sh", "unshuffled_original_si", "unshuffled_original_sk", "unshuffled_original_sl", "unshuffled_original_so", "unshuffled_original_sq", "unshuffled_original_sr", "unshuffled_original_su", "unshuffled_original_sv", "unshuffled_original_sw", "unshuffled_original_ta", "unshuffled_original_te", "unshuffled_original_tg", "unshuffled_original_th", "unshuffled_original_tk", "unshuffled_original_tl", "unshuffled_original_tr", "unshuffled_original_tt", "unshuffled_original_tyv", "unshuffled_original_ug", "unshuffled_original_uk", "unshuffled_original_ur", "unshuffled_original_uz", "unshuffled_original_vec", "unshuffled_original_vi", "unshuffled_original_vo", "unshuffled_original_wa", "unshuffled_original_war", "unshuffled_original_wuu", "unshuffled_original_xal", "unshuffled_original_xmf", "unshuffled_original_yi", "unshuffled_original_yo", "unshuffled_original_yue", "unshuffled_original_zh"], "dataset_info": [{"config_name": "unshuffled_deduplicated_af", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 171320914, "num_examples": 130640}], "download_size": 65989254, "dataset_size": 171320914}, {"config_name": "unshuffled_deduplicated_als", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2915912, "num_examples": 4518}], "download_size": 1263294, "dataset_size": 2915912}, {"config_name": "unshuffled_deduplicated_arz", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 34893248, "num_examples": 79928}], "download_size": 10027493, "dataset_size": 34893248}, {"config_name": "unshuffled_deduplicated_an", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 842246, "num_examples": 2025}], "download_size": 133373, "dataset_size": 842246}, {"config_name": "unshuffled_deduplicated_ast", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2150022, "num_examples": 5343}], "download_size": 856177, "dataset_size": 2150022}, {"config_name": "unshuffled_deduplicated_ba", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 93623739, "num_examples": 27050}], "download_size": 25983491, "dataset_size": 93623739}, {"config_name": "unshuffled_deduplicated_am", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 215618603, "num_examples": 43102}], "download_size": 61347279, "dataset_size": 215618603}, {"config_name": "unshuffled_deduplicated_as", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 73989818, "num_examples": 9212}], "download_size": 15513004, "dataset_size": 73989818}, {"config_name": "unshuffled_deduplicated_azb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 20001183, "num_examples": 9985}], "download_size": 5191704, "dataset_size": 20001183}, {"config_name": "unshuffled_deduplicated_be", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1077152244, "num_examples": 307405}], "download_size": 306700943, "dataset_size": 1077152244}, {"config_name": "unshuffled_deduplicated_bo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 144506264, "num_examples": 15762}], "download_size": 22365048, "dataset_size": 144506264}, {"config_name": "unshuffled_deduplicated_bxr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11325, "num_examples": 36}], "download_size": 3666, "dataset_size": 11325}, {"config_name": "unshuffled_deduplicated_ceb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24439249, "num_examples": 26145}], "download_size": 7124786, "dataset_size": 24439249}, {"config_name": "unshuffled_deduplicated_az", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1526935070, "num_examples": 626796}], "download_size": 521744076, "dataset_size": 1526935070}, {"config_name": "unshuffled_deduplicated_bcl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 900, "num_examples": 1}], "download_size": 594, "dataset_size": 900}, {"config_name": "unshuffled_deduplicated_cy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 140412555, "num_examples": 98225}], "download_size": 53629697, "dataset_size": 140412555}, {"config_name": "unshuffled_deduplicated_dsb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7589, "num_examples": 37}], "download_size": 3640, "dataset_size": 7589}, {"config_name": "unshuffled_deduplicated_bn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6233041155, "num_examples": 1114481}], "download_size": 1257218381, "dataset_size": 6233041155}, {"config_name": "unshuffled_deduplicated_bs", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 125977, "num_examples": 702}], "download_size": 38669, "dataset_size": 125977}, {"config_name": "unshuffled_deduplicated_ce", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7021674, "num_examples": 2984}], "download_size": 1862792, "dataset_size": 7021674}, {"config_name": "unshuffled_deduplicated_cv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 27359554, "num_examples": 10130}], "download_size": 7461982, "dataset_size": 27359554}, {"config_name": "unshuffled_deduplicated_diq", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 161, "num_examples": 1}], "download_size": 331, "dataset_size": 161}, {"config_name": "unshuffled_deduplicated_eml", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24657, "num_examples": 80}], "download_size": 10055, "dataset_size": 24657}, {"config_name": "unshuffled_deduplicated_et", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2434152666, "num_examples": 1172041}], "download_size": 966785545, "dataset_size": 2434152666}, {"config_name": "unshuffled_deduplicated_bg", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14420684170, "num_examples": 3398679}], "download_size": 3848659853, "dataset_size": 14420684170}, {"config_name": "unshuffled_deduplicated_bpy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1725535, "num_examples": 1770}], "download_size": 191472, "dataset_size": 1725535}, {"config_name": "unshuffled_deduplicated_ca", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4544123629, "num_examples": 2458067}], "download_size": 1734548117, "dataset_size": 4544123629}, {"config_name": "unshuffled_deduplicated_ckb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 237229156, "num_examples": 68210}], "download_size": 60319928, "dataset_size": 237229156}, {"config_name": "unshuffled_deduplicated_ar", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33468271639, "num_examples": 9006977}], "download_size": 9667185012, "dataset_size": 33468271639}, {"config_name": "unshuffled_deduplicated_av", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 334755, "num_examples": 360}], "download_size": 75341, "dataset_size": 334755}, {"config_name": "unshuffled_deduplicated_bar", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 551, "num_examples": 4}], "download_size": 354, "dataset_size": 551}, {"config_name": "unshuffled_deduplicated_bh", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 35216, "num_examples": 82}], "download_size": 6003, "dataset_size": 35216}, {"config_name": "unshuffled_deduplicated_br", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16712284, "num_examples": 14724}], "download_size": 6468062, "dataset_size": 16712284}, {"config_name": "unshuffled_deduplicated_cbk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 535, "num_examples": 1}], "download_size": 247, "dataset_size": 535}, {"config_name": "unshuffled_deduplicated_da", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10204168604, "num_examples": 4771098}], "download_size": 3816376656, "dataset_size": 10204168604}, {"config_name": "unshuffled_deduplicated_dv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 82122241, "num_examples": 17024}], "download_size": 16836170, "dataset_size": 82122241}, {"config_name": "unshuffled_deduplicated_eo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 239597935, "num_examples": 84752}], "download_size": 92858714, "dataset_size": 239597935}, {"config_name": "unshuffled_deduplicated_fa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 39986583410, "num_examples": 8203495}], "download_size": 10459318520, "dataset_size": 39986583410}, {"config_name": "unshuffled_deduplicated_fy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 26562554, "num_examples": 20661}], "download_size": 10270434, "dataset_size": 26562554}, {"config_name": "unshuffled_deduplicated_gn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24545, "num_examples": 68}], "download_size": 9566, "dataset_size": 24545}, {"config_name": "unshuffled_deduplicated_cs", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 25590158564, "num_examples": 12308039}], "download_size": 10494256383, "dataset_size": 25590158564}, {"config_name": "unshuffled_deduplicated_hi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9550345517, "num_examples": 1909387}], "download_size": 2007441283, "dataset_size": 9550345517}, {"config_name": "unshuffled_deduplicated_hu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19027456462, "num_examples": 6582908}], "download_size": 7368098962, "dataset_size": 19027456462}, {"config_name": "unshuffled_deduplicated_ie", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1688, "num_examples": 11}], "download_size": 649, "dataset_size": 1688}, {"config_name": "unshuffled_deduplicated_fr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 147774253219, "num_examples": 59448891}], "download_size": 55462770729, "dataset_size": 147774253219}, {"config_name": "unshuffled_deduplicated_gd", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1339050, "num_examples": 3883}], "download_size": 420601, "dataset_size": 1339050}, {"config_name": "unshuffled_deduplicated_gu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 758319353, "num_examples": 169834}], "download_size": 162974870, "dataset_size": 758319353}, {"config_name": "unshuffled_deduplicated_hsb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1821734, "num_examples": 3084}], "download_size": 728158, "dataset_size": 1821734}, {"config_name": "unshuffled_deduplicated_ia", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 373710, "num_examples": 529}], "download_size": 52722, "dataset_size": 373710}, {"config_name": "unshuffled_deduplicated_io", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 139493, "num_examples": 617}], "download_size": 42813, "dataset_size": 139493}, {"config_name": "unshuffled_deduplicated_jbo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 700428, "num_examples": 617}], "download_size": 203506, "dataset_size": 700428}, {"config_name": "unshuffled_deduplicated_km", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 609886370, "num_examples": 108346}], "download_size": 114480044, "dataset_size": 609886370}, {"config_name": "unshuffled_deduplicated_ku", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 62855449, "num_examples": 29054}], "download_size": 23343869, "dataset_size": 62855449}, {"config_name": "unshuffled_deduplicated_la", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8867995, "num_examples": 18808}], "download_size": 3421499, "dataset_size": 8867995}, {"config_name": "unshuffled_deduplicated_lmo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 458386, "num_examples": 1374}], "download_size": 106048, "dataset_size": 458386}, {"config_name": "unshuffled_deduplicated_lv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1895693807, "num_examples": 843195}], "download_size": 710448932, "dataset_size": 1895693807}, {"config_name": "unshuffled_deduplicated_min", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 318749, "num_examples": 166}], "download_size": 10233, "dataset_size": 318749}, {"config_name": "unshuffled_deduplicated_mr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1487944837, "num_examples": 212556}], "download_size": 299680349, "dataset_size": 1487944837}, {"config_name": "unshuffled_deduplicated_mwl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1121, "num_examples": 7}], "download_size": 797, "dataset_size": 1121}, {"config_name": "unshuffled_deduplicated_nah", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11540, "num_examples": 58}], "download_size": 2868, "dataset_size": 11540}, {"config_name": "unshuffled_deduplicated_new", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4226557, "num_examples": 2126}], "download_size": 830767, "dataset_size": 4226557}, {"config_name": "unshuffled_deduplicated_oc", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3938772, "num_examples": 6485}], "download_size": 1338194, "dataset_size": 3938772}, {"config_name": "unshuffled_deduplicated_pam", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 319, "num_examples": 1}], "download_size": 366, "dataset_size": 319}, {"config_name": "unshuffled_deduplicated_ps", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 254360032, "num_examples": 67921}], "download_size": 71823163, "dataset_size": 254360032}, {"config_name": "unshuffled_deduplicated_it", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 73843292670, "num_examples": 28522082}], "download_size": 27931571784, "dataset_size": 73843292670}, {"config_name": "unshuffled_deduplicated_ka", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1982841952, "num_examples": 372158}], "download_size": 377220437, "dataset_size": 1982841952}, {"config_name": "unshuffled_deduplicated_ro", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11601264185, "num_examples": 5044757}], "download_size": 4478423935, "dataset_size": 11601264185}, {"config_name": "unshuffled_deduplicated_scn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2990, "num_examples": 17}], "download_size": 1620, "dataset_size": 2990}, {"config_name": "unshuffled_deduplicated_ko", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11956006533, "num_examples": 3675420}], "download_size": 4462788278, "dataset_size": 11956006533}, {"config_name": "unshuffled_deduplicated_kw", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14971, "num_examples": 68}], "download_size": 6195, "dataset_size": 14971}, {"config_name": "unshuffled_deduplicated_lez", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3075326, "num_examples": 1381}], "download_size": 763936, "dataset_size": 3075326}, {"config_name": "unshuffled_deduplicated_lrc", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 65291, "num_examples": 72}], "download_size": 16272, "dataset_size": 65291}, {"config_name": "unshuffled_deduplicated_mg", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13516085, "num_examples": 13343}], "download_size": 4303472, "dataset_size": 13516085}, {"config_name": "unshuffled_deduplicated_ml", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2685637627, "num_examples": 453904}], "download_size": 496801596, "dataset_size": 2685637627}, {"config_name": "unshuffled_deduplicated_ms", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 45064684, "num_examples": 183443}], "download_size": 16391407, "dataset_size": 45064684}, {"config_name": "unshuffled_deduplicated_myv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1224, "num_examples": 5}], "download_size": 705, "dataset_size": 1224}, {"config_name": "unshuffled_deduplicated_nds", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13360483, "num_examples": 8714}], "download_size": 5271194, "dataset_size": 13360483}, {"config_name": "unshuffled_deduplicated_nn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 57286159, "num_examples": 109118}], "download_size": 23583774, "dataset_size": 57286159}, {"config_name": "unshuffled_deduplicated_os", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10962689, "num_examples": 2559}], "download_size": 2829131, "dataset_size": 10962689}, {"config_name": "unshuffled_deduplicated_pms", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1996853, "num_examples": 2859}], "download_size": 716837, "dataset_size": 1996853}, {"config_name": "unshuffled_deduplicated_qu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 72587, "num_examples": 411}], "download_size": 17501, "dataset_size": 72587}, {"config_name": "unshuffled_deduplicated_sa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 38236039, "num_examples": 7121}], "download_size": 7268337, "dataset_size": 38236039}, {"config_name": "unshuffled_deduplicated_sk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4768416160, "num_examples": 2820821}], "download_size": 1960409934, "dataset_size": 4768416160}, {"config_name": "unshuffled_deduplicated_sh", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6184582, "num_examples": 17610}], "download_size": 1445894, "dataset_size": 6184582}, {"config_name": "unshuffled_deduplicated_so", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16269, "num_examples": 42}], "download_size": 2109, "dataset_size": 16269}, {"config_name": "unshuffled_deduplicated_sr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2358255234, "num_examples": 645747}], "download_size": 665025000, "dataset_size": 2358255234}, {"config_name": "unshuffled_deduplicated_ta", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5477003981, "num_examples": 833101}], "download_size": 971118176, "dataset_size": 5477003981}, {"config_name": "unshuffled_deduplicated_tk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7092199, "num_examples": 4694}], "download_size": 2219582, "dataset_size": 7092199}, {"config_name": "unshuffled_deduplicated_tyv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8319, "num_examples": 24}], "download_size": 2976, "dataset_size": 8319}, {"config_name": "unshuffled_deduplicated_uz", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11834927, "num_examples": 15074}], "download_size": 4300299, "dataset_size": 11834927}, {"config_name": "unshuffled_deduplicated_wa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 214337, "num_examples": 677}], "download_size": 79130, "dataset_size": 214337}, {"config_name": "unshuffled_deduplicated_xmf", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4617445, "num_examples": 2418}], "download_size": 943151, "dataset_size": 4617445}, {"config_name": "unshuffled_deduplicated_sv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 26239415574, "num_examples": 11014487}], "download_size": 10185393483, "dataset_size": 26239415574}, {"config_name": "unshuffled_deduplicated_tg", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 261233997, "num_examples": 56259}], "download_size": 62908723, "dataset_size": 261233997}, {"config_name": "unshuffled_deduplicated_de", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 155723559907, "num_examples": 62398034}], "download_size": 60797849113, "dataset_size": 155723559907}, {"config_name": "unshuffled_deduplicated_tr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 28375018927, "num_examples": 11596446}], "download_size": 10390754678, "dataset_size": 28375018927}, {"config_name": "unshuffled_deduplicated_el", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 28689398676, "num_examples": 6521169}], "download_size": 7907952068, "dataset_size": 28689398676}, {"config_name": "unshuffled_deduplicated_uk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 29791312367, "num_examples": 7782375}], "download_size": 8037737457, "dataset_size": 29791312367}, {"config_name": "unshuffled_deduplicated_vi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33528331774, "num_examples": 9897709}], "download_size": 10711506712, "dataset_size": 33528331774}, {"config_name": "unshuffled_deduplicated_wuu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33253, "num_examples": 64}], "download_size": 7273, "dataset_size": 33253}, {"config_name": "unshuffled_deduplicated_yo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 27169, "num_examples": 49}], "download_size": 8925, "dataset_size": 27169}, {"config_name": "unshuffled_original_als", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5297910, "num_examples": 7324}], "download_size": 1489734, "dataset_size": 5297910}, {"config_name": "unshuffled_original_arz", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 70132423, "num_examples": 158113}], "download_size": 15891255, "dataset_size": 70132423}, {"config_name": "unshuffled_original_az", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2964781192, "num_examples": 912330}], "download_size": 927763846, "dataset_size": 2964781192}, {"config_name": "unshuffled_original_bcl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 901, "num_examples": 1}], "download_size": 581, "dataset_size": 901}, {"config_name": "unshuffled_original_bn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10771945233, "num_examples": 1675515}], "download_size": 2139944099, "dataset_size": 10771945233}, {"config_name": "unshuffled_original_bs", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 482740, "num_examples": 2143}], "download_size": 56419, "dataset_size": 482740}, {"config_name": "unshuffled_original_ce", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8735740, "num_examples": 4042}], "download_size": 2089184, "dataset_size": 8735740}, {"config_name": "unshuffled_original_cv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 41047029, "num_examples": 20281}], "download_size": 9400068, "dataset_size": 41047029}, {"config_name": "unshuffled_original_diq", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 162, "num_examples": 1}], "download_size": 318, "dataset_size": 162}, {"config_name": "unshuffled_original_eml", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 26099, "num_examples": 84}], "download_size": 10071, "dataset_size": 26099}, {"config_name": "unshuffled_original_et", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5174800705, "num_examples": 2093621}], "download_size": 1881328631, "dataset_size": 5174800705}, {"config_name": "unshuffled_deduplicated_zh", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 267614324325, "num_examples": 41708901}], "download_size": 99982781539, "dataset_size": 267614324325}, {"config_name": "unshuffled_original_an", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1329433, "num_examples": 2449}], "download_size": 148184, "dataset_size": 1329433}, {"config_name": "unshuffled_original_ast", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2539238, "num_examples": 6999}], "download_size": 920730, "dataset_size": 2539238}, {"config_name": "unshuffled_original_ba", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 133704014, "num_examples": 42551}], "download_size": 33215002, "dataset_size": 133704014}, {"config_name": "unshuffled_original_bg", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33753811450, "num_examples": 5869686}], "download_size": 8336964541, "dataset_size": 33753811450}, {"config_name": "unshuffled_original_bpy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4347467, "num_examples": 6046}], "download_size": 336974, "dataset_size": 4347467}, {"config_name": "unshuffled_original_ca", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8623251470, "num_examples": 4390754}], "download_size": 3101954304, "dataset_size": 8623251470}, {"config_name": "unshuffled_original_ckb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 510965919, "num_examples": 103639}], "download_size": 111884006, "dataset_size": 510965919}, {"config_name": "unshuffled_deduplicated_es", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 160418075023, "num_examples": 56326016}], "download_size": 60464970319, "dataset_size": 160418075023}, {"config_name": "unshuffled_original_da", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16756455589, "num_examples": 7664010}], "download_size": 6000579388, "dataset_size": 16756455589}, {"config_name": "unshuffled_original_dv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 131628992, "num_examples": 21018}], "download_size": 24914404, "dataset_size": 131628992}, {"config_name": "unshuffled_original_eo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 314188336, "num_examples": 121168}], "download_size": 117076019, "dataset_size": 314188336}, {"config_name": "unshuffled_deduplicated_fi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13945067515, "num_examples": 5326443}], "download_size": 5380047103, "dataset_size": 13945067515}, {"config_name": "unshuffled_deduplicated_ga", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 63370688, "num_examples": 46493}], "download_size": 22218633, "dataset_size": 63370688}, {"config_name": "unshuffled_deduplicated_gom", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1863089, "num_examples": 484}], "download_size": 377051, "dataset_size": 1863089}, {"config_name": "unshuffled_deduplicated_hr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 118047678, "num_examples": 321484}], "download_size": 46731365, "dataset_size": 118047678}, {"config_name": "unshuffled_deduplicated_hy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1559114836, "num_examples": 396093}], "download_size": 393620208, "dataset_size": 1559114836}, {"config_name": "unshuffled_deduplicated_ilo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 667896, "num_examples": 1578}], "download_size": 230065, "dataset_size": 667896}, {"config_name": "unshuffled_original_fa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 84209448803, "num_examples": 13704702}], "download_size": 20956409096, "dataset_size": 84209448803}, {"config_name": "unshuffled_original_fy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 36238452, "num_examples": 33053}], "download_size": 12409774, "dataset_size": 36238452}, {"config_name": "unshuffled_original_gn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 37427, "num_examples": 106}], "download_size": 9761, "dataset_size": 37427}, {"config_name": "unshuffled_original_hi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17929286362, "num_examples": 3264660}], "download_size": 3656636848, "dataset_size": 17929286362}, {"config_name": "unshuffled_original_hu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 43074893842, "num_examples": 11197780}], "download_size": 15693847091, "dataset_size": 43074893842}, {"config_name": "unshuffled_original_ie", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 25355, "num_examples": 101}], "download_size": 783, "dataset_size": 25355}, {"config_name": "unshuffled_deduplicated_ja", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 113315056833, "num_examples": 39496439}], "download_size": 40801218295, "dataset_size": 113315056833}, {"config_name": "unshuffled_deduplicated_kk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1583064520, "num_examples": 338073}], "download_size": 389111715, "dataset_size": 1583064520}, {"config_name": "unshuffled_deduplicated_krc", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2412731, "num_examples": 1377}], "download_size": 615982, "dataset_size": 2412731}, {"config_name": "unshuffled_deduplicated_ky", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 407576051, "num_examples": 86561}], "download_size": 106219565, "dataset_size": 407576051}, {"config_name": "unshuffled_deduplicated_li", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 28176, "num_examples": 118}], "download_size": 11724, "dataset_size": 28176}, {"config_name": "unshuffled_deduplicated_lt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4185372402, "num_examples": 1737411}], "download_size": 1653025558, "dataset_size": 4185372402}, {"config_name": "unshuffled_deduplicated_mhr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6247177, "num_examples": 2515}], "download_size": 1622076, "dataset_size": 6247177}, {"config_name": "unshuffled_deduplicated_mn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 880883961, "num_examples": 197878}], "download_size": 219516471, "dataset_size": 880883961}, {"config_name": "unshuffled_deduplicated_mt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17539926, "num_examples": 16383}], "download_size": 5898934, "dataset_size": 17539926}, {"config_name": "unshuffled_deduplicated_mzn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 626534, "num_examples": 917}], "download_size": 157541, "dataset_size": 626534}, {"config_name": "unshuffled_deduplicated_ne", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1239170286, "num_examples": 219334}], "download_size": 240627361, "dataset_size": 1239170286}, {"config_name": "unshuffled_deduplicated_no", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5077919278, "num_examples": 3229940}], "download_size": 1960828800, "dataset_size": 5077919278}, {"config_name": "unshuffled_deduplicated_pa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 482461302, "num_examples": 87235}], "download_size": 102390579, "dataset_size": 482461302}, {"config_name": "unshuffled_deduplicated_pnb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9416915, "num_examples": 3463}], "download_size": 2579976, "dataset_size": 9416915}, {"config_name": "unshuffled_deduplicated_rm", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6932, "num_examples": 34}], "download_size": 2679, "dataset_size": 6932}, {"config_name": "unshuffled_deduplicated_sah", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 27293316, "num_examples": 8555}], "download_size": 7020207, "dataset_size": 27293316}, {"config_name": "unshuffled_deduplicated_si", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 841460012, "num_examples": 120684}], "download_size": 175610997, "dataset_size": 841460012}, {"config_name": "unshuffled_deduplicated_sq", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1208425681, "num_examples": 461598}], "download_size": 445358539, "dataset_size": 1208425681}, {"config_name": "unshuffled_deduplicated_sw", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8747758, "num_examples": 24803}], "download_size": 2946034, "dataset_size": 8747758}, {"config_name": "unshuffled_deduplicated_th", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17082022564, "num_examples": 3749826}], "download_size": 3536468931, "dataset_size": 17082022564}, {"config_name": "unshuffled_deduplicated_tt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 320641922, "num_examples": 82738}], "download_size": 85893621, "dataset_size": 320641922}, {"config_name": "unshuffled_deduplicated_ur", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1819253063, "num_examples": 428674}], "download_size": 483593818, "dataset_size": 1819253063}, {"config_name": "unshuffled_deduplicated_vo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2098461, "num_examples": 3317}], "download_size": 301687, "dataset_size": 2098461}, {"config_name": "unshuffled_deduplicated_xal", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 114574, "num_examples": 36}], "download_size": 31863, "dataset_size": 114574}, {"config_name": "unshuffled_deduplicated_yue", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2267, "num_examples": 7}], "download_size": 646, "dataset_size": 2267}, {"config_name": "unshuffled_original_am", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 378060369, "num_examples": 83663}], "download_size": 102789518, "dataset_size": 378060369}, {"config_name": "unshuffled_original_as", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 117733678, "num_examples": 14985}], "download_size": 21437245, "dataset_size": 117733678}, {"config_name": "unshuffled_original_azb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 28469069, "num_examples": 15446}], "download_size": 6641415, "dataset_size": 28469069}, {"config_name": "unshuffled_original_be", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1877972506, "num_examples": 586031}], "download_size": 498295673, "dataset_size": 1877972506}, {"config_name": "unshuffled_original_bo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 195400209, "num_examples": 26795}], "download_size": 28940995, "dataset_size": 195400209}, {"config_name": "unshuffled_original_bxr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13376, "num_examples": 42}], "download_size": 3688, "dataset_size": 13376}, {"config_name": "unshuffled_original_ceb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 40964537, "num_examples": 56248}], "download_size": 11070392, "dataset_size": 40964537}, {"config_name": "unshuffled_original_cy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 224933804, "num_examples": 157698}], "download_size": 81736037, "dataset_size": 224933804}, {"config_name": "unshuffled_original_dsb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13761, "num_examples": 65}], "download_size": 3753, "dataset_size": 13761}, {"config_name": "unshuffled_original_fr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 303190338653, "num_examples": 96742378}], "download_size": 105324330228, "dataset_size": 303190338653}, {"config_name": "unshuffled_original_gd", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2022000, "num_examples": 5799}], "download_size": 525253, "dataset_size": 2022000}, {"config_name": "unshuffled_original_gu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1094814909, "num_examples": 240691}], "download_size": 232021129, "dataset_size": 1094814909}, {"config_name": "unshuffled_original_hsb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4482886, "num_examples": 7959}], "download_size": 1389826, "dataset_size": 4482886}, {"config_name": "unshuffled_original_ia", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 689455, "num_examples": 1040}], "download_size": 83325, "dataset_size": 689455}, {"config_name": "unshuffled_original_io", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 158808, "num_examples": 694}], "download_size": 44548, "dataset_size": 158808}, {"config_name": "unshuffled_original_jbo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 763027, "num_examples": 832}], "download_size": 212962, "dataset_size": 763027}, {"config_name": "unshuffled_original_km", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1102616385, "num_examples": 159363}], "download_size": 193286621, "dataset_size": 1102616385}, {"config_name": "unshuffled_original_ku", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 99062676, "num_examples": 46535}], "download_size": 33376537, "dataset_size": 99062676}, {"config_name": "unshuffled_original_la", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 27801400, "num_examples": 94588}], "download_size": 5458131, "dataset_size": 27801400}, {"config_name": "unshuffled_original_lmo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 470001, "num_examples": 1401}], "download_size": 109759, "dataset_size": 470001}, {"config_name": "unshuffled_original_lv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4266812625, "num_examples": 1593820}], "download_size": 1486675302, "dataset_size": 4266812625}, {"config_name": "unshuffled_original_min", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 624991, "num_examples": 220}], "download_size": 12379, "dataset_size": 624991}, {"config_name": "unshuffled_original_mr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2816455519, "num_examples": 326804}], "download_size": 525303459, "dataset_size": 2816455519}, {"config_name": "unshuffled_original_mwl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1273, "num_examples": 8}], "download_size": 789, "dataset_size": 1273}, {"config_name": "unshuffled_original_nah", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12070, "num_examples": 61}], "download_size": 2857, "dataset_size": 12070}, {"config_name": "unshuffled_original_new", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5766053, "num_examples": 4696}], "download_size": 1031042, "dataset_size": 5766053}, {"config_name": "unshuffled_original_oc", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6127539, "num_examples": 10709}], "download_size": 1574956, "dataset_size": 6127539}, {"config_name": "unshuffled_original_pam", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 800, "num_examples": 3}], "download_size": 364, "dataset_size": 800}, {"config_name": "unshuffled_original_ps", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 379515973, "num_examples": 98216}], "download_size": 103659691, "dataset_size": 379515973}, {"config_name": "unshuffled_original_ro", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 26869251055, "num_examples": 9387265}], "download_size": 9534521905, "dataset_size": 26869251055}, {"config_name": "unshuffled_original_scn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3573, "num_examples": 21}], "download_size": 1614, "dataset_size": 3573}, {"config_name": "unshuffled_original_sk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9808179461, "num_examples": 5492194}], "download_size": 3708313186, "dataset_size": 9808179461}, {"config_name": "unshuffled_original_sr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4131922671, "num_examples": 1013619}], "download_size": 1081129678, "dataset_size": 4131922671}, {"config_name": "unshuffled_original_ta", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9934879441, "num_examples": 1263280}], "download_size": 1737252172, "dataset_size": 9934879441}, {"config_name": "unshuffled_original_tk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10662991, "num_examples": 6456}], "download_size": 2956150, "dataset_size": 10662991}, {"config_name": "unshuffled_original_tyv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12219, "num_examples": 34}], "download_size": 3034, "dataset_size": 12219}, {"config_name": "unshuffled_original_uz", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21464779, "num_examples": 27537}], "download_size": 5775644, "dataset_size": 21464779}, {"config_name": "unshuffled_original_wa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 291400, "num_examples": 1001}], "download_size": 89942, "dataset_size": 291400}, {"config_name": "unshuffled_original_xmf", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6120123, "num_examples": 3783}], "download_size": 1048265, "dataset_size": 6120123}, {"config_name": "unshuffled_original_it", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 147378116499, "num_examples": 46981781}], "download_size": 52157691650, "dataset_size": 147378116499}, {"config_name": "unshuffled_original_ka", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3768832240, "num_examples": 563916}], "download_size": 680732710, "dataset_size": 3768832240}, {"config_name": "unshuffled_original_ko", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 25292102197, "num_examples": 7345075}], "download_size": 8807937093, "dataset_size": 25292102197}, {"config_name": "unshuffled_original_kw", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 47016, "num_examples": 203}], "download_size": 6715, "dataset_size": 47016}, {"config_name": "unshuffled_original_lez", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3378104, "num_examples": 1485}], "download_size": 825648, "dataset_size": 3378104}, {"config_name": "unshuffled_original_lrc", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 78347, "num_examples": 88}], "download_size": 16573, "dataset_size": 78347}, {"config_name": "unshuffled_original_mg", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21789998, "num_examples": 17957}], "download_size": 6213316, "dataset_size": 21789998}, {"config_name": "unshuffled_original_ml", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5244279375, "num_examples": 603937}], "download_size": 938681749, "dataset_size": 5244279375}, {"config_name": "unshuffled_original_ms", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 122326270, "num_examples": 534016}], "download_size": 28458804, "dataset_size": 122326270}, {"config_name": "unshuffled_original_myv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1436, "num_examples": 6}], "download_size": 691, "dataset_size": 1436}, {"config_name": "unshuffled_original_nds", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18238189, "num_examples": 18174}], "download_size": 6744705, "dataset_size": 18238189}, {"config_name": "unshuffled_original_nn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 90838777, "num_examples": 185884}], "download_size": 32863375, "dataset_size": 90838777}, {"config_name": "unshuffled_original_os", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12893477, "num_examples": 5213}], "download_size": 3096133, "dataset_size": 12893477}, {"config_name": "unshuffled_original_pms", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2154710, "num_examples": 3225}], "download_size": 756400, "dataset_size": 2154710}, {"config_name": "unshuffled_original_qu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 85032, "num_examples": 452}], "download_size": 17931, "dataset_size": 85032}, {"config_name": "unshuffled_original_sa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 97055224, "num_examples": 14291}], "download_size": 17517475, "dataset_size": 97055224}, {"config_name": "unshuffled_original_sh", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 25841505, "num_examples": 36700}], "download_size": 3457359, "dataset_size": 25841505}, {"config_name": "unshuffled_original_so", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 63785, "num_examples": 156}], "download_size": 2478, "dataset_size": 63785}, {"config_name": "unshuffled_original_sv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 47000933560, "num_examples": 17395625}], "download_size": 17182697021, "dataset_size": 47000933560}, {"config_name": "unshuffled_original_tg", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 397436494, "num_examples": 89002}], "download_size": 90972727, "dataset_size": 397436494}, {"config_name": "unshuffled_original_tr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 63581153419, "num_examples": 18535253}], "download_size": 21961561999, "dataset_size": 63581153419}, {"config_name": "unshuffled_original_uk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 56439494556, "num_examples": 12973467}], "download_size": 14419203733, "dataset_size": 56439494556}, {"config_name": "unshuffled_original_vi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 72226388484, "num_examples": 14898250}], "download_size": 21503594095, "dataset_size": 72226388484}, {"config_name": "unshuffled_original_wuu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 114041, "num_examples": 214}], "download_size": 8780, "dataset_size": 114041}, {"config_name": "unshuffled_original_yo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 58546, "num_examples": 214}], "download_size": 9550, "dataset_size": 58546}, {"config_name": "unshuffled_original_zh", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 545607539477, "num_examples": 60137667}], "download_size": 206003993405, "dataset_size": 545607539477}, {"config_name": "unshuffled_deduplicated_en", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1297616499791, "num_examples": 304230423}], "download_size": 496496144465, "dataset_size": 1297616499791}, {"config_name": "unshuffled_deduplicated_eu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 360674267, "num_examples": 256513}], "download_size": 134683484, "dataset_size": 360674267}, {"config_name": "unshuffled_deduplicated_frr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4500, "num_examples": 7}], "download_size": 540, "dataset_size": 4500}, {"config_name": "unshuffled_deduplicated_gl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 404922022, "num_examples": 284320}], "download_size": 155851883, "dataset_size": 404922022}, {"config_name": "unshuffled_deduplicated_he", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10451408409, "num_examples": 2375030}], "download_size": 3043383695, "dataset_size": 10451408409}, {"config_name": "unshuffled_deduplicated_ht", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3439, "num_examples": 9}], "download_size": 594, "dataset_size": 3439}, {"config_name": "unshuffled_deduplicated_id", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16964948727, "num_examples": 9948521}], "download_size": 5995510660, "dataset_size": 16964948727}, {"config_name": "unshuffled_deduplicated_is", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 891047926, "num_examples": 389515}], "download_size": 332871764, "dataset_size": 891047926}, {"config_name": "unshuffled_deduplicated_jv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 609713, "num_examples": 1163}], "download_size": 208165, "dataset_size": 609713}, {"config_name": "unshuffled_deduplicated_kn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1080985653, "num_examples": 251064}], "download_size": 215526836, "dataset_size": 1080985653}, {"config_name": "unshuffled_deduplicated_kv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1200609, "num_examples": 924}], "download_size": 327479, "dataset_size": 1200609}, {"config_name": "unshuffled_deduplicated_lb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21242773, "num_examples": 21735}], "download_size": 8300328, "dataset_size": 21242773}, {"config_name": "unshuffled_deduplicated_lo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 119015146, "num_examples": 32652}], "download_size": 23634237, "dataset_size": 119015146}, {"config_name": "unshuffled_deduplicated_mai", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10721, "num_examples": 25}], "download_size": 2267, "dataset_size": 10721}, {"config_name": "unshuffled_deduplicated_mk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1186605123, "num_examples": 299457}], "download_size": 303118518, "dataset_size": 1186605123}, {"config_name": "unshuffled_deduplicated_mrj", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1096428, "num_examples": 669}], "download_size": 289048, "dataset_size": 1096428}, {"config_name": "unshuffled_deduplicated_my", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1112006614, "num_examples": 136639}], "download_size": 207136614, "dataset_size": 1112006614}, {"config_name": "unshuffled_deduplicated_nap", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13782, "num_examples": 55}], "download_size": 4965, "dataset_size": 13782}, {"config_name": "unshuffled_deduplicated_nl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 41726089054, "num_examples": 20812149}], "download_size": 15734167112, "dataset_size": 41726089054}, {"config_name": "unshuffled_deduplicated_or", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 197401878, "num_examples": 44230}], "download_size": 38726721, "dataset_size": 197401878}, {"config_name": "unshuffled_deduplicated_pl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 50387595763, "num_examples": 20682611}], "download_size": 20189161328, "dataset_size": 50387595763}, {"config_name": "unshuffled_deduplicated_pt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 68162434231, "num_examples": 26920397}], "download_size": 25997795946, "dataset_size": 68162434231}, {"config_name": "unshuffled_deduplicated_ru", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 611031071327, "num_examples": 115954598}], "download_size": 166677136024, "dataset_size": 611031071327}, {"config_name": "unshuffled_deduplicated_sd", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 275327037, "num_examples": 33925}], "download_size": 74169753, "dataset_size": 275327037}, {"config_name": "unshuffled_deduplicated_sl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1311219223, "num_examples": 886223}], "download_size": 523218283, "dataset_size": 1311219223}, {"config_name": "unshuffled_deduplicated_su", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 149921, "num_examples": 511}], "download_size": 53164, "dataset_size": 149921}, {"config_name": "unshuffled_deduplicated_te", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1694004428, "num_examples": 312644}], "download_size": 342429224, "dataset_size": 1694004428}, {"config_name": "unshuffled_deduplicated_tl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 429427446, "num_examples": 294132}], "download_size": 151342433, "dataset_size": 429427446}, {"config_name": "unshuffled_deduplicated_ug", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 86344782, "num_examples": 15503}], "download_size": 20527752, "dataset_size": 86344782}, {"config_name": "unshuffled_deduplicated_vec", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17303, "num_examples": 64}], "download_size": 7647, "dataset_size": 17303}, {"config_name": "unshuffled_deduplicated_war", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2338532, "num_examples": 9161}], "download_size": 546586, "dataset_size": 2338532}, {"config_name": "unshuffled_deduplicated_yi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 87935052, "num_examples": 32919}], "download_size": 22197718, "dataset_size": 87935052}, {"config_name": "unshuffled_original_af", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 254076274, "num_examples": 201117}], "download_size": 85795254, "dataset_size": 254076274}, {"config_name": "unshuffled_original_ar", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 87935768938, "num_examples": 16365602}], "download_size": 22232546836, "dataset_size": 87935768938}, {"config_name": "unshuffled_original_av", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 423603, "num_examples": 456}], "download_size": 84767, "dataset_size": 423603}, {"config_name": "unshuffled_original_bar", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 555, "num_examples": 4}], "download_size": 341, "dataset_size": 555}, {"config_name": "unshuffled_original_bh", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 116514, "num_examples": 336}], "download_size": 7615, "dataset_size": 116514}, {"config_name": "unshuffled_original_br", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 30203875, "num_examples": 37085}], "download_size": 9178158, "dataset_size": 30203875}, {"config_name": "unshuffled_original_cbk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 536, "num_examples": 1}], "download_size": 234, "dataset_size": 536}, {"config_name": "unshuffled_original_cs", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 57080142860, "num_examples": 21001388}], "download_size": 21716697253, "dataset_size": 57080142860}, {"config_name": "unshuffled_original_de", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 331224484023, "num_examples": 104913504}], "download_size": 119506267566, "dataset_size": 331224484023}, {"config_name": "unshuffled_original_el", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 66273231642, "num_examples": 10425596}], "download_size": 17309601342, "dataset_size": 66273231642}, {"config_name": "unshuffled_original_es", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 298492270636, "num_examples": 88199221}], "download_size": 106039137656, "dataset_size": 298492270636}, {"config_name": "unshuffled_original_fi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 28571419204, "num_examples": 8557453}], "download_size": 9970837279, "dataset_size": 28571419204}, {"config_name": "unshuffled_original_ga", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 92369035, "num_examples": 83223}], "download_size": 29262282, "dataset_size": 92369035}, {"config_name": "unshuffled_original_gom", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2257169, "num_examples": 640}], "download_size": 442950, "dataset_size": 2257169}, {"config_name": "unshuffled_original_hr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 243829069, "num_examples": 582219}], "download_size": 79417804, "dataset_size": 243829069}, {"config_name": "unshuffled_original_hy", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3939672772, "num_examples": 659430}], "download_size": 897364024, "dataset_size": 3939672772}, {"config_name": "unshuffled_original_ilo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 925809, "num_examples": 2638}], "download_size": 267451, "dataset_size": 925809}, {"config_name": "unshuffled_original_ja", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 232216718556, "num_examples": 62721527}], "download_size": 79564645083, "dataset_size": 232216718556}, {"config_name": "unshuffled_original_kk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2833778199, "num_examples": 524591}], "download_size": 615067761, "dataset_size": 2833778199}, {"config_name": "unshuffled_original_krc", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2688672, "num_examples": 1581}], "download_size": 656496, "dataset_size": 2688672}, {"config_name": "unshuffled_original_ky", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 630794622, "num_examples": 146993}], "download_size": 152636608, "dataset_size": 630794622}, {"config_name": "unshuffled_original_li", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 31312, "num_examples": 137}], "download_size": 11793, "dataset_size": 31312}, {"config_name": "unshuffled_original_lt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9445278312, "num_examples": 2977757}], "download_size": 3439789726, "dataset_size": 9445278312}, {"config_name": "unshuffled_original_mhr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7553453, "num_examples": 3212}], "download_size": 1834912, "dataset_size": 7553453}, {"config_name": "unshuffled_original_mn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2332897881, "num_examples": 395605}], "download_size": 472357548, "dataset_size": 2332897881}, {"config_name": "unshuffled_original_mt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24470330, "num_examples": 26598}], "download_size": 7533204, "dataset_size": 24470330}, {"config_name": "unshuffled_original_mzn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 720229, "num_examples": 1055}], "download_size": 177817, "dataset_size": 720229}, {"config_name": "unshuffled_original_ne", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1866852959, "num_examples": 299938}], "download_size": 355291639, "dataset_size": 1866852959}, {"config_name": "unshuffled_original_no", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8652054976, "num_examples": 5546211}], "download_size": 3106155643, "dataset_size": 8652054976}, {"config_name": "unshuffled_original_pa", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 801167879, "num_examples": 127467}], "download_size": 164207256, "dataset_size": 801167879}, {"config_name": "unshuffled_original_pnb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12039418, "num_examples": 4599}], "download_size": 3215579, "dataset_size": 12039418}, {"config_name": "unshuffled_original_rm", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8027, "num_examples": 41}], "download_size": 2691, "dataset_size": 8027}, {"config_name": "unshuffled_original_sah", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 43817239, "num_examples": 22301}], "download_size": 9079982, "dataset_size": 43817239}, {"config_name": "unshuffled_original_si", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1469374795, "num_examples": 203082}], "download_size": 310935021, "dataset_size": 1469374795}, {"config_name": "unshuffled_original_sq", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2440834375, "num_examples": 672077}], "download_size": 861831806, "dataset_size": 2440834375}, {"config_name": "unshuffled_original_sw", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14073775, "num_examples": 41986}], "download_size": 3712739, "dataset_size": 14073775}, {"config_name": "unshuffled_original_th", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 38289228753, "num_examples": 6064129}], "download_size": 7377469078, "dataset_size": 38289228753}, {"config_name": "unshuffled_original_tt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 703412782, "num_examples": 135923}], "download_size": 151056507, "dataset_size": 703412782}, {"config_name": "unshuffled_original_ur", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2802270961, "num_examples": 638596}], "download_size": 712607161, "dataset_size": 2802270961}, {"config_name": "unshuffled_original_vo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2118909, "num_examples": 3366}], "download_size": 307184, "dataset_size": 2118909}, {"config_name": "unshuffled_original_xal", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 116043, "num_examples": 39}], "download_size": 32117, "dataset_size": 116043}, {"config_name": "unshuffled_original_yue", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3899, "num_examples": 11}], "download_size": 647, "dataset_size": 3899}, {"config_name": "unshuffled_original_en", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2525437912097, "num_examples": 455994980}], "download_size": 903830686146, "dataset_size": 2525437912097}, {"config_name": "unshuffled_original_eu", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 894836188, "num_examples": 506883}], "download_size": 248190119, "dataset_size": 894836188}, {"config_name": "unshuffled_original_frr", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4507, "num_examples": 7}], "download_size": 527, "dataset_size": 4507}, {"config_name": "unshuffled_original_gl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 656477422, "num_examples": 544388}], "download_size": 235384299, "dataset_size": 656477422}, {"config_name": "unshuffled_original_he", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21113706929, "num_examples": 3808397}], "download_size": 5660026441, "dataset_size": 21113706929}, {"config_name": "unshuffled_original_ht", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4083, "num_examples": 13}], "download_size": 590, "dataset_size": 4083}, {"config_name": "unshuffled_original_id", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 32317679452, "num_examples": 16236463}], "download_size": 10596988488, "dataset_size": 32317679452}, {"config_name": "unshuffled_original_is", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1524936467, "num_examples": 625673}], "download_size": 533034495, "dataset_size": 1524936467}, {"config_name": "unshuffled_original_jv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 691812, "num_examples": 1445}], "download_size": 219246, "dataset_size": 691812}, {"config_name": "unshuffled_original_kn", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1763625096, "num_examples": 350363}], "download_size": 342155433, "dataset_size": 1763625096}, {"config_name": "unshuffled_original_kv", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2379758, "num_examples": 1549}], "download_size": 400725, "dataset_size": 2379758}, {"config_name": "unshuffled_original_lb", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 30595156, "num_examples": 34807}], "download_size": 10725552, "dataset_size": 30595156}, {"config_name": "unshuffled_original_lo", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 182361509, "num_examples": 52910}], "download_size": 33916738, "dataset_size": 182361509}, {"config_name": "unshuffled_original_mai", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 325990, "num_examples": 123}], "download_size": 5563, "dataset_size": 325990}, {"config_name": "unshuffled_original_mk", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2202480390, "num_examples": 437871}], "download_size": 508239918, "dataset_size": 2202480390}, {"config_name": "unshuffled_original_mrj", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1165977, "num_examples": 757}], "download_size": 303447, "dataset_size": 1165977}, {"config_name": "unshuffled_original_my", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2021872493, "num_examples": 232329}], "download_size": 369850157, "dataset_size": 2021872493}, {"config_name": "unshuffled_original_nap", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17839, "num_examples": 73}], "download_size": 5023, "dataset_size": 17839}, {"config_name": "unshuffled_original_nl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 83230965323, "num_examples": 34682142}], "download_size": 29352811750, "dataset_size": 83230965323}, {"config_name": "unshuffled_original_or", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 260151226, "num_examples": 59463}], "download_size": 49834443, "dataset_size": 260151226}, {"config_name": "unshuffled_original_pl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 117121370605, "num_examples": 35440972}], "download_size": 42884898947, "dataset_size": 117121370605}, {"config_name": "unshuffled_original_pt", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 132635490139, "num_examples": 42114520}], "download_size": 47257949300, "dataset_size": 132635490139}, {"config_name": "unshuffled_original_ru", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1241627166551, "num_examples": 161836003}], "download_size": 319755378587, "dataset_size": 1241627166551}, {"config_name": "unshuffled_original_sd", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 364256869, "num_examples": 44280}], "download_size": 90621520, "dataset_size": 364256869}, {"config_name": "unshuffled_original_sl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2675665926, "num_examples": 1746604}], "download_size": 956197026, "dataset_size": 2675665926}, {"config_name": "unshuffled_original_su", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 225627, "num_examples": 805}], "download_size": 59643, "dataset_size": 225627}, {"config_name": "unshuffled_original_te", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2611548765, "num_examples": 475703}], "download_size": 522470115, "dataset_size": 2611548765}, {"config_name": "unshuffled_original_tl", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 606295665, "num_examples": 458206}], "download_size": 204895159, "dataset_size": 606295665}, {"config_name": "unshuffled_original_ug", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 127419368, "num_examples": 22255}], "download_size": 27923925, "dataset_size": 127419368}, {"config_name": "unshuffled_original_vec", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19182, "num_examples": 73}], "download_size": 7672, "dataset_size": 19182}, {"config_name": "unshuffled_original_war", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2682430, "num_examples": 9760}], "download_size": 644576, "dataset_size": 2682430}, {"config_name": "unshuffled_original_yi", "features": [{"name": "id", "dtype": "int64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 147601654, "num_examples": 59364}], "download_size": 33337157, "dataset_size": 147601654}]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-100K<n<1M #size_categories-100M<n<1B #size_categories-10K<n<100K #size_categories-10M<n<100M #size_categories-1K<n<10K #size_categories-1M<n<10M #size_categories-n<1K #source_datasets-original #language-Afrikaans #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Avaric #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Russia Buriat #language-Catalan #language-Chavacano #language-Chechen #language-Cebuano #language-Central Kurdish #language-Czech #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dimli (individual language) #language-Lower Sorbian #language-Dhivehi #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-French #language-Northern Frisian #language-Western Frisian #language-Irish #language-Scottish Gaelic #language-Galician #language-Guarani #language-Goan Konkani #language-Gujarati #language-Hebrew #language-Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Iloko #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Lojban #language-Javanese #language-Georgian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Karachay-Balkar #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Luxembourgish #language-Lezghian #language-Limburgan #language-Lombard #language-Lao #language-Northern Luri #language-Lithuanian #language-Latvian #language-Maithili #language-Malagasy #language-Eastern Mari #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-nah #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Occitan (post 1500) #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pampanga #language-Polish #language-Piemontese #language-Western Panjabi #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Romanian #language-Russian #language-Sanskrit #language-Yakut #language-Sicilian #language-Sindhi #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Somali #language-Albanian #language-Serbian #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Tamil #language-Telugu #language-Tajik #language-Thai #language-Turkmen #language-Tagalog #language-Turkish #language-Tatar #language-Tuvinian #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venetian #language-Vietnamese #language-Volapük #language-Walloon #language-Waray (Philippines) #language-Wu Chinese #language-Kalmyk #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #license-cc0-1.0 #arxiv-2010.14571 #region-us 

Dataset Card for "oscar"
========================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository:
* Paper:
* Point of Contact:


### Dataset Summary


OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture. Data is distributed by language in both original and deduplicated form.


The version here is the original OSCAR 2019 release: URL


For more recent versions, visit the oscar-corpus organization on the Hub:


* OSCAR 22.01 (released in January 2022): oscar-corpus/OSCAR-2201
* OSCAR 21.09 (released in September 2021): oscar-corpus/OSCAR-2109


### Supported Tasks and Leaderboards


OSCAR is mainly inteded to pretrain language models and word represantations.


### Languages


All the data is distributed by language, both the original and the deduplicated versions of the data are available. 166 different languages are available. The table in subsection Data Splits Sample Size provides the language code for each subcorpus as well as the number of words (space separated tokens), lines and sizes for both the original and the deduplicated versions of OSCAR.


Dataset Structure
-----------------


We show detailed information for all the configurations of the dataset.


### Data Instances



Click to expand the Data/size information for each language (deduplicated)
#### unshuffled\_deduplicated\_af


* Size of downloaded dataset files: 65.99 MB
* Size of the generated dataset: 172.30 MB
* Total amount of disk used: 238.29 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_als


* Size of downloaded dataset files: 1.26 MB
* Size of the generated dataset: 2.96 MB
* Total amount of disk used: 4.22 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_am


* Size of downloaded dataset files: 61.35 MB
* Size of the generated dataset: 216.15 MB
* Total amount of disk used: 277.50 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_an


* Size of downloaded dataset files: 0.14 MB
* Size of the generated dataset: 0.85 MB
* Total amount of disk used: 0.99 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ar


* Size of downloaded dataset files: 9.67 GB
* Size of the generated dataset: 33.57 GB
* Total amount of disk used: 43.23 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_arz


* Size of downloaded dataset files: 10.02 MB
* Size of the generated dataset: 35.91 MB
* Total amount of disk used: 45.94 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_as


* Size of downloaded dataset files: 15.51 MB
* Size of the generated dataset: 74.07 MB
* Total amount of disk used: 89.58 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ast


* Size of downloaded dataset files: 0.86 MB
* Size of the generated dataset: 2.17 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_av


* Size of downloaded dataset files: 0.07 MB
* Size of the generated dataset: 0.34 MB
* Total amount of disk used: 0.41 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_az


* Size of downloaded dataset files: 521.74 MB
* Size of the generated dataset: 1.53 GB
* Total amount of disk used: 2.05 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_azb


* Size of downloaded dataset files: 5.19 MB
* Size of the generated dataset: 20.08 MB
* Total amount of disk used: 25.27 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ba


* Size of downloaded dataset files: 25.98 MB
* Size of the generated dataset: 93.84 MB
* Total amount of disk used: 119.82 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_be


* Size of downloaded dataset files: 306.70 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.39 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bg


* Size of downloaded dataset files: 3.85 GB
* Size of the generated dataset: 14.45 GB
* Total amount of disk used: 18.30 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bn


* Size of downloaded dataset files: 1.26 GB
* Size of the generated dataset: 6.24 GB
* Total amount of disk used: 7.50 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bo


* Size of downloaded dataset files: 22.37 MB
* Size of the generated dataset: 144.65 MB
* Total amount of disk used: 167.02 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bpy


* Size of downloaded dataset files: 0.19 MB
* Size of the generated dataset: 1.78 MB
* Total amount of disk used: 1.97 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_br


* Size of downloaded dataset files: 6.47 MB
* Size of the generated dataset: 17.00 MB
* Total amount of disk used: 23.47 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bs


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.15 MB
* Total amount of disk used: 0.18 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ca


* Size of downloaded dataset files: 1.73 GB
* Size of the generated dataset: 4.57 GB
* Total amount of disk used: 6.30 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ce


* Size of downloaded dataset files: 1.87 MB
* Size of the generated dataset: 7.04 MB
* Total amount of disk used: 8.90 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ceb


* Size of downloaded dataset files: 7.12 MB
* Size of the generated dataset: 24.83 MB
* Total amount of disk used: 31.95 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ckb


* Size of downloaded dataset files: 60.32 MB
* Size of the generated dataset: 237.72 MB
* Total amount of disk used: 298.05 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_cs


* Size of downloaded dataset files: 10.49 GB
* Size of the generated dataset: 25.71 GB
* Total amount of disk used: 36.20 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_cv


* Size of downloaded dataset files: 7.47 MB
* Size of the generated dataset: 27.49 MB
* Total amount of disk used: 34.95 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_cy


* Size of downloaded dataset files: 53.63 MB
* Size of the generated dataset: 141.22 MB
* Total amount of disk used: 194.86 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_da


* Size of downloaded dataset files: 3.82 GB
* Size of the generated dataset: 10.24 GB
* Total amount of disk used: 14.06 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_de


* Size of downloaded dataset files: 60.80 GB
* Size of the generated dataset: 156.30 GB
* Total amount of disk used: 217.10 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_dv


* Size of downloaded dataset files: 16.84 MB
* Size of the generated dataset: 82.19 MB
* Total amount of disk used: 99.03 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_el


* Size of downloaded dataset files: 7.91 GB
* Size of the generated dataset: 28.74 GB
* Total amount of disk used: 36.65 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_en


* Size of downloaded dataset files: 496.50 GB
* Size of the generated dataset: 1299.75 GB
* Total amount of disk used: 1796.24 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_eo


* Size of downloaded dataset files: 92.86 MB
* Size of the generated dataset: 240.12 MB
* Total amount of disk used: 332.99 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_es


* Size of downloaded dataset files: 60.46 GB
* Size of the generated dataset: 160.86 GB
* Total amount of disk used: 221.32 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_et


* Size of downloaded dataset files: 966.79 MB
* Size of the generated dataset: 2.45 GB
* Total amount of disk used: 3.41 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_eu


* Size of downloaded dataset files: 134.68 MB
* Size of the generated dataset: 363.93 MB
* Total amount of disk used: 498.61 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_fa


* Size of downloaded dataset files: 10.46 GB
* Size of the generated dataset: 40.06 GB
* Total amount of disk used: 50.52 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_fi


* Size of downloaded dataset files: 5.38 GB
* Size of the generated dataset: 13.99 GB
* Total amount of disk used: 19.37 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_fr


* Size of downloaded dataset files: 55.46 GB
* Size of the generated dataset: 148.28 GB
* Total amount of disk used: 203.75 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_fy


* Size of downloaded dataset files: 10.27 MB
* Size of the generated dataset: 26.73 MB
* Total amount of disk used: 37.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ga


* Size of downloaded dataset files: 22.22 MB
* Size of the generated dataset: 63.86 MB
* Total amount of disk used: 86.08 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_gd


* Size of downloaded dataset files: 0.42 MB
* Size of the generated dataset: 1.36 MB
* Total amount of disk used: 1.78 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_gl


* Size of downloaded dataset files: 155.85 MB
* Size of the generated dataset: 408.34 MB
* Total amount of disk used: 564.19 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_gom


* Size of downloaded dataset files: 0.38 MB
* Size of the generated dataset: 1.87 MB
* Total amount of disk used: 2.24 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_gu


* Size of downloaded dataset files: 162.97 MB
* Size of the generated dataset: 759.34 MB
* Total amount of disk used: 922.32 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_he


* Size of downloaded dataset files: 3.04 GB
* Size of the generated dataset: 10.47 GB
* Total amount of disk used: 13.51 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_hi


* Size of downloaded dataset files: 2.01 GB
* Size of the generated dataset: 9.57 GB
* Total amount of disk used: 11.58 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_hr


* Size of downloaded dataset files: 46.74 MB
* Size of the generated dataset: 121.50 MB
* Total amount of disk used: 168.23 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_hsb


* Size of downloaded dataset files: 0.72 MB
* Size of the generated dataset: 1.89 MB
* Total amount of disk used: 2.61 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_hu


* Size of downloaded dataset files: 7.37 GB
* Size of the generated dataset: 19.09 GB
* Total amount of disk used: 26.46 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_hy


* Size of downloaded dataset files: 393.62 MB
* Size of the generated dataset: 1.56 GB
* Total amount of disk used: 1.96 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ia


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.38 MB
* Total amount of disk used: 0.43 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_id


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 17.05 GB
* Total amount of disk used: 23.05 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ilo


* Size of downloaded dataset files: 0.23 MB
* Size of the generated dataset: 0.68 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.14 MB
* Total amount of disk used: 0.19 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_is


* Size of downloaded dataset files: 332.87 MB
* Size of the generated dataset: 894.28 MB
* Total amount of disk used: 1.23 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_it


* Size of downloaded dataset files: 27.93 GB
* Size of the generated dataset: 74.09 GB
* Total amount of disk used: 102.03 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ja


* Size of downloaded dataset files: 40.80 GB
* Size of the generated dataset: 113.63 GB
* Total amount of disk used: 154.44 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_jbo


* Size of downloaded dataset files: 0.20 MB
* Size of the generated dataset: 0.70 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_jv


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.62 MB
* Total amount of disk used: 0.82 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ka


* Size of downloaded dataset files: 377.23 MB
* Size of the generated dataset: 1.99 GB
* Total amount of disk used: 2.36 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_kk


* Size of downloaded dataset files: 389.12 MB
* Size of the generated dataset: 1.59 GB
* Total amount of disk used: 1.97 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_km


* Size of downloaded dataset files: 114.48 MB
* Size of the generated dataset: 610.61 MB
* Total amount of disk used: 725.09 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_kn


* Size of downloaded dataset files: 215.52 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ko


* Size of downloaded dataset files: 4.46 GB
* Size of the generated dataset: 12.00 GB
* Total amount of disk used: 16.47 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_krc


* Size of downloaded dataset files: 0.62 MB
* Size of the generated dataset: 2.41 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ku


* Size of downloaded dataset files: 23.34 MB
* Size of the generated dataset: 63.09 MB
* Total amount of disk used: 86.43 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_kv


* Size of downloaded dataset files: 0.33 MB
* Size of the generated dataset: 1.21 MB
* Total amount of disk used: 1.54 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ky


* Size of downloaded dataset files: 106.22 MB
* Size of the generated dataset: 408.40 MB
* Total amount of disk used: 514.61 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_la


* Size of downloaded dataset files: 3.42 MB
* Size of the generated dataset: 9.79 MB
* Total amount of disk used: 13.22 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lb


* Size of downloaded dataset files: 8.30 MB
* Size of the generated dataset: 21.42 MB
* Total amount of disk used: 29.72 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lez


* Size of downloaded dataset files: 0.77 MB
* Size of the generated dataset: 3.08 MB
* Total amount of disk used: 3.84 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.46 MB
* Total amount of disk used: 0.57 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lo


* Size of downloaded dataset files: 23.63 MB
* Size of the generated dataset: 119.29 MB
* Total amount of disk used: 142.92 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.08 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lt


* Size of downloaded dataset files: 1.65 GB
* Size of the generated dataset: 4.20 GB
* Total amount of disk used: 5.86 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_lv


* Size of downloaded dataset files: 710.45 MB
* Size of the generated dataset: 1.91 GB
* Total amount of disk used: 2.62 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mai


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mg


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 13.59 MB
* Total amount of disk used: 17.89 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mhr


* Size of downloaded dataset files: 1.63 MB
* Size of the generated dataset: 6.26 MB
* Total amount of disk used: 7.89 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.31 MB
* Total amount of disk used: 0.33 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mk


* Size of downloaded dataset files: 303.12 MB
* Size of the generated dataset: 1.19 GB
* Total amount of disk used: 1.49 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ml


* Size of downloaded dataset files: 496.80 MB
* Size of the generated dataset: 2.69 GB
* Total amount of disk used: 3.18 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mn


* Size of downloaded dataset files: 219.52 MB
* Size of the generated dataset: 883.46 MB
* Total amount of disk used: 1.10 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mr


* Size of downloaded dataset files: 299.68 MB
* Size of the generated dataset: 1.49 GB
* Total amount of disk used: 1.79 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mrj


* Size of downloaded dataset files: 0.29 MB
* Size of the generated dataset: 1.10 MB
* Total amount of disk used: 1.38 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ms


* Size of downloaded dataset files: 16.39 MB
* Size of the generated dataset: 49.45 MB
* Total amount of disk used: 65.85 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mt


* Size of downloaded dataset files: 5.90 MB
* Size of the generated dataset: 17.68 MB
* Total amount of disk used: 23.58 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_my


* Size of downloaded dataset files: 207.14 MB
* Size of the generated dataset: 1.11 GB
* Total amount of disk used: 1.32 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_mzn


* Size of downloaded dataset files: 0.16 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.79 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_nds


* Size of downloaded dataset files: 5.27 MB
* Size of the generated dataset: 13.48 MB
* Total amount of disk used: 18.76 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ne


* Size of downloaded dataset files: 240.63 MB
* Size of the generated dataset: 1.24 GB
* Total amount of disk used: 1.48 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_new


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 4.26 MB
* Total amount of disk used: 5.09 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_nl


* Size of downloaded dataset files: 15.73 GB
* Size of the generated dataset: 41.91 GB
* Total amount of disk used: 57.65 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_nn


* Size of downloaded dataset files: 23.58 MB
* Size of the generated dataset: 58.32 MB
* Total amount of disk used: 81.90 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_no


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 5.11 GB
* Total amount of disk used: 7.07 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_oc


* Size of downloaded dataset files: 1.34 MB
* Size of the generated dataset: 4.00 MB
* Total amount of disk used: 5.34 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_or


* Size of downloaded dataset files: 38.72 MB
* Size of the generated dataset: 197.63 MB
* Total amount of disk used: 236.36 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_os


* Size of downloaded dataset files: 2.83 MB
* Size of the generated dataset: 11.00 MB
* Total amount of disk used: 13.83 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_pa


* Size of downloaded dataset files: 102.39 MB
* Size of the generated dataset: 483.04 MB
* Total amount of disk used: 585.42 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_pl


* Size of downloaded dataset files: 20.19 GB
* Size of the generated dataset: 50.59 GB
* Total amount of disk used: 70.78 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_pms


* Size of downloaded dataset files: 0.71 MB
* Size of the generated dataset: 2.00 MB
* Total amount of disk used: 2.72 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_pnb


* Size of downloaded dataset files: 2.58 MB
* Size of the generated dataset: 9.44 MB
* Total amount of disk used: 12.02 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ps


* Size of downloaded dataset files: 71.83 MB
* Size of the generated dataset: 254.79 MB
* Total amount of disk used: 326.61 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_pt


* Size of downloaded dataset files: 26.00 GB
* Size of the generated dataset: 68.37 GB
* Total amount of disk used: 94.37 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ro


* Size of downloaded dataset files: 4.48 GB
* Size of the generated dataset: 11.66 GB
* Total amount of disk used: 16.14 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ru


* Size of downloaded dataset files: 166.68 GB
* Size of the generated dataset: 611.70 GB
* Total amount of disk used: 778.38 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sa


* Size of downloaded dataset files: 7.27 MB
* Size of the generated dataset: 38.33 MB
* Total amount of disk used: 45.60 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sah


* Size of downloaded dataset files: 7.01 MB
* Size of the generated dataset: 27.46 MB
* Total amount of disk used: 34.49 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sd


* Size of downloaded dataset files: 74.17 MB
* Size of the generated dataset: 275.48 MB
* Total amount of disk used: 349.66 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sh


* Size of downloaded dataset files: 1.45 MB
* Size of the generated dataset: 6.44 MB
* Total amount of disk used: 7.87 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_si


* Size of downloaded dataset files: 175.62 MB
* Size of the generated dataset: 842.57 MB
* Total amount of disk used: 1.02 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sk


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 4.80 GB
* Total amount of disk used: 6.76 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sl


* Size of downloaded dataset files: 523.22 MB
* Size of the generated dataset: 1.32 GB
* Total amount of disk used: 1.85 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sq


* Size of downloaded dataset files: 445.36 MB
* Size of the generated dataset: 1.21 GB
* Total amount of disk used: 1.66 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sr


* Size of downloaded dataset files: 665.03 MB
* Size of the generated dataset: 2.36 GB
* Total amount of disk used: 3.03 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_su


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.21 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sv


* Size of downloaded dataset files: 10.19 GB
* Size of the generated dataset: 26.33 GB
* Total amount of disk used: 36.51 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_sw


* Size of downloaded dataset files: 2.95 MB
* Size of the generated dataset: 8.98 MB
* Total amount of disk used: 11.92 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ta


* Size of downloaded dataset files: 971.12 MB
* Size of the generated dataset: 5.48 GB
* Total amount of disk used: 6.45 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_te


* Size of downloaded dataset files: 342.43 MB
* Size of the generated dataset: 1.70 GB
* Total amount of disk used: 2.04 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_tg


* Size of downloaded dataset files: 62.90 MB
* Size of the generated dataset: 261.68 MB
* Total amount of disk used: 324.60 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_th


* Size of downloaded dataset files: 3.54 GB
* Size of the generated dataset: 17.11 GB
* Total amount of disk used: 20.65 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_tk


* Size of downloaded dataset files: 2.22 MB
* Size of the generated dataset: 7.12 MB
* Total amount of disk used: 9.34 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_tl


* Size of downloaded dataset files: 151.34 MB
* Size of the generated dataset: 431.69 MB
* Total amount of disk used: 583.04 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_tr


* Size of downloaded dataset files: 10.39 GB
* Size of the generated dataset: 28.47 GB
* Total amount of disk used: 38.86 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_tt


* Size of downloaded dataset files: 85.89 MB
* Size of the generated dataset: 321.37 MB
* Total amount of disk used: 407.26 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ug


* Size of downloaded dataset files: 20.53 MB
* Size of the generated dataset: 86.44 MB
* Total amount of disk used: 106.97 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_uk


* Size of downloaded dataset files: 8.04 GB
* Size of the generated dataset: 29.86 GB
* Total amount of disk used: 37.90 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_ur


* Size of downloaded dataset files: 483.59 MB
* Size of the generated dataset: 1.82 GB
* Total amount of disk used: 2.31 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_uz


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 12.00 MB
* Total amount of disk used: 16.29 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_vi


* Size of downloaded dataset files: 10.71 GB
* Size of the generated dataset: 33.60 GB
* Total amount of disk used: 44.31 GB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.10 MB
* Total amount of disk used: 2.40 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_wa


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.22 MB
* Total amount of disk used: 0.29 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_war


* Size of downloaded dataset files: 0.55 MB
* Size of the generated dataset: 2.36 MB
* Total amount of disk used: 2.90 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_xmf


* Size of downloaded dataset files: 0.94 MB
* Size of the generated dataset: 4.63 MB
* Total amount of disk used: 5.58 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_yi


* Size of downloaded dataset files: 22.20 MB
* Size of the generated dataset: 88.29 MB
* Total amount of disk used: 110.49 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_deduplicated\_zh


* Size of downloaded dataset files: 99.98 GB
* Size of the generated dataset: 267.88 GB
* Total amount of disk used: 367.86 GB


An example of 'train' looks as follows.




Click to expand the Data/size information for each language (original)
#### unshuffled\_original\_af


* Size of downloaded dataset files: 85.79 MB
* Size of the generated dataset: 254.08 MB
* Total amount of disk used: 339.87 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_als


* Size of downloaded dataset files: 1.49 MB
* Size of the generated dataset: 5.30 MB
* Total amount of disk used: 6.78 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_am


* Size of downloaded dataset files: 102.79 MB
* Size of the generated dataset: 378.06 MB
* Total amount of disk used: 480.85 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_an


* Size of downloaded dataset files: 0.15 MB
* Size of the generated dataset: 1.33 MB
* Total amount of disk used: 1.48 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ar


* Size of downloaded dataset files: 22.23 GB
* Size of the generated dataset: 87.94 GB
* Total amount of disk used: 110.17 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_arz


* Size of downloaded dataset files: 15.90 MB
* Size of the generated dataset: 70.13 MB
* Total amount of disk used: 86.03 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_as


* Size of downloaded dataset files: 21.43 MB
* Size of the generated dataset: 117.73 MB
* Total amount of disk used: 139.17 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ast


* Size of downloaded dataset files: 0.92 MB
* Size of the generated dataset: 2.54 MB
* Total amount of disk used: 3.46 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_av


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.42 MB
* Total amount of disk used: 0.50 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_az


* Size of downloaded dataset files: 927.76 MB
* Size of the generated dataset: 2.96 GB
* Total amount of disk used: 3.89 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_azb


* Size of downloaded dataset files: 6.64 MB
* Size of the generated dataset: 28.47 MB
* Total amount of disk used: 35.11 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ba


* Size of downloaded dataset files: 33.22 MB
* Size of the generated dataset: 133.70 MB
* Total amount of disk used: 166.92 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_be


* Size of downloaded dataset files: 498.29 MB
* Size of the generated dataset: 1.88 GB
* Total amount of disk used: 2.38 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_bg


* Size of downloaded dataset files: 8.34 GB
* Size of the generated dataset: 33.75 GB
* Total amount of disk used: 42.09 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_bn


* Size of downloaded dataset files: 2.14 GB
* Size of the generated dataset: 10.77 GB
* Total amount of disk used: 12.91 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_bo


* Size of downloaded dataset files: 28.94 MB
* Size of the generated dataset: 195.40 MB
* Total amount of disk used: 224.34 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_bpy


* Size of downloaded dataset files: 0.34 MB
* Size of the generated dataset: 4.35 MB
* Total amount of disk used: 4.69 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_br


* Size of downloaded dataset files: 9.18 MB
* Size of the generated dataset: 30.20 MB
* Total amount of disk used: 39.38 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_bs


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.48 MB
* Total amount of disk used: 0.53 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ca


* Size of downloaded dataset files: 3.10 GB
* Size of the generated dataset: 8.62 GB
* Total amount of disk used: 11.73 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ce


* Size of downloaded dataset files: 2.09 MB
* Size of the generated dataset: 8.73 MB
* Total amount of disk used: 10.82 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ceb


* Size of downloaded dataset files: 11.07 MB
* Size of the generated dataset: 40.97 MB
* Total amount of disk used: 52.03 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ckb


* Size of downloaded dataset files: 111.88 MB
* Size of the generated dataset: 510.97 MB
* Total amount of disk used: 622.85 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_cs


* Size of downloaded dataset files: 21.72 GB
* Size of the generated dataset: 57.08 GB
* Total amount of disk used: 78.80 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_cv


* Size of downloaded dataset files: 9.40 MB
* Size of the generated dataset: 41.05 MB
* Total amount of disk used: 50.45 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_cy


* Size of downloaded dataset files: 81.74 MB
* Size of the generated dataset: 224.93 MB
* Total amount of disk used: 306.67 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_da


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 16.76 GB
* Total amount of disk used: 22.76 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_de


* Size of downloaded dataset files: 119.51 GB
* Size of the generated dataset: 331.22 GB
* Total amount of disk used: 450.73 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_dv


* Size of downloaded dataset files: 24.91 MB
* Size of the generated dataset: 131.63 MB
* Total amount of disk used: 156.54 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_el


* Size of downloaded dataset files: 17.31 GB
* Size of the generated dataset: 66.27 GB
* Total amount of disk used: 83.58 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_en


* Size of downloaded dataset files: 903.83 GB
* Size of the generated dataset: 2525.44 GB
* Total amount of disk used: 3429.27 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_eo


* Size of downloaded dataset files: 117.07 MB
* Size of the generated dataset: 314.18 MB
* Total amount of disk used: 431.27 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_es


* Size of downloaded dataset files: 106.04 GB
* Size of the generated dataset: 298.49 GB
* Total amount of disk used: 404.53 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_et


* Size of downloaded dataset files: 1.88 GB
* Size of the generated dataset: 5.17 GB
* Total amount of disk used: 7.06 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_eu


* Size of downloaded dataset files: 248.19 MB
* Size of the generated dataset: 894.83 MB
* Total amount of disk used: 1.14 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_fa


* Size of downloaded dataset files: 20.96 GB
* Size of the generated dataset: 84.21 GB
* Total amount of disk used: 105.17 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_fi


* Size of downloaded dataset files: 9.97 GB
* Size of the generated dataset: 28.57 GB
* Total amount of disk used: 38.54 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_fr


* Size of downloaded dataset files: 105.32 GB
* Size of the generated dataset: 303.19 GB
* Total amount of disk used: 408.51 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_fy


* Size of downloaded dataset files: 12.40 MB
* Size of the generated dataset: 36.24 MB
* Total amount of disk used: 48.64 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ga


* Size of downloaded dataset files: 29.27 MB
* Size of the generated dataset: 92.37 MB
* Total amount of disk used: 121.63 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_gd


* Size of downloaded dataset files: 0.52 MB
* Size of the generated dataset: 2.02 MB
* Total amount of disk used: 2.55 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_gl


* Size of downloaded dataset files: 235.38 MB
* Size of the generated dataset: 656.48 MB
* Total amount of disk used: 891.87 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_gom


* Size of downloaded dataset files: 0.44 MB
* Size of the generated dataset: 2.25 MB
* Total amount of disk used: 2.71 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_gu


* Size of downloaded dataset files: 232.02 MB
* Size of the generated dataset: 1.09 GB
* Total amount of disk used: 1.33 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_he


* Size of downloaded dataset files: 5.66 GB
* Size of the generated dataset: 21.11 GB
* Total amount of disk used: 26.77 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_hi


* Size of downloaded dataset files: 3.66 GB
* Size of the generated dataset: 17.93 GB
* Total amount of disk used: 21.59 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_hr


* Size of downloaded dataset files: 79.42 MB
* Size of the generated dataset: 243.83 MB
* Total amount of disk used: 323.24 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_hsb


* Size of downloaded dataset files: 1.39 MB
* Size of the generated dataset: 4.49 MB
* Total amount of disk used: 5.87 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_hu


* Size of downloaded dataset files: 15.69 GB
* Size of the generated dataset: 43.07 GB
* Total amount of disk used: 58.77 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_hy


* Size of downloaded dataset files: 897.36 MB
* Size of the generated dataset: 3.94 GB
* Total amount of disk used: 4.84 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ia


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.78 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_id


* Size of downloaded dataset files: 10.60 GB
* Size of the generated dataset: 32.32 GB
* Total amount of disk used: 42.91 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ilo


* Size of downloaded dataset files: 0.27 MB
* Size of the generated dataset: 0.92 MB
* Total amount of disk used: 1.20 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.20 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_is


* Size of downloaded dataset files: 533.03 MB
* Size of the generated dataset: 1.52 GB
* Total amount of disk used: 2.06 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_it


* Size of downloaded dataset files: 52.16 GB
* Size of the generated dataset: 147.38 GB
* Total amount of disk used: 199.54 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ja


* Size of downloaded dataset files: 79.56 GB
* Size of the generated dataset: 232.22 GB
* Total amount of disk used: 311.78 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_jbo


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.77 MB
* Total amount of disk used: 0.98 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_jv


* Size of downloaded dataset files: 0.22 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ka


* Size of downloaded dataset files: 680.74 MB
* Size of the generated dataset: 3.77 GB
* Total amount of disk used: 4.45 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_kk


* Size of downloaded dataset files: 615.06 MB
* Size of the generated dataset: 2.83 GB
* Total amount of disk used: 3.45 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_km


* Size of downloaded dataset files: 193.28 MB
* Size of the generated dataset: 1.10 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_kn


* Size of downloaded dataset files: 342.15 MB
* Size of the generated dataset: 1.76 GB
* Total amount of disk used: 2.11 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ko


* Size of downloaded dataset files: 8.81 GB
* Size of the generated dataset: 25.29 GB
* Total amount of disk used: 34.10 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_krc


* Size of downloaded dataset files: 0.66 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.34 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ku


* Size of downloaded dataset files: 33.38 MB
* Size of the generated dataset: 99.06 MB
* Total amount of disk used: 132.44 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_kv


* Size of downloaded dataset files: 0.40 MB
* Size of the generated dataset: 2.38 MB
* Total amount of disk used: 2.78 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ky


* Size of downloaded dataset files: 152.64 MB
* Size of the generated dataset: 630.79 MB
* Total amount of disk used: 783.43 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_la


* Size of downloaded dataset files: 5.46 MB
* Size of the generated dataset: 27.80 MB
* Total amount of disk used: 33.26 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_lb


* Size of downloaded dataset files: 10.73 MB
* Size of the generated dataset: 30.60 MB
* Total amount of disk used: 41.32 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_lez


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 3.38 MB
* Total amount of disk used: 4.20 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.47 MB
* Total amount of disk used: 0.58 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_lo


* Size of downloaded dataset files: 33.92 MB
* Size of the generated dataset: 182.36 MB
* Total amount of disk used: 216.28 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_lt


* Size of downloaded dataset files: 3.44 GB
* Size of the generated dataset: 9.45 GB
* Total amount of disk used: 12.89 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_lv


* Size of downloaded dataset files: 1.49 GB
* Size of the generated dataset: 4.27 GB
* Total amount of disk used: 5.75 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_mai


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.33 MB
* Total amount of disk used: 0.34 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_mg


* Size of downloaded dataset files: 6.22 MB
* Size of the generated dataset: 21.79 MB
* Total amount of disk used: 28.01 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_mhr


* Size of downloaded dataset files: 1.84 MB
* Size of the generated dataset: 7.55 MB
* Total amount of disk used: 9.38 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.64 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_mk


* Size of downloaded dataset files: 508.24 MB
* Size of the generated dataset: 2.20 GB
* Total amount of disk used: 2.71 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ml


* Size of downloaded dataset files: 938.69 MB
* Size of the generated dataset: 5.24 GB
* Total amount of disk used: 6.18 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_mn


* Size of downloaded dataset files: 472.36 MB
* Size of the generated dataset: 2.33 GB
* Total amount of disk used: 2.81 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_mr


* Size of downloaded dataset files: 525.31 MB
* Size of the generated dataset: 2.82 GB
* Total amount of disk used: 3.34 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_mrj


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 1.16 MB
* Total amount of disk used: 1.47 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ms


* Size of downloaded dataset files: 28.46 MB
* Size of the generated dataset: 122.33 MB
* Total amount of disk used: 150.79 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_mt


* Size of downloaded dataset files: 7.53 MB
* Size of the generated dataset: 24.47 MB
* Total amount of disk used: 32.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_my


* Size of downloaded dataset files: 369.85 MB
* Size of the generated dataset: 2.02 GB
* Total amount of disk used: 2.39 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_mzn


* Size of downloaded dataset files: 0.18 MB
* Size of the generated dataset: 0.72 MB
* Total amount of disk used: 0.90 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_nds


* Size of downloaded dataset files: 6.74 MB
* Size of the generated dataset: 18.23 MB
* Total amount of disk used: 24.99 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ne


* Size of downloaded dataset files: 355.29 MB
* Size of the generated dataset: 1.87 GB
* Total amount of disk used: 2.22 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_new


* Size of downloaded dataset files: 1.03 MB
* Size of the generated dataset: 5.77 MB
* Total amount of disk used: 6.79 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_nl


* Size of downloaded dataset files: 29.35 GB
* Size of the generated dataset: 83.23 GB
* Total amount of disk used: 112.58 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_nn


* Size of downloaded dataset files: 32.86 MB
* Size of the generated dataset: 90.84 MB
* Total amount of disk used: 123.70 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_no


* Size of downloaded dataset files: 3.11 GB
* Size of the generated dataset: 8.65 GB
* Total amount of disk used: 11.76 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_oc


* Size of downloaded dataset files: 1.57 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.71 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_or


* Size of downloaded dataset files: 49.84 MB
* Size of the generated dataset: 260.15 MB
* Total amount of disk used: 309.99 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_os


* Size of downloaded dataset files: 3.09 MB
* Size of the generated dataset: 12.90 MB
* Total amount of disk used: 15.99 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_pa


* Size of downloaded dataset files: 164.21 MB
* Size of the generated dataset: 801.16 MB
* Total amount of disk used: 965.37 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_pl


* Size of downloaded dataset files: 42.88 GB
* Size of the generated dataset: 117.12 GB
* Total amount of disk used: 160.01 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_pms


* Size of downloaded dataset files: 0.75 MB
* Size of the generated dataset: 2.15 MB
* Total amount of disk used: 2.92 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_pnb


* Size of downloaded dataset files: 3.22 MB
* Size of the generated dataset: 12.04 MB
* Total amount of disk used: 15.26 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ps


* Size of downloaded dataset files: 103.66 MB
* Size of the generated dataset: 379.51 MB
* Total amount of disk used: 483.17 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_pt


* Size of downloaded dataset files: 47.26 GB
* Size of the generated dataset: 132.64 GB
* Total amount of disk used: 179.89 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.08 MB
* Total amount of disk used: 0.10 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ro


* Size of downloaded dataset files: 9.53 GB
* Size of the generated dataset: 26.87 GB
* Total amount of disk used: 36.40 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ru


* Size of downloaded dataset files: 319.76 GB
* Size of the generated dataset: 1241.63 GB
* Total amount of disk used: 1561.38 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_sa


* Size of downloaded dataset files: 17.52 MB
* Size of the generated dataset: 97.06 MB
* Total amount of disk used: 114.58 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_sah


* Size of downloaded dataset files: 9.08 MB
* Size of the generated dataset: 43.82 MB
* Total amount of disk used: 52.90 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_sd


* Size of downloaded dataset files: 90.62 MB
* Size of the generated dataset: 364.25 MB
* Total amount of disk used: 454.88 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_sh


* Size of downloaded dataset files: 3.46 MB
* Size of the generated dataset: 25.84 MB
* Total amount of disk used: 29.30 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_si


* Size of downloaded dataset files: 310.93 MB
* Size of the generated dataset: 1.47 GB
* Total amount of disk used: 1.78 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_sk


* Size of downloaded dataset files: 3.71 GB
* Size of the generated dataset: 9.81 GB
* Total amount of disk used: 13.52 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_sl


* Size of downloaded dataset files: 956.20 MB
* Size of the generated dataset: 2.68 GB
* Total amount of disk used: 3.63 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_sq


* Size of downloaded dataset files: 861.84 MB
* Size of the generated dataset: 2.44 GB
* Total amount of disk used: 3.30 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_sr


* Size of downloaded dataset files: 1.08 GB
* Size of the generated dataset: 4.13 GB
* Total amount of disk used: 5.21 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_su


* Size of downloaded dataset files: 0.06 MB
* Size of the generated dataset: 0.23 MB
* Total amount of disk used: 0.28 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_sv


* Size of downloaded dataset files: 17.18 GB
* Size of the generated dataset: 47.00 GB
* Total amount of disk used: 64.18 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_sw


* Size of downloaded dataset files: 3.71 MB
* Size of the generated dataset: 14.07 MB
* Total amount of disk used: 17.78 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ta


* Size of downloaded dataset files: 1.74 GB
* Size of the generated dataset: 9.93 GB
* Total amount of disk used: 11.67 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_te


* Size of downloaded dataset files: 522.47 MB
* Size of the generated dataset: 2.61 GB
* Total amount of disk used: 3.13 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_tg


* Size of downloaded dataset files: 90.97 MB
* Size of the generated dataset: 397.43 MB
* Total amount of disk used: 488.41 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_th


* Size of downloaded dataset files: 7.38 GB
* Size of the generated dataset: 38.29 GB
* Total amount of disk used: 45.67 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_tk


* Size of downloaded dataset files: 2.96 MB
* Size of the generated dataset: 10.66 MB
* Total amount of disk used: 13.62 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_tl


* Size of downloaded dataset files: 204.89 MB
* Size of the generated dataset: 606.30 MB
* Total amount of disk used: 811.19 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_tr


* Size of downloaded dataset files: 21.96 GB
* Size of the generated dataset: 63.58 GB
* Total amount of disk used: 85.54 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_tt


* Size of downloaded dataset files: 151.06 MB
* Size of the generated dataset: 703.42 MB
* Total amount of disk used: 854.47 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_ug


* Size of downloaded dataset files: 27.92 MB
* Size of the generated dataset: 127.42 MB
* Total amount of disk used: 155.35 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_uk


* Size of downloaded dataset files: 14.42 GB
* Size of the generated dataset: 56.44 GB
* Total amount of disk used: 70.86 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_ur


* Size of downloaded dataset files: 712.61 MB
* Size of the generated dataset: 2.80 GB
* Total amount of disk used: 3.51 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_uz


* Size of downloaded dataset files: 5.78 MB
* Size of the generated dataset: 21.46 MB
* Total amount of disk used: 27.24 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_vi


* Size of downloaded dataset files: 21.50 GB
* Size of the generated dataset: 72.23 GB
* Total amount of disk used: 93.73 GB


An example of 'train' looks as follows.


#### unshuffled\_original\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.12 MB
* Total amount of disk used: 2.42 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_wa


* Size of downloaded dataset files: 0.09 MB
* Size of the generated dataset: 0.29 MB
* Total amount of disk used: 0.38 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_war


* Size of downloaded dataset files: 0.64 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.32 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_xmf


* Size of downloaded dataset files: 1.05 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.17 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_yi


* Size of downloaded dataset files: 33.33 MB
* Size of the generated dataset: 147.60 MB
* Total amount of disk used: 180.94 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.


#### unshuffled\_original\_zh


* Size of downloaded dataset files: 206.00 GB
* Size of the generated dataset: 545.61 GB
* Total amount of disk used: 751.61 GB


An example of 'train' looks as follows.



### Data Fields


The data fields are the same among all configs.


* 'id': a 'int64' feature.
* 'text': a 'string' feature.


### Data Splits



Click to expand the number of samples per configuration


Dataset Creation
----------------


### Curation Rationale


OSCAR was constructed new pipeline derived from the fastText's one, called *goclassy*. Goclassy reuses the fastText linear classifier and the pre-trained fastText model for language recognition, but it completely rewrites and parallelises their pipeline in an asynchronous manner.


The order of operations is more or less the same as in the fastText pre-processing pipeline but instead of clustering multiple operations into a single blocking process, a worker is launched for each operation but bounding the number of possible parallel operations at a given time by the number of available threads instead of the number of CPUs. Goclassy is implemented in the Go programming language so it lets the Go runtime handle the scheduling of the processes. Thus the goclassy's pipeline one does not have to wait for a whole WET file to download, decompress and classify in order to start downloading and processing the next one, a new file will start downloading and processing as soon as the scheduler is able to allocate a new process.


Filtering and cleaning processes at line level are done before feeding each line to the classifier. Lines shorter than 100 UTF-8 characters and lines containing invalid UTF-8 characters are discarted and are not classified. After all files are proccesed the deduplicated versions are constructed and everything is then splitted in shards and compressed.


### Source Data


#### Initial Data Collection and Normalization


Common Crawl is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable. Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metdata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected nofollow and URL policies.


Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics.


To construct OSCAR the WET files of Common Crawl were used. These contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format and is stored on Amazon Web Services. In the case of OSCAR, the November 2018 snapshot was used. It surpasses 20TB of uncompressed data and contains more than 50 thousand plain text files where each file consists of the plain text from multiple websites along its metadata header.


#### Who are the source language producers?


The data comes from multiple web pages in a large variety of languages.


### Annotations


The dataset does not contain any additional annotations.


#### Annotation process


N/A


#### Who are the annotators?


N/A


### Personal and Sensitive Information


Being constructed from Common Crawl, Personal and sensitive information might be present. This must be considered before training deep learning models with OSCAR, specially in the case of text-generation models.


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


OSCAR is intended to bring more data to a wide variety of lanuages, the aim of the corpus is to make large amounts of data available to lower resource languages in order to facilitate the pre-training of state-of-the-art language modeling architectures.


### Discussion of Biases


OSCAR is not properly filtered yet and this can be reflected on the models trained with it. Care is advised specially concerning biases of the resulting models.


### Other Known Limitations


The fastText linear classifier is limed both in performance and the variety of languages it can recognize, so the quality of some OSCAR sub-corpora might be lower than expected, specially for the lowest-resource langiuages. Some audits have already been done by third parties.


Additional Information
----------------------


### Dataset Curators


The corpus was put together by Pedro J. Ortiz, Benoît Sagot, and Laurent Romary, during work done at Inria, particularly at the ALMAnaCH team.


### Licensing Information



```
These data are released under this licensing scheme
We do not own any of the text from which these data has been extracted.
We license the actual packaging of these data under the Creative Commons CC0 license ("no rights reserved") URL
To the extent possible under law, Inria has waived all copyright and related or neighboring rights to OSCAR
This work is published from: France.

Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
* Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
* Clearly identify the copyrighted work claimed to be infringed.
* Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

We will comply to legitimate requests by removing the affected sources from the next release of the corpus.

```

### Contributions


Thanks to @pjox and @lhoestq for adding this dataset.



### Dataset Summary


OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture. Data is distributed by language in both original and deduplicated form.


The version here is the original OSCAR 2019 release: URL


For more recent versions, visit the oscar-corpus organization on the Hub:


* OSCAR 22.01 (released in January 2022): oscar-corpus/OSCAR-2201
* OSCAR 21.09 (released in September 2021): oscar-corpus/OSCAR-2109 ### Supported Tasks and Leaderboards


OSCAR is mainly inteded to pretrain language models and word represantations. ### Languages


All the data is distributed by language, both the original and the deduplicated versions of the data are available. 166 different languages are available. The table in subsection Data Splits Sample Size provides the language code for each subcorpus as well as the number of words (space separated tokens), lines and sizes for both the original and the deduplicated versions of OSCAR.


Dataset Structure
-----------------


We show detailed information for all the configurations of the dataset. ### Data Instances



Click to expand the Data/size information for each language (deduplicated) #### unshuffled\_deduplicated\_af


* Size of downloaded dataset files: 65.99 MB
* Size of the generated dataset: 172.30 MB
* Total amount of disk used: 238.29 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_als


* Size of downloaded dataset files: 1.26 MB
* Size of the generated dataset: 2.96 MB
* Total amount of disk used: 4.22 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_am


* Size of downloaded dataset files: 61.35 MB
* Size of the generated dataset: 216.15 MB
* Total amount of disk used: 277.50 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_an


* Size of downloaded dataset files: 0.14 MB
* Size of the generated dataset: 0.85 MB
* Total amount of disk used: 0.99 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ar


* Size of downloaded dataset files: 9.67 GB
* Size of the generated dataset: 33.57 GB
* Total amount of disk used: 43.23 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_arz


* Size of downloaded dataset files: 10.02 MB
* Size of the generated dataset: 35.91 MB
* Total amount of disk used: 45.94 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_as


* Size of downloaded dataset files: 15.51 MB
* Size of the generated dataset: 74.07 MB
* Total amount of disk used: 89.58 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ast


* Size of downloaded dataset files: 0.86 MB
* Size of the generated dataset: 2.17 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_av


* Size of downloaded dataset files: 0.07 MB
* Size of the generated dataset: 0.34 MB
* Total amount of disk used: 0.41 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_az


* Size of downloaded dataset files: 521.74 MB
* Size of the generated dataset: 1.53 GB
* Total amount of disk used: 2.05 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_azb


* Size of downloaded dataset files: 5.19 MB
* Size of the generated dataset: 20.08 MB
* Total amount of disk used: 25.27 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ba


* Size of downloaded dataset files: 25.98 MB
* Size of the generated dataset: 93.84 MB
* Total amount of disk used: 119.82 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_be


* Size of downloaded dataset files: 306.70 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.39 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bg


* Size of downloaded dataset files: 3.85 GB
* Size of the generated dataset: 14.45 GB
* Total amount of disk used: 18.30 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bn


* Size of downloaded dataset files: 1.26 GB
* Size of the generated dataset: 6.24 GB
* Total amount of disk used: 7.50 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bo


* Size of downloaded dataset files: 22.37 MB
* Size of the generated dataset: 144.65 MB
* Total amount of disk used: 167.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bpy


* Size of downloaded dataset files: 0.19 MB
* Size of the generated dataset: 1.78 MB
* Total amount of disk used: 1.97 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_br


* Size of downloaded dataset files: 6.47 MB
* Size of the generated dataset: 17.00 MB
* Total amount of disk used: 23.47 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bs


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.15 MB
* Total amount of disk used: 0.18 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ca


* Size of downloaded dataset files: 1.73 GB
* Size of the generated dataset: 4.57 GB
* Total amount of disk used: 6.30 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ce


* Size of downloaded dataset files: 1.87 MB
* Size of the generated dataset: 7.04 MB
* Total amount of disk used: 8.90 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ceb


* Size of downloaded dataset files: 7.12 MB
* Size of the generated dataset: 24.83 MB
* Total amount of disk used: 31.95 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ckb


* Size of downloaded dataset files: 60.32 MB
* Size of the generated dataset: 237.72 MB
* Total amount of disk used: 298.05 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cs


* Size of downloaded dataset files: 10.49 GB
* Size of the generated dataset: 25.71 GB
* Total amount of disk used: 36.20 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cv


* Size of downloaded dataset files: 7.47 MB
* Size of the generated dataset: 27.49 MB
* Total amount of disk used: 34.95 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cy


* Size of downloaded dataset files: 53.63 MB
* Size of the generated dataset: 141.22 MB
* Total amount of disk used: 194.86 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_da


* Size of downloaded dataset files: 3.82 GB
* Size of the generated dataset: 10.24 GB
* Total amount of disk used: 14.06 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_de


* Size of downloaded dataset files: 60.80 GB
* Size of the generated dataset: 156.30 GB
* Total amount of disk used: 217.10 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_dv


* Size of downloaded dataset files: 16.84 MB
* Size of the generated dataset: 82.19 MB
* Total amount of disk used: 99.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_el


* Size of downloaded dataset files: 7.91 GB
* Size of the generated dataset: 28.74 GB
* Total amount of disk used: 36.65 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_en


* Size of downloaded dataset files: 496.50 GB
* Size of the generated dataset: 1299.75 GB
* Total amount of disk used: 1796.24 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_eo


* Size of downloaded dataset files: 92.86 MB
* Size of the generated dataset: 240.12 MB
* Total amount of disk used: 332.99 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_es


* Size of downloaded dataset files: 60.46 GB
* Size of the generated dataset: 160.86 GB
* Total amount of disk used: 221.32 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_et


* Size of downloaded dataset files: 966.79 MB
* Size of the generated dataset: 2.45 GB
* Total amount of disk used: 3.41 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_eu


* Size of downloaded dataset files: 134.68 MB
* Size of the generated dataset: 363.93 MB
* Total amount of disk used: 498.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fa


* Size of downloaded dataset files: 10.46 GB
* Size of the generated dataset: 40.06 GB
* Total amount of disk used: 50.52 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fi


* Size of downloaded dataset files: 5.38 GB
* Size of the generated dataset: 13.99 GB
* Total amount of disk used: 19.37 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fr


* Size of downloaded dataset files: 55.46 GB
* Size of the generated dataset: 148.28 GB
* Total amount of disk used: 203.75 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fy


* Size of downloaded dataset files: 10.27 MB
* Size of the generated dataset: 26.73 MB
* Total amount of disk used: 37.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ga


* Size of downloaded dataset files: 22.22 MB
* Size of the generated dataset: 63.86 MB
* Total amount of disk used: 86.08 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gd


* Size of downloaded dataset files: 0.42 MB
* Size of the generated dataset: 1.36 MB
* Total amount of disk used: 1.78 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gl


* Size of downloaded dataset files: 155.85 MB
* Size of the generated dataset: 408.34 MB
* Total amount of disk used: 564.19 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gom


* Size of downloaded dataset files: 0.38 MB
* Size of the generated dataset: 1.87 MB
* Total amount of disk used: 2.24 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gu


* Size of downloaded dataset files: 162.97 MB
* Size of the generated dataset: 759.34 MB
* Total amount of disk used: 922.32 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_he


* Size of downloaded dataset files: 3.04 GB
* Size of the generated dataset: 10.47 GB
* Total amount of disk used: 13.51 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hi


* Size of downloaded dataset files: 2.01 GB
* Size of the generated dataset: 9.57 GB
* Total amount of disk used: 11.58 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hr


* Size of downloaded dataset files: 46.74 MB
* Size of the generated dataset: 121.50 MB
* Total amount of disk used: 168.23 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hsb


* Size of downloaded dataset files: 0.72 MB
* Size of the generated dataset: 1.89 MB
* Total amount of disk used: 2.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hu


* Size of downloaded dataset files: 7.37 GB
* Size of the generated dataset: 19.09 GB
* Total amount of disk used: 26.46 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hy


* Size of downloaded dataset files: 393.62 MB
* Size of the generated dataset: 1.56 GB
* Total amount of disk used: 1.96 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ia


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.38 MB
* Total amount of disk used: 0.43 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_id


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 17.05 GB
* Total amount of disk used: 23.05 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ilo


* Size of downloaded dataset files: 0.23 MB
* Size of the generated dataset: 0.68 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.14 MB
* Total amount of disk used: 0.19 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_is


* Size of downloaded dataset files: 332.87 MB
* Size of the generated dataset: 894.28 MB
* Total amount of disk used: 1.23 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_it


* Size of downloaded dataset files: 27.93 GB
* Size of the generated dataset: 74.09 GB
* Total amount of disk used: 102.03 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ja


* Size of downloaded dataset files: 40.80 GB
* Size of the generated dataset: 113.63 GB
* Total amount of disk used: 154.44 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_jbo


* Size of downloaded dataset files: 0.20 MB
* Size of the generated dataset: 0.70 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_jv


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.62 MB
* Total amount of disk used: 0.82 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ka


* Size of downloaded dataset files: 377.23 MB
* Size of the generated dataset: 1.99 GB
* Total amount of disk used: 2.36 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kk


* Size of downloaded dataset files: 389.12 MB
* Size of the generated dataset: 1.59 GB
* Total amount of disk used: 1.97 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_km


* Size of downloaded dataset files: 114.48 MB
* Size of the generated dataset: 610.61 MB
* Total amount of disk used: 725.09 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kn


* Size of downloaded dataset files: 215.52 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ko


* Size of downloaded dataset files: 4.46 GB
* Size of the generated dataset: 12.00 GB
* Total amount of disk used: 16.47 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_krc


* Size of downloaded dataset files: 0.62 MB
* Size of the generated dataset: 2.41 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ku


* Size of downloaded dataset files: 23.34 MB
* Size of the generated dataset: 63.09 MB
* Total amount of disk used: 86.43 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kv


* Size of downloaded dataset files: 0.33 MB
* Size of the generated dataset: 1.21 MB
* Total amount of disk used: 1.54 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ky


* Size of downloaded dataset files: 106.22 MB
* Size of the generated dataset: 408.40 MB
* Total amount of disk used: 514.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_la


* Size of downloaded dataset files: 3.42 MB
* Size of the generated dataset: 9.79 MB
* Total amount of disk used: 13.22 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lb


* Size of downloaded dataset files: 8.30 MB
* Size of the generated dataset: 21.42 MB
* Total amount of disk used: 29.72 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lez


* Size of downloaded dataset files: 0.77 MB
* Size of the generated dataset: 3.08 MB
* Total amount of disk used: 3.84 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.46 MB
* Total amount of disk used: 0.57 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lo


* Size of downloaded dataset files: 23.63 MB
* Size of the generated dataset: 119.29 MB
* Total amount of disk used: 142.92 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.08 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lt


* Size of downloaded dataset files: 1.65 GB
* Size of the generated dataset: 4.20 GB
* Total amount of disk used: 5.86 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lv


* Size of downloaded dataset files: 710.45 MB
* Size of the generated dataset: 1.91 GB
* Total amount of disk used: 2.62 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mai


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mg


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 13.59 MB
* Total amount of disk used: 17.89 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mhr


* Size of downloaded dataset files: 1.63 MB
* Size of the generated dataset: 6.26 MB
* Total amount of disk used: 7.89 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.31 MB
* Total amount of disk used: 0.33 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mk


* Size of downloaded dataset files: 303.12 MB
* Size of the generated dataset: 1.19 GB
* Total amount of disk used: 1.49 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ml


* Size of downloaded dataset files: 496.80 MB
* Size of the generated dataset: 2.69 GB
* Total amount of disk used: 3.18 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mn


* Size of downloaded dataset files: 219.52 MB
* Size of the generated dataset: 883.46 MB
* Total amount of disk used: 1.10 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mr


* Size of downloaded dataset files: 299.68 MB
* Size of the generated dataset: 1.49 GB
* Total amount of disk used: 1.79 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mrj


* Size of downloaded dataset files: 0.29 MB
* Size of the generated dataset: 1.10 MB
* Total amount of disk used: 1.38 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ms


* Size of downloaded dataset files: 16.39 MB
* Size of the generated dataset: 49.45 MB
* Total amount of disk used: 65.85 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mt


* Size of downloaded dataset files: 5.90 MB
* Size of the generated dataset: 17.68 MB
* Total amount of disk used: 23.58 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_my


* Size of downloaded dataset files: 207.14 MB
* Size of the generated dataset: 1.11 GB
* Total amount of disk used: 1.32 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mzn


* Size of downloaded dataset files: 0.16 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.79 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nds


* Size of downloaded dataset files: 5.27 MB
* Size of the generated dataset: 13.48 MB
* Total amount of disk used: 18.76 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ne


* Size of downloaded dataset files: 240.63 MB
* Size of the generated dataset: 1.24 GB
* Total amount of disk used: 1.48 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_new


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 4.26 MB
* Total amount of disk used: 5.09 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nl


* Size of downloaded dataset files: 15.73 GB
* Size of the generated dataset: 41.91 GB
* Total amount of disk used: 57.65 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nn


* Size of downloaded dataset files: 23.58 MB
* Size of the generated dataset: 58.32 MB
* Total amount of disk used: 81.90 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_no


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 5.11 GB
* Total amount of disk used: 7.07 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_oc


* Size of downloaded dataset files: 1.34 MB
* Size of the generated dataset: 4.00 MB
* Total amount of disk used: 5.34 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_or


* Size of downloaded dataset files: 38.72 MB
* Size of the generated dataset: 197.63 MB
* Total amount of disk used: 236.36 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_os


* Size of downloaded dataset files: 2.83 MB
* Size of the generated dataset: 11.00 MB
* Total amount of disk used: 13.83 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pa


* Size of downloaded dataset files: 102.39 MB
* Size of the generated dataset: 483.04 MB
* Total amount of disk used: 585.42 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pl


* Size of downloaded dataset files: 20.19 GB
* Size of the generated dataset: 50.59 GB
* Total amount of disk used: 70.78 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pms


* Size of downloaded dataset files: 0.71 MB
* Size of the generated dataset: 2.00 MB
* Total amount of disk used: 2.72 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pnb


* Size of downloaded dataset files: 2.58 MB
* Size of the generated dataset: 9.44 MB
* Total amount of disk used: 12.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ps


* Size of downloaded dataset files: 71.83 MB
* Size of the generated dataset: 254.79 MB
* Total amount of disk used: 326.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pt


* Size of downloaded dataset files: 26.00 GB
* Size of the generated dataset: 68.37 GB
* Total amount of disk used: 94.37 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ro


* Size of downloaded dataset files: 4.48 GB
* Size of the generated dataset: 11.66 GB
* Total amount of disk used: 16.14 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ru


* Size of downloaded dataset files: 166.68 GB
* Size of the generated dataset: 611.70 GB
* Total amount of disk used: 778.38 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sa


* Size of downloaded dataset files: 7.27 MB
* Size of the generated dataset: 38.33 MB
* Total amount of disk used: 45.60 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sah


* Size of downloaded dataset files: 7.01 MB
* Size of the generated dataset: 27.46 MB
* Total amount of disk used: 34.49 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sd


* Size of downloaded dataset files: 74.17 MB
* Size of the generated dataset: 275.48 MB
* Total amount of disk used: 349.66 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sh


* Size of downloaded dataset files: 1.45 MB
* Size of the generated dataset: 6.44 MB
* Total amount of disk used: 7.87 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_si


* Size of downloaded dataset files: 175.62 MB
* Size of the generated dataset: 842.57 MB
* Total amount of disk used: 1.02 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sk


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 4.80 GB
* Total amount of disk used: 6.76 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sl


* Size of downloaded dataset files: 523.22 MB
* Size of the generated dataset: 1.32 GB
* Total amount of disk used: 1.85 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sq


* Size of downloaded dataset files: 445.36 MB
* Size of the generated dataset: 1.21 GB
* Total amount of disk used: 1.66 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sr


* Size of downloaded dataset files: 665.03 MB
* Size of the generated dataset: 2.36 GB
* Total amount of disk used: 3.03 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_su


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.21 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sv


* Size of downloaded dataset files: 10.19 GB
* Size of the generated dataset: 26.33 GB
* Total amount of disk used: 36.51 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sw


* Size of downloaded dataset files: 2.95 MB
* Size of the generated dataset: 8.98 MB
* Total amount of disk used: 11.92 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ta


* Size of downloaded dataset files: 971.12 MB
* Size of the generated dataset: 5.48 GB
* Total amount of disk used: 6.45 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_te


* Size of downloaded dataset files: 342.43 MB
* Size of the generated dataset: 1.70 GB
* Total amount of disk used: 2.04 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tg


* Size of downloaded dataset files: 62.90 MB
* Size of the generated dataset: 261.68 MB
* Total amount of disk used: 324.60 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_th


* Size of downloaded dataset files: 3.54 GB
* Size of the generated dataset: 17.11 GB
* Total amount of disk used: 20.65 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tk


* Size of downloaded dataset files: 2.22 MB
* Size of the generated dataset: 7.12 MB
* Total amount of disk used: 9.34 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tl


* Size of downloaded dataset files: 151.34 MB
* Size of the generated dataset: 431.69 MB
* Total amount of disk used: 583.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tr


* Size of downloaded dataset files: 10.39 GB
* Size of the generated dataset: 28.47 GB
* Total amount of disk used: 38.86 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tt


* Size of downloaded dataset files: 85.89 MB
* Size of the generated dataset: 321.37 MB
* Total amount of disk used: 407.26 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ug


* Size of downloaded dataset files: 20.53 MB
* Size of the generated dataset: 86.44 MB
* Total amount of disk used: 106.97 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_uk


* Size of downloaded dataset files: 8.04 GB
* Size of the generated dataset: 29.86 GB
* Total amount of disk used: 37.90 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ur


* Size of downloaded dataset files: 483.59 MB
* Size of the generated dataset: 1.82 GB
* Total amount of disk used: 2.31 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_uz


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 12.00 MB
* Total amount of disk used: 16.29 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_vi


* Size of downloaded dataset files: 10.71 GB
* Size of the generated dataset: 33.60 GB
* Total amount of disk used: 44.31 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.10 MB
* Total amount of disk used: 2.40 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_wa


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.22 MB
* Total amount of disk used: 0.29 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_war


* Size of downloaded dataset files: 0.55 MB
* Size of the generated dataset: 2.36 MB
* Total amount of disk used: 2.90 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_xmf


* Size of downloaded dataset files: 0.94 MB
* Size of the generated dataset: 4.63 MB
* Total amount of disk used: 5.58 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_yi


* Size of downloaded dataset files: 22.20 MB
* Size of the generated dataset: 88.29 MB
* Total amount of disk used: 110.49 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_zh


* Size of downloaded dataset files: 99.98 GB
* Size of the generated dataset: 267.88 GB
* Total amount of disk used: 367.86 GB


An example of 'train' looks as follows.




Click to expand the Data/size information for each language (original) #### unshuffled\_original\_af


* Size of downloaded dataset files: 85.79 MB
* Size of the generated dataset: 254.08 MB
* Total amount of disk used: 339.87 MB


An example of 'train' looks as follows. #### unshuffled\_original\_als


* Size of downloaded dataset files: 1.49 MB
* Size of the generated dataset: 5.30 MB
* Total amount of disk used: 6.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_am


* Size of downloaded dataset files: 102.79 MB
* Size of the generated dataset: 378.06 MB
* Total amount of disk used: 480.85 MB


An example of 'train' looks as follows. #### unshuffled\_original\_an


* Size of downloaded dataset files: 0.15 MB
* Size of the generated dataset: 1.33 MB
* Total amount of disk used: 1.48 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ar


* Size of downloaded dataset files: 22.23 GB
* Size of the generated dataset: 87.94 GB
* Total amount of disk used: 110.17 GB


An example of 'train' looks as follows. #### unshuffled\_original\_arz


* Size of downloaded dataset files: 15.90 MB
* Size of the generated dataset: 70.13 MB
* Total amount of disk used: 86.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_as


* Size of downloaded dataset files: 21.43 MB
* Size of the generated dataset: 117.73 MB
* Total amount of disk used: 139.17 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ast


* Size of downloaded dataset files: 0.92 MB
* Size of the generated dataset: 2.54 MB
* Total amount of disk used: 3.46 MB


An example of 'train' looks as follows. #### unshuffled\_original\_av


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.42 MB
* Total amount of disk used: 0.50 MB


An example of 'train' looks as follows. #### unshuffled\_original\_az


* Size of downloaded dataset files: 927.76 MB
* Size of the generated dataset: 2.96 GB
* Total amount of disk used: 3.89 GB


An example of 'train' looks as follows. #### unshuffled\_original\_azb


* Size of downloaded dataset files: 6.64 MB
* Size of the generated dataset: 28.47 MB
* Total amount of disk used: 35.11 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ba


* Size of downloaded dataset files: 33.22 MB
* Size of the generated dataset: 133.70 MB
* Total amount of disk used: 166.92 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_be


* Size of downloaded dataset files: 498.29 MB
* Size of the generated dataset: 1.88 GB
* Total amount of disk used: 2.38 GB


An example of 'train' looks as follows. #### unshuffled\_original\_bg


* Size of downloaded dataset files: 8.34 GB
* Size of the generated dataset: 33.75 GB
* Total amount of disk used: 42.09 GB


An example of 'train' looks as follows. #### unshuffled\_original\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bn


* Size of downloaded dataset files: 2.14 GB
* Size of the generated dataset: 10.77 GB
* Total amount of disk used: 12.91 GB


An example of 'train' looks as follows. #### unshuffled\_original\_bo


* Size of downloaded dataset files: 28.94 MB
* Size of the generated dataset: 195.40 MB
* Total amount of disk used: 224.34 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bpy


* Size of downloaded dataset files: 0.34 MB
* Size of the generated dataset: 4.35 MB
* Total amount of disk used: 4.69 MB


An example of 'train' looks as follows. #### unshuffled\_original\_br


* Size of downloaded dataset files: 9.18 MB
* Size of the generated dataset: 30.20 MB
* Total amount of disk used: 39.38 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bs


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.48 MB
* Total amount of disk used: 0.53 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ca


* Size of downloaded dataset files: 3.10 GB
* Size of the generated dataset: 8.62 GB
* Total amount of disk used: 11.73 GB


An example of 'train' looks as follows. #### unshuffled\_original\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ce


* Size of downloaded dataset files: 2.09 MB
* Size of the generated dataset: 8.73 MB
* Total amount of disk used: 10.82 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ceb


* Size of downloaded dataset files: 11.07 MB
* Size of the generated dataset: 40.97 MB
* Total amount of disk used: 52.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ckb


* Size of downloaded dataset files: 111.88 MB
* Size of the generated dataset: 510.97 MB
* Total amount of disk used: 622.85 MB


An example of 'train' looks as follows. #### unshuffled\_original\_cs


* Size of downloaded dataset files: 21.72 GB
* Size of the generated dataset: 57.08 GB
* Total amount of disk used: 78.80 GB


An example of 'train' looks as follows. #### unshuffled\_original\_cv


* Size of downloaded dataset files: 9.40 MB
* Size of the generated dataset: 41.05 MB
* Total amount of disk used: 50.45 MB


An example of 'train' looks as follows. #### unshuffled\_original\_cy


* Size of downloaded dataset files: 81.74 MB
* Size of the generated dataset: 224.93 MB
* Total amount of disk used: 306.67 MB


An example of 'train' looks as follows. #### unshuffled\_original\_da


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 16.76 GB
* Total amount of disk used: 22.76 GB


An example of 'train' looks as follows. #### unshuffled\_original\_de


* Size of downloaded dataset files: 119.51 GB
* Size of the generated dataset: 331.22 GB
* Total amount of disk used: 450.73 GB


An example of 'train' looks as follows. #### unshuffled\_original\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_dv


* Size of downloaded dataset files: 24.91 MB
* Size of the generated dataset: 131.63 MB
* Total amount of disk used: 156.54 MB


An example of 'train' looks as follows. #### unshuffled\_original\_el


* Size of downloaded dataset files: 17.31 GB
* Size of the generated dataset: 66.27 GB
* Total amount of disk used: 83.58 GB


An example of 'train' looks as follows. #### unshuffled\_original\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_en


* Size of downloaded dataset files: 903.83 GB
* Size of the generated dataset: 2525.44 GB
* Total amount of disk used: 3429.27 GB


An example of 'train' looks as follows. #### unshuffled\_original\_eo


* Size of downloaded dataset files: 117.07 MB
* Size of the generated dataset: 314.18 MB
* Total amount of disk used: 431.27 MB


An example of 'train' looks as follows. #### unshuffled\_original\_es


* Size of downloaded dataset files: 106.04 GB
* Size of the generated dataset: 298.49 GB
* Total amount of disk used: 404.53 GB


An example of 'train' looks as follows. #### unshuffled\_original\_et


* Size of downloaded dataset files: 1.88 GB
* Size of the generated dataset: 5.17 GB
* Total amount of disk used: 7.06 GB


An example of 'train' looks as follows. #### unshuffled\_original\_eu


* Size of downloaded dataset files: 248.19 MB
* Size of the generated dataset: 894.83 MB
* Total amount of disk used: 1.14 GB


An example of 'train' looks as follows. #### unshuffled\_original\_fa


* Size of downloaded dataset files: 20.96 GB
* Size of the generated dataset: 84.21 GB
* Total amount of disk used: 105.17 GB


An example of 'train' looks as follows. #### unshuffled\_original\_fi


* Size of downloaded dataset files: 9.97 GB
* Size of the generated dataset: 28.57 GB
* Total amount of disk used: 38.54 GB


An example of 'train' looks as follows. #### unshuffled\_original\_fr


* Size of downloaded dataset files: 105.32 GB
* Size of the generated dataset: 303.19 GB
* Total amount of disk used: 408.51 GB


An example of 'train' looks as follows. #### unshuffled\_original\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_fy


* Size of downloaded dataset files: 12.40 MB
* Size of the generated dataset: 36.24 MB
* Total amount of disk used: 48.64 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ga


* Size of downloaded dataset files: 29.27 MB
* Size of the generated dataset: 92.37 MB
* Total amount of disk used: 121.63 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gd


* Size of downloaded dataset files: 0.52 MB
* Size of the generated dataset: 2.02 MB
* Total amount of disk used: 2.55 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gl


* Size of downloaded dataset files: 235.38 MB
* Size of the generated dataset: 656.48 MB
* Total amount of disk used: 891.87 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gom


* Size of downloaded dataset files: 0.44 MB
* Size of the generated dataset: 2.25 MB
* Total amount of disk used: 2.71 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gu


* Size of downloaded dataset files: 232.02 MB
* Size of the generated dataset: 1.09 GB
* Total amount of disk used: 1.33 GB


An example of 'train' looks as follows. #### unshuffled\_original\_he


* Size of downloaded dataset files: 5.66 GB
* Size of the generated dataset: 21.11 GB
* Total amount of disk used: 26.77 GB


An example of 'train' looks as follows. #### unshuffled\_original\_hi


* Size of downloaded dataset files: 3.66 GB
* Size of the generated dataset: 17.93 GB
* Total amount of disk used: 21.59 GB


An example of 'train' looks as follows. #### unshuffled\_original\_hr


* Size of downloaded dataset files: 79.42 MB
* Size of the generated dataset: 243.83 MB
* Total amount of disk used: 323.24 MB


An example of 'train' looks as follows. #### unshuffled\_original\_hsb


* Size of downloaded dataset files: 1.39 MB
* Size of the generated dataset: 4.49 MB
* Total amount of disk used: 5.87 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_hu


* Size of downloaded dataset files: 15.69 GB
* Size of the generated dataset: 43.07 GB
* Total amount of disk used: 58.77 GB


An example of 'train' looks as follows. #### unshuffled\_original\_hy


* Size of downloaded dataset files: 897.36 MB
* Size of the generated dataset: 3.94 GB
* Total amount of disk used: 4.84 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ia


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_id


* Size of downloaded dataset files: 10.60 GB
* Size of the generated dataset: 32.32 GB
* Total amount of disk used: 42.91 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ilo


* Size of downloaded dataset files: 0.27 MB
* Size of the generated dataset: 0.92 MB
* Total amount of disk used: 1.20 MB


An example of 'train' looks as follows. #### unshuffled\_original\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.20 MB


An example of 'train' looks as follows. #### unshuffled\_original\_is


* Size of downloaded dataset files: 533.03 MB
* Size of the generated dataset: 1.52 GB
* Total amount of disk used: 2.06 GB


An example of 'train' looks as follows. #### unshuffled\_original\_it


* Size of downloaded dataset files: 52.16 GB
* Size of the generated dataset: 147.38 GB
* Total amount of disk used: 199.54 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ja


* Size of downloaded dataset files: 79.56 GB
* Size of the generated dataset: 232.22 GB
* Total amount of disk used: 311.78 GB


An example of 'train' looks as follows. #### unshuffled\_original\_jbo


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.77 MB
* Total amount of disk used: 0.98 MB


An example of 'train' looks as follows. #### unshuffled\_original\_jv


* Size of downloaded dataset files: 0.22 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ka


* Size of downloaded dataset files: 680.74 MB
* Size of the generated dataset: 3.77 GB
* Total amount of disk used: 4.45 GB


An example of 'train' looks as follows. #### unshuffled\_original\_kk


* Size of downloaded dataset files: 615.06 MB
* Size of the generated dataset: 2.83 GB
* Total amount of disk used: 3.45 GB


An example of 'train' looks as follows. #### unshuffled\_original\_km


* Size of downloaded dataset files: 193.28 MB
* Size of the generated dataset: 1.10 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows. #### unshuffled\_original\_kn


* Size of downloaded dataset files: 342.15 MB
* Size of the generated dataset: 1.76 GB
* Total amount of disk used: 2.11 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ko


* Size of downloaded dataset files: 8.81 GB
* Size of the generated dataset: 25.29 GB
* Total amount of disk used: 34.10 GB


An example of 'train' looks as follows. #### unshuffled\_original\_krc


* Size of downloaded dataset files: 0.66 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.34 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ku


* Size of downloaded dataset files: 33.38 MB
* Size of the generated dataset: 99.06 MB
* Total amount of disk used: 132.44 MB


An example of 'train' looks as follows. #### unshuffled\_original\_kv


* Size of downloaded dataset files: 0.40 MB
* Size of the generated dataset: 2.38 MB
* Total amount of disk used: 2.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ky


* Size of downloaded dataset files: 152.64 MB
* Size of the generated dataset: 630.79 MB
* Total amount of disk used: 783.43 MB


An example of 'train' looks as follows. #### unshuffled\_original\_la


* Size of downloaded dataset files: 5.46 MB
* Size of the generated dataset: 27.80 MB
* Total amount of disk used: 33.26 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lb


* Size of downloaded dataset files: 10.73 MB
* Size of the generated dataset: 30.60 MB
* Total amount of disk used: 41.32 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lez


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 3.38 MB
* Total amount of disk used: 4.20 MB


An example of 'train' looks as follows. #### unshuffled\_original\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.47 MB
* Total amount of disk used: 0.58 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lo


* Size of downloaded dataset files: 33.92 MB
* Size of the generated dataset: 182.36 MB
* Total amount of disk used: 216.28 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lt


* Size of downloaded dataset files: 3.44 GB
* Size of the generated dataset: 9.45 GB
* Total amount of disk used: 12.89 GB


An example of 'train' looks as follows. #### unshuffled\_original\_lv


* Size of downloaded dataset files: 1.49 GB
* Size of the generated dataset: 4.27 GB
* Total amount of disk used: 5.75 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mai


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.33 MB
* Total amount of disk used: 0.34 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mg


* Size of downloaded dataset files: 6.22 MB
* Size of the generated dataset: 21.79 MB
* Total amount of disk used: 28.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mhr


* Size of downloaded dataset files: 1.84 MB
* Size of the generated dataset: 7.55 MB
* Total amount of disk used: 9.38 MB


An example of 'train' looks as follows. #### unshuffled\_original\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.64 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mk


* Size of downloaded dataset files: 508.24 MB
* Size of the generated dataset: 2.20 GB
* Total amount of disk used: 2.71 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ml


* Size of downloaded dataset files: 938.69 MB
* Size of the generated dataset: 5.24 GB
* Total amount of disk used: 6.18 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mn


* Size of downloaded dataset files: 472.36 MB
* Size of the generated dataset: 2.33 GB
* Total amount of disk used: 2.81 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mr


* Size of downloaded dataset files: 525.31 MB
* Size of the generated dataset: 2.82 GB
* Total amount of disk used: 3.34 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mrj


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 1.16 MB
* Total amount of disk used: 1.47 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ms


* Size of downloaded dataset files: 28.46 MB
* Size of the generated dataset: 122.33 MB
* Total amount of disk used: 150.79 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mt


* Size of downloaded dataset files: 7.53 MB
* Size of the generated dataset: 24.47 MB
* Total amount of disk used: 32.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_my


* Size of downloaded dataset files: 369.85 MB
* Size of the generated dataset: 2.02 GB
* Total amount of disk used: 2.39 GB


An example of 'train' looks as follows. #### unshuffled\_original\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mzn


* Size of downloaded dataset files: 0.18 MB
* Size of the generated dataset: 0.72 MB
* Total amount of disk used: 0.90 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nds


* Size of downloaded dataset files: 6.74 MB
* Size of the generated dataset: 18.23 MB
* Total amount of disk used: 24.99 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ne


* Size of downloaded dataset files: 355.29 MB
* Size of the generated dataset: 1.87 GB
* Total amount of disk used: 2.22 GB


An example of 'train' looks as follows. #### unshuffled\_original\_new


* Size of downloaded dataset files: 1.03 MB
* Size of the generated dataset: 5.77 MB
* Total amount of disk used: 6.79 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nl


* Size of downloaded dataset files: 29.35 GB
* Size of the generated dataset: 83.23 GB
* Total amount of disk used: 112.58 GB


An example of 'train' looks as follows. #### unshuffled\_original\_nn


* Size of downloaded dataset files: 32.86 MB
* Size of the generated dataset: 90.84 MB
* Total amount of disk used: 123.70 MB


An example of 'train' looks as follows. #### unshuffled\_original\_no


* Size of downloaded dataset files: 3.11 GB
* Size of the generated dataset: 8.65 GB
* Total amount of disk used: 11.76 GB


An example of 'train' looks as follows. #### unshuffled\_original\_oc


* Size of downloaded dataset files: 1.57 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.71 MB


An example of 'train' looks as follows. #### unshuffled\_original\_or


* Size of downloaded dataset files: 49.84 MB
* Size of the generated dataset: 260.15 MB
* Total amount of disk used: 309.99 MB


An example of 'train' looks as follows. #### unshuffled\_original\_os


* Size of downloaded dataset files: 3.09 MB
* Size of the generated dataset: 12.90 MB
* Total amount of disk used: 15.99 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pa


* Size of downloaded dataset files: 164.21 MB
* Size of the generated dataset: 801.16 MB
* Total amount of disk used: 965.37 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pl


* Size of downloaded dataset files: 42.88 GB
* Size of the generated dataset: 117.12 GB
* Total amount of disk used: 160.01 GB


An example of 'train' looks as follows. #### unshuffled\_original\_pms


* Size of downloaded dataset files: 0.75 MB
* Size of the generated dataset: 2.15 MB
* Total amount of disk used: 2.92 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pnb


* Size of downloaded dataset files: 3.22 MB
* Size of the generated dataset: 12.04 MB
* Total amount of disk used: 15.26 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ps


* Size of downloaded dataset files: 103.66 MB
* Size of the generated dataset: 379.51 MB
* Total amount of disk used: 483.17 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pt


* Size of downloaded dataset files: 47.26 GB
* Size of the generated dataset: 132.64 GB
* Total amount of disk used: 179.89 GB


An example of 'train' looks as follows. #### unshuffled\_original\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.08 MB
* Total amount of disk used: 0.10 MB


An example of 'train' looks as follows. #### unshuffled\_original\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ro


* Size of downloaded dataset files: 9.53 GB
* Size of the generated dataset: 26.87 GB
* Total amount of disk used: 36.40 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ru


* Size of downloaded dataset files: 319.76 GB
* Size of the generated dataset: 1241.63 GB
* Total amount of disk used: 1561.38 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sa


* Size of downloaded dataset files: 17.52 MB
* Size of the generated dataset: 97.06 MB
* Total amount of disk used: 114.58 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sah


* Size of downloaded dataset files: 9.08 MB
* Size of the generated dataset: 43.82 MB
* Total amount of disk used: 52.90 MB


An example of 'train' looks as follows. #### unshuffled\_original\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sd


* Size of downloaded dataset files: 90.62 MB
* Size of the generated dataset: 364.25 MB
* Total amount of disk used: 454.88 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sh


* Size of downloaded dataset files: 3.46 MB
* Size of the generated dataset: 25.84 MB
* Total amount of disk used: 29.30 MB


An example of 'train' looks as follows. #### unshuffled\_original\_si


* Size of downloaded dataset files: 310.93 MB
* Size of the generated dataset: 1.47 GB
* Total amount of disk used: 1.78 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sk


* Size of downloaded dataset files: 3.71 GB
* Size of the generated dataset: 9.81 GB
* Total amount of disk used: 13.52 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sl


* Size of downloaded dataset files: 956.20 MB
* Size of the generated dataset: 2.68 GB
* Total amount of disk used: 3.63 GB


An example of 'train' looks as follows. #### unshuffled\_original\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sq


* Size of downloaded dataset files: 861.84 MB
* Size of the generated dataset: 2.44 GB
* Total amount of disk used: 3.30 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sr


* Size of downloaded dataset files: 1.08 GB
* Size of the generated dataset: 4.13 GB
* Total amount of disk used: 5.21 GB


An example of 'train' looks as follows. #### unshuffled\_original\_su


* Size of downloaded dataset files: 0.06 MB
* Size of the generated dataset: 0.23 MB
* Total amount of disk used: 0.28 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sv


* Size of downloaded dataset files: 17.18 GB
* Size of the generated dataset: 47.00 GB
* Total amount of disk used: 64.18 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sw


* Size of downloaded dataset files: 3.71 MB
* Size of the generated dataset: 14.07 MB
* Total amount of disk used: 17.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ta


* Size of downloaded dataset files: 1.74 GB
* Size of the generated dataset: 9.93 GB
* Total amount of disk used: 11.67 GB


An example of 'train' looks as follows. #### unshuffled\_original\_te


* Size of downloaded dataset files: 522.47 MB
* Size of the generated dataset: 2.61 GB
* Total amount of disk used: 3.13 GB


An example of 'train' looks as follows. #### unshuffled\_original\_tg


* Size of downloaded dataset files: 90.97 MB
* Size of the generated dataset: 397.43 MB
* Total amount of disk used: 488.41 MB


An example of 'train' looks as follows. #### unshuffled\_original\_th


* Size of downloaded dataset files: 7.38 GB
* Size of the generated dataset: 38.29 GB
* Total amount of disk used: 45.67 GB


An example of 'train' looks as follows. #### unshuffled\_original\_tk


* Size of downloaded dataset files: 2.96 MB
* Size of the generated dataset: 10.66 MB
* Total amount of disk used: 13.62 MB


An example of 'train' looks as follows. #### unshuffled\_original\_tl


* Size of downloaded dataset files: 204.89 MB
* Size of the generated dataset: 606.30 MB
* Total amount of disk used: 811.19 MB


An example of 'train' looks as follows. #### unshuffled\_original\_tr


* Size of downloaded dataset files: 21.96 GB
* Size of the generated dataset: 63.58 GB
* Total amount of disk used: 85.54 GB


An example of 'train' looks as follows. #### unshuffled\_original\_tt


* Size of downloaded dataset files: 151.06 MB
* Size of the generated dataset: 703.42 MB
* Total amount of disk used: 854.47 MB


An example of 'train' looks as follows. #### unshuffled\_original\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ug


* Size of downloaded dataset files: 27.92 MB
* Size of the generated dataset: 127.42 MB
* Total amount of disk used: 155.35 MB


An example of 'train' looks as follows. #### unshuffled\_original\_uk


* Size of downloaded dataset files: 14.42 GB
* Size of the generated dataset: 56.44 GB
* Total amount of disk used: 70.86 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ur


* Size of downloaded dataset files: 712.61 MB
* Size of the generated dataset: 2.80 GB
* Total amount of disk used: 3.51 GB


An example of 'train' looks as follows. #### unshuffled\_original\_uz


* Size of downloaded dataset files: 5.78 MB
* Size of the generated dataset: 21.46 MB
* Total amount of disk used: 27.24 MB


An example of 'train' looks as follows. #### unshuffled\_original\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_vi


* Size of downloaded dataset files: 21.50 GB
* Size of the generated dataset: 72.23 GB
* Total amount of disk used: 93.73 GB


An example of 'train' looks as follows. #### unshuffled\_original\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.12 MB
* Total amount of disk used: 2.42 MB


An example of 'train' looks as follows. #### unshuffled\_original\_wa


* Size of downloaded dataset files: 0.09 MB
* Size of the generated dataset: 0.29 MB
* Total amount of disk used: 0.38 MB


An example of 'train' looks as follows. #### unshuffled\_original\_war


* Size of downloaded dataset files: 0.64 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.32 MB


An example of 'train' looks as follows. #### unshuffled\_original\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows. #### unshuffled\_original\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows. #### unshuffled\_original\_xmf


* Size of downloaded dataset files: 1.05 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.17 MB


An example of 'train' looks as follows. #### unshuffled\_original\_yi


* Size of downloaded dataset files: 33.33 MB
* Size of the generated dataset: 147.60 MB
* Total amount of disk used: 180.94 MB


An example of 'train' looks as follows. #### unshuffled\_original\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows. #### unshuffled\_original\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_zh


* Size of downloaded dataset files: 206.00 GB
* Size of the generated dataset: 545.61 GB
* Total amount of disk used: 751.61 GB


An example of 'train' looks as follows. ### Data Fields


The data fields are the same among all configs.


* 'id': a 'int64' feature.
* 'text': a 'string' feature. ### Data Splits



Click to expand the number of samples per configuration


Dataset Creation
---------------- ### Curation Rationale


OSCAR was constructed new pipeline derived from the fastText's one, called *goclassy*. Goclassy reuses the fastText linear classifier and the pre-trained fastText model for language recognition, but it completely rewrites and parallelises their pipeline in an asynchronous manner.


The order of operations is more or less the same as in the fastText pre-processing pipeline but instead of clustering multiple operations into a single blocking process, a worker is launched for each operation but bounding the number of possible parallel operations at a given time by the number of available threads instead of the number of CPUs. Goclassy is implemented in the Go programming language so it lets the Go runtime handle the scheduling of the processes. Thus the goclassy's pipeline one does not have to wait for a whole WET file to download, decompress and classify in order to start downloading and processing the next one, a new file will start downloading and processing as soon as the scheduler is able to allocate a new process.


Filtering and cleaning processes at line level are done before feeding each line to the classifier. Lines shorter than 100 UTF-8 characters and lines containing invalid UTF-8 characters are discarted and are not classified. After all files are proccesed the deduplicated versions are constructed and everything is then splitted in shards and compressed. ### Source Data #### Initial Data Collection and Normalization


Common Crawl is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable. Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metdata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected nofollow and URL policies.


Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics.


To construct OSCAR the WET files of Common Crawl were used. These contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format and is stored on Amazon Web Services. In the case of OSCAR, the November 2018 snapshot was used. It surpasses 20TB of uncompressed data and contains more than 50 thousand plain text files where each file consists of the plain text from multiple websites along its metadata header. #### Who are the source language producers?


The data comes from multiple web pages in a large variety of languages. ### Annotations


The dataset does not contain any additional annotations. #### Annotation process


N/A #### Who are the annotators?


N/A ### Personal and Sensitive Information


Being constructed from Common Crawl, Personal and sensitive information might be present. This must be considered before training deep learning models with OSCAR, specially in the case of text-generation models.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


OSCAR is intended to bring more data to a wide variety of lanuages, the aim of the corpus is to make large amounts of data available to lower resource languages in order to facilitate the pre-training of state-of-the-art language modeling architectures. ### Discussion of Biases


OSCAR is not properly filtered yet and this can be reflected on the models trained with it. Care is advised specially concerning biases of the resulting models. ### Other Known Limitations


The fastText linear classifier is limed both in performance and the variety of languages it can recognize, so the quality of some OSCAR sub-corpora might be lower than expected, specially for the lowest-resource langiuages. Some audits have already been done by third parties.


Additional Information
---------------------- ### Dataset Curators


The corpus was put together by Pedro J. Ortiz, Benoît Sagot, and Laurent Romary, during work done at Inria, particularly at the ALMAnaCH team. ### Licensing Information



```
These data are released under this licensing scheme
We do not own any of the text from which these data has been extracted.
We license the actual packaging of these data under the Creative Commons CC0 license ("no rights reserved") URL
To the extent possible under law, Inria has waived all copyright and related or neighboring rights to OSCAR
This work is published from: France.

Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
* Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
* Clearly identify the copyrighted work claimed to be infringed.
* Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

We will comply to legitimate requests by removing the affected sources from the next release of the corpus.

``` ### Contributions


Thanks to @pjox and @lhoestq for adding this dataset.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-100K<n<1M #size_categories-100M<n<1B #size_categories-10K<n<100K #size_categories-10M<n<100M #size_categories-1K<n<10K #size_categories-1M<n<10M #size_categories-n<1K #source_datasets-original #language-Afrikaans #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Avaric #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Russia Buriat #language-Catalan #language-Chavacano #language-Chechen #language-Cebuano #language-Central Kurdish #language-Czech #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dimli (individual language) #language-Lower Sorbian #language-Dhivehi #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-French #language-Northern Frisian #language-Western Frisian #language-Irish #language-Scottish Gaelic #language-Galician #language-Guarani #language-Goan Konkani #language-Gujarati #language-Hebrew #language-Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Iloko #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Lojban #language-Javanese #language-Georgian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Karachay-Balkar #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Luxembourgish #language-Lezghian #language-Limburgan #language-Lombard #language-Lao #language-Northern Luri #language-Lithuanian #language-Latvian #language-Maithili #language-Malagasy #language-Eastern Mari #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-nah #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Occitan (post 1500) #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pampanga #language-Polish #language-Piemontese #language-Western Panjabi #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Romanian #language-Russian #language-Sanskrit #language-Yakut #language-Sicilian #language-Sindhi #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Somali #language-Albanian #language-Serbian #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Tamil #language-Telugu #language-Tajik #language-Thai #language-Turkmen #language-Tagalog #language-Turkish #language-Tatar #language-Tuvinian #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venetian #language-Vietnamese #language-Volapük #language-Walloon #language-Waray (Philippines) #language-Wu Chinese #language-Kalmyk #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #license-cc0-1.0 #arxiv-2010.14571 #region-us 
 ### Dataset Summary


OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture. Data is distributed by language in both original and deduplicated form.


The version here is the original OSCAR 2019 release: URL


For more recent versions, visit the oscar-corpus organization on the Hub:


* OSCAR 22.01 (released in January 2022): oscar-corpus/OSCAR-2201
* OSCAR 21.09 (released in September 2021): oscar-corpus/OSCAR-2109 ### Supported Tasks and Leaderboards


OSCAR is mainly inteded to pretrain language models and word represantations. ### Languages


All the data is distributed by language, both the original and the deduplicated versions of the data are available. 166 different languages are available. The table in subsection Data Splits Sample Size provides the language code for each subcorpus as well as the number of words (space separated tokens), lines and sizes for both the original and the deduplicated versions of OSCAR.


Dataset Structure
-----------------


We show detailed information for all the configurations of the dataset. ### Data Instances



Click to expand the Data/size information for each language (deduplicated) #### unshuffled\_deduplicated\_af


* Size of downloaded dataset files: 65.99 MB
* Size of the generated dataset: 172.30 MB
* Total amount of disk used: 238.29 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_als


* Size of downloaded dataset files: 1.26 MB
* Size of the generated dataset: 2.96 MB
* Total amount of disk used: 4.22 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_am


* Size of downloaded dataset files: 61.35 MB
* Size of the generated dataset: 216.15 MB
* Total amount of disk used: 277.50 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_an


* Size of downloaded dataset files: 0.14 MB
* Size of the generated dataset: 0.85 MB
* Total amount of disk used: 0.99 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ar


* Size of downloaded dataset files: 9.67 GB
* Size of the generated dataset: 33.57 GB
* Total amount of disk used: 43.23 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_arz


* Size of downloaded dataset files: 10.02 MB
* Size of the generated dataset: 35.91 MB
* Total amount of disk used: 45.94 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_as


* Size of downloaded dataset files: 15.51 MB
* Size of the generated dataset: 74.07 MB
* Total amount of disk used: 89.58 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ast


* Size of downloaded dataset files: 0.86 MB
* Size of the generated dataset: 2.17 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_av


* Size of downloaded dataset files: 0.07 MB
* Size of the generated dataset: 0.34 MB
* Total amount of disk used: 0.41 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_az


* Size of downloaded dataset files: 521.74 MB
* Size of the generated dataset: 1.53 GB
* Total amount of disk used: 2.05 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_azb


* Size of downloaded dataset files: 5.19 MB
* Size of the generated dataset: 20.08 MB
* Total amount of disk used: 25.27 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ba


* Size of downloaded dataset files: 25.98 MB
* Size of the generated dataset: 93.84 MB
* Total amount of disk used: 119.82 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_be


* Size of downloaded dataset files: 306.70 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.39 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bg


* Size of downloaded dataset files: 3.85 GB
* Size of the generated dataset: 14.45 GB
* Total amount of disk used: 18.30 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bn


* Size of downloaded dataset files: 1.26 GB
* Size of the generated dataset: 6.24 GB
* Total amount of disk used: 7.50 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bo


* Size of downloaded dataset files: 22.37 MB
* Size of the generated dataset: 144.65 MB
* Total amount of disk used: 167.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bpy


* Size of downloaded dataset files: 0.19 MB
* Size of the generated dataset: 1.78 MB
* Total amount of disk used: 1.97 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_br


* Size of downloaded dataset files: 6.47 MB
* Size of the generated dataset: 17.00 MB
* Total amount of disk used: 23.47 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bs


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.15 MB
* Total amount of disk used: 0.18 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ca


* Size of downloaded dataset files: 1.73 GB
* Size of the generated dataset: 4.57 GB
* Total amount of disk used: 6.30 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ce


* Size of downloaded dataset files: 1.87 MB
* Size of the generated dataset: 7.04 MB
* Total amount of disk used: 8.90 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ceb


* Size of downloaded dataset files: 7.12 MB
* Size of the generated dataset: 24.83 MB
* Total amount of disk used: 31.95 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ckb


* Size of downloaded dataset files: 60.32 MB
* Size of the generated dataset: 237.72 MB
* Total amount of disk used: 298.05 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cs


* Size of downloaded dataset files: 10.49 GB
* Size of the generated dataset: 25.71 GB
* Total amount of disk used: 36.20 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cv


* Size of downloaded dataset files: 7.47 MB
* Size of the generated dataset: 27.49 MB
* Total amount of disk used: 34.95 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_cy


* Size of downloaded dataset files: 53.63 MB
* Size of the generated dataset: 141.22 MB
* Total amount of disk used: 194.86 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_da


* Size of downloaded dataset files: 3.82 GB
* Size of the generated dataset: 10.24 GB
* Total amount of disk used: 14.06 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_de


* Size of downloaded dataset files: 60.80 GB
* Size of the generated dataset: 156.30 GB
* Total amount of disk used: 217.10 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_dv


* Size of downloaded dataset files: 16.84 MB
* Size of the generated dataset: 82.19 MB
* Total amount of disk used: 99.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_el


* Size of downloaded dataset files: 7.91 GB
* Size of the generated dataset: 28.74 GB
* Total amount of disk used: 36.65 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_en


* Size of downloaded dataset files: 496.50 GB
* Size of the generated dataset: 1299.75 GB
* Total amount of disk used: 1796.24 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_eo


* Size of downloaded dataset files: 92.86 MB
* Size of the generated dataset: 240.12 MB
* Total amount of disk used: 332.99 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_es


* Size of downloaded dataset files: 60.46 GB
* Size of the generated dataset: 160.86 GB
* Total amount of disk used: 221.32 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_et


* Size of downloaded dataset files: 966.79 MB
* Size of the generated dataset: 2.45 GB
* Total amount of disk used: 3.41 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_eu


* Size of downloaded dataset files: 134.68 MB
* Size of the generated dataset: 363.93 MB
* Total amount of disk used: 498.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fa


* Size of downloaded dataset files: 10.46 GB
* Size of the generated dataset: 40.06 GB
* Total amount of disk used: 50.52 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fi


* Size of downloaded dataset files: 5.38 GB
* Size of the generated dataset: 13.99 GB
* Total amount of disk used: 19.37 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fr


* Size of downloaded dataset files: 55.46 GB
* Size of the generated dataset: 148.28 GB
* Total amount of disk used: 203.75 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_fy


* Size of downloaded dataset files: 10.27 MB
* Size of the generated dataset: 26.73 MB
* Total amount of disk used: 37.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ga


* Size of downloaded dataset files: 22.22 MB
* Size of the generated dataset: 63.86 MB
* Total amount of disk used: 86.08 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gd


* Size of downloaded dataset files: 0.42 MB
* Size of the generated dataset: 1.36 MB
* Total amount of disk used: 1.78 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gl


* Size of downloaded dataset files: 155.85 MB
* Size of the generated dataset: 408.34 MB
* Total amount of disk used: 564.19 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gom


* Size of downloaded dataset files: 0.38 MB
* Size of the generated dataset: 1.87 MB
* Total amount of disk used: 2.24 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_gu


* Size of downloaded dataset files: 162.97 MB
* Size of the generated dataset: 759.34 MB
* Total amount of disk used: 922.32 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_he


* Size of downloaded dataset files: 3.04 GB
* Size of the generated dataset: 10.47 GB
* Total amount of disk used: 13.51 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hi


* Size of downloaded dataset files: 2.01 GB
* Size of the generated dataset: 9.57 GB
* Total amount of disk used: 11.58 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hr


* Size of downloaded dataset files: 46.74 MB
* Size of the generated dataset: 121.50 MB
* Total amount of disk used: 168.23 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hsb


* Size of downloaded dataset files: 0.72 MB
* Size of the generated dataset: 1.89 MB
* Total amount of disk used: 2.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hu


* Size of downloaded dataset files: 7.37 GB
* Size of the generated dataset: 19.09 GB
* Total amount of disk used: 26.46 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_hy


* Size of downloaded dataset files: 393.62 MB
* Size of the generated dataset: 1.56 GB
* Total amount of disk used: 1.96 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ia


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.38 MB
* Total amount of disk used: 0.43 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_id


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 17.05 GB
* Total amount of disk used: 23.05 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ilo


* Size of downloaded dataset files: 0.23 MB
* Size of the generated dataset: 0.68 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.14 MB
* Total amount of disk used: 0.19 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_is


* Size of downloaded dataset files: 332.87 MB
* Size of the generated dataset: 894.28 MB
* Total amount of disk used: 1.23 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_it


* Size of downloaded dataset files: 27.93 GB
* Size of the generated dataset: 74.09 GB
* Total amount of disk used: 102.03 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ja


* Size of downloaded dataset files: 40.80 GB
* Size of the generated dataset: 113.63 GB
* Total amount of disk used: 154.44 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_jbo


* Size of downloaded dataset files: 0.20 MB
* Size of the generated dataset: 0.70 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_jv


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.62 MB
* Total amount of disk used: 0.82 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ka


* Size of downloaded dataset files: 377.23 MB
* Size of the generated dataset: 1.99 GB
* Total amount of disk used: 2.36 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kk


* Size of downloaded dataset files: 389.12 MB
* Size of the generated dataset: 1.59 GB
* Total amount of disk used: 1.97 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_km


* Size of downloaded dataset files: 114.48 MB
* Size of the generated dataset: 610.61 MB
* Total amount of disk used: 725.09 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kn


* Size of downloaded dataset files: 215.52 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ko


* Size of downloaded dataset files: 4.46 GB
* Size of the generated dataset: 12.00 GB
* Total amount of disk used: 16.47 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_krc


* Size of downloaded dataset files: 0.62 MB
* Size of the generated dataset: 2.41 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ku


* Size of downloaded dataset files: 23.34 MB
* Size of the generated dataset: 63.09 MB
* Total amount of disk used: 86.43 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kv


* Size of downloaded dataset files: 0.33 MB
* Size of the generated dataset: 1.21 MB
* Total amount of disk used: 1.54 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ky


* Size of downloaded dataset files: 106.22 MB
* Size of the generated dataset: 408.40 MB
* Total amount of disk used: 514.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_la


* Size of downloaded dataset files: 3.42 MB
* Size of the generated dataset: 9.79 MB
* Total amount of disk used: 13.22 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lb


* Size of downloaded dataset files: 8.30 MB
* Size of the generated dataset: 21.42 MB
* Total amount of disk used: 29.72 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lez


* Size of downloaded dataset files: 0.77 MB
* Size of the generated dataset: 3.08 MB
* Total amount of disk used: 3.84 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.46 MB
* Total amount of disk used: 0.57 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lo


* Size of downloaded dataset files: 23.63 MB
* Size of the generated dataset: 119.29 MB
* Total amount of disk used: 142.92 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.08 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lt


* Size of downloaded dataset files: 1.65 GB
* Size of the generated dataset: 4.20 GB
* Total amount of disk used: 5.86 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_lv


* Size of downloaded dataset files: 710.45 MB
* Size of the generated dataset: 1.91 GB
* Total amount of disk used: 2.62 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mai


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mg


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 13.59 MB
* Total amount of disk used: 17.89 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mhr


* Size of downloaded dataset files: 1.63 MB
* Size of the generated dataset: 6.26 MB
* Total amount of disk used: 7.89 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.31 MB
* Total amount of disk used: 0.33 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mk


* Size of downloaded dataset files: 303.12 MB
* Size of the generated dataset: 1.19 GB
* Total amount of disk used: 1.49 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ml


* Size of downloaded dataset files: 496.80 MB
* Size of the generated dataset: 2.69 GB
* Total amount of disk used: 3.18 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mn


* Size of downloaded dataset files: 219.52 MB
* Size of the generated dataset: 883.46 MB
* Total amount of disk used: 1.10 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mr


* Size of downloaded dataset files: 299.68 MB
* Size of the generated dataset: 1.49 GB
* Total amount of disk used: 1.79 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mrj


* Size of downloaded dataset files: 0.29 MB
* Size of the generated dataset: 1.10 MB
* Total amount of disk used: 1.38 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ms


* Size of downloaded dataset files: 16.39 MB
* Size of the generated dataset: 49.45 MB
* Total amount of disk used: 65.85 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mt


* Size of downloaded dataset files: 5.90 MB
* Size of the generated dataset: 17.68 MB
* Total amount of disk used: 23.58 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_my


* Size of downloaded dataset files: 207.14 MB
* Size of the generated dataset: 1.11 GB
* Total amount of disk used: 1.32 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_mzn


* Size of downloaded dataset files: 0.16 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.79 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nds


* Size of downloaded dataset files: 5.27 MB
* Size of the generated dataset: 13.48 MB
* Total amount of disk used: 18.76 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ne


* Size of downloaded dataset files: 240.63 MB
* Size of the generated dataset: 1.24 GB
* Total amount of disk used: 1.48 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_new


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 4.26 MB
* Total amount of disk used: 5.09 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nl


* Size of downloaded dataset files: 15.73 GB
* Size of the generated dataset: 41.91 GB
* Total amount of disk used: 57.65 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_nn


* Size of downloaded dataset files: 23.58 MB
* Size of the generated dataset: 58.32 MB
* Total amount of disk used: 81.90 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_no


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 5.11 GB
* Total amount of disk used: 7.07 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_oc


* Size of downloaded dataset files: 1.34 MB
* Size of the generated dataset: 4.00 MB
* Total amount of disk used: 5.34 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_or


* Size of downloaded dataset files: 38.72 MB
* Size of the generated dataset: 197.63 MB
* Total amount of disk used: 236.36 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_os


* Size of downloaded dataset files: 2.83 MB
* Size of the generated dataset: 11.00 MB
* Total amount of disk used: 13.83 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pa


* Size of downloaded dataset files: 102.39 MB
* Size of the generated dataset: 483.04 MB
* Total amount of disk used: 585.42 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pl


* Size of downloaded dataset files: 20.19 GB
* Size of the generated dataset: 50.59 GB
* Total amount of disk used: 70.78 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pms


* Size of downloaded dataset files: 0.71 MB
* Size of the generated dataset: 2.00 MB
* Total amount of disk used: 2.72 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pnb


* Size of downloaded dataset files: 2.58 MB
* Size of the generated dataset: 9.44 MB
* Total amount of disk used: 12.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ps


* Size of downloaded dataset files: 71.83 MB
* Size of the generated dataset: 254.79 MB
* Total amount of disk used: 326.61 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_pt


* Size of downloaded dataset files: 26.00 GB
* Size of the generated dataset: 68.37 GB
* Total amount of disk used: 94.37 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ro


* Size of downloaded dataset files: 4.48 GB
* Size of the generated dataset: 11.66 GB
* Total amount of disk used: 16.14 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ru


* Size of downloaded dataset files: 166.68 GB
* Size of the generated dataset: 611.70 GB
* Total amount of disk used: 778.38 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sa


* Size of downloaded dataset files: 7.27 MB
* Size of the generated dataset: 38.33 MB
* Total amount of disk used: 45.60 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sah


* Size of downloaded dataset files: 7.01 MB
* Size of the generated dataset: 27.46 MB
* Total amount of disk used: 34.49 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sd


* Size of downloaded dataset files: 74.17 MB
* Size of the generated dataset: 275.48 MB
* Total amount of disk used: 349.66 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sh


* Size of downloaded dataset files: 1.45 MB
* Size of the generated dataset: 6.44 MB
* Total amount of disk used: 7.87 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_si


* Size of downloaded dataset files: 175.62 MB
* Size of the generated dataset: 842.57 MB
* Total amount of disk used: 1.02 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sk


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 4.80 GB
* Total amount of disk used: 6.76 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sl


* Size of downloaded dataset files: 523.22 MB
* Size of the generated dataset: 1.32 GB
* Total amount of disk used: 1.85 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sq


* Size of downloaded dataset files: 445.36 MB
* Size of the generated dataset: 1.21 GB
* Total amount of disk used: 1.66 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sr


* Size of downloaded dataset files: 665.03 MB
* Size of the generated dataset: 2.36 GB
* Total amount of disk used: 3.03 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_su


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.21 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sv


* Size of downloaded dataset files: 10.19 GB
* Size of the generated dataset: 26.33 GB
* Total amount of disk used: 36.51 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_sw


* Size of downloaded dataset files: 2.95 MB
* Size of the generated dataset: 8.98 MB
* Total amount of disk used: 11.92 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ta


* Size of downloaded dataset files: 971.12 MB
* Size of the generated dataset: 5.48 GB
* Total amount of disk used: 6.45 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_te


* Size of downloaded dataset files: 342.43 MB
* Size of the generated dataset: 1.70 GB
* Total amount of disk used: 2.04 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tg


* Size of downloaded dataset files: 62.90 MB
* Size of the generated dataset: 261.68 MB
* Total amount of disk used: 324.60 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_th


* Size of downloaded dataset files: 3.54 GB
* Size of the generated dataset: 17.11 GB
* Total amount of disk used: 20.65 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tk


* Size of downloaded dataset files: 2.22 MB
* Size of the generated dataset: 7.12 MB
* Total amount of disk used: 9.34 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tl


* Size of downloaded dataset files: 151.34 MB
* Size of the generated dataset: 431.69 MB
* Total amount of disk used: 583.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tr


* Size of downloaded dataset files: 10.39 GB
* Size of the generated dataset: 28.47 GB
* Total amount of disk used: 38.86 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tt


* Size of downloaded dataset files: 85.89 MB
* Size of the generated dataset: 321.37 MB
* Total amount of disk used: 407.26 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ug


* Size of downloaded dataset files: 20.53 MB
* Size of the generated dataset: 86.44 MB
* Total amount of disk used: 106.97 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_uk


* Size of downloaded dataset files: 8.04 GB
* Size of the generated dataset: 29.86 GB
* Total amount of disk used: 37.90 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_ur


* Size of downloaded dataset files: 483.59 MB
* Size of the generated dataset: 1.82 GB
* Total amount of disk used: 2.31 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_uz


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 12.00 MB
* Total amount of disk used: 16.29 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_vi


* Size of downloaded dataset files: 10.71 GB
* Size of the generated dataset: 33.60 GB
* Total amount of disk used: 44.31 GB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.10 MB
* Total amount of disk used: 2.40 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_wa


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.22 MB
* Total amount of disk used: 0.29 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_war


* Size of downloaded dataset files: 0.55 MB
* Size of the generated dataset: 2.36 MB
* Total amount of disk used: 2.90 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_xmf


* Size of downloaded dataset files: 0.94 MB
* Size of the generated dataset: 4.63 MB
* Total amount of disk used: 5.58 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_yi


* Size of downloaded dataset files: 22.20 MB
* Size of the generated dataset: 88.29 MB
* Total amount of disk used: 110.49 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_deduplicated\_zh


* Size of downloaded dataset files: 99.98 GB
* Size of the generated dataset: 267.88 GB
* Total amount of disk used: 367.86 GB


An example of 'train' looks as follows.




Click to expand the Data/size information for each language (original) #### unshuffled\_original\_af


* Size of downloaded dataset files: 85.79 MB
* Size of the generated dataset: 254.08 MB
* Total amount of disk used: 339.87 MB


An example of 'train' looks as follows. #### unshuffled\_original\_als


* Size of downloaded dataset files: 1.49 MB
* Size of the generated dataset: 5.30 MB
* Total amount of disk used: 6.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_am


* Size of downloaded dataset files: 102.79 MB
* Size of the generated dataset: 378.06 MB
* Total amount of disk used: 480.85 MB


An example of 'train' looks as follows. #### unshuffled\_original\_an


* Size of downloaded dataset files: 0.15 MB
* Size of the generated dataset: 1.33 MB
* Total amount of disk used: 1.48 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ar


* Size of downloaded dataset files: 22.23 GB
* Size of the generated dataset: 87.94 GB
* Total amount of disk used: 110.17 GB


An example of 'train' looks as follows. #### unshuffled\_original\_arz


* Size of downloaded dataset files: 15.90 MB
* Size of the generated dataset: 70.13 MB
* Total amount of disk used: 86.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_as


* Size of downloaded dataset files: 21.43 MB
* Size of the generated dataset: 117.73 MB
* Total amount of disk used: 139.17 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ast


* Size of downloaded dataset files: 0.92 MB
* Size of the generated dataset: 2.54 MB
* Total amount of disk used: 3.46 MB


An example of 'train' looks as follows. #### unshuffled\_original\_av


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.42 MB
* Total amount of disk used: 0.50 MB


An example of 'train' looks as follows. #### unshuffled\_original\_az


* Size of downloaded dataset files: 927.76 MB
* Size of the generated dataset: 2.96 GB
* Total amount of disk used: 3.89 GB


An example of 'train' looks as follows. #### unshuffled\_original\_azb


* Size of downloaded dataset files: 6.64 MB
* Size of the generated dataset: 28.47 MB
* Total amount of disk used: 35.11 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ba


* Size of downloaded dataset files: 33.22 MB
* Size of the generated dataset: 133.70 MB
* Total amount of disk used: 166.92 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_be


* Size of downloaded dataset files: 498.29 MB
* Size of the generated dataset: 1.88 GB
* Total amount of disk used: 2.38 GB


An example of 'train' looks as follows. #### unshuffled\_original\_bg


* Size of downloaded dataset files: 8.34 GB
* Size of the generated dataset: 33.75 GB
* Total amount of disk used: 42.09 GB


An example of 'train' looks as follows. #### unshuffled\_original\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bn


* Size of downloaded dataset files: 2.14 GB
* Size of the generated dataset: 10.77 GB
* Total amount of disk used: 12.91 GB


An example of 'train' looks as follows. #### unshuffled\_original\_bo


* Size of downloaded dataset files: 28.94 MB
* Size of the generated dataset: 195.40 MB
* Total amount of disk used: 224.34 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bpy


* Size of downloaded dataset files: 0.34 MB
* Size of the generated dataset: 4.35 MB
* Total amount of disk used: 4.69 MB


An example of 'train' looks as follows. #### unshuffled\_original\_br


* Size of downloaded dataset files: 9.18 MB
* Size of the generated dataset: 30.20 MB
* Total amount of disk used: 39.38 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bs


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.48 MB
* Total amount of disk used: 0.53 MB


An example of 'train' looks as follows. #### unshuffled\_original\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ca


* Size of downloaded dataset files: 3.10 GB
* Size of the generated dataset: 8.62 GB
* Total amount of disk used: 11.73 GB


An example of 'train' looks as follows. #### unshuffled\_original\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ce


* Size of downloaded dataset files: 2.09 MB
* Size of the generated dataset: 8.73 MB
* Total amount of disk used: 10.82 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ceb


* Size of downloaded dataset files: 11.07 MB
* Size of the generated dataset: 40.97 MB
* Total amount of disk used: 52.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ckb


* Size of downloaded dataset files: 111.88 MB
* Size of the generated dataset: 510.97 MB
* Total amount of disk used: 622.85 MB


An example of 'train' looks as follows. #### unshuffled\_original\_cs


* Size of downloaded dataset files: 21.72 GB
* Size of the generated dataset: 57.08 GB
* Total amount of disk used: 78.80 GB


An example of 'train' looks as follows. #### unshuffled\_original\_cv


* Size of downloaded dataset files: 9.40 MB
* Size of the generated dataset: 41.05 MB
* Total amount of disk used: 50.45 MB


An example of 'train' looks as follows. #### unshuffled\_original\_cy


* Size of downloaded dataset files: 81.74 MB
* Size of the generated dataset: 224.93 MB
* Total amount of disk used: 306.67 MB


An example of 'train' looks as follows. #### unshuffled\_original\_da


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 16.76 GB
* Total amount of disk used: 22.76 GB


An example of 'train' looks as follows. #### unshuffled\_original\_de


* Size of downloaded dataset files: 119.51 GB
* Size of the generated dataset: 331.22 GB
* Total amount of disk used: 450.73 GB


An example of 'train' looks as follows. #### unshuffled\_original\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_dv


* Size of downloaded dataset files: 24.91 MB
* Size of the generated dataset: 131.63 MB
* Total amount of disk used: 156.54 MB


An example of 'train' looks as follows. #### unshuffled\_original\_el


* Size of downloaded dataset files: 17.31 GB
* Size of the generated dataset: 66.27 GB
* Total amount of disk used: 83.58 GB


An example of 'train' looks as follows. #### unshuffled\_original\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_en


* Size of downloaded dataset files: 903.83 GB
* Size of the generated dataset: 2525.44 GB
* Total amount of disk used: 3429.27 GB


An example of 'train' looks as follows. #### unshuffled\_original\_eo


* Size of downloaded dataset files: 117.07 MB
* Size of the generated dataset: 314.18 MB
* Total amount of disk used: 431.27 MB


An example of 'train' looks as follows. #### unshuffled\_original\_es


* Size of downloaded dataset files: 106.04 GB
* Size of the generated dataset: 298.49 GB
* Total amount of disk used: 404.53 GB


An example of 'train' looks as follows. #### unshuffled\_original\_et


* Size of downloaded dataset files: 1.88 GB
* Size of the generated dataset: 5.17 GB
* Total amount of disk used: 7.06 GB


An example of 'train' looks as follows. #### unshuffled\_original\_eu


* Size of downloaded dataset files: 248.19 MB
* Size of the generated dataset: 894.83 MB
* Total amount of disk used: 1.14 GB


An example of 'train' looks as follows. #### unshuffled\_original\_fa


* Size of downloaded dataset files: 20.96 GB
* Size of the generated dataset: 84.21 GB
* Total amount of disk used: 105.17 GB


An example of 'train' looks as follows. #### unshuffled\_original\_fi


* Size of downloaded dataset files: 9.97 GB
* Size of the generated dataset: 28.57 GB
* Total amount of disk used: 38.54 GB


An example of 'train' looks as follows. #### unshuffled\_original\_fr


* Size of downloaded dataset files: 105.32 GB
* Size of the generated dataset: 303.19 GB
* Total amount of disk used: 408.51 GB


An example of 'train' looks as follows. #### unshuffled\_original\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_fy


* Size of downloaded dataset files: 12.40 MB
* Size of the generated dataset: 36.24 MB
* Total amount of disk used: 48.64 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ga


* Size of downloaded dataset files: 29.27 MB
* Size of the generated dataset: 92.37 MB
* Total amount of disk used: 121.63 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gd


* Size of downloaded dataset files: 0.52 MB
* Size of the generated dataset: 2.02 MB
* Total amount of disk used: 2.55 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gl


* Size of downloaded dataset files: 235.38 MB
* Size of the generated dataset: 656.48 MB
* Total amount of disk used: 891.87 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gom


* Size of downloaded dataset files: 0.44 MB
* Size of the generated dataset: 2.25 MB
* Total amount of disk used: 2.71 MB


An example of 'train' looks as follows. #### unshuffled\_original\_gu


* Size of downloaded dataset files: 232.02 MB
* Size of the generated dataset: 1.09 GB
* Total amount of disk used: 1.33 GB


An example of 'train' looks as follows. #### unshuffled\_original\_he


* Size of downloaded dataset files: 5.66 GB
* Size of the generated dataset: 21.11 GB
* Total amount of disk used: 26.77 GB


An example of 'train' looks as follows. #### unshuffled\_original\_hi


* Size of downloaded dataset files: 3.66 GB
* Size of the generated dataset: 17.93 GB
* Total amount of disk used: 21.59 GB


An example of 'train' looks as follows. #### unshuffled\_original\_hr


* Size of downloaded dataset files: 79.42 MB
* Size of the generated dataset: 243.83 MB
* Total amount of disk used: 323.24 MB


An example of 'train' looks as follows. #### unshuffled\_original\_hsb


* Size of downloaded dataset files: 1.39 MB
* Size of the generated dataset: 4.49 MB
* Total amount of disk used: 5.87 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_hu


* Size of downloaded dataset files: 15.69 GB
* Size of the generated dataset: 43.07 GB
* Total amount of disk used: 58.77 GB


An example of 'train' looks as follows. #### unshuffled\_original\_hy


* Size of downloaded dataset files: 897.36 MB
* Size of the generated dataset: 3.94 GB
* Total amount of disk used: 4.84 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ia


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_id


* Size of downloaded dataset files: 10.60 GB
* Size of the generated dataset: 32.32 GB
* Total amount of disk used: 42.91 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ilo


* Size of downloaded dataset files: 0.27 MB
* Size of the generated dataset: 0.92 MB
* Total amount of disk used: 1.20 MB


An example of 'train' looks as follows. #### unshuffled\_original\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.20 MB


An example of 'train' looks as follows. #### unshuffled\_original\_is


* Size of downloaded dataset files: 533.03 MB
* Size of the generated dataset: 1.52 GB
* Total amount of disk used: 2.06 GB


An example of 'train' looks as follows. #### unshuffled\_original\_it


* Size of downloaded dataset files: 52.16 GB
* Size of the generated dataset: 147.38 GB
* Total amount of disk used: 199.54 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ja


* Size of downloaded dataset files: 79.56 GB
* Size of the generated dataset: 232.22 GB
* Total amount of disk used: 311.78 GB


An example of 'train' looks as follows. #### unshuffled\_original\_jbo


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.77 MB
* Total amount of disk used: 0.98 MB


An example of 'train' looks as follows. #### unshuffled\_original\_jv


* Size of downloaded dataset files: 0.22 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ka


* Size of downloaded dataset files: 680.74 MB
* Size of the generated dataset: 3.77 GB
* Total amount of disk used: 4.45 GB


An example of 'train' looks as follows. #### unshuffled\_original\_kk


* Size of downloaded dataset files: 615.06 MB
* Size of the generated dataset: 2.83 GB
* Total amount of disk used: 3.45 GB


An example of 'train' looks as follows. #### unshuffled\_original\_km


* Size of downloaded dataset files: 193.28 MB
* Size of the generated dataset: 1.10 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows. #### unshuffled\_original\_kn


* Size of downloaded dataset files: 342.15 MB
* Size of the generated dataset: 1.76 GB
* Total amount of disk used: 2.11 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ko


* Size of downloaded dataset files: 8.81 GB
* Size of the generated dataset: 25.29 GB
* Total amount of disk used: 34.10 GB


An example of 'train' looks as follows. #### unshuffled\_original\_krc


* Size of downloaded dataset files: 0.66 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.34 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ku


* Size of downloaded dataset files: 33.38 MB
* Size of the generated dataset: 99.06 MB
* Total amount of disk used: 132.44 MB


An example of 'train' looks as follows. #### unshuffled\_original\_kv


* Size of downloaded dataset files: 0.40 MB
* Size of the generated dataset: 2.38 MB
* Total amount of disk used: 2.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ky


* Size of downloaded dataset files: 152.64 MB
* Size of the generated dataset: 630.79 MB
* Total amount of disk used: 783.43 MB


An example of 'train' looks as follows. #### unshuffled\_original\_la


* Size of downloaded dataset files: 5.46 MB
* Size of the generated dataset: 27.80 MB
* Total amount of disk used: 33.26 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lb


* Size of downloaded dataset files: 10.73 MB
* Size of the generated dataset: 30.60 MB
* Total amount of disk used: 41.32 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lez


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 3.38 MB
* Total amount of disk used: 4.20 MB


An example of 'train' looks as follows. #### unshuffled\_original\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.47 MB
* Total amount of disk used: 0.58 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lo


* Size of downloaded dataset files: 33.92 MB
* Size of the generated dataset: 182.36 MB
* Total amount of disk used: 216.28 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows. #### unshuffled\_original\_lt


* Size of downloaded dataset files: 3.44 GB
* Size of the generated dataset: 9.45 GB
* Total amount of disk used: 12.89 GB


An example of 'train' looks as follows. #### unshuffled\_original\_lv


* Size of downloaded dataset files: 1.49 GB
* Size of the generated dataset: 4.27 GB
* Total amount of disk used: 5.75 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mai


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.33 MB
* Total amount of disk used: 0.34 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mg


* Size of downloaded dataset files: 6.22 MB
* Size of the generated dataset: 21.79 MB
* Total amount of disk used: 28.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mhr


* Size of downloaded dataset files: 1.84 MB
* Size of the generated dataset: 7.55 MB
* Total amount of disk used: 9.38 MB


An example of 'train' looks as follows. #### unshuffled\_original\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.64 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mk


* Size of downloaded dataset files: 508.24 MB
* Size of the generated dataset: 2.20 GB
* Total amount of disk used: 2.71 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ml


* Size of downloaded dataset files: 938.69 MB
* Size of the generated dataset: 5.24 GB
* Total amount of disk used: 6.18 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mn


* Size of downloaded dataset files: 472.36 MB
* Size of the generated dataset: 2.33 GB
* Total amount of disk used: 2.81 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mr


* Size of downloaded dataset files: 525.31 MB
* Size of the generated dataset: 2.82 GB
* Total amount of disk used: 3.34 GB


An example of 'train' looks as follows. #### unshuffled\_original\_mrj


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 1.16 MB
* Total amount of disk used: 1.47 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ms


* Size of downloaded dataset files: 28.46 MB
* Size of the generated dataset: 122.33 MB
* Total amount of disk used: 150.79 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mt


* Size of downloaded dataset files: 7.53 MB
* Size of the generated dataset: 24.47 MB
* Total amount of disk used: 32.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_my


* Size of downloaded dataset files: 369.85 MB
* Size of the generated dataset: 2.02 GB
* Total amount of disk used: 2.39 GB


An example of 'train' looks as follows. #### unshuffled\_original\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_mzn


* Size of downloaded dataset files: 0.18 MB
* Size of the generated dataset: 0.72 MB
* Total amount of disk used: 0.90 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nds


* Size of downloaded dataset files: 6.74 MB
* Size of the generated dataset: 18.23 MB
* Total amount of disk used: 24.99 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ne


* Size of downloaded dataset files: 355.29 MB
* Size of the generated dataset: 1.87 GB
* Total amount of disk used: 2.22 GB


An example of 'train' looks as follows. #### unshuffled\_original\_new


* Size of downloaded dataset files: 1.03 MB
* Size of the generated dataset: 5.77 MB
* Total amount of disk used: 6.79 MB


An example of 'train' looks as follows. #### unshuffled\_original\_nl


* Size of downloaded dataset files: 29.35 GB
* Size of the generated dataset: 83.23 GB
* Total amount of disk used: 112.58 GB


An example of 'train' looks as follows. #### unshuffled\_original\_nn


* Size of downloaded dataset files: 32.86 MB
* Size of the generated dataset: 90.84 MB
* Total amount of disk used: 123.70 MB


An example of 'train' looks as follows. #### unshuffled\_original\_no


* Size of downloaded dataset files: 3.11 GB
* Size of the generated dataset: 8.65 GB
* Total amount of disk used: 11.76 GB


An example of 'train' looks as follows. #### unshuffled\_original\_oc


* Size of downloaded dataset files: 1.57 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.71 MB


An example of 'train' looks as follows. #### unshuffled\_original\_or


* Size of downloaded dataset files: 49.84 MB
* Size of the generated dataset: 260.15 MB
* Total amount of disk used: 309.99 MB


An example of 'train' looks as follows. #### unshuffled\_original\_os


* Size of downloaded dataset files: 3.09 MB
* Size of the generated dataset: 12.90 MB
* Total amount of disk used: 15.99 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pa


* Size of downloaded dataset files: 164.21 MB
* Size of the generated dataset: 801.16 MB
* Total amount of disk used: 965.37 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pl


* Size of downloaded dataset files: 42.88 GB
* Size of the generated dataset: 117.12 GB
* Total amount of disk used: 160.01 GB


An example of 'train' looks as follows. #### unshuffled\_original\_pms


* Size of downloaded dataset files: 0.75 MB
* Size of the generated dataset: 2.15 MB
* Total amount of disk used: 2.92 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pnb


* Size of downloaded dataset files: 3.22 MB
* Size of the generated dataset: 12.04 MB
* Total amount of disk used: 15.26 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ps


* Size of downloaded dataset files: 103.66 MB
* Size of the generated dataset: 379.51 MB
* Total amount of disk used: 483.17 MB


An example of 'train' looks as follows. #### unshuffled\_original\_pt


* Size of downloaded dataset files: 47.26 GB
* Size of the generated dataset: 132.64 GB
* Total amount of disk used: 179.89 GB


An example of 'train' looks as follows. #### unshuffled\_original\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.08 MB
* Total amount of disk used: 0.10 MB


An example of 'train' looks as follows. #### unshuffled\_original\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ro


* Size of downloaded dataset files: 9.53 GB
* Size of the generated dataset: 26.87 GB
* Total amount of disk used: 36.40 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ru


* Size of downloaded dataset files: 319.76 GB
* Size of the generated dataset: 1241.63 GB
* Total amount of disk used: 1561.38 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sa


* Size of downloaded dataset files: 17.52 MB
* Size of the generated dataset: 97.06 MB
* Total amount of disk used: 114.58 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sah


* Size of downloaded dataset files: 9.08 MB
* Size of the generated dataset: 43.82 MB
* Total amount of disk used: 52.90 MB


An example of 'train' looks as follows. #### unshuffled\_original\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sd


* Size of downloaded dataset files: 90.62 MB
* Size of the generated dataset: 364.25 MB
* Total amount of disk used: 454.88 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sh


* Size of downloaded dataset files: 3.46 MB
* Size of the generated dataset: 25.84 MB
* Total amount of disk used: 29.30 MB


An example of 'train' looks as follows. #### unshuffled\_original\_si


* Size of downloaded dataset files: 310.93 MB
* Size of the generated dataset: 1.47 GB
* Total amount of disk used: 1.78 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sk


* Size of downloaded dataset files: 3.71 GB
* Size of the generated dataset: 9.81 GB
* Total amount of disk used: 13.52 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sl


* Size of downloaded dataset files: 956.20 MB
* Size of the generated dataset: 2.68 GB
* Total amount of disk used: 3.63 GB


An example of 'train' looks as follows. #### unshuffled\_original\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sq


* Size of downloaded dataset files: 861.84 MB
* Size of the generated dataset: 2.44 GB
* Total amount of disk used: 3.30 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sr


* Size of downloaded dataset files: 1.08 GB
* Size of the generated dataset: 4.13 GB
* Total amount of disk used: 5.21 GB


An example of 'train' looks as follows. #### unshuffled\_original\_su


* Size of downloaded dataset files: 0.06 MB
* Size of the generated dataset: 0.23 MB
* Total amount of disk used: 0.28 MB


An example of 'train' looks as follows. #### unshuffled\_original\_sv


* Size of downloaded dataset files: 17.18 GB
* Size of the generated dataset: 47.00 GB
* Total amount of disk used: 64.18 GB


An example of 'train' looks as follows. #### unshuffled\_original\_sw


* Size of downloaded dataset files: 3.71 MB
* Size of the generated dataset: 14.07 MB
* Total amount of disk used: 17.78 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ta


* Size of downloaded dataset files: 1.74 GB
* Size of the generated dataset: 9.93 GB
* Total amount of disk used: 11.67 GB


An example of 'train' looks as follows. #### unshuffled\_original\_te


* Size of downloaded dataset files: 522.47 MB
* Size of the generated dataset: 2.61 GB
* Total amount of disk used: 3.13 GB


An example of 'train' looks as follows. #### unshuffled\_original\_tg


* Size of downloaded dataset files: 90.97 MB
* Size of the generated dataset: 397.43 MB
* Total amount of disk used: 488.41 MB


An example of 'train' looks as follows. #### unshuffled\_original\_th


* Size of downloaded dataset files: 7.38 GB
* Size of the generated dataset: 38.29 GB
* Total amount of disk used: 45.67 GB


An example of 'train' looks as follows. #### unshuffled\_original\_tk


* Size of downloaded dataset files: 2.96 MB
* Size of the generated dataset: 10.66 MB
* Total amount of disk used: 13.62 MB


An example of 'train' looks as follows. #### unshuffled\_original\_tl


* Size of downloaded dataset files: 204.89 MB
* Size of the generated dataset: 606.30 MB
* Total amount of disk used: 811.19 MB


An example of 'train' looks as follows. #### unshuffled\_original\_tr


* Size of downloaded dataset files: 21.96 GB
* Size of the generated dataset: 63.58 GB
* Total amount of disk used: 85.54 GB


An example of 'train' looks as follows. #### unshuffled\_original\_tt


* Size of downloaded dataset files: 151.06 MB
* Size of the generated dataset: 703.42 MB
* Total amount of disk used: 854.47 MB


An example of 'train' looks as follows. #### unshuffled\_original\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. #### unshuffled\_original\_ug


* Size of downloaded dataset files: 27.92 MB
* Size of the generated dataset: 127.42 MB
* Total amount of disk used: 155.35 MB


An example of 'train' looks as follows. #### unshuffled\_original\_uk


* Size of downloaded dataset files: 14.42 GB
* Size of the generated dataset: 56.44 GB
* Total amount of disk used: 70.86 GB


An example of 'train' looks as follows. #### unshuffled\_original\_ur


* Size of downloaded dataset files: 712.61 MB
* Size of the generated dataset: 2.80 GB
* Total amount of disk used: 3.51 GB


An example of 'train' looks as follows. #### unshuffled\_original\_uz


* Size of downloaded dataset files: 5.78 MB
* Size of the generated dataset: 21.46 MB
* Total amount of disk used: 27.24 MB


An example of 'train' looks as follows. #### unshuffled\_original\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows. #### unshuffled\_original\_vi


* Size of downloaded dataset files: 21.50 GB
* Size of the generated dataset: 72.23 GB
* Total amount of disk used: 93.73 GB


An example of 'train' looks as follows. #### unshuffled\_original\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.12 MB
* Total amount of disk used: 2.42 MB


An example of 'train' looks as follows. #### unshuffled\_original\_wa


* Size of downloaded dataset files: 0.09 MB
* Size of the generated dataset: 0.29 MB
* Total amount of disk used: 0.38 MB


An example of 'train' looks as follows. #### unshuffled\_original\_war


* Size of downloaded dataset files: 0.64 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.32 MB


An example of 'train' looks as follows. #### unshuffled\_original\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows. #### unshuffled\_original\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows. #### unshuffled\_original\_xmf


* Size of downloaded dataset files: 1.05 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.17 MB


An example of 'train' looks as follows. #### unshuffled\_original\_yi


* Size of downloaded dataset files: 33.33 MB
* Size of the generated dataset: 147.60 MB
* Total amount of disk used: 180.94 MB


An example of 'train' looks as follows. #### unshuffled\_original\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows. #### unshuffled\_original\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. #### unshuffled\_original\_zh


* Size of downloaded dataset files: 206.00 GB
* Size of the generated dataset: 545.61 GB
* Total amount of disk used: 751.61 GB


An example of 'train' looks as follows. ### Data Fields


The data fields are the same among all configs.


* 'id': a 'int64' feature.
* 'text': a 'string' feature. ### Data Splits



Click to expand the number of samples per configuration


Dataset Creation
---------------- ### Curation Rationale


OSCAR was constructed new pipeline derived from the fastText's one, called *goclassy*. Goclassy reuses the fastText linear classifier and the pre-trained fastText model for language recognition, but it completely rewrites and parallelises their pipeline in an asynchronous manner.


The order of operations is more or less the same as in the fastText pre-processing pipeline but instead of clustering multiple operations into a single blocking process, a worker is launched for each operation but bounding the number of possible parallel operations at a given time by the number of available threads instead of the number of CPUs. Goclassy is implemented in the Go programming language so it lets the Go runtime handle the scheduling of the processes. Thus the goclassy's pipeline one does not have to wait for a whole WET file to download, decompress and classify in order to start downloading and processing the next one, a new file will start downloading and processing as soon as the scheduler is able to allocate a new process.


Filtering and cleaning processes at line level are done before feeding each line to the classifier. Lines shorter than 100 UTF-8 characters and lines containing invalid UTF-8 characters are discarted and are not classified. After all files are proccesed the deduplicated versions are constructed and everything is then splitted in shards and compressed. ### Source Data #### Initial Data Collection and Normalization


Common Crawl is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable. Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metdata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected nofollow and URL policies.


Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics.


To construct OSCAR the WET files of Common Crawl were used. These contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format and is stored on Amazon Web Services. In the case of OSCAR, the November 2018 snapshot was used. It surpasses 20TB of uncompressed data and contains more than 50 thousand plain text files where each file consists of the plain text from multiple websites along its metadata header. #### Who are the source language producers?


The data comes from multiple web pages in a large variety of languages. ### Annotations


The dataset does not contain any additional annotations. #### Annotation process


N/A #### Who are the annotators?


N/A ### Personal and Sensitive Information


Being constructed from Common Crawl, Personal and sensitive information might be present. This must be considered before training deep learning models with OSCAR, specially in the case of text-generation models.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


OSCAR is intended to bring more data to a wide variety of lanuages, the aim of the corpus is to make large amounts of data available to lower resource languages in order to facilitate the pre-training of state-of-the-art language modeling architectures. ### Discussion of Biases


OSCAR is not properly filtered yet and this can be reflected on the models trained with it. Care is advised specially concerning biases of the resulting models. ### Other Known Limitations


The fastText linear classifier is limed both in performance and the variety of languages it can recognize, so the quality of some OSCAR sub-corpora might be lower than expected, specially for the lowest-resource langiuages. Some audits have already been done by third parties.


Additional Information
---------------------- ### Dataset Curators


The corpus was put together by Pedro J. Ortiz, Benoît Sagot, and Laurent Romary, during work done at Inria, particularly at the ALMAnaCH team. ### Licensing Information



```
These data are released under this licensing scheme
We do not own any of the text from which these data has been extracted.
We license the actual packaging of these data under the Creative Commons CC0 license ("no rights reserved") URL
To the extent possible under law, Inria has waived all copyright and related or neighboring rights to OSCAR
This work is published from: France.

Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
* Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
* Clearly identify the copyrighted work claimed to be infringed.
* Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

We will comply to legitimate requests by removing the affected sources from the next release of the corpus.

``` ### Contributions


Thanks to @pjox and @lhoestq for adding this dataset.
passage:  passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-100K<n<1M #size_categories-100M<n<1B #size_categories-10K<n<100K #size_categories-10M<n<100M #size_categories-1K<n<10K #size_categories-1M<n<10M #size_categories-n<1K #source_datasets-original #language-Afrikaans #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Avaric #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Russia Buriat #language-Catalan #language-Chavacano #language-Chechen #language-Cebuano #language-Central Kurdish #language-Czech #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dimli (individual language) #language-Lower Sorbian #language-Dhivehi #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-French #language-Northern Frisian #language-Western Frisian #language-Irish #language-Scottish Gaelic #language-Galician #language-Guarani #language-Goan Konkani #language-Gujarati #language-Hebrew #language-Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Iloko #language-Ido #language-Icelandic #language-Italian #language-Japanese #language-Lojban #language-Javanese #language-Georgian #language-Kazakh #language-Khmer #language-Kannada #language-Korean #language-Karachay-Balkar #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Luxembourgish #language-Lezghian #language-Limburgan #language-Lombard #language-Lao #language-Northern Luri #language-Lithuanian #language-Latvian #language-Maithili #language-Malagasy #language-Eastern Mari #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-nah #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Occitan (post 1500) #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pampanga #language-Polish #language-Piemontese #language-Western Panjabi #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Romanian #language-Russian #language-Sanskrit #language-Yakut #language-Sicilian #language-Sindhi #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Somali #language-Albanian #language-Serbian #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Tamil #language-Telugu #language-Tajik #language-Thai #language-Turkmen #language-Tagalog #language-Turkish #language-Tatar #language-Tuvinian #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venetian #language-Vietnamese #language-Volapük #language-Walloon #language-Waray (Philippines) #language-Wu Chinese #language-Kalmyk #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Chinese #license-cc0-1.0 #arxiv-2010.14571 #region-us 
### Dataset Summary


OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture. Data is distributed by language in both original and deduplicated form.


The version here is the original OSCAR 2019 release: URL


For more recent versions, visit the oscar-corpus organization on the Hub:


* OSCAR 22.01 (released in January 2022): oscar-corpus/OSCAR-2201
* OSCAR 21.09 (released in September 2021): oscar-corpus/OSCAR-2109### Supported Tasks and Leaderboards


OSCAR is mainly inteded to pretrain language models and word represantations.### Languages


All the data is distributed by language, both the original and the deduplicated versions of the data are available. 166 different languages are available. The table in subsection Data Splits Sample Size provides the language code for each subcorpus as well as the number of words (space separated tokens), lines and sizes for both the original and the deduplicated versions of OSCAR.


Dataset Structure
-----------------


We show detailed information for all the configurations of the dataset.### Data Instances



Click to expand the Data/size information for each language (deduplicated)#### unshuffled\_deduplicated\_af


* Size of downloaded dataset files: 65.99 MB
* Size of the generated dataset: 172.30 MB
* Total amount of disk used: 238.29 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_als


* Size of downloaded dataset files: 1.26 MB
* Size of the generated dataset: 2.96 MB
* Total amount of disk used: 4.22 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_am


* Size of downloaded dataset files: 61.35 MB
* Size of the generated dataset: 216.15 MB
* Total amount of disk used: 277.50 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_an


* Size of downloaded dataset files: 0.14 MB
* Size of the generated dataset: 0.85 MB
* Total amount of disk used: 0.99 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ar


* Size of downloaded dataset files: 9.67 GB
* Size of the generated dataset: 33.57 GB
* Total amount of disk used: 43.23 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_arz


* Size of downloaded dataset files: 10.02 MB
* Size of the generated dataset: 35.91 MB
* Total amount of disk used: 45.94 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_as


* Size of downloaded dataset files: 15.51 MB
* Size of the generated dataset: 74.07 MB
* Total amount of disk used: 89.58 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ast


* Size of downloaded dataset files: 0.86 MB
* Size of the generated dataset: 2.17 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_av


* Size of downloaded dataset files: 0.07 MB
* Size of the generated dataset: 0.34 MB
* Total amount of disk used: 0.41 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_az


* Size of downloaded dataset files: 521.74 MB
* Size of the generated dataset: 1.53 GB
* Total amount of disk used: 2.05 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_azb


* Size of downloaded dataset files: 5.19 MB
* Size of the generated dataset: 20.08 MB
* Total amount of disk used: 25.27 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ba


* Size of downloaded dataset files: 25.98 MB
* Size of the generated dataset: 93.84 MB
* Total amount of disk used: 119.82 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_be


* Size of downloaded dataset files: 306.70 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.39 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bg


* Size of downloaded dataset files: 3.85 GB
* Size of the generated dataset: 14.45 GB
* Total amount of disk used: 18.30 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bn


* Size of downloaded dataset files: 1.26 GB
* Size of the generated dataset: 6.24 GB
* Total amount of disk used: 7.50 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bo


* Size of downloaded dataset files: 22.37 MB
* Size of the generated dataset: 144.65 MB
* Total amount of disk used: 167.02 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bpy


* Size of downloaded dataset files: 0.19 MB
* Size of the generated dataset: 1.78 MB
* Total amount of disk used: 1.97 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_br


* Size of downloaded dataset files: 6.47 MB
* Size of the generated dataset: 17.00 MB
* Total amount of disk used: 23.47 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_bs


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.15 MB
* Total amount of disk used: 0.18 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ca


* Size of downloaded dataset files: 1.73 GB
* Size of the generated dataset: 4.57 GB
* Total amount of disk used: 6.30 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ce


* Size of downloaded dataset files: 1.87 MB
* Size of the generated dataset: 7.04 MB
* Total amount of disk used: 8.90 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ceb


* Size of downloaded dataset files: 7.12 MB
* Size of the generated dataset: 24.83 MB
* Total amount of disk used: 31.95 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ckb


* Size of downloaded dataset files: 60.32 MB
* Size of the generated dataset: 237.72 MB
* Total amount of disk used: 298.05 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_cs


* Size of downloaded dataset files: 10.49 GB
* Size of the generated dataset: 25.71 GB
* Total amount of disk used: 36.20 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_cv


* Size of downloaded dataset files: 7.47 MB
* Size of the generated dataset: 27.49 MB
* Total amount of disk used: 34.95 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_cy


* Size of downloaded dataset files: 53.63 MB
* Size of the generated dataset: 141.22 MB
* Total amount of disk used: 194.86 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_da


* Size of downloaded dataset files: 3.82 GB
* Size of the generated dataset: 10.24 GB
* Total amount of disk used: 14.06 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_de


* Size of downloaded dataset files: 60.80 GB
* Size of the generated dataset: 156.30 GB
* Total amount of disk used: 217.10 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_dv


* Size of downloaded dataset files: 16.84 MB
* Size of the generated dataset: 82.19 MB
* Total amount of disk used: 99.03 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_el


* Size of downloaded dataset files: 7.91 GB
* Size of the generated dataset: 28.74 GB
* Total amount of disk used: 36.65 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_en


* Size of downloaded dataset files: 496.50 GB
* Size of the generated dataset: 1299.75 GB
* Total amount of disk used: 1796.24 GB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_eo


* Size of downloaded dataset files: 92.86 MB
* Size of the generated dataset: 240.12 MB
* Total amount of disk used: 332.99 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_es


* Size of downloaded dataset files: 60.46 GB
* Size of the generated dataset: 160.86 GB
* Total amount of disk used: 221.32 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_et


* Size of downloaded dataset files: 966.79 MB
* Size of the generated dataset: 2.45 GB
* Total amount of disk used: 3.41 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_eu


* Size of downloaded dataset files: 134.68 MB
* Size of the generated dataset: 363.93 MB
* Total amount of disk used: 498.61 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_fa


* Size of downloaded dataset files: 10.46 GB
* Size of the generated dataset: 40.06 GB
* Total amount of disk used: 50.52 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_fi


* Size of downloaded dataset files: 5.38 GB
* Size of the generated dataset: 13.99 GB
* Total amount of disk used: 19.37 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_fr


* Size of downloaded dataset files: 55.46 GB
* Size of the generated dataset: 148.28 GB
* Total amount of disk used: 203.75 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_fy


* Size of downloaded dataset files: 10.27 MB
* Size of the generated dataset: 26.73 MB
* Total amount of disk used: 37.00 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_ga


* Size of downloaded dataset files: 22.22 MB
* Size of the generated dataset: 63.86 MB
* Total amount of disk used: 86.08 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_gd


* Size of downloaded dataset files: 0.42 MB
* Size of the generated dataset: 1.36 MB
* Total amount of disk used: 1.78 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_gl


* Size of downloaded dataset files: 155.85 MB
* Size of the generated dataset: 408.34 MB
* Total amount of disk used: 564.19 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_gom


* Size of downloaded dataset files: 0.38 MB
* Size of the generated dataset: 1.87 MB
* Total amount of disk used: 2.24 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_gu


* Size of downloaded dataset files: 162.97 MB
* Size of the generated dataset: 759.34 MB
* Total amount of disk used: 922.32 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_he


* Size of downloaded dataset files: 3.04 GB
* Size of the generated dataset: 10.47 GB
* Total amount of disk used: 13.51 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_hi


* Size of downloaded dataset files: 2.01 GB
* Size of the generated dataset: 9.57 GB
* Total amount of disk used: 11.58 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_hr


* Size of downloaded dataset files: 46.74 MB
* Size of the generated dataset: 121.50 MB
* Total amount of disk used: 168.23 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_hsb


* Size of downloaded dataset files: 0.72 MB
* Size of the generated dataset: 1.89 MB
* Total amount of disk used: 2.61 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_hu


* Size of downloaded dataset files: 7.37 GB
* Size of the generated dataset: 19.09 GB
* Total amount of disk used: 26.46 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_hy


* Size of downloaded dataset files: 393.62 MB
* Size of the generated dataset: 1.56 GB
* Total amount of disk used: 1.96 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ia


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.38 MB
* Total amount of disk used: 0.43 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_id


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 17.05 GB
* Total amount of disk used: 23.05 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ilo


* Size of downloaded dataset files: 0.23 MB
* Size of the generated dataset: 0.68 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.14 MB
* Total amount of disk used: 0.19 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_is


* Size of downloaded dataset files: 332.87 MB
* Size of the generated dataset: 894.28 MB
* Total amount of disk used: 1.23 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_it


* Size of downloaded dataset files: 27.93 GB
* Size of the generated dataset: 74.09 GB
* Total amount of disk used: 102.03 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ja


* Size of downloaded dataset files: 40.80 GB
* Size of the generated dataset: 113.63 GB
* Total amount of disk used: 154.44 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_jbo


* Size of downloaded dataset files: 0.20 MB
* Size of the generated dataset: 0.70 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_jv


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.62 MB
* Total amount of disk used: 0.82 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ka


* Size of downloaded dataset files: 377.23 MB
* Size of the generated dataset: 1.99 GB
* Total amount of disk used: 2.36 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_kk


* Size of downloaded dataset files: 389.12 MB
* Size of the generated dataset: 1.59 GB
* Total amount of disk used: 1.97 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_km


* Size of downloaded dataset files: 114.48 MB
* Size of the generated dataset: 610.61 MB
* Total amount of disk used: 725.09 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_kn


* Size of downloaded dataset files: 215.52 MB
* Size of the generated dataset: 1.08 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ko


* Size of downloaded dataset files: 4.46 GB
* Size of the generated dataset: 12.00 GB
* Total amount of disk used: 16.47 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_krc


* Size of downloaded dataset files: 0.62 MB
* Size of the generated dataset: 2.41 MB
* Total amount of disk used: 3.03 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ku


* Size of downloaded dataset files: 23.34 MB
* Size of the generated dataset: 63.09 MB
* Total amount of disk used: 86.43 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_kv


* Size of downloaded dataset files: 0.33 MB
* Size of the generated dataset: 1.21 MB
* Total amount of disk used: 1.54 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ky


* Size of downloaded dataset files: 106.22 MB
* Size of the generated dataset: 408.40 MB
* Total amount of disk used: 514.61 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_la


* Size of downloaded dataset files: 3.42 MB
* Size of the generated dataset: 9.79 MB
* Total amount of disk used: 13.22 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_lb


* Size of downloaded dataset files: 8.30 MB
* Size of the generated dataset: 21.42 MB
* Total amount of disk used: 29.72 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_lez


* Size of downloaded dataset files: 0.77 MB
* Size of the generated dataset: 3.08 MB
* Total amount of disk used: 3.84 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.46 MB
* Total amount of disk used: 0.57 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_lo


* Size of downloaded dataset files: 23.63 MB
* Size of the generated dataset: 119.29 MB
* Total amount of disk used: 142.92 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.08 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_lt


* Size of downloaded dataset files: 1.65 GB
* Size of the generated dataset: 4.20 GB
* Total amount of disk used: 5.86 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_lv


* Size of downloaded dataset files: 710.45 MB
* Size of the generated dataset: 1.91 GB
* Total amount of disk used: 2.62 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mai


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mg


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 13.59 MB
* Total amount of disk used: 17.89 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_mhr


* Size of downloaded dataset files: 1.63 MB
* Size of the generated dataset: 6.26 MB
* Total amount of disk used: 7.89 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.31 MB
* Total amount of disk used: 0.33 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mk


* Size of downloaded dataset files: 303.12 MB
* Size of the generated dataset: 1.19 GB
* Total amount of disk used: 1.49 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ml


* Size of downloaded dataset files: 496.80 MB
* Size of the generated dataset: 2.69 GB
* Total amount of disk used: 3.18 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mn


* Size of downloaded dataset files: 219.52 MB
* Size of the generated dataset: 883.46 MB
* Total amount of disk used: 1.10 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mr


* Size of downloaded dataset files: 299.68 MB
* Size of the generated dataset: 1.49 GB
* Total amount of disk used: 1.79 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mrj


* Size of downloaded dataset files: 0.29 MB
* Size of the generated dataset: 1.10 MB
* Total amount of disk used: 1.38 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ms


* Size of downloaded dataset files: 16.39 MB
* Size of the generated dataset: 49.45 MB
* Total amount of disk used: 65.85 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mt


* Size of downloaded dataset files: 5.90 MB
* Size of the generated dataset: 17.68 MB
* Total amount of disk used: 23.58 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_my


* Size of downloaded dataset files: 207.14 MB
* Size of the generated dataset: 1.11 GB
* Total amount of disk used: 1.32 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_mzn


* Size of downloaded dataset files: 0.16 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.79 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_nds


* Size of downloaded dataset files: 5.27 MB
* Size of the generated dataset: 13.48 MB
* Total amount of disk used: 18.76 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ne


* Size of downloaded dataset files: 240.63 MB
* Size of the generated dataset: 1.24 GB
* Total amount of disk used: 1.48 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_new


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 4.26 MB
* Total amount of disk used: 5.09 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_nl


* Size of downloaded dataset files: 15.73 GB
* Size of the generated dataset: 41.91 GB
* Total amount of disk used: 57.65 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_nn


* Size of downloaded dataset files: 23.58 MB
* Size of the generated dataset: 58.32 MB
* Total amount of disk used: 81.90 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_no


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 5.11 GB
* Total amount of disk used: 7.07 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_oc


* Size of downloaded dataset files: 1.34 MB
* Size of the generated dataset: 4.00 MB
* Total amount of disk used: 5.34 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_or


* Size of downloaded dataset files: 38.72 MB
* Size of the generated dataset: 197.63 MB
* Total amount of disk used: 236.36 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_os


* Size of downloaded dataset files: 2.83 MB
* Size of the generated dataset: 11.00 MB
* Total amount of disk used: 13.83 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_pa


* Size of downloaded dataset files: 102.39 MB
* Size of the generated dataset: 483.04 MB
* Total amount of disk used: 585.42 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_pl


* Size of downloaded dataset files: 20.19 GB
* Size of the generated dataset: 50.59 GB
* Total amount of disk used: 70.78 GB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_pms


* Size of downloaded dataset files: 0.71 MB
* Size of the generated dataset: 2.00 MB
* Total amount of disk used: 2.72 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_pnb


* Size of downloaded dataset files: 2.58 MB
* Size of the generated dataset: 9.44 MB
* Total amount of disk used: 12.02 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ps


* Size of downloaded dataset files: 71.83 MB
* Size of the generated dataset: 254.79 MB
* Total amount of disk used: 326.61 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_pt


* Size of downloaded dataset files: 26.00 GB
* Size of the generated dataset: 68.37 GB
* Total amount of disk used: 94.37 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ro


* Size of downloaded dataset files: 4.48 GB
* Size of the generated dataset: 11.66 GB
* Total amount of disk used: 16.14 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ru


* Size of downloaded dataset files: 166.68 GB
* Size of the generated dataset: 611.70 GB
* Total amount of disk used: 778.38 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sa


* Size of downloaded dataset files: 7.27 MB
* Size of the generated dataset: 38.33 MB
* Total amount of disk used: 45.60 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_sah


* Size of downloaded dataset files: 7.01 MB
* Size of the generated dataset: 27.46 MB
* Total amount of disk used: 34.49 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sd


* Size of downloaded dataset files: 74.17 MB
* Size of the generated dataset: 275.48 MB
* Total amount of disk used: 349.66 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sh


* Size of downloaded dataset files: 1.45 MB
* Size of the generated dataset: 6.44 MB
* Total amount of disk used: 7.87 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_si


* Size of downloaded dataset files: 175.62 MB
* Size of the generated dataset: 842.57 MB
* Total amount of disk used: 1.02 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sk


* Size of downloaded dataset files: 1.96 GB
* Size of the generated dataset: 4.80 GB
* Total amount of disk used: 6.76 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sl


* Size of downloaded dataset files: 523.22 MB
* Size of the generated dataset: 1.32 GB
* Total amount of disk used: 1.85 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sq


* Size of downloaded dataset files: 445.36 MB
* Size of the generated dataset: 1.21 GB
* Total amount of disk used: 1.66 GB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_sr


* Size of downloaded dataset files: 665.03 MB
* Size of the generated dataset: 2.36 GB
* Total amount of disk used: 3.03 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_su


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.21 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sv


* Size of downloaded dataset files: 10.19 GB
* Size of the generated dataset: 26.33 GB
* Total amount of disk used: 36.51 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_sw


* Size of downloaded dataset files: 2.95 MB
* Size of the generated dataset: 8.98 MB
* Total amount of disk used: 11.92 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ta


* Size of downloaded dataset files: 971.12 MB
* Size of the generated dataset: 5.48 GB
* Total amount of disk used: 6.45 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_te


* Size of downloaded dataset files: 342.43 MB
* Size of the generated dataset: 1.70 GB
* Total amount of disk used: 2.04 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_tg


* Size of downloaded dataset files: 62.90 MB
* Size of the generated dataset: 261.68 MB
* Total amount of disk used: 324.60 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_th


* Size of downloaded dataset files: 3.54 GB
* Size of the generated dataset: 17.11 GB
* Total amount of disk used: 20.65 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_tk


* Size of downloaded dataset files: 2.22 MB
* Size of the generated dataset: 7.12 MB
* Total amount of disk used: 9.34 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_tl


* Size of downloaded dataset files: 151.34 MB
* Size of the generated dataset: 431.69 MB
* Total amount of disk used: 583.04 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_tr


* Size of downloaded dataset files: 10.39 GB
* Size of the generated dataset: 28.47 GB
* Total amount of disk used: 38.86 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_tt


* Size of downloaded dataset files: 85.89 MB
* Size of the generated dataset: 321.37 MB
* Total amount of disk used: 407.26 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ug


* Size of downloaded dataset files: 20.53 MB
* Size of the generated dataset: 86.44 MB
* Total amount of disk used: 106.97 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_uk


* Size of downloaded dataset files: 8.04 GB
* Size of the generated dataset: 29.86 GB
* Total amount of disk used: 37.90 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_ur


* Size of downloaded dataset files: 483.59 MB
* Size of the generated dataset: 1.82 GB
* Total amount of disk used: 2.31 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_uz


* Size of downloaded dataset files: 4.30 MB
* Size of the generated dataset: 12.00 MB
* Total amount of disk used: 16.29 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_vi


* Size of downloaded dataset files: 10.71 GB
* Size of the generated dataset: 33.60 GB
* Total amount of disk used: 44.31 GB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.10 MB
* Total amount of disk used: 2.40 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_wa


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.22 MB
* Total amount of disk used: 0.29 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_war


* Size of downloaded dataset files: 0.55 MB
* Size of the generated dataset: 2.36 MB
* Total amount of disk used: 2.90 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_xmf


* Size of downloaded dataset files: 0.94 MB
* Size of the generated dataset: 4.63 MB
* Total amount of disk used: 5.58 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_yi


* Size of downloaded dataset files: 22.20 MB
* Size of the generated dataset: 88.29 MB
* Total amount of disk used: 110.49 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows. passage: #### unshuffled\_deduplicated\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_deduplicated\_zh


* Size of downloaded dataset files: 99.98 GB
* Size of the generated dataset: 267.88 GB
* Total amount of disk used: 367.86 GB


An example of 'train' looks as follows.




Click to expand the Data/size information for each language (original)#### unshuffled\_original\_af


* Size of downloaded dataset files: 85.79 MB
* Size of the generated dataset: 254.08 MB
* Total amount of disk used: 339.87 MB


An example of 'train' looks as follows.#### unshuffled\_original\_als


* Size of downloaded dataset files: 1.49 MB
* Size of the generated dataset: 5.30 MB
* Total amount of disk used: 6.78 MB


An example of 'train' looks as follows.#### unshuffled\_original\_am


* Size of downloaded dataset files: 102.79 MB
* Size of the generated dataset: 378.06 MB
* Total amount of disk used: 480.85 MB


An example of 'train' looks as follows.#### unshuffled\_original\_an


* Size of downloaded dataset files: 0.15 MB
* Size of the generated dataset: 1.33 MB
* Total amount of disk used: 1.48 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ar


* Size of downloaded dataset files: 22.23 GB
* Size of the generated dataset: 87.94 GB
* Total amount of disk used: 110.17 GB


An example of 'train' looks as follows.#### unshuffled\_original\_arz


* Size of downloaded dataset files: 15.90 MB
* Size of the generated dataset: 70.13 MB
* Total amount of disk used: 86.03 MB


An example of 'train' looks as follows.#### unshuffled\_original\_as


* Size of downloaded dataset files: 21.43 MB
* Size of the generated dataset: 117.73 MB
* Total amount of disk used: 139.17 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_ast


* Size of downloaded dataset files: 0.92 MB
* Size of the generated dataset: 2.54 MB
* Total amount of disk used: 3.46 MB


An example of 'train' looks as follows.#### unshuffled\_original\_av


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.42 MB
* Total amount of disk used: 0.50 MB


An example of 'train' looks as follows.#### unshuffled\_original\_az


* Size of downloaded dataset files: 927.76 MB
* Size of the generated dataset: 2.96 GB
* Total amount of disk used: 3.89 GB


An example of 'train' looks as follows.#### unshuffled\_original\_azb


* Size of downloaded dataset files: 6.64 MB
* Size of the generated dataset: 28.47 MB
* Total amount of disk used: 35.11 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ba


* Size of downloaded dataset files: 33.22 MB
* Size of the generated dataset: 133.70 MB
* Total amount of disk used: 166.92 MB


An example of 'train' looks as follows.#### unshuffled\_original\_bar


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_bcl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_be


* Size of downloaded dataset files: 498.29 MB
* Size of the generated dataset: 1.88 GB
* Total amount of disk used: 2.38 GB


An example of 'train' looks as follows.#### unshuffled\_original\_bg


* Size of downloaded dataset files: 8.34 GB
* Size of the generated dataset: 33.75 GB
* Total amount of disk used: 42.09 GB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_bh


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows.#### unshuffled\_original\_bn


* Size of downloaded dataset files: 2.14 GB
* Size of the generated dataset: 10.77 GB
* Total amount of disk used: 12.91 GB


An example of 'train' looks as follows.#### unshuffled\_original\_bo


* Size of downloaded dataset files: 28.94 MB
* Size of the generated dataset: 195.40 MB
* Total amount of disk used: 224.34 MB


An example of 'train' looks as follows.#### unshuffled\_original\_bpy


* Size of downloaded dataset files: 0.34 MB
* Size of the generated dataset: 4.35 MB
* Total amount of disk used: 4.69 MB


An example of 'train' looks as follows.#### unshuffled\_original\_br


* Size of downloaded dataset files: 9.18 MB
* Size of the generated dataset: 30.20 MB
* Total amount of disk used: 39.38 MB


An example of 'train' looks as follows.#### unshuffled\_original\_bs


* Size of downloaded dataset files: 0.05 MB
* Size of the generated dataset: 0.48 MB
* Total amount of disk used: 0.53 MB


An example of 'train' looks as follows.#### unshuffled\_original\_bxr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ca


* Size of downloaded dataset files: 3.10 GB
* Size of the generated dataset: 8.62 GB
* Total amount of disk used: 11.73 GB


An example of 'train' looks as follows.#### unshuffled\_original\_cbk


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_ce


* Size of downloaded dataset files: 2.09 MB
* Size of the generated dataset: 8.73 MB
* Total amount of disk used: 10.82 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ceb


* Size of downloaded dataset files: 11.07 MB
* Size of the generated dataset: 40.97 MB
* Total amount of disk used: 52.03 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ckb


* Size of downloaded dataset files: 111.88 MB
* Size of the generated dataset: 510.97 MB
* Total amount of disk used: 622.85 MB


An example of 'train' looks as follows.#### unshuffled\_original\_cs


* Size of downloaded dataset files: 21.72 GB
* Size of the generated dataset: 57.08 GB
* Total amount of disk used: 78.80 GB


An example of 'train' looks as follows.#### unshuffled\_original\_cv


* Size of downloaded dataset files: 9.40 MB
* Size of the generated dataset: 41.05 MB
* Total amount of disk used: 50.45 MB


An example of 'train' looks as follows.#### unshuffled\_original\_cy


* Size of downloaded dataset files: 81.74 MB
* Size of the generated dataset: 224.93 MB
* Total amount of disk used: 306.67 MB


An example of 'train' looks as follows.#### unshuffled\_original\_da


* Size of downloaded dataset files: 6.00 GB
* Size of the generated dataset: 16.76 GB
* Total amount of disk used: 22.76 GB


An example of 'train' looks as follows.#### unshuffled\_original\_de


* Size of downloaded dataset files: 119.51 GB
* Size of the generated dataset: 331.22 GB
* Total amount of disk used: 450.73 GB


An example of 'train' looks as follows.#### unshuffled\_original\_diq


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_dsb


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_original\_dv


* Size of downloaded dataset files: 24.91 MB
* Size of the generated dataset: 131.63 MB
* Total amount of disk used: 156.54 MB


An example of 'train' looks as follows.#### unshuffled\_original\_el


* Size of downloaded dataset files: 17.31 GB
* Size of the generated dataset: 66.27 GB
* Total amount of disk used: 83.58 GB


An example of 'train' looks as follows.#### unshuffled\_original\_eml


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.#### unshuffled\_original\_en


* Size of downloaded dataset files: 903.83 GB
* Size of the generated dataset: 2525.44 GB
* Total amount of disk used: 3429.27 GB


An example of 'train' looks as follows.#### unshuffled\_original\_eo


* Size of downloaded dataset files: 117.07 MB
* Size of the generated dataset: 314.18 MB
* Total amount of disk used: 431.27 MB


An example of 'train' looks as follows.#### unshuffled\_original\_es


* Size of downloaded dataset files: 106.04 GB
* Size of the generated dataset: 298.49 GB
* Total amount of disk used: 404.53 GB


An example of 'train' looks as follows.#### unshuffled\_original\_et


* Size of downloaded dataset files: 1.88 GB
* Size of the generated dataset: 5.17 GB
* Total amount of disk used: 7.06 GB


An example of 'train' looks as follows.#### unshuffled\_original\_eu


* Size of downloaded dataset files: 248.19 MB
* Size of the generated dataset: 894.83 MB
* Total amount of disk used: 1.14 GB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_fa


* Size of downloaded dataset files: 20.96 GB
* Size of the generated dataset: 84.21 GB
* Total amount of disk used: 105.17 GB


An example of 'train' looks as follows.#### unshuffled\_original\_fi


* Size of downloaded dataset files: 9.97 GB
* Size of the generated dataset: 28.57 GB
* Total amount of disk used: 38.54 GB


An example of 'train' looks as follows.#### unshuffled\_original\_fr


* Size of downloaded dataset files: 105.32 GB
* Size of the generated dataset: 303.19 GB
* Total amount of disk used: 408.51 GB


An example of 'train' looks as follows.#### unshuffled\_original\_frr


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_fy


* Size of downloaded dataset files: 12.40 MB
* Size of the generated dataset: 36.24 MB
* Total amount of disk used: 48.64 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ga


* Size of downloaded dataset files: 29.27 MB
* Size of the generated dataset: 92.37 MB
* Total amount of disk used: 121.63 MB


An example of 'train' looks as follows.#### unshuffled\_original\_gd


* Size of downloaded dataset files: 0.52 MB
* Size of the generated dataset: 2.02 MB
* Total amount of disk used: 2.55 MB


An example of 'train' looks as follows.#### unshuffled\_original\_gl


* Size of downloaded dataset files: 235.38 MB
* Size of the generated dataset: 656.48 MB
* Total amount of disk used: 891.87 MB


An example of 'train' looks as follows.#### unshuffled\_original\_gn


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_gom


* Size of downloaded dataset files: 0.44 MB
* Size of the generated dataset: 2.25 MB
* Total amount of disk used: 2.71 MB


An example of 'train' looks as follows.#### unshuffled\_original\_gu


* Size of downloaded dataset files: 232.02 MB
* Size of the generated dataset: 1.09 GB
* Total amount of disk used: 1.33 GB


An example of 'train' looks as follows.#### unshuffled\_original\_he


* Size of downloaded dataset files: 5.66 GB
* Size of the generated dataset: 21.11 GB
* Total amount of disk used: 26.77 GB


An example of 'train' looks as follows.#### unshuffled\_original\_hi


* Size of downloaded dataset files: 3.66 GB
* Size of the generated dataset: 17.93 GB
* Total amount of disk used: 21.59 GB


An example of 'train' looks as follows.#### unshuffled\_original\_hr


* Size of downloaded dataset files: 79.42 MB
* Size of the generated dataset: 243.83 MB
* Total amount of disk used: 323.24 MB


An example of 'train' looks as follows.#### unshuffled\_original\_hsb


* Size of downloaded dataset files: 1.39 MB
* Size of the generated dataset: 4.49 MB
* Total amount of disk used: 5.87 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ht


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_hu


* Size of downloaded dataset files: 15.69 GB
* Size of the generated dataset: 43.07 GB
* Total amount of disk used: 58.77 GB


An example of 'train' looks as follows.#### unshuffled\_original\_hy


* Size of downloaded dataset files: 897.36 MB
* Size of the generated dataset: 3.94 GB
* Total amount of disk used: 4.84 GB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_ia


* Size of downloaded dataset files: 0.08 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.78 MB


An example of 'train' looks as follows.#### unshuffled\_original\_id


* Size of downloaded dataset files: 10.60 GB
* Size of the generated dataset: 32.32 GB
* Total amount of disk used: 42.91 GB


An example of 'train' looks as follows.#### unshuffled\_original\_ie


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ilo


* Size of downloaded dataset files: 0.27 MB
* Size of the generated dataset: 0.92 MB
* Total amount of disk used: 1.20 MB


An example of 'train' looks as follows.#### unshuffled\_original\_io


* Size of downloaded dataset files: 0.04 MB
* Size of the generated dataset: 0.16 MB
* Total amount of disk used: 0.20 MB


An example of 'train' looks as follows.#### unshuffled\_original\_is


* Size of downloaded dataset files: 533.03 MB
* Size of the generated dataset: 1.52 GB
* Total amount of disk used: 2.06 GB


An example of 'train' looks as follows.#### unshuffled\_original\_it


* Size of downloaded dataset files: 52.16 GB
* Size of the generated dataset: 147.38 GB
* Total amount of disk used: 199.54 GB


An example of 'train' looks as follows.#### unshuffled\_original\_ja


* Size of downloaded dataset files: 79.56 GB
* Size of the generated dataset: 232.22 GB
* Total amount of disk used: 311.78 GB


An example of 'train' looks as follows.#### unshuffled\_original\_jbo


* Size of downloaded dataset files: 0.21 MB
* Size of the generated dataset: 0.77 MB
* Total amount of disk used: 0.98 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_jv


* Size of downloaded dataset files: 0.22 MB
* Size of the generated dataset: 0.69 MB
* Total amount of disk used: 0.91 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ka


* Size of downloaded dataset files: 680.74 MB
* Size of the generated dataset: 3.77 GB
* Total amount of disk used: 4.45 GB


An example of 'train' looks as follows.#### unshuffled\_original\_kk


* Size of downloaded dataset files: 615.06 MB
* Size of the generated dataset: 2.83 GB
* Total amount of disk used: 3.45 GB


An example of 'train' looks as follows.#### unshuffled\_original\_km


* Size of downloaded dataset files: 193.28 MB
* Size of the generated dataset: 1.10 GB
* Total amount of disk used: 1.30 GB


An example of 'train' looks as follows.#### unshuffled\_original\_kn


* Size of downloaded dataset files: 342.15 MB
* Size of the generated dataset: 1.76 GB
* Total amount of disk used: 2.11 GB


An example of 'train' looks as follows.#### unshuffled\_original\_ko


* Size of downloaded dataset files: 8.81 GB
* Size of the generated dataset: 25.29 GB
* Total amount of disk used: 34.10 GB


An example of 'train' looks as follows.#### unshuffled\_original\_krc


* Size of downloaded dataset files: 0.66 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.34 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ku


* Size of downloaded dataset files: 33.38 MB
* Size of the generated dataset: 99.06 MB
* Total amount of disk used: 132.44 MB


An example of 'train' looks as follows.#### unshuffled\_original\_kv


* Size of downloaded dataset files: 0.40 MB
* Size of the generated dataset: 2.38 MB
* Total amount of disk used: 2.78 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_kw


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.04 MB
* Total amount of disk used: 0.05 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ky


* Size of downloaded dataset files: 152.64 MB
* Size of the generated dataset: 630.79 MB
* Total amount of disk used: 783.43 MB


An example of 'train' looks as follows.#### unshuffled\_original\_la


* Size of downloaded dataset files: 5.46 MB
* Size of the generated dataset: 27.80 MB
* Total amount of disk used: 33.26 MB


An example of 'train' looks as follows.#### unshuffled\_original\_lb


* Size of downloaded dataset files: 10.73 MB
* Size of the generated dataset: 30.60 MB
* Total amount of disk used: 41.32 MB


An example of 'train' looks as follows.#### unshuffled\_original\_lez


* Size of downloaded dataset files: 0.83 MB
* Size of the generated dataset: 3.38 MB
* Total amount of disk used: 4.20 MB


An example of 'train' looks as follows.#### unshuffled\_original\_li


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.03 MB
* Total amount of disk used: 0.04 MB


An example of 'train' looks as follows.#### unshuffled\_original\_lmo


* Size of downloaded dataset files: 0.10 MB
* Size of the generated dataset: 0.47 MB
* Total amount of disk used: 0.58 MB


An example of 'train' looks as follows.#### unshuffled\_original\_lo


* Size of downloaded dataset files: 33.92 MB
* Size of the generated dataset: 182.36 MB
* Total amount of disk used: 216.28 MB


An example of 'train' looks as follows.#### unshuffled\_original\_lrc


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.07 MB
* Total amount of disk used: 0.09 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_lt


* Size of downloaded dataset files: 3.44 GB
* Size of the generated dataset: 9.45 GB
* Total amount of disk used: 12.89 GB


An example of 'train' looks as follows.#### unshuffled\_original\_lv


* Size of downloaded dataset files: 1.49 GB
* Size of the generated dataset: 4.27 GB
* Total amount of disk used: 5.75 GB


An example of 'train' looks as follows.#### unshuffled\_original\_mai


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.33 MB
* Total amount of disk used: 0.34 MB


An example of 'train' looks as follows.#### unshuffled\_original\_mg


* Size of downloaded dataset files: 6.22 MB
* Size of the generated dataset: 21.79 MB
* Total amount of disk used: 28.01 MB


An example of 'train' looks as follows.#### unshuffled\_original\_mhr


* Size of downloaded dataset files: 1.84 MB
* Size of the generated dataset: 7.55 MB
* Total amount of disk used: 9.38 MB


An example of 'train' looks as follows.#### unshuffled\_original\_min


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.63 MB
* Total amount of disk used: 0.64 MB


An example of 'train' looks as follows.#### unshuffled\_original\_mk


* Size of downloaded dataset files: 508.24 MB
* Size of the generated dataset: 2.20 GB
* Total amount of disk used: 2.71 GB


An example of 'train' looks as follows.#### unshuffled\_original\_ml


* Size of downloaded dataset files: 938.69 MB
* Size of the generated dataset: 5.24 GB
* Total amount of disk used: 6.18 GB


An example of 'train' looks as follows.#### unshuffled\_original\_mn


* Size of downloaded dataset files: 472.36 MB
* Size of the generated dataset: 2.33 GB
* Total amount of disk used: 2.81 GB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_mr


* Size of downloaded dataset files: 525.31 MB
* Size of the generated dataset: 2.82 GB
* Total amount of disk used: 3.34 GB


An example of 'train' looks as follows.#### unshuffled\_original\_mrj


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 1.16 MB
* Total amount of disk used: 1.47 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ms


* Size of downloaded dataset files: 28.46 MB
* Size of the generated dataset: 122.33 MB
* Total amount of disk used: 150.79 MB


An example of 'train' looks as follows.#### unshuffled\_original\_mt


* Size of downloaded dataset files: 7.53 MB
* Size of the generated dataset: 24.47 MB
* Total amount of disk used: 32.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_mwl


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_my


* Size of downloaded dataset files: 369.85 MB
* Size of the generated dataset: 2.02 GB
* Total amount of disk used: 2.39 GB


An example of 'train' looks as follows.#### unshuffled\_original\_myv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_mzn


* Size of downloaded dataset files: 0.18 MB
* Size of the generated dataset: 0.72 MB
* Total amount of disk used: 0.90 MB


An example of 'train' looks as follows.#### unshuffled\_original\_nah


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_nap


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.02 MB


An example of 'train' looks as follows.#### unshuffled\_original\_nds


* Size of downloaded dataset files: 6.74 MB
* Size of the generated dataset: 18.23 MB
* Total amount of disk used: 24.99 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ne


* Size of downloaded dataset files: 355.29 MB
* Size of the generated dataset: 1.87 GB
* Total amount of disk used: 2.22 GB


An example of 'train' looks as follows.#### unshuffled\_original\_new


* Size of downloaded dataset files: 1.03 MB
* Size of the generated dataset: 5.77 MB
* Total amount of disk used: 6.79 MB


An example of 'train' looks as follows.#### unshuffled\_original\_nl


* Size of downloaded dataset files: 29.35 GB
* Size of the generated dataset: 83.23 GB
* Total amount of disk used: 112.58 GB


An example of 'train' looks as follows.#### unshuffled\_original\_nn


* Size of downloaded dataset files: 32.86 MB
* Size of the generated dataset: 90.84 MB
* Total amount of disk used: 123.70 MB


An example of 'train' looks as follows.#### unshuffled\_original\_no


* Size of downloaded dataset files: 3.11 GB
* Size of the generated dataset: 8.65 GB
* Total amount of disk used: 11.76 GB


An example of 'train' looks as follows.#### unshuffled\_original\_oc


* Size of downloaded dataset files: 1.57 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.71 MB


An example of 'train' looks as follows.#### unshuffled\_original\_or


* Size of downloaded dataset files: 49.84 MB
* Size of the generated dataset: 260.15 MB
* Total amount of disk used: 309.99 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_os


* Size of downloaded dataset files: 3.09 MB
* Size of the generated dataset: 12.90 MB
* Total amount of disk used: 15.99 MB


An example of 'train' looks as follows.#### unshuffled\_original\_pa


* Size of downloaded dataset files: 164.21 MB
* Size of the generated dataset: 801.16 MB
* Total amount of disk used: 965.37 MB


An example of 'train' looks as follows.#### unshuffled\_original\_pam


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_pl


* Size of downloaded dataset files: 42.88 GB
* Size of the generated dataset: 117.12 GB
* Total amount of disk used: 160.01 GB


An example of 'train' looks as follows.#### unshuffled\_original\_pms


* Size of downloaded dataset files: 0.75 MB
* Size of the generated dataset: 2.15 MB
* Total amount of disk used: 2.92 MB


An example of 'train' looks as follows.#### unshuffled\_original\_pnb


* Size of downloaded dataset files: 3.22 MB
* Size of the generated dataset: 12.04 MB
* Total amount of disk used: 15.26 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ps


* Size of downloaded dataset files: 103.66 MB
* Size of the generated dataset: 379.51 MB
* Total amount of disk used: 483.17 MB


An example of 'train' looks as follows.#### unshuffled\_original\_pt


* Size of downloaded dataset files: 47.26 GB
* Size of the generated dataset: 132.64 GB
* Total amount of disk used: 179.89 GB


An example of 'train' looks as follows.#### unshuffled\_original\_qu


* Size of downloaded dataset files: 0.02 MB
* Size of the generated dataset: 0.08 MB
* Total amount of disk used: 0.10 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_rm


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ro


* Size of downloaded dataset files: 9.53 GB
* Size of the generated dataset: 26.87 GB
* Total amount of disk used: 36.40 GB


An example of 'train' looks as follows.#### unshuffled\_original\_ru


* Size of downloaded dataset files: 319.76 GB
* Size of the generated dataset: 1241.63 GB
* Total amount of disk used: 1561.38 GB


An example of 'train' looks as follows.#### unshuffled\_original\_sa


* Size of downloaded dataset files: 17.52 MB
* Size of the generated dataset: 97.06 MB
* Total amount of disk used: 114.58 MB


An example of 'train' looks as follows.#### unshuffled\_original\_sah


* Size of downloaded dataset files: 9.08 MB
* Size of the generated dataset: 43.82 MB
* Total amount of disk used: 52.90 MB


An example of 'train' looks as follows.#### unshuffled\_original\_scn


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_sd


* Size of downloaded dataset files: 90.62 MB
* Size of the generated dataset: 364.25 MB
* Total amount of disk used: 454.88 MB


An example of 'train' looks as follows.#### unshuffled\_original\_sh


* Size of downloaded dataset files: 3.46 MB
* Size of the generated dataset: 25.84 MB
* Total amount of disk used: 29.30 MB


An example of 'train' looks as follows.#### unshuffled\_original\_si


* Size of downloaded dataset files: 310.93 MB
* Size of the generated dataset: 1.47 GB
* Total amount of disk used: 1.78 GB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_sk


* Size of downloaded dataset files: 3.71 GB
* Size of the generated dataset: 9.81 GB
* Total amount of disk used: 13.52 GB


An example of 'train' looks as follows.#### unshuffled\_original\_sl


* Size of downloaded dataset files: 956.20 MB
* Size of the generated dataset: 2.68 GB
* Total amount of disk used: 3.63 GB


An example of 'train' looks as follows.#### unshuffled\_original\_so


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows.#### unshuffled\_original\_sq


* Size of downloaded dataset files: 861.84 MB
* Size of the generated dataset: 2.44 GB
* Total amount of disk used: 3.30 GB


An example of 'train' looks as follows.#### unshuffled\_original\_sr


* Size of downloaded dataset files: 1.08 GB
* Size of the generated dataset: 4.13 GB
* Total amount of disk used: 5.21 GB


An example of 'train' looks as follows.#### unshuffled\_original\_su


* Size of downloaded dataset files: 0.06 MB
* Size of the generated dataset: 0.23 MB
* Total amount of disk used: 0.28 MB


An example of 'train' looks as follows.#### unshuffled\_original\_sv


* Size of downloaded dataset files: 17.18 GB
* Size of the generated dataset: 47.00 GB
* Total amount of disk used: 64.18 GB


An example of 'train' looks as follows.#### unshuffled\_original\_sw


* Size of downloaded dataset files: 3.71 MB
* Size of the generated dataset: 14.07 MB
* Total amount of disk used: 17.78 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ta


* Size of downloaded dataset files: 1.74 GB
* Size of the generated dataset: 9.93 GB
* Total amount of disk used: 11.67 GB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_te


* Size of downloaded dataset files: 522.47 MB
* Size of the generated dataset: 2.61 GB
* Total amount of disk used: 3.13 GB


An example of 'train' looks as follows.#### unshuffled\_original\_tg


* Size of downloaded dataset files: 90.97 MB
* Size of the generated dataset: 397.43 MB
* Total amount of disk used: 488.41 MB


An example of 'train' looks as follows.#### unshuffled\_original\_th


* Size of downloaded dataset files: 7.38 GB
* Size of the generated dataset: 38.29 GB
* Total amount of disk used: 45.67 GB


An example of 'train' looks as follows.#### unshuffled\_original\_tk


* Size of downloaded dataset files: 2.96 MB
* Size of the generated dataset: 10.66 MB
* Total amount of disk used: 13.62 MB


An example of 'train' looks as follows.#### unshuffled\_original\_tl


* Size of downloaded dataset files: 204.89 MB
* Size of the generated dataset: 606.30 MB
* Total amount of disk used: 811.19 MB


An example of 'train' looks as follows.#### unshuffled\_original\_tr


* Size of downloaded dataset files: 21.96 GB
* Size of the generated dataset: 63.58 GB
* Total amount of disk used: 85.54 GB


An example of 'train' looks as follows.#### unshuffled\_original\_tt


* Size of downloaded dataset files: 151.06 MB
* Size of the generated dataset: 703.42 MB
* Total amount of disk used: 854.47 MB


An example of 'train' looks as follows.#### unshuffled\_original\_tyv


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.01 MB
* Total amount of disk used: 0.01 MB


An example of 'train' looks as follows.#### unshuffled\_original\_ug


* Size of downloaded dataset files: 27.92 MB
* Size of the generated dataset: 127.42 MB
* Total amount of disk used: 155.35 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_uk


* Size of downloaded dataset files: 14.42 GB
* Size of the generated dataset: 56.44 GB
* Total amount of disk used: 70.86 GB


An example of 'train' looks as follows.#### unshuffled\_original\_ur


* Size of downloaded dataset files: 712.61 MB
* Size of the generated dataset: 2.80 GB
* Total amount of disk used: 3.51 GB


An example of 'train' looks as follows.#### unshuffled\_original\_uz


* Size of downloaded dataset files: 5.78 MB
* Size of the generated dataset: 21.46 MB
* Total amount of disk used: 27.24 MB


An example of 'train' looks as follows.#### unshuffled\_original\_vec


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.02 MB
* Total amount of disk used: 0.03 MB


An example of 'train' looks as follows.#### unshuffled\_original\_vi


* Size of downloaded dataset files: 21.50 GB
* Size of the generated dataset: 72.23 GB
* Total amount of disk used: 93.73 GB


An example of 'train' looks as follows.#### unshuffled\_original\_vo


* Size of downloaded dataset files: 0.30 MB
* Size of the generated dataset: 2.12 MB
* Total amount of disk used: 2.42 MB


An example of 'train' looks as follows.#### unshuffled\_original\_wa


* Size of downloaded dataset files: 0.09 MB
* Size of the generated dataset: 0.29 MB
* Total amount of disk used: 0.38 MB


An example of 'train' looks as follows.#### unshuffled\_original\_war


* Size of downloaded dataset files: 0.64 MB
* Size of the generated dataset: 2.68 MB
* Total amount of disk used: 3.32 MB


An example of 'train' looks as follows.#### unshuffled\_original\_wuu


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.13 MB


An example of 'train' looks as follows. passage: #### unshuffled\_original\_xal


* Size of downloaded dataset files: 0.03 MB
* Size of the generated dataset: 0.12 MB
* Total amount of disk used: 0.15 MB


An example of 'train' looks as follows.#### unshuffled\_original\_xmf


* Size of downloaded dataset files: 1.05 MB
* Size of the generated dataset: 6.12 MB
* Total amount of disk used: 7.17 MB


An example of 'train' looks as follows.#### unshuffled\_original\_yi


* Size of downloaded dataset files: 33.33 MB
* Size of the generated dataset: 147.60 MB
* Total amount of disk used: 180.94 MB


An example of 'train' looks as follows.#### unshuffled\_original\_yo


* Size of downloaded dataset files: 0.01 MB
* Size of the generated dataset: 0.06 MB
* Total amount of disk used: 0.06 MB


An example of 'train' looks as follows.#### unshuffled\_original\_yue


* Size of downloaded dataset files: 0.00 MB
* Size of the generated dataset: 0.00 MB
* Total amount of disk used: 0.00 MB


An example of 'train' looks as follows.#### unshuffled\_original\_zh


* Size of downloaded dataset files: 206.00 GB
* Size of the generated dataset: 545.61 GB
* Total amount of disk used: 751.61 GB


An example of 'train' looks as follows.### Data Fields


The data fields are the same among all configs.


* 'id': a 'int64' feature.
* 'text': a 'string' feature.### Data Splits



Click to expand the number of samples per configuration


Dataset Creation
---------------- passage: ### Curation Rationale


OSCAR was constructed new pipeline derived from the fastText's one, called *goclassy*. Goclassy reuses the fastText linear classifier and the pre-trained fastText model for language recognition, but it completely rewrites and parallelises their pipeline in an asynchronous manner.


The order of operations is more or less the same as in the fastText pre-processing pipeline but instead of clustering multiple operations into a single blocking process, a worker is launched for each operation but bounding the number of possible parallel operations at a given time by the number of available threads instead of the number of CPUs. Goclassy is implemented in the Go programming language so it lets the Go runtime handle the scheduling of the processes. Thus the goclassy's pipeline one does not have to wait for a whole WET file to download, decompress and classify in order to start downloading and processing the next one, a new file will start downloading and processing as soon as the scheduler is able to allocate a new process.


Filtering and cleaning processes at line level are done before feeding each line to the classifier. Lines shorter than 100 UTF-8 characters and lines containing invalid UTF-8 characters are discarted and are not classified. After all files are proccesed the deduplicated versions are constructed and everything is then splitted in shards and compressed.### Source Data#### Initial Data Collection and Normalization


Common Crawl is a non-profit foundation which produces and maintains an open repository of web crawled data that is both accessible and analysable. Common Crawl's complete web archive consists of petabytes of data collected over 8 years of web crawling. The repository contains raw web page HTML data (WARC files), metdata extracts (WAT files) and plain text extracts (WET files). The organisation's crawlers has always respected nofollow and URL policies.


Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web pages written in a large variety of languages and covering all possible types of topics.


To construct OSCAR the WET files of Common Crawl were used. These contain the extracted plain texts from the websites mostly converted to UTF-8, as well as headers containing the metatada of each crawled document. Each WET file comes compressed in gzip format and is stored on Amazon Web Services. In the case of OSCAR, the November 2018 snapshot was used. It surpasses 20TB of uncompressed data and contains more than 50 thousand plain text files where each file consists of the plain text from multiple websites along its metadata header.#### Who are the source language producers?


The data comes from multiple web pages in a large variety of languages.### Annotations


The dataset does not contain any additional annotations.#### Annotation process


N/A#### Who are the annotators?


N/A### Personal and Sensitive Information


Being constructed from Common Crawl, Personal and sensitive information might be present. This must be considered before training deep learning models with OSCAR, specially in the case of text-generation models.


Considerations for Using the Data
---------------------------------### Social Impact of Dataset


OSCAR is intended to bring more data to a wide variety of lanuages, the aim of the corpus is to make large amounts of data available to lower resource languages in order to facilitate the pre-training of state-of-the-art language modeling architectures.