
# Dataset Card for Wikimedia Wikipedia

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [https://dumps.wikimedia.org](https://dumps.wikimedia.org)
- **Repository:**
- **Paper:**
- **Point of Contact:**

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.

The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them per date and language this way:
```python
from datasets import load_dataset

ds = load_dataset("wikimedia/wikipedia", "20231101.en")
```

#### Data Visualization
Click the [Nomic Atlas](https://atlas.nomic.ai/map/475c26d7-b142-4795-9887-02b6eeb18dc0/0d312be6-a3bb-4586-b6b7-53dcd0cbefa5) map below to visualize the 6.4 million samples in the `20231101.en` split.

<a href="https://atlas.nomic.ai/map/475c26d7-b142-4795-9887-02b6eeb18dc0/0d312be6-a3bb-4586-b6b7-53dcd0cbefa5">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/6480c476cacb1c4a0696eeb8/sZNN6Vubc0Oue83vKaJUu.webp" alt="Nomic-Atlas Wikipedia Map" width="25%"/>
</a>

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of languages here: https://meta.wikimedia.org/wiki/List_of_Wikipedias

## Dataset Structure

### Data Instances

An example looks as follows:
```
{'id': '1',
 'url': 'https://simple.wikipedia.org/wiki/April',
 'title': 'April',
 'text': 'April is the fourth month...'
}
```

### Data Fields

The data fields are the same among all configurations:
- `id` (`str`): ID of the article.
- `url` (`str`): URL of the article.
- `title` (`str`): Title of the article.
- `text` (`str`): Text content of the article.

### Data Splits

All configurations contain a single `train` split.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

The dataset is built from the Wikipedia dumps: https://dumps.wikimedia.org

You can find the full list of languages and dates here: https://dumps.wikimedia.org/backup-index.html

The articles have been parsed using the [`mwparserfromhell`](https://mwparserfromhell.readthedocs.io) tool.

When uploading the data files for the 20231101 dump, we noticed that the Wikimedia Dumps website does not contain this date dump
for the "bbc", "dga", nor "zgh" Wikipedias. We have reported the issue to the Wikimedia Phabricator: https://phabricator.wikimedia.org/T351761

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Copyright licensing information: https://dumps.wikimedia.org/legal.html

All original textual content is licensed under the [GNU Free Documentation License](https://www.gnu.org/licenses/fdl-1.3.html) (GFDL)
and the [Creative Commons Attribution-Share-Alike 3.0 License](https://creativecommons.org/licenses/by-sa/3.0/).
Some text may be available only under the Creative Commons license; see their [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) for details.
Text written by some authors may be released under additional licenses or into the public domain.

### Citation Information

```
@ONLINE{wikidump,
    author = "Wikimedia Foundation",
    title  = "Wikimedia Downloads",
    url    = "https://dumps.wikimedia.org"
}
```
task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M size_categories:1M<n<10M language:ab language:ace language:ady language:af language:alt language:am language:ami language:an language:ang language:anp language:ar language:arc language:ary language:arz language:as language:ast language:atj language:av language:avk language:awa language:ay language:az language:azb language:ba language:ban language:bar language:bbc language:bcl language:be language:bg language:bh language:bi language:bjn language:blk language:bm language:bn language:bo language:bpy language:br language:bs language:bug language:bxr language:ca language:cbk language:cdo language:ce language:ceb language:ch language:chr language:chy language:ckb language:co language:cr language:crh language:cs language:csb language:cu language:cv language:cy language:da language:dag language:de language:dga language:din language:diq language:dsb language:dty language:dv language:dz language:ee language:el language:eml language:en language:eo language:es language:et language:eu language:ext language:fa language:fat language:ff language:fi language:fj language:fo language:fon language:fr language:frp language:frr language:fur language:fy language:ga language:gag language:gan language:gcr language:gd language:gl language:glk language:gn language:gom language:gor language:got language:gpe language:gsw language:gu language:guc language:gur language:guw language:gv language:ha language:hak language:haw language:hbs language:he language:hi language:hif language:hr language:hsb language:ht language:hu language:hy language:hyw language:ia language:id language:ie language:ig language:ik language:ilo language:inh language:io language:is language:it language:iu language:ja language:jam language:jbo language:jv language:ka language:kaa language:kab language:kbd language:kbp language:kcg language:kg language:ki language:kk language:kl language:km language:kn language:ko language:koi language:krc language:ks language:ksh language:ku language:kv language:kw language:ky language:la language:lad language:lb language:lbe language:lez language:lfn language:lg language:li language:lij language:lld language:lmo language:ln language:lo language:lt language:ltg language:lv language:lzh language:mad language:mai language:map language:mdf language:mg language:mhr language:mi language:min language:mk language:ml language:mn language:mni language:mnw language:mr language:mrj language:ms language:mt language:mwl language:my language:myv language:mzn language:nah language:nan language:nap language:nds language:ne language:new language:nia language:nl language:nn language:no language:nov language:nqo language:nrf language:nso language:nv language:ny language:oc language:olo language:om language:or language:os language:pa language:pag language:pam language:pap language:pcd language:pcm language:pdc language:pfl language:pi language:pih language:pl language:pms language:pnb language:pnt language:ps language:pt language:pwn language:qu language:rm language:rmy language:rn language:ro language:ru language:rue language:rup language:rw language:sa language:sah language:sat language:sc language:scn language:sco language:sd language:se language:sg language:sgs language:shi language:shn language:si language:sk language:skr language:sl language:sm language:smn language:sn language:so language:sq language:sr language:srn language:ss language:st language:stq language:su language:sv language:sw language:szl language:szy language:ta language:tay language:tcy language:te language:tet language:tg language:th language:ti language:tk language:tl language:tly language:tn language:to language:tpi language:tr language:trv language:ts language:tt language:tum language:tw language:ty language:tyv language:udm language:ug language:uk language:ur language:uz language:ve language:vec language:vep language:vi language:vls language:vo language:vro language:wa language:war language:wo language:wuu language:xal language:xh language:xmf language:yi language:yo language:yue language:za language:zea language:zgh language:zh language:zu license:cc-by-sa-3.0 license:gfdl region:us
{"language": ["ab", "ace", "ady", "af", "alt", "am", "ami", "an", "ang", "anp", "ar", "arc", "ary", "arz", "as", "ast", "atj", "av", "avk", "awa", "ay", "az", "azb", "ba", "ban", "bar", "bbc", "bcl", "be", "bg", "bh", "bi", "bjn", "blk", "bm", "bn", "bo", "bpy", "br", "bs", "bug", "bxr", "ca", "cbk", "cdo", "ce", "ceb", "ch", "chr", "chy", "ckb", "co", "cr", "crh", "cs", "csb", "cu", "cv", "cy", "da", "dag", "de", "dga", "din", "diq", "dsb", "dty", "dv", "dz", "ee", "el", "eml", "en", "eo", "es", "et", "eu", "ext", "fa", "fat", "ff", "fi", "fj", "fo", "fon", "fr", "frp", "frr", "fur", "fy", "ga", "gag", "gan", "gcr", "gd", "gl", "glk", "gn", "gom", "gor", "got", "gpe", "gsw", "gu", "guc", "gur", "guw", "gv", "ha", "hak", "haw", "hbs", "he", "hi", "hif", "hr", "hsb", "ht", "hu", "hy", "hyw", "ia", "id", "ie", "ig", "ik", "ilo", "inh", "io", "is", "it", "iu", "ja", "jam", "jbo", "jv", "ka", "kaa", "kab", "kbd", "kbp", "kcg", "kg", "ki", "kk", "kl", "km", "kn", "ko", "koi", "krc", "ks", "ksh", "ku", "kv", "kw", "ky", "la", "lad", "lb", "lbe", "lez", "lfn", "lg", "li", "lij", "lld", "lmo", "ln", "lo", "lt", "ltg", "lv", "lzh", "mad", "mai", "map", "mdf", "mg", "mhr", "mi", "min", "mk", "ml", "mn", "mni", "mnw", "mr", "mrj", "ms", "mt", "mwl", "my", "myv", "mzn", "nah", "nan", "nap", "nds", "ne", "new", "nia", "nl", "nn", "no", "nov", "nqo", "nrf", "nso", "nv", "ny", "oc", "olo", "om", "or", "os", "pa", "pag", "pam", "pap", "pcd", "pcm", "pdc", "pfl", "pi", "pih", "pl", "pms", "pnb", "pnt", "ps", "pt", "pwn", "qu", "rm", "rmy", "rn", "ro", "ru", "rue", "rup", "rw", "sa", "sah", "sat", "sc", "scn", "sco", "sd", "se", "sg", "sgs", "shi", "shn", "si", "sk", "skr", "sl", "sm", "smn", "sn", "so", "sq", "sr", "srn", "ss", "st", "stq", "su", "sv", "sw", "szl", "szy", "ta", "tay", "tcy", "te", "tet", "tg", "th", "ti", "tk", "tl", "tly", "tn", "to", "tpi", "tr", "trv", "ts", "tt", "tum", "tw", "ty", "tyv", "udm", "ug", "uk", "ur", "uz", "ve", "vec", "vep", "vi", "vls", "vo", "vro", "wa", "war", "wo", "wuu", "xal", "xh", "xmf", "yi", "yo", "yue", "za", "zea", "zgh", "zh", "zu"], "license": ["cc-by-sa-3.0", "gfdl"], "size_categories": ["n<1K", "1K<n<10K", "10K<n<100K", "100K<n<1M", "1M<n<10M"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "configs": [{"config_name": "20231101.ab", "data_files": [{"split": "train", "path": "20231101.ab/train-*"}]}, {"config_name": "20231101.ace", "data_files": [{"split": "train", "path": "20231101.ace/train-*"}]}, {"config_name": "20231101.ady", "data_files": [{"split": "train", "path": "20231101.ady/train-*"}]}, {"config_name": "20231101.af", "data_files": [{"split": "train", "path": "20231101.af/train-*"}]}, {"config_name": "20231101.als", "data_files": [{"split": "train", "path": "20231101.als/train-*"}]}, {"config_name": "20231101.alt", "data_files": [{"split": "train", "path": "20231101.alt/train-*"}]}, {"config_name": "20231101.am", "data_files": [{"split": "train", "path": "20231101.am/train-*"}]}, {"config_name": "20231101.ami", "data_files": [{"split": "train", "path": "20231101.ami/train-*"}]}, {"config_name": "20231101.an", "data_files": [{"split": "train", "path": "20231101.an/train-*"}]}, {"config_name": "20231101.ang", "data_files": [{"split": "train", "path": "20231101.ang/train-*"}]}, {"config_name": "20231101.anp", "data_files": [{"split": "train", "path": "20231101.anp/train-*"}]}, {"config_name": "20231101.ar", "data_files": [{"split": "train", "path": "20231101.ar/train-*"}]}, {"config_name": "20231101.arc", "data_files": [{"split": "train", "path": "20231101.arc/train-*"}]}, {"config_name": "20231101.ary", "data_files": [{"split": "train", "path": "20231101.ary/train-*"}]}, {"config_name": "20231101.arz", "data_files": [{"split": "train", "path": "20231101.arz/train-*"}]}, {"config_name": "20231101.as", "data_files": [{"split": "train", "path": "20231101.as/train-*"}]}, {"config_name": "20231101.ast", "data_files": [{"split": "train", "path": "20231101.ast/train-*"}]}, {"config_name": "20231101.atj", "data_files": [{"split": "train", "path": "20231101.atj/train-*"}]}, {"config_name": "20231101.av", "data_files": [{"split": "train", "path": "20231101.av/train-*"}]}, {"config_name": "20231101.avk", "data_files": [{"split": "train", "path": "20231101.avk/train-*"}]}, {"config_name": "20231101.awa", "data_files": [{"split": "train", "path": "20231101.awa/train-*"}]}, {"config_name": "20231101.ay", "data_files": [{"split": "train", "path": "20231101.ay/train-*"}]}, {"config_name": "20231101.az", "data_files": [{"split": "train", "path": "20231101.az/train-*"}]}, {"config_name": "20231101.azb", "data_files": [{"split": "train", "path": "20231101.azb/train-*"}]}, {"config_name": "20231101.ba", "data_files": [{"split": "train", "path": "20231101.ba/train-*"}]}, {"config_name": "20231101.ban", "data_files": [{"split": "train", "path": "20231101.ban/train-*"}]}, {"config_name": "20231101.bar", "data_files": [{"split": "train", "path": "20231101.bar/train-*"}]}, {"config_name": "20231101.bat-smg", "data_files": [{"split": "train", "path": "20231101.bat-smg/train-*"}]}, {"config_name": "20231101.bcl", "data_files": [{"split": "train", "path": "20231101.bcl/train-*"}]}, {"config_name": "20231101.be", "data_files": [{"split": "train", "path": "20231101.be/train-*"}]}, {"config_name": "20231101.be-x-old", "data_files": [{"split": "train", "path": "20231101.be-x-old/train-*"}]}, {"config_name": "20231101.bg", "data_files": [{"split": "train", "path": "20231101.bg/train-*"}]}, {"config_name": "20231101.bh", "data_files": [{"split": "train", "path": "20231101.bh/train-*"}]}, {"config_name": "20231101.bi", "data_files": [{"split": "train", "path": "20231101.bi/train-*"}]}, {"config_name": "20231101.bjn", "data_files": [{"split": "train", "path": "20231101.bjn/train-*"}]}, {"config_name": "20231101.blk", "data_files": [{"split": "train", "path": "20231101.blk/train-*"}]}, {"config_name": "20231101.bm", "data_files": [{"split": "train", "path": "20231101.bm/train-*"}]}, {"config_name": "20231101.bn", "data_files": [{"split": "train", "path": "20231101.bn/train-*"}]}, {"config_name": "20231101.bo", "data_files": [{"split": "train", "path": "20231101.bo/train-*"}]}, {"config_name": "20231101.bpy", "data_files": [{"split": "train", "path": "20231101.bpy/train-*"}]}, {"config_name": "20231101.br", "data_files": [{"split": "train", "path": "20231101.br/train-*"}]}, {"config_name": "20231101.bs", "data_files": [{"split": "train", "path": "20231101.bs/train-*"}]}, {"config_name": "20231101.bug", "data_files": [{"split": "train", "path": "20231101.bug/train-*"}]}, {"config_name": "20231101.bxr", "data_files": [{"split": "train", "path": "20231101.bxr/train-*"}]}, {"config_name": "20231101.ca", "data_files": [{"split": "train", "path": "20231101.ca/train-*"}]}, {"config_name": "20231101.cbk-zam", "data_files": [{"split": "train", "path": "20231101.cbk-zam/train-*"}]}, {"config_name": "20231101.cdo", "data_files": [{"split": "train", "path": "20231101.cdo/train-*"}]}, {"config_name": "20231101.ce", "data_files": [{"split": "train", "path": "20231101.ce/train-*"}]}, {"config_name": "20231101.ceb", "data_files": [{"split": "train", "path": "20231101.ceb/train-*"}]}, {"config_name": "20231101.ch", "data_files": [{"split": "train", "path": "20231101.ch/train-*"}]}, {"config_name": "20231101.chr", "data_files": [{"split": "train", "path": "20231101.chr/train-*"}]}, {"config_name": "20231101.chy", "data_files": [{"split": "train", "path": "20231101.chy/train-*"}]}, {"config_name": "20231101.ckb", "data_files": [{"split": "train", "path": "20231101.ckb/train-*"}]}, {"config_name": "20231101.co", "data_files": [{"split": "train", "path": "20231101.co/train-*"}]}, {"config_name": "20231101.cr", "data_files": [{"split": "train", "path": "20231101.cr/train-*"}]}, {"config_name": "20231101.crh", "data_files": [{"split": "train", "path": "20231101.crh/train-*"}]}, {"config_name": "20231101.cs", "data_files": [{"split": "train", "path": "20231101.cs/train-*"}]}, {"config_name": "20231101.csb", "data_files": [{"split": "train", "path": "20231101.csb/train-*"}]}, {"config_name": "20231101.cu", "data_files": [{"split": "train", "path": "20231101.cu/train-*"}]}, {"config_name": "20231101.cv", "data_files": [{"split": "train", "path": "20231101.cv/train-*"}]}, {"config_name": "20231101.cy", "data_files": [{"split": "train", "path": "20231101.cy/train-*"}]}, {"config_name": "20231101.da", "data_files": [{"split": "train", "path": "20231101.da/train-*"}]}, {"config_name": "20231101.dag", "data_files": [{"split": "train", "path": "20231101.dag/train-*"}]}, {"config_name": "20231101.de", "data_files": [{"split": "train", "path": "20231101.de/train-*"}]}, {"config_name": "20231101.din", "data_files": [{"split": "train", "path": "20231101.din/train-*"}]}, {"config_name": "20231101.diq", "data_files": [{"split": "train", "path": "20231101.diq/train-*"}]}, {"config_name": "20231101.dsb", "data_files": [{"split": "train", "path": "20231101.dsb/train-*"}]}, {"config_name": "20231101.dty", "data_files": [{"split": "train", "path": "20231101.dty/train-*"}]}, {"config_name": "20231101.dv", "data_files": [{"split": "train", "path": "20231101.dv/train-*"}]}, {"config_name": "20231101.dz", "data_files": [{"split": "train", "path": "20231101.dz/train-*"}]}, {"config_name": "20231101.ee", "data_files": [{"split": "train", "path": "20231101.ee/train-*"}]}, {"config_name": "20231101.el", "data_files": [{"split": "train", "path": "20231101.el/train-*"}]}, {"config_name": "20231101.eml", "data_files": [{"split": "train", "path": "20231101.eml/train-*"}]}, {"config_name": "20231101.en", "data_files": [{"split": "train", "path": "20231101.en/train-*"}]}, {"config_name": "20231101.eo", "data_files": [{"split": "train", "path": "20231101.eo/train-*"}]}, {"config_name": "20231101.es", "data_files": [{"split": "train", "path": "20231101.es/train-*"}]}, {"config_name": "20231101.et", "data_files": [{"split": "train", "path": "20231101.et/train-*"}]}, {"config_name": "20231101.eu", "data_files": [{"split": "train", "path": "20231101.eu/train-*"}]}, {"config_name": "20231101.ext", "data_files": [{"split": "train", "path": "20231101.ext/train-*"}]}, {"config_name": "20231101.fa", "data_files": [{"split": "train", "path": "20231101.fa/train-*"}]}, {"config_name": "20231101.fat", "data_files": [{"split": "train", "path": "20231101.fat/train-*"}]}, {"config_name": "20231101.ff", "data_files": [{"split": "train", "path": "20231101.ff/train-*"}]}, {"config_name": "20231101.fi", "data_files": [{"split": "train", "path": "20231101.fi/train-*"}]}, {"config_name": "20231101.fiu-vro", "data_files": [{"split": "train", "path": "20231101.fiu-vro/train-*"}]}, {"config_name": "20231101.fj", "data_files": [{"split": "train", "path": "20231101.fj/train-*"}]}, {"config_name": "20231101.fo", "data_files": [{"split": "train", "path": "20231101.fo/train-*"}]}, {"config_name": "20231101.fon", "data_files": [{"split": "train", "path": "20231101.fon/train-*"}]}, {"config_name": "20231101.fr", "data_files": [{"split": "train", "path": "20231101.fr/train-*"}]}, {"config_name": "20231101.frp", "data_files": [{"split": "train", "path": "20231101.frp/train-*"}]}, {"config_name": "20231101.frr", "data_files": [{"split": "train", "path": "20231101.frr/train-*"}]}, {"config_name": "20231101.fur", "data_files": [{"split": "train", "path": "20231101.fur/train-*"}]}, {"config_name": "20231101.fy", "data_files": [{"split": "train", "path": "20231101.fy/train-*"}]}, {"config_name": "20231101.ga", "data_files": [{"split": "train", "path": "20231101.ga/train-*"}]}, {"config_name": "20231101.gag", "data_files": [{"split": "train", "path": "20231101.gag/train-*"}]}, {"config_name": "20231101.gan", "data_files": [{"split": "train", "path": "20231101.gan/train-*"}]}, {"config_name": "20231101.gcr", "data_files": [{"split": "train", "path": "20231101.gcr/train-*"}]}, {"config_name": "20231101.gd", "data_files": [{"split": "train", "path": "20231101.gd/train-*"}]}, {"config_name": "20231101.gl", "data_files": [{"split": "train", "path": "20231101.gl/train-*"}]}, {"config_name": "20231101.glk", "data_files": [{"split": "train", "path": "20231101.glk/train-*"}]}, {"config_name": "20231101.gn", "data_files": [{"split": "train", "path": "20231101.gn/train-*"}]}, {"config_name": "20231101.gom", "data_files": [{"split": "train", "path": "20231101.gom/train-*"}]}, {"config_name": "20231101.gor", "data_files": [{"split": "train", "path": "20231101.gor/train-*"}]}, {"config_name": "20231101.got", "data_files": [{"split": "train", "path": "20231101.got/train-*"}]}, {"config_name": "20231101.gpe", "data_files": [{"split": "train", "path": "20231101.gpe/train-*"}]}, {"config_name": "20231101.gu", "data_files": [{"split": "train", "path": "20231101.gu/train-*"}]}, {"config_name": "20231101.guc", "data_files": [{"split": "train", "path": "20231101.guc/train-*"}]}, {"config_name": "20231101.gur", "data_files": [{"split": "train", "path": "20231101.gur/train-*"}]}, {"config_name": "20231101.guw", "data_files": [{"split": "train", "path": "20231101.guw/train-*"}]}, {"config_name": "20231101.gv", "data_files": [{"split": "train", "path": "20231101.gv/train-*"}]}, {"config_name": "20231101.ha", "data_files": [{"split": "train", "path": "20231101.ha/train-*"}]}, {"config_name": "20231101.hak", "data_files": [{"split": "train", "path": "20231101.hak/train-*"}]}, {"config_name": "20231101.haw", "data_files": [{"split": "train", "path": "20231101.haw/train-*"}]}, {"config_name": "20231101.he", "data_files": [{"split": "train", "path": "20231101.he/train-*"}]}, {"config_name": "20231101.hi", "data_files": [{"split": "train", "path": "20231101.hi/train-*"}]}, {"config_name": "20231101.hif", "data_files": [{"split": "train", "path": "20231101.hif/train-*"}]}, {"config_name": "20231101.hr", "data_files": [{"split": "train", "path": "20231101.hr/train-*"}]}, {"config_name": "20231101.hsb", "data_files": [{"split": "train", "path": "20231101.hsb/train-*"}]}, {"config_name": "20231101.ht", "data_files": [{"split": "train", "path": "20231101.ht/train-*"}]}, {"config_name": "20231101.hu", "data_files": [{"split": "train", "path": "20231101.hu/train-*"}]}, {"config_name": "20231101.hy", "data_files": [{"split": "train", "path": "20231101.hy/train-*"}]}, {"config_name": "20231101.hyw", "data_files": [{"split": "train", "path": "20231101.hyw/train-*"}]}, {"config_name": "20231101.ia", "data_files": [{"split": "train", "path": "20231101.ia/train-*"}]}, {"config_name": "20231101.id", "data_files": [{"split": "train", "path": "20231101.id/train-*"}]}, {"config_name": "20231101.ie", "data_files": [{"split": "train", "path": "20231101.ie/train-*"}]}, {"config_name": "20231101.ig", "data_files": [{"split": "train", "path": "20231101.ig/train-*"}]}, {"config_name": "20231101.ik", "data_files": [{"split": "train", "path": "20231101.ik/train-*"}]}, {"config_name": "20231101.ilo", "data_files": [{"split": "train", "path": "20231101.ilo/train-*"}]}, {"config_name": "20231101.inh", "data_files": [{"split": "train", "path": "20231101.inh/train-*"}]}, {"config_name": "20231101.io", "data_files": [{"split": "train", "path": "20231101.io/train-*"}]}, {"config_name": "20231101.is", "data_files": [{"split": "train", "path": "20231101.is/train-*"}]}, {"config_name": "20231101.it", "data_files": [{"split": "train", "path": "20231101.it/train-*"}]}, {"config_name": "20231101.iu", "data_files": [{"split": "train", "path": "20231101.iu/train-*"}]}, {"config_name": "20231101.ja", "data_files": [{"split": "train", "path": "20231101.ja/train-*"}]}, {"config_name": "20231101.jam", "data_files": [{"split": "train", "path": "20231101.jam/train-*"}]}, {"config_name": "20231101.jbo", "data_files": [{"split": "train", "path": "20231101.jbo/train-*"}]}, {"config_name": "20231101.jv", "data_files": [{"split": "train", "path": "20231101.jv/train-*"}]}, {"config_name": "20231101.ka", "data_files": [{"split": "train", "path": "20231101.ka/train-*"}]}, {"config_name": "20231101.kaa", "data_files": [{"split": "train", "path": "20231101.kaa/train-*"}]}, {"config_name": "20231101.kab", "data_files": [{"split": "train", "path": "20231101.kab/train-*"}]}, {"config_name": "20231101.kbd", "data_files": [{"split": "train", "path": "20231101.kbd/train-*"}]}, {"config_name": "20231101.kbp", "data_files": [{"split": "train", "path": "20231101.kbp/train-*"}]}, {"config_name": "20231101.kcg", "data_files": [{"split": "train", "path": "20231101.kcg/train-*"}]}, {"config_name": "20231101.kg", "data_files": [{"split": "train", "path": "20231101.kg/train-*"}]}, {"config_name": "20231101.ki", "data_files": [{"split": "train", "path": "20231101.ki/train-*"}]}, {"config_name": "20231101.kk", "data_files": [{"split": "train", "path": "20231101.kk/train-*"}]}, {"config_name": "20231101.kl", "data_files": [{"split": "train", "path": "20231101.kl/train-*"}]}, {"config_name": "20231101.km", "data_files": [{"split": "train", "path": "20231101.km/train-*"}]}, {"config_name": "20231101.kn", "data_files": [{"split": "train", "path": "20231101.kn/train-*"}]}, {"config_name": "20231101.ko", "data_files": [{"split": "train", "path": "20231101.ko/train-*"}]}, {"config_name": "20231101.koi", "data_files": [{"split": "train", "path": "20231101.koi/train-*"}]}, {"config_name": "20231101.krc", "data_files": [{"split": "train", "path": "20231101.krc/train-*"}]}, {"config_name": "20231101.ks", "data_files": [{"split": "train", "path": "20231101.ks/train-*"}]}, {"config_name": "20231101.ksh", "data_files": [{"split": "train", "path": "20231101.ksh/train-*"}]}, {"config_name": "20231101.ku", "data_files": [{"split": "train", "path": "20231101.ku/train-*"}]}, {"config_name": "20231101.kv", "data_files": [{"split": "train", "path": "20231101.kv/train-*"}]}, {"config_name": "20231101.kw", "data_files": [{"split": "train", "path": "20231101.kw/train-*"}]}, {"config_name": "20231101.ky", "data_files": [{"split": "train", "path": "20231101.ky/train-*"}]}, {"config_name": "20231101.la", "data_files": [{"split": "train", "path": "20231101.la/train-*"}]}, {"config_name": "20231101.lad", "data_files": [{"split": "train", "path": "20231101.lad/train-*"}]}, {"config_name": "20231101.lb", "data_files": [{"split": "train", "path": "20231101.lb/train-*"}]}, {"config_name": "20231101.lbe", "data_files": [{"split": "train", "path": "20231101.lbe/train-*"}]}, {"config_name": "20231101.lez", "data_files": [{"split": "train", "path": "20231101.lez/train-*"}]}, {"config_name": "20231101.lfn", "data_files": [{"split": "train", "path": "20231101.lfn/train-*"}]}, {"config_name": "20231101.lg", "data_files": [{"split": "train", "path": "20231101.lg/train-*"}]}, {"config_name": "20231101.li", "data_files": [{"split": "train", "path": "20231101.li/train-*"}]}, {"config_name": "20231101.lij", "data_files": [{"split": "train", "path": "20231101.lij/train-*"}]}, {"config_name": "20231101.lld", "data_files": [{"split": "train", "path": "20231101.lld/train-*"}]}, {"config_name": "20231101.lmo", "data_files": [{"split": "train", "path": "20231101.lmo/train-*"}]}, {"config_name": "20231101.ln", "data_files": [{"split": "train", "path": "20231101.ln/train-*"}]}, {"config_name": "20231101.lo", "data_files": [{"split": "train", "path": "20231101.lo/train-*"}]}, {"config_name": "20231101.lt", "data_files": [{"split": "train", "path": "20231101.lt/train-*"}]}, {"config_name": "20231101.ltg", "data_files": [{"split": "train", "path": "20231101.ltg/train-*"}]}, {"config_name": "20231101.lv", "data_files": [{"split": "train", "path": "20231101.lv/train-*"}]}, {"config_name": "20231101.mad", "data_files": [{"split": "train", "path": "20231101.mad/train-*"}]}, {"config_name": "20231101.mai", "data_files": [{"split": "train", "path": "20231101.mai/train-*"}]}, {"config_name": "20231101.map-bms", "data_files": [{"split": "train", "path": "20231101.map-bms/train-*"}]}, {"config_name": "20231101.mdf", "data_files": [{"split": "train", "path": "20231101.mdf/train-*"}]}, {"config_name": "20231101.mg", "data_files": [{"split": "train", "path": "20231101.mg/train-*"}]}, {"config_name": "20231101.mhr", "data_files": [{"split": "train", "path": "20231101.mhr/train-*"}]}, {"config_name": "20231101.mi", "data_files": [{"split": "train", "path": "20231101.mi/train-*"}]}, {"config_name": "20231101.min", "data_files": [{"split": "train", "path": "20231101.min/train-*"}]}, {"config_name": "20231101.mk", "data_files": [{"split": "train", "path": "20231101.mk/train-*"}]}, {"config_name": "20231101.ml", "data_files": [{"split": "train", "path": "20231101.ml/train-*"}]}, {"config_name": "20231101.mn", "data_files": [{"split": "train", "path": "20231101.mn/train-*"}]}, {"config_name": "20231101.mni", "data_files": [{"split": "train", "path": "20231101.mni/train-*"}]}, {"config_name": "20231101.mnw", "data_files": [{"split": "train", "path": "20231101.mnw/train-*"}]}, {"config_name": "20231101.mr", "data_files": [{"split": "train", "path": "20231101.mr/train-*"}]}, {"config_name": "20231101.mrj", "data_files": [{"split": "train", "path": "20231101.mrj/train-*"}]}, {"config_name": "20231101.ms", "data_files": [{"split": "train", "path": "20231101.ms/train-*"}]}, {"config_name": "20231101.mt", "data_files": [{"split": "train", "path": "20231101.mt/train-*"}]}, {"config_name": "20231101.mwl", "data_files": [{"split": "train", "path": "20231101.mwl/train-*"}]}, {"config_name": "20231101.my", "data_files": [{"split": "train", "path": "20231101.my/train-*"}]}, {"config_name": "20231101.myv", "data_files": [{"split": "train", "path": "20231101.myv/train-*"}]}, {"config_name": "20231101.mzn", "data_files": [{"split": "train", "path": "20231101.mzn/train-*"}]}, {"config_name": "20231101.nah", "data_files": [{"split": "train", "path": "20231101.nah/train-*"}]}, {"config_name": "20231101.nap", "data_files": [{"split": "train", "path": "20231101.nap/train-*"}]}, {"config_name": "20231101.nds", "data_files": [{"split": "train", "path": "20231101.nds/train-*"}]}, {"config_name": "20231101.nds-nl", "data_files": [{"split": "train", "path": "20231101.nds-nl/train-*"}]}, {"config_name": "20231101.ne", "data_files": [{"split": "train", "path": "20231101.ne/train-*"}]}, {"config_name": "20231101.new", "data_files": [{"split": "train", "path": "20231101.new/train-*"}]}, {"config_name": "20231101.nia", "data_files": [{"split": "train", "path": "20231101.nia/train-*"}]}, {"config_name": "20231101.nl", "data_files": [{"split": "train", "path": "20231101.nl/train-*"}]}, {"config_name": "20231101.nn", "data_files": [{"split": "train", "path": "20231101.nn/train-*"}]}, {"config_name": "20231101.no", "data_files": [{"split": "train", "path": "20231101.no/train-*"}]}, {"config_name": "20231101.nov", "data_files": [{"split": "train", "path": "20231101.nov/train-*"}]}, {"config_name": "20231101.nqo", "data_files": [{"split": "train", "path": "20231101.nqo/train-*"}]}, {"config_name": "20231101.nrm", "data_files": [{"split": "train", "path": "20231101.nrm/train-*"}]}, {"config_name": "20231101.nso", "data_files": [{"split": "train", "path": "20231101.nso/train-*"}]}, {"config_name": "20231101.nv", "data_files": [{"split": "train", "path": "20231101.nv/train-*"}]}, {"config_name": "20231101.ny", "data_files": [{"split": "train", "path": "20231101.ny/train-*"}]}, {"config_name": "20231101.oc", "data_files": [{"split": "train", "path": "20231101.oc/train-*"}]}, {"config_name": "20231101.olo", "data_files": [{"split": "train", "path": "20231101.olo/train-*"}]}, {"config_name": "20231101.om", "data_files": [{"split": "train", "path": "20231101.om/train-*"}]}, {"config_name": "20231101.or", "data_files": [{"split": "train", "path": "20231101.or/train-*"}]}, {"config_name": "20231101.os", "data_files": [{"split": "train", "path": "20231101.os/train-*"}]}, {"config_name": "20231101.pa", "data_files": [{"split": "train", "path": "20231101.pa/train-*"}]}, {"config_name": "20231101.pag", "data_files": [{"split": "train", "path": "20231101.pag/train-*"}]}, {"config_name": "20231101.pam", "data_files": [{"split": "train", "path": "20231101.pam/train-*"}]}, {"config_name": "20231101.pap", "data_files": [{"split": "train", "path": "20231101.pap/train-*"}]}, {"config_name": "20231101.pcd", "data_files": [{"split": "train", "path": "20231101.pcd/train-*"}]}, {"config_name": "20231101.pcm", "data_files": [{"split": "train", "path": "20231101.pcm/train-*"}]}, {"config_name": "20231101.pdc", "data_files": [{"split": "train", "path": "20231101.pdc/train-*"}]}, {"config_name": "20231101.pfl", "data_files": [{"split": "train", "path": "20231101.pfl/train-*"}]}, {"config_name": "20231101.pi", "data_files": [{"split": "train", "path": "20231101.pi/train-*"}]}, {"config_name": "20231101.pih", "data_files": [{"split": "train", "path": "20231101.pih/train-*"}]}, {"config_name": "20231101.pl", "data_files": [{"split": "train", "path": "20231101.pl/train-*"}]}, {"config_name": "20231101.pms", "data_files": [{"split": "train", "path": "20231101.pms/train-*"}]}, {"config_name": "20231101.pnb", "data_files": [{"split": "train", "path": "20231101.pnb/train-*"}]}, {"config_name": "20231101.pnt", "data_files": [{"split": "train", "path": "20231101.pnt/train-*"}]}, {"config_name": "20231101.ps", "data_files": [{"split": "train", "path": "20231101.ps/train-*"}]}, {"config_name": "20231101.pt", "data_files": [{"split": "train", "path": "20231101.pt/train-*"}]}, {"config_name": "20231101.pwn", "data_files": [{"split": "train", "path": "20231101.pwn/train-*"}]}, {"config_name": "20231101.qu", "data_files": [{"split": "train", "path": "20231101.qu/train-*"}]}, {"config_name": "20231101.rm", "data_files": [{"split": "train", "path": "20231101.rm/train-*"}]}, {"config_name": "20231101.rmy", "data_files": [{"split": "train", "path": "20231101.rmy/train-*"}]}, {"config_name": "20231101.rn", "data_files": [{"split": "train", "path": "20231101.rn/train-*"}]}, {"config_name": "20231101.ro", "data_files": [{"split": "train", "path": "20231101.ro/train-*"}]}, {"config_name": "20231101.roa-rup", "data_files": [{"split": "train", "path": "20231101.roa-rup/train-*"}]}, {"config_name": "20231101.roa-tara", "data_files": [{"split": "train", "path": "20231101.roa-tara/train-*"}]}, {"config_name": "20231101.ru", "data_files": [{"split": "train", "path": "20231101.ru/train-*"}]}, {"config_name": "20231101.rue", "data_files": [{"split": "train", "path": "20231101.rue/train-*"}]}, {"config_name": "20231101.rw", "data_files": [{"split": "train", "path": "20231101.rw/train-*"}]}, {"config_name": "20231101.sa", "data_files": [{"split": "train", "path": "20231101.sa/train-*"}]}, {"config_name": "20231101.sah", "data_files": [{"split": "train", "path": "20231101.sah/train-*"}]}, {"config_name": "20231101.sat", "data_files": [{"split": "train", "path": "20231101.sat/train-*"}]}, {"config_name": "20231101.sc", "data_files": [{"split": "train", "path": "20231101.sc/train-*"}]}, {"config_name": "20231101.scn", "data_files": [{"split": "train", "path": "20231101.scn/train-*"}]}, {"config_name": "20231101.sco", "data_files": [{"split": "train", "path": "20231101.sco/train-*"}]}, {"config_name": "20231101.sd", "data_files": [{"split": "train", "path": "20231101.sd/train-*"}]}, {"config_name": "20231101.se", "data_files": [{"split": "train", "path": "20231101.se/train-*"}]}, {"config_name": "20231101.sg", "data_files": [{"split": "train", "path": "20231101.sg/train-*"}]}, {"config_name": "20231101.sh", "data_files": [{"split": "train", "path": "20231101.sh/train-*"}]}, {"config_name": "20231101.shi", "data_files": [{"split": "train", "path": "20231101.shi/train-*"}]}, {"config_name": "20231101.shn", "data_files": [{"split": "train", "path": "20231101.shn/train-*"}]}, {"config_name": "20231101.si", "data_files": [{"split": "train", "path": "20231101.si/train-*"}]}, {"config_name": "20231101.simple", "data_files": [{"split": "train", "path": "20231101.simple/train-*"}]}, {"config_name": "20231101.sk", "data_files": [{"split": "train", "path": "20231101.sk/train-*"}]}, {"config_name": "20231101.skr", "data_files": [{"split": "train", "path": "20231101.skr/train-*"}]}, {"config_name": "20231101.sl", "data_files": [{"split": "train", "path": "20231101.sl/train-*"}]}, {"config_name": "20231101.sm", "data_files": [{"split": "train", "path": "20231101.sm/train-*"}]}, {"config_name": "20231101.smn", "data_files": [{"split": "train", "path": "20231101.smn/train-*"}]}, {"config_name": "20231101.sn", "data_files": [{"split": "train", "path": "20231101.sn/train-*"}]}, {"config_name": "20231101.so", "data_files": [{"split": "train", "path": "20231101.so/train-*"}]}, {"config_name": "20231101.sq", "data_files": [{"split": "train", "path": "20231101.sq/train-*"}]}, {"config_name": "20231101.sr", "data_files": [{"split": "train", "path": "20231101.sr/train-*"}]}, {"config_name": "20231101.srn", "data_files": [{"split": "train", "path": "20231101.srn/train-*"}]}, {"config_name": "20231101.ss", "data_files": [{"split": "train", "path": "20231101.ss/train-*"}]}, {"config_name": "20231101.st", "data_files": [{"split": "train", "path": "20231101.st/train-*"}]}, {"config_name": "20231101.stq", "data_files": [{"split": "train", "path": "20231101.stq/train-*"}]}, {"config_name": "20231101.su", "data_files": [{"split": "train", "path": "20231101.su/train-*"}]}, {"config_name": "20231101.sv", "data_files": [{"split": "train", "path": "20231101.sv/train-*"}]}, {"config_name": "20231101.sw", "data_files": [{"split": "train", "path": "20231101.sw/train-*"}]}, {"config_name": "20231101.szl", "data_files": [{"split": "train", "path": "20231101.szl/train-*"}]}, {"config_name": "20231101.szy", "data_files": [{"split": "train", "path": "20231101.szy/train-*"}]}, {"config_name": "20231101.ta", "data_files": [{"split": "train", "path": "20231101.ta/train-*"}]}, {"config_name": "20231101.tay", "data_files": [{"split": "train", "path": "20231101.tay/train-*"}]}, {"config_name": "20231101.tcy", "data_files": [{"split": "train", "path": "20231101.tcy/train-*"}]}, {"config_name": "20231101.te", "data_files": [{"split": "train", "path": "20231101.te/train-*"}]}, {"config_name": "20231101.tet", "data_files": [{"split": "train", "path": "20231101.tet/train-*"}]}, {"config_name": "20231101.tg", "data_files": [{"split": "train", "path": "20231101.tg/train-*"}]}, {"config_name": "20231101.th", "data_files": [{"split": "train", "path": "20231101.th/train-*"}]}, {"config_name": "20231101.ti", "data_files": [{"split": "train", "path": "20231101.ti/train-*"}]}, {"config_name": "20231101.tk", "data_files": [{"split": "train", "path": "20231101.tk/train-*"}]}, {"config_name": "20231101.tl", "data_files": [{"split": "train", "path": "20231101.tl/train-*"}]}, {"config_name": "20231101.tly", "data_files": [{"split": "train", "path": "20231101.tly/train-*"}]}, {"config_name": "20231101.tn", "data_files": [{"split": "train", "path": "20231101.tn/train-*"}]}, {"config_name": "20231101.to", "data_files": [{"split": "train", "path": "20231101.to/train-*"}]}, {"config_name": "20231101.tpi", "data_files": [{"split": "train", "path": "20231101.tpi/train-*"}]}, {"config_name": "20231101.tr", "data_files": [{"split": "train", "path": "20231101.tr/train-*"}]}, {"config_name": "20231101.trv", "data_files": [{"split": "train", "path": "20231101.trv/train-*"}]}, {"config_name": "20231101.ts", "data_files": [{"split": "train", "path": "20231101.ts/train-*"}]}, {"config_name": "20231101.tt", "data_files": [{"split": "train", "path": "20231101.tt/train-*"}]}, {"config_name": "20231101.tum", "data_files": [{"split": "train", "path": "20231101.tum/train-*"}]}, {"config_name": "20231101.tw", "data_files": [{"split": "train", "path": "20231101.tw/train-*"}]}, {"config_name": "20231101.ty", "data_files": [{"split": "train", "path": "20231101.ty/train-*"}]}, {"config_name": "20231101.tyv", "data_files": [{"split": "train", "path": "20231101.tyv/train-*"}]}, {"config_name": "20231101.udm", "data_files": [{"split": "train", "path": "20231101.udm/train-*"}]}, {"config_name": "20231101.ug", "data_files": [{"split": "train", "path": "20231101.ug/train-*"}]}, {"config_name": "20231101.uk", "data_files": [{"split": "train", "path": "20231101.uk/train-*"}]}, {"config_name": "20231101.ur", "data_files": [{"split": "train", "path": "20231101.ur/train-*"}]}, {"config_name": "20231101.uz", "data_files": [{"split": "train", "path": "20231101.uz/train-*"}]}, {"config_name": "20231101.ve", "data_files": [{"split": "train", "path": "20231101.ve/train-*"}]}, {"config_name": "20231101.vec", "data_files": [{"split": "train", "path": "20231101.vec/train-*"}]}, {"config_name": "20231101.vep", "data_files": [{"split": "train", "path": "20231101.vep/train-*"}]}, {"config_name": "20231101.vi", "data_files": [{"split": "train", "path": "20231101.vi/train-*"}]}, {"config_name": "20231101.vls", "data_files": [{"split": "train", "path": "20231101.vls/train-*"}]}, {"config_name": "20231101.vo", "data_files": [{"split": "train", "path": "20231101.vo/train-*"}]}, {"config_name": "20231101.wa", "data_files": [{"split": "train", "path": "20231101.wa/train-*"}]}, {"config_name": "20231101.war", "data_files": [{"split": "train", "path": "20231101.war/train-*"}]}, {"config_name": "20231101.wo", "data_files": [{"split": "train", "path": "20231101.wo/train-*"}]}, {"config_name": "20231101.wuu", "data_files": [{"split": "train", "path": "20231101.wuu/train-*"}]}, {"config_name": "20231101.xal", "data_files": [{"split": "train", "path": "20231101.xal/train-*"}]}, {"config_name": "20231101.xh", "data_files": [{"split": "train", "path": "20231101.xh/train-*"}]}, {"config_name": "20231101.xmf", "data_files": [{"split": "train", "path": "20231101.xmf/train-*"}]}, {"config_name": "20231101.yi", "data_files": [{"split": "train", "path": "20231101.yi/train-*"}]}, {"config_name": "20231101.yo", "data_files": [{"split": "train", "path": "20231101.yo/train-*"}]}, {"config_name": "20231101.za", "data_files": [{"split": "train", "path": "20231101.za/train-*"}]}, {"config_name": "20231101.zea", "data_files": [{"split": "train", "path": "20231101.zea/train-*"}]}, {"config_name": "20231101.zh", "data_files": [{"split": "train", "path": "20231101.zh/train-*"}]}, {"config_name": "20231101.zh-classical", "data_files": [{"split": "train", "path": "20231101.zh-classical/train-*"}]}, {"config_name": "20231101.zh-min-nan", "data_files": [{"split": "train", "path": "20231101.zh-min-nan/train-*"}]}, {"config_name": "20231101.zh-yue", "data_files": [{"split": "train", "path": "20231101.zh-yue/train-*"}]}, {"config_name": "20231101.zu", "data_files": [{"split": "train", "path": "20231101.zu/train-*"}]}], "dataset_info": [{"config_name": "20231101.ab", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4334455, "num_examples": 6152}], "download_size": 1237796, "dataset_size": 4334455}, {"config_name": "20231101.ace", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5065801, "num_examples": 13003}], "download_size": 1574258, "dataset_size": 5065801}, {"config_name": "20231101.ady", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 765030, "num_examples": 706}], "download_size": 347450, "dataset_size": 765030}, {"config_name": "20231101.af", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 226672176, "num_examples": 112518}], "download_size": 124485544, "dataset_size": 226672176}, {"config_name": "20231101.als", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 81450196, "num_examples": 30013}], "download_size": 49452211, "dataset_size": 81450196}, {"config_name": "20231101.alt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6819963, "num_examples": 1087}], "download_size": 2910477, "dataset_size": 6819963}, {"config_name": "20231101.am", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24218002, "num_examples": 13906}], "download_size": 10720027, "dataset_size": 24218002}, {"config_name": "20231101.ami", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4460174, "num_examples": 1628}], "download_size": 2261859, "dataset_size": 4460174}, {"config_name": "20231101.an", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 57572050, "num_examples": 44249}], "download_size": 29573020, "dataset_size": 57572050}, {"config_name": "20231101.ang", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2913906, "num_examples": 4121}], "download_size": 1789811, "dataset_size": 2913906}, {"config_name": "20231101.anp", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9226211, "num_examples": 2749}], "download_size": 3355979, "dataset_size": 9226211}, {"config_name": "20231101.ar", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3124486159, "num_examples": 1219201}], "download_size": 1323304271, "dataset_size": 3124486159}, {"config_name": "20231101.arc", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 849731, "num_examples": 1936}], "download_size": 369584, "dataset_size": 849731}, {"config_name": "20231101.ary", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12049878, "num_examples": 8087}], "download_size": 4672257, "dataset_size": 12049878}, {"config_name": "20231101.arz", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1402294447, "num_examples": 1620194}], "download_size": 317231585, "dataset_size": 1402294447}, {"config_name": "20231101.as", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 90312333, "num_examples": 12338}], "download_size": 34581561, "dataset_size": 90312333}, {"config_name": "20231101.ast", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 470575521, "num_examples": 133419}], "download_size": 271196430, "dataset_size": 470575521}, {"config_name": "20231101.atj", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1012467, "num_examples": 1971}], "download_size": 513962, "dataset_size": 1012467}, {"config_name": "20231101.av", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6084045, "num_examples": 3426}], "download_size": 2573436, "dataset_size": 6084045}, {"config_name": "20231101.avk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 32119428, "num_examples": 28353}], "download_size": 7984474, "dataset_size": 32119428}, {"config_name": "20231101.awa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3703396, "num_examples": 3679}], "download_size": 1269824, "dataset_size": 3703396}, {"config_name": "20231101.ay", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4395813, "num_examples": 5384}], "download_size": 1756131, "dataset_size": 4395813}, {"config_name": "20231101.az", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 433663157, "num_examples": 196158}], "download_size": 230064038, "dataset_size": 433663157}, {"config_name": "20231101.azb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 187041147, "num_examples": 243376}], "download_size": 46739926, "dataset_size": 187041147}, {"config_name": "20231101.ba", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 297738837, "num_examples": 63319}], "download_size": 122595805, "dataset_size": 297738837}, {"config_name": "20231101.ban", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18012727, "num_examples": 20986}], "download_size": 6715876, "dataset_size": 18012727}, {"config_name": "20231101.bar", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 36317102, "num_examples": 27096}], "download_size": 21799389, "dataset_size": 36317102}, {"config_name": "20231101.bat-smg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7212849, "num_examples": 17221}], "download_size": 3348765, "dataset_size": 7212849}, {"config_name": "20231101.bcl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 20394331, "num_examples": 15743}], "download_size": 11369234, "dataset_size": 20394331}, {"config_name": "20231101.be", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 624718980, "num_examples": 236165}], "download_size": 284921288, "dataset_size": 624718980}, {"config_name": "20231101.be-x-old", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 252510447, "num_examples": 84361}], "download_size": 114318588, "dataset_size": 252510447}, {"config_name": "20231101.bg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1103334425, "num_examples": 294275}], "download_size": 512344058, "dataset_size": 1103334425}, {"config_name": "20231101.bh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16675295, "num_examples": 8612}], "download_size": 5880458, "dataset_size": 16675295}, {"config_name": "20231101.bi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 404249, "num_examples": 1548}], "download_size": 203610, "dataset_size": 404249}, {"config_name": "20231101.bjn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6884860, "num_examples": 10519}], "download_size": 3323032, "dataset_size": 6884860}, {"config_name": "20231101.blk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 26566991, "num_examples": 2946}], "download_size": 8028430, "dataset_size": 26566991}, {"config_name": "20231101.bm", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 623659, "num_examples": 1258}], "download_size": 343812, "dataset_size": 623659}, {"config_name": "20231101.bn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 962624238, "num_examples": 143069}], "download_size": 343885999, "dataset_size": 962624238}, {"config_name": "20231101.bo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 132723880, "num_examples": 12881}], "download_size": 38851784, "dataset_size": 132723880}, {"config_name": "20231101.bpy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 42975314, "num_examples": 25165}], "download_size": 6568483, "dataset_size": 42975314}, {"config_name": "20231101.br", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 85635744, "num_examples": 84340}], "download_size": 49768597, "dataset_size": 85635744}, {"config_name": "20231101.bs", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 193734399, "num_examples": 92596}], "download_size": 107858627, "dataset_size": 193734399}, {"config_name": "20231101.bug", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3434889, "num_examples": 15880}], "download_size": 817034, "dataset_size": 3434889}, {"config_name": "20231101.bxr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6687172, "num_examples": 2791}], "download_size": 3078699, "dataset_size": 6687172}, {"config_name": "20231101.ca", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1958810542, "num_examples": 737409}], "download_size": 1116799343, "dataset_size": 1958810542}, {"config_name": "20231101.cbk-zam", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2061944, "num_examples": 3285}], "download_size": 825899, "dataset_size": 2061944}, {"config_name": "20231101.cdo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5109207, "num_examples": 16449}], "download_size": 1982914, "dataset_size": 5109207}, {"config_name": "20231101.ce", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 730387049, "num_examples": 601271}], "download_size": 88393330, "dataset_size": 730387049}, {"config_name": "20231101.ceb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4568256711, "num_examples": 6122708}], "download_size": 828085216, "dataset_size": 4568256711}, {"config_name": "20231101.ch", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 178002, "num_examples": 576}], "download_size": 89277, "dataset_size": 178002}, {"config_name": "20231101.chr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 767618, "num_examples": 1113}], "download_size": 343140, "dataset_size": 767618}, {"config_name": "20231101.chy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 148139, "num_examples": 802}], "download_size": 75865, "dataset_size": 148139}, {"config_name": "20231101.ckb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 107150420, "num_examples": 52024}], "download_size": 42964544, "dataset_size": 107150420}, {"config_name": "20231101.co", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11104243, "num_examples": 7799}], "download_size": 5794731, "dataset_size": 11104243}, {"config_name": "20231101.cr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 57257, "num_examples": 187}], "download_size": 36081, "dataset_size": 57257}, {"config_name": "20231101.crh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9689171, "num_examples": 27691}], "download_size": 3654461, "dataset_size": 9689171}, {"config_name": "20231101.cs", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1566286962, "num_examples": 534044}], "download_size": 976484249, "dataset_size": 1566286962}, {"config_name": "20231101.csb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3748643, "num_examples": 5480}], "download_size": 2055233, "dataset_size": 3748643}, {"config_name": "20231101.cu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 981592, "num_examples": 1235}], "download_size": 398252, "dataset_size": 981592}, {"config_name": "20231101.cv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 81873026, "num_examples": 51863}], "download_size": 29640641, "dataset_size": 81873026}, {"config_name": "20231101.cy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 305837783, "num_examples": 279455}], "download_size": 112257456, "dataset_size": 305837783}, {"config_name": "20231101.da", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 547068330, "num_examples": 295347}], "download_size": 327688122, "dataset_size": 547068330}, {"config_name": "20231101.dag", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21618973, "num_examples": 10071}], "download_size": 9026986, "dataset_size": 21618973}, {"config_name": "20231101.de", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9622925305, "num_examples": 2845308}], "download_size": 5771317942, "dataset_size": 9622925305}, {"config_name": "20231101.din", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 564398, "num_examples": 512}], "download_size": 340530, "dataset_size": 564398}, {"config_name": "20231101.diq", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19671441, "num_examples": 41775}], "download_size": 7616839, "dataset_size": 19671441}, {"config_name": "20231101.dsb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3315228, "num_examples": 3379}], "download_size": 1931937, "dataset_size": 3315228}, {"config_name": "20231101.dty", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7030648, "num_examples": 3632}], "download_size": 2521250, "dataset_size": 7030648}, {"config_name": "20231101.dv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13934393, "num_examples": 4352}], "download_size": 5283133, "dataset_size": 13934393}, {"config_name": "20231101.dz", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8855969, "num_examples": 788}], "download_size": 2583520, "dataset_size": 8855969}, {"config_name": "20231101.ee", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 898491, "num_examples": 1181}], "download_size": 492813, "dataset_size": 898491}, {"config_name": "20231101.el", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1345589075, "num_examples": 226834}], "download_size": 637372489, "dataset_size": 1345589075}, {"config_name": "20231101.eml", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3625415, "num_examples": 12961}], "download_size": 1689575, "dataset_size": 3625415}, {"config_name": "20231101.en", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 20200062385, "num_examples": 6407814}], "download_size": 11630929031, "dataset_size": 20200062385}, {"config_name": "20231101.eo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 523113804, "num_examples": 344851}], "download_size": 297738138, "dataset_size": 523113804}, {"config_name": "20231101.es", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6033536133, "num_examples": 1841155}], "download_size": 3493595869, "dataset_size": 6033536133}, {"config_name": "20231101.et", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 440177170, "num_examples": 240397}], "download_size": 265444734, "dataset_size": 440177170}, {"config_name": "20231101.eu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 565567318, "num_examples": 416347}], "download_size": 270355505, "dataset_size": 565567318}, {"config_name": "20231101.ext", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4389633, "num_examples": 3785}], "download_size": 2761099, "dataset_size": 4389633}, {"config_name": "20231101.fa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1899154938, "num_examples": 979869}], "download_size": 759368283, "dataset_size": 1899154938}, {"config_name": "20231101.fat", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2032812, "num_examples": 1122}], "download_size": 1124684, "dataset_size": 2032812}, {"config_name": "20231101.ff", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1867995, "num_examples": 2419}], "download_size": 1087702, "dataset_size": 1867995}, {"config_name": "20231101.fi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1146146663, "num_examples": 561598}], "download_size": 680512230, "dataset_size": 1146146663}, {"config_name": "20231101.fiu-vro", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4636361, "num_examples": 6590}], "download_size": 2434159, "dataset_size": 4636361}, {"config_name": "20231101.fj", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 604791, "num_examples": 1294}], "download_size": 328059, "dataset_size": 604791}, {"config_name": "20231101.fo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15415249, "num_examples": 14080}], "download_size": 8857239, "dataset_size": 15415249}, {"config_name": "20231101.fon", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 592216, "num_examples": 705}], "download_size": 317444, "dataset_size": 592216}, {"config_name": "20231101.fr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8065794826, "num_examples": 2564646}], "download_size": 4614488286, "dataset_size": 8065794826}, {"config_name": "20231101.frp", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3676441, "num_examples": 5766}], "download_size": 1914046, "dataset_size": 3676441}, {"config_name": "20231101.frr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10819914, "num_examples": 18666}], "download_size": 5317694, "dataset_size": 10819914}, {"config_name": "20231101.fur", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4090412, "num_examples": 4001}], "download_size": 2421238, "dataset_size": 4090412}, {"config_name": "20231101.fy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 134196708, "num_examples": 52416}], "download_size": 76002257, "dataset_size": 134196708}, {"config_name": "20231101.ga", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 60640820, "num_examples": 59156}], "download_size": 34136733, "dataset_size": 60640820}, {"config_name": "20231101.gag", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2428849, "num_examples": 2968}], "download_size": 1331866, "dataset_size": 2428849}, {"config_name": "20231101.gan", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2915229, "num_examples": 6743}], "download_size": 1508844, "dataset_size": 2915229}, {"config_name": "20231101.gcr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2338277, "num_examples": 2399}], "download_size": 1345482, "dataset_size": 2338277}, {"config_name": "20231101.gd", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14051607, "num_examples": 15979}], "download_size": 7190137, "dataset_size": 14051607}, {"config_name": "20231101.gl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 493905881, "num_examples": 200092}], "download_size": 291104907, "dataset_size": 493905881}, {"config_name": "20231101.glk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6086185, "num_examples": 7049}], "download_size": 2382997, "dataset_size": 6086185}, {"config_name": "20231101.gn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6921948, "num_examples": 5519}], "download_size": 3806548, "dataset_size": 6921948}, {"config_name": "20231101.gom", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 30889533, "num_examples": 4259}], "download_size": 11306217, "dataset_size": 30889533}, {"config_name": "20231101.gor", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6369540, "num_examples": 15359}], "download_size": 2101154, "dataset_size": 6369540}, {"config_name": "20231101.got", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1533770, "num_examples": 1013}], "download_size": 636307, "dataset_size": 1533770}, {"config_name": "20231101.gpe", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2017667, "num_examples": 1110}], "download_size": 1141261, "dataset_size": 2017667}, {"config_name": "20231101.gu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 121282557, "num_examples": 30445}], "download_size": 39554078, "dataset_size": 121282557}, {"config_name": "20231101.guc", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 978923, "num_examples": 679}], "download_size": 578311, "dataset_size": 978923}, {"config_name": "20231101.gur", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2325435, "num_examples": 1383}], "download_size": 1068954, "dataset_size": 2325435}, {"config_name": "20231101.guw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1913143, "num_examples": 1312}], "download_size": 1042328, "dataset_size": 1913143}, {"config_name": "20231101.gv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6307253, "num_examples": 6206}], "download_size": 3347095, "dataset_size": 6307253}, {"config_name": "20231101.ha", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 77906472, "num_examples": 36492}], "download_size": 43131815, "dataset_size": 77906472}, {"config_name": "20231101.hak", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4523680, "num_examples": 10246}], "download_size": 1878558, "dataset_size": 4523680}, {"config_name": "20231101.haw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1677790, "num_examples": 2612}], "download_size": 696781, "dataset_size": 1677790}, {"config_name": "20231101.he", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1950200381, "num_examples": 333874}], "download_size": 979183998, "dataset_size": 1950200381}, {"config_name": "20231101.hi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 672817362, "num_examples": 163093}], "download_size": 237834604, "dataset_size": 672817362}, {"config_name": "20231101.hif", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5685329, "num_examples": 10986}], "download_size": 2715682, "dataset_size": 5685329}, {"config_name": "20231101.hr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 443636903, "num_examples": 202848}], "download_size": 275245343, "dataset_size": 443636903}, {"config_name": "20231101.hsb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15667118, "num_examples": 13957}], "download_size": 7437491, "dataset_size": 15667118}, {"config_name": "20231101.ht", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 55088040, "num_examples": 70159}], "download_size": 21993952, "dataset_size": 55088040}, {"config_name": "20231101.hu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1515899113, "num_examples": 532427}], "download_size": 904857314, "dataset_size": 1515899113}, {"config_name": "20231101.hy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1179459973, "num_examples": 303036}], "download_size": 490121120, "dataset_size": 1179459973}, {"config_name": "20231101.hyw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 59564550, "num_examples": 11725}], "download_size": 27450541, "dataset_size": 59564550}, {"config_name": "20231101.ia", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16409449, "num_examples": 28247}], "download_size": 8237640, "dataset_size": 16409449}, {"config_name": "20231101.id", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1125928594, "num_examples": 665622}], "download_size": 583801799, "dataset_size": 1125928594}, {"config_name": "20231101.ie", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6737711, "num_examples": 11877}], "download_size": 3019044, "dataset_size": 6737711}, {"config_name": "20231101.ig", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 66086115, "num_examples": 22908}], "download_size": 34663540, "dataset_size": 66086115}, {"config_name": "20231101.ik", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 199773, "num_examples": 846}], "download_size": 115758, "dataset_size": 199773}, {"config_name": "20231101.ilo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16854494, "num_examples": 15371}], "download_size": 7352572, "dataset_size": 16854494}, {"config_name": "20231101.inh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2727253, "num_examples": 2123}], "download_size": 1279524, "dataset_size": 2727253}, {"config_name": "20231101.io", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 38735196, "num_examples": 40930}], "download_size": 17106040, "dataset_size": 38735196}, {"config_name": "20231101.is", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 87856729, "num_examples": 57453}], "download_size": 52286137, "dataset_size": 87856729}, {"config_name": "20231101.it", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4924856310, "num_examples": 1833639}], "download_size": 2931265519, "dataset_size": 4924856310}, {"config_name": "20231101.iu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 291185, "num_examples": 562}], "download_size": 136987, "dataset_size": 291185}, {"config_name": "20231101.ja", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7039610767, "num_examples": 1389467}], "download_size": 3941998526, "dataset_size": 7039610767}, {"config_name": "20231101.jam", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1142348, "num_examples": 1780}], "download_size": 702664, "dataset_size": 1142348}, {"config_name": "20231101.jbo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2523538, "num_examples": 1394}], "download_size": 890356, "dataset_size": 2523538}, {"config_name": "20231101.jv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 72786688, "num_examples": 73380}], "download_size": 36852134, "dataset_size": 72786688}, {"config_name": "20231101.ka", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 699872960, "num_examples": 169602}], "download_size": 239987665, "dataset_size": 699872960}, {"config_name": "20231101.kaa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5139436, "num_examples": 4074}], "download_size": 2913134, "dataset_size": 5139436}, {"config_name": "20231101.kab", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4392542, "num_examples": 5830}], "download_size": 2580584, "dataset_size": 4392542}, {"config_name": "20231101.kbd", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3014575, "num_examples": 1670}], "download_size": 1304580, "dataset_size": 3014575}, {"config_name": "20231101.kbp", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3584563, "num_examples": 1931}], "download_size": 1806400, "dataset_size": 3584563}, {"config_name": "20231101.kcg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 914665, "num_examples": 1151}], "download_size": 513904, "dataset_size": 914665}, {"config_name": "20231101.kg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 390163, "num_examples": 1329}], "download_size": 209059, "dataset_size": 390163}, {"config_name": "20231101.ki", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 760980, "num_examples": 1668}], "download_size": 427003, "dataset_size": 760980}, {"config_name": "20231101.kk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 497917145, "num_examples": 238615}], "download_size": 180750520, "dataset_size": 497917145}, {"config_name": "20231101.kl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 313658, "num_examples": 301}], "download_size": 193719, "dataset_size": 313658}, {"config_name": "20231101.km", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 103252582, "num_examples": 11994}], "download_size": 35567417, "dataset_size": 103252582}, {"config_name": "20231101.kn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 402848197, "num_examples": 31437}], "download_size": 147156434, "dataset_size": 402848197}, {"config_name": "20231101.ko", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1412099944, "num_examples": 647897}], "download_size": 782677061, "dataset_size": 1412099944}, {"config_name": "20231101.koi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5103799, "num_examples": 3504}], "download_size": 1888392, "dataset_size": 5103799}, {"config_name": "20231101.krc", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4589808, "num_examples": 2100}], "download_size": 2022144, "dataset_size": 4589808}, {"config_name": "20231101.ks", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2868186, "num_examples": 4307}], "download_size": 1094458, "dataset_size": 2868186}, {"config_name": "20231101.ksh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3117003, "num_examples": 2945}], "download_size": 2009928, "dataset_size": 3117003}, {"config_name": "20231101.ku", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 44523131, "num_examples": 63076}], "download_size": 22938233, "dataset_size": 44523131}, {"config_name": "20231101.kv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9245577, "num_examples": 5595}], "download_size": 3690978, "dataset_size": 9245577}, {"config_name": "20231101.kw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4687165, "num_examples": 6995}], "download_size": 2711398, "dataset_size": 4687165}, {"config_name": "20231101.ky", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 166911089, "num_examples": 79438}], "download_size": 63947035, "dataset_size": 166911089}, {"config_name": "20231101.la", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 141080163, "num_examples": 138263}], "download_size": 76588430, "dataset_size": 141080163}, {"config_name": "20231101.lad", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4901343, "num_examples": 3663}], "download_size": 2754531, "dataset_size": 4901343}, {"config_name": "20231101.lb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 88826996, "num_examples": 62414}], "download_size": 50515020, "dataset_size": 88826996}, {"config_name": "20231101.lbe", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 745140, "num_examples": 1279}], "download_size": 304394, "dataset_size": 745140}, {"config_name": "20231101.lez", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9794637, "num_examples": 4264}], "download_size": 3864848, "dataset_size": 9794637}, {"config_name": "20231101.lfn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8870685, "num_examples": 4832}], "download_size": 5207546, "dataset_size": 8870685}, {"config_name": "20231101.lg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6891539, "num_examples": 4048}], "download_size": 3708097, "dataset_size": 6891539}, {"config_name": "20231101.li", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 29633678, "num_examples": 14849}], "download_size": 17727918, "dataset_size": 29633678}, {"config_name": "20231101.lij", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11448686, "num_examples": 11203}], "download_size": 6255409, "dataset_size": 11448686}, {"config_name": "20231101.lld", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 50163974, "num_examples": 180677}], "download_size": 13866243, "dataset_size": 50163974}, {"config_name": "20231101.lmo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 43496783, "num_examples": 73510}], "download_size": 19142356, "dataset_size": 43496783}, {"config_name": "20231101.ln", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2035050, "num_examples": 3534}], "download_size": 1122138, "dataset_size": 2035050}, {"config_name": "20231101.lo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15283258, "num_examples": 5014}], "download_size": 5646554, "dataset_size": 15283258}, {"config_name": "20231101.lt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 336559824, "num_examples": 211292}], "download_size": 194873569, "dataset_size": 336559824}, {"config_name": "20231101.ltg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 915364, "num_examples": 1070}], "download_size": 530299, "dataset_size": 915364}, {"config_name": "20231101.lv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 227272112, "num_examples": 123413}], "download_size": 129739227, "dataset_size": 227272112}, {"config_name": "20231101.mad", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1596836, "num_examples": 1192}], "download_size": 908630, "dataset_size": 1596836}, {"config_name": "20231101.mai", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21562856, "num_examples": 14714}], "download_size": 6180231, "dataset_size": 21562856}, {"config_name": "20231101.map-bms", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5341068, "num_examples": 13580}], "download_size": 2377123, "dataset_size": 5341068}, {"config_name": "20231101.mdf", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4694770, "num_examples": 4257}], "download_size": 1725294, "dataset_size": 4694770}, {"config_name": "20231101.mg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 73767229, "num_examples": 96316}], "download_size": 22117304, "dataset_size": 73767229}, {"config_name": "20231101.mhr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19249450, "num_examples": 11347}], "download_size": 6902162, "dataset_size": 19249450}, {"config_name": "20231101.mi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4169094, "num_examples": 7919}], "download_size": 1044444, "dataset_size": 4169094}, {"config_name": "20231101.min", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 118995918, "num_examples": 227143}], "download_size": 25691303, "dataset_size": 118995918}, {"config_name": "20231101.mk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 651422351, "num_examples": 139559}], "download_size": 271265486, "dataset_size": 651422351}, {"config_name": "20231101.ml", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 494135127, "num_examples": 85791}], "download_size": 183071274, "dataset_size": 494135127}, {"config_name": "20231101.mn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 91943210, "num_examples": 24048}], "download_size": 41521786, "dataset_size": 91943210}, {"config_name": "20231101.mni", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9820483, "num_examples": 10894}], "download_size": 2208525, "dataset_size": 9820483}, {"config_name": "20231101.mnw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 47237206, "num_examples": 3295}], "download_size": 13765461, "dataset_size": 47237206}, {"config_name": "20231101.mr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 261879018, "num_examples": 94133}], "download_size": 81991233, "dataset_size": 261879018}, {"config_name": "20231101.mrj", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8732281, "num_examples": 10542}], "download_size": 3283618, "dataset_size": 8732281}, {"config_name": "20231101.ms", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 423352360, "num_examples": 368628}], "download_size": 210149264, "dataset_size": 423352360}, {"config_name": "20231101.mt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 32009639, "num_examples": 5743}], "download_size": 18686521, "dataset_size": 32009639}, {"config_name": "20231101.mwl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19353725, "num_examples": 4500}], "download_size": 11521563, "dataset_size": 19353725}, {"config_name": "20231101.my", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 314417700, "num_examples": 109310}], "download_size": 85497205, "dataset_size": 314417700}, {"config_name": "20231101.myv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11145865, "num_examples": 7958}], "download_size": 4600620, "dataset_size": 11145865}, {"config_name": "20231101.mzn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16335757, "num_examples": 18717}], "download_size": 5419390, "dataset_size": 16335757}, {"config_name": "20231101.nah", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2503320, "num_examples": 6218}], "download_size": 1191779, "dataset_size": 2503320}, {"config_name": "20231101.nap", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6395706, "num_examples": 14884}], "download_size": 3188122, "dataset_size": 6395706}, {"config_name": "20231101.nds", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 92990126, "num_examples": 84285}], "download_size": 48106879, "dataset_size": 92990126}, {"config_name": "20231101.nds-nl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13582403, "num_examples": 7847}], "download_size": 8354427, "dataset_size": 13582403}, {"config_name": "20231101.ne", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 109032486, "num_examples": 32885}], "download_size": 37548833, "dataset_size": 109032486}, {"config_name": "20231101.new", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 159095610, "num_examples": 73003}], "download_size": 20517810, "dataset_size": 159095610}, {"config_name": "20231101.nia", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2117902, "num_examples": 1714}], "download_size": 1086670, "dataset_size": 2117902}, {"config_name": "20231101.nl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2646316266, "num_examples": 2135977}], "download_size": 1436843432, "dataset_size": 2646316266}, {"config_name": "20231101.nn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 237467406, "num_examples": 167653}], "download_size": 134751873, "dataset_size": 237467406}, {"config_name": "20231101.no", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1033188011, "num_examples": 617937}], "download_size": 590970350, "dataset_size": 1033188011}, {"config_name": "20231101.nov", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 965640, "num_examples": 1693}], "download_size": 493500, "dataset_size": 965640}, {"config_name": "20231101.nqo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8261058, "num_examples": 1580}], "download_size": 3508645, "dataset_size": 8261058}, {"config_name": "20231101.nrm", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3216817, "num_examples": 4902}], "download_size": 1507257, "dataset_size": 3216817}, {"config_name": "20231101.nso", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2796467, "num_examples": 8650}], "download_size": 936349, "dataset_size": 2796467}, {"config_name": "20231101.nv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16993060, "num_examples": 22460}], "download_size": 3304031, "dataset_size": 16993060}, {"config_name": "20231101.ny", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1691825, "num_examples": 1129}], "download_size": 938621, "dataset_size": 1691825}, {"config_name": "20231101.oc", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 120092607, "num_examples": 89101}], "download_size": 64043588, "dataset_size": 120092607}, {"config_name": "20231101.olo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3173332, "num_examples": 4640}], "download_size": 1724315, "dataset_size": 3173332}, {"config_name": "20231101.om", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3604768, "num_examples": 1970}], "download_size": 1982849, "dataset_size": 3604768}, {"config_name": "20231101.or", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 75078226, "num_examples": 17375}], "download_size": 26706212, "dataset_size": 75078226}, {"config_name": "20231101.os", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13182881, "num_examples": 17663}], "download_size": 5572799, "dataset_size": 13182881}, {"config_name": "20231101.pa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 212972877, "num_examples": 51423}], "download_size": 81452929, "dataset_size": 212972877}, {"config_name": "20231101.pag", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1391816, "num_examples": 2665}], "download_size": 455808, "dataset_size": 1391816}, {"config_name": "20231101.pam", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 8294902, "num_examples": 9006}], "download_size": 4277038, "dataset_size": 8294902}, {"config_name": "20231101.pap", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4251480, "num_examples": 3520}], "download_size": 2435005, "dataset_size": 4251480}, {"config_name": "20231101.pcd", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5704321, "num_examples": 5717}], "download_size": 3145572, "dataset_size": 5704321}, {"config_name": "20231101.pcm", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1886987, "num_examples": 1238}], "download_size": 1160762, "dataset_size": 1886987}, {"config_name": "20231101.pdc", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1225978, "num_examples": 2176}], "download_size": 698254, "dataset_size": 1225978}, {"config_name": "20231101.pfl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3694464, "num_examples": 2762}], "download_size": 1971214, "dataset_size": 3694464}, {"config_name": "20231101.pi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1144100, "num_examples": 3057}], "download_size": 200764, "dataset_size": 1144100}, {"config_name": "20231101.pih", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 278139, "num_examples": 934}], "download_size": 177092, "dataset_size": 278139}, {"config_name": "20231101.pl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2950148809, "num_examples": 1587721}], "download_size": 1765059986, "dataset_size": 2950148809}, {"config_name": "20231101.pms", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 34340217, "num_examples": 67980}], "download_size": 12008880, "dataset_size": 34340217}, {"config_name": "20231101.pnb", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 304117649, "num_examples": 72307}], "download_size": 133266242, "dataset_size": 304117649}, {"config_name": "20231101.pnt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 630636, "num_examples": 533}], "download_size": 275639, "dataset_size": 630636}, {"config_name": "20231101.ps", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 114259737, "num_examples": 20529}], "download_size": 53312545, "dataset_size": 114259737}, {"config_name": "20231101.pt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2758783436, "num_examples": 1112246}], "download_size": 1579641059, "dataset_size": 2758783436}, {"config_name": "20231101.pwn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 811954, "num_examples": 408}], "download_size": 444109, "dataset_size": 811954}, {"config_name": "20231101.qu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16828457, "num_examples": 24196}], "download_size": 7688106, "dataset_size": 16828457}, {"config_name": "20231101.rm", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18053014, "num_examples": 3822}], "download_size": 10483970, "dataset_size": 18053014}, {"config_name": "20231101.rmy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 611778, "num_examples": 1279}], "download_size": 356457, "dataset_size": 611778}, {"config_name": "20231101.rn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 530318, "num_examples": 819}], "download_size": 301252, "dataset_size": 530318}, {"config_name": "20231101.ro", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 847410736, "num_examples": 442389}], "download_size": 466937380, "dataset_size": 847410736}, {"config_name": "20231101.roa-rup", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1687829, "num_examples": 1432}], "download_size": 951677, "dataset_size": 1687829}, {"config_name": "20231101.roa-tara", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7470331, "num_examples": 9367}], "download_size": 4003095, "dataset_size": 7470331}, {"config_name": "20231101.ru", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10277958919, "num_examples": 1945063}], "download_size": 4876849588, "dataset_size": 10277958919}, {"config_name": "20231101.rue", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13128572, "num_examples": 8759}], "download_size": 6346106, "dataset_size": 13128572}, {"config_name": "20231101.rw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11898854, "num_examples": 8063}], "download_size": 6623388, "dataset_size": 11898854}, {"config_name": "20231101.sa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 69854997, "num_examples": 12156}], "download_size": 23850161, "dataset_size": 69854997}, {"config_name": "20231101.sah", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 48562374, "num_examples": 17098}], "download_size": 21675888, "dataset_size": 48562374}, {"config_name": "20231101.sat", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 45247783, "num_examples": 9767}], "download_size": 15428584, "dataset_size": 45247783}, {"config_name": "20231101.sc", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12776438, "num_examples": 7586}], "download_size": 7711996, "dataset_size": 12776438}, {"config_name": "20231101.scn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17685098, "num_examples": 26530}], "download_size": 10223816, "dataset_size": 17685098}, {"config_name": "20231101.sco", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 42808738, "num_examples": 35276}], "download_size": 24287944, "dataset_size": 42808738}, {"config_name": "20231101.sd", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 37021659, "num_examples": 16928}], "download_size": 17591997, "dataset_size": 37021659}, {"config_name": "20231101.se", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3600527, "num_examples": 8043}], "download_size": 1816006, "dataset_size": 3600527}, {"config_name": "20231101.sg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 140127, "num_examples": 564}], "download_size": 72486, "dataset_size": 140127}, {"config_name": "20231101.sh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 569225870, "num_examples": 458392}], "download_size": 266379293, "dataset_size": 569225870}, {"config_name": "20231101.shi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2369002, "num_examples": 1779}], "download_size": 1359828, "dataset_size": 2369002}, {"config_name": "20231101.shn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33553593, "num_examples": 13945}], "download_size": 8163231, "dataset_size": 33553593}, {"config_name": "20231101.si", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 138806443, "num_examples": 23065}], "download_size": 54229127, "dataset_size": 138806443}, {"config_name": "20231101.simple", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 291254232, "num_examples": 241787}], "download_size": 156885218, "dataset_size": 291254232}, {"config_name": "20231101.sk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 416804817, "num_examples": 242235}], "download_size": 239513292, "dataset_size": 416804817}, {"config_name": "20231101.skr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 22705446, "num_examples": 5819}], "download_size": 9978607, "dataset_size": 22705446}, {"config_name": "20231101.sl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 454829910, "num_examples": 183006}], "download_size": 267485569, "dataset_size": 454829910}, {"config_name": "20231101.sm", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 902927, "num_examples": 1151}], "download_size": 492349, "dataset_size": 902927}, {"config_name": "20231101.smn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5764244, "num_examples": 5383}], "download_size": 2813872, "dataset_size": 5764244}, {"config_name": "20231101.sn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9790528, "num_examples": 11621}], "download_size": 4979456, "dataset_size": 9790528}, {"config_name": "20231101.so", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13663784, "num_examples": 9021}], "download_size": 7940363, "dataset_size": 13663784}, {"config_name": "20231101.sq", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 208779652, "num_examples": 104854}], "download_size": 116945494, "dataset_size": 208779652}, {"config_name": "20231101.sr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1721596392, "num_examples": 676605}], "download_size": 697391786, "dataset_size": 1721596392}, {"config_name": "20231101.srn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 649317, "num_examples": 1219}], "download_size": 215103, "dataset_size": 649317}, {"config_name": "20231101.ss", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1076102, "num_examples": 945}], "download_size": 600997, "dataset_size": 1076102}, {"config_name": "20231101.st", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 968161, "num_examples": 1099}], "download_size": 530165, "dataset_size": 968161}, {"config_name": "20231101.stq", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4942784, "num_examples": 4134}], "download_size": 2884429, "dataset_size": 4942784}, {"config_name": "20231101.su", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 48066965, "num_examples": 61555}], "download_size": 19806020, "dataset_size": 48066965}, {"config_name": "20231101.sv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2153690744, "num_examples": 2574513}], "download_size": 974261228, "dataset_size": 2153690744}, {"config_name": "20231101.sw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 73119299, "num_examples": 78587}], "download_size": 35936177, "dataset_size": 73119299}, {"config_name": "20231101.szl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21439309, "num_examples": 57035}], "download_size": 7347967, "dataset_size": 21439309}, {"config_name": "20231101.szy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11355780, "num_examples": 4885}], "download_size": 6192815, "dataset_size": 11355780}, {"config_name": "20231101.ta", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 810734099, "num_examples": 160651}], "download_size": 265652020, "dataset_size": 810734099}, {"config_name": "20231101.tay", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2974229, "num_examples": 2747}], "download_size": 1232811, "dataset_size": 2974229}, {"config_name": "20231101.tcy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12166612, "num_examples": 2202}], "download_size": 4611006, "dataset_size": 12166612}, {"config_name": "20231101.te", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 730376585, "num_examples": 87854}], "download_size": 215097076, "dataset_size": 730376585}, {"config_name": "20231101.tet", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1466200, "num_examples": 1468}], "download_size": 744390, "dataset_size": 1466200}, {"config_name": "20231101.tg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 148256281, "num_examples": 110962}], "download_size": 49825647, "dataset_size": 148256281}, {"config_name": "20231101.th", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1014547923, "num_examples": 159719}], "download_size": 371916105, "dataset_size": 1014547923}, {"config_name": "20231101.ti", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 729995, "num_examples": 435}], "download_size": 363723, "dataset_size": 729995}, {"config_name": "20231101.tk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13326412, "num_examples": 7918}], "download_size": 7383654, "dataset_size": 13326412}, {"config_name": "20231101.tl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 85794472, "num_examples": 45341}], "download_size": 45797527, "dataset_size": 85794472}, {"config_name": "20231101.tly", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2590482, "num_examples": 8086}], "download_size": 1070456, "dataset_size": 2590482}, {"config_name": "20231101.tn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4380768, "num_examples": 1585}], "download_size": 1708110, "dataset_size": 4380768}, {"config_name": "20231101.to", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1090611, "num_examples": 1887}], "download_size": 518244, "dataset_size": 1090611}, {"config_name": "20231101.tpi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 460420, "num_examples": 1399}], "download_size": 241908, "dataset_size": 460420}, {"config_name": "20231101.tr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 997254242, "num_examples": 534988}], "download_size": 552923659, "dataset_size": 997254242}, {"config_name": "20231101.trv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4971204, "num_examples": 1880}], "download_size": 2706664, "dataset_size": 4971204}, {"config_name": "20231101.ts", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 847032, "num_examples": 785}], "download_size": 455648, "dataset_size": 847032}, {"config_name": "20231101.tt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 681325421, "num_examples": 501116}], "download_size": 129141056, "dataset_size": 681325421}, {"config_name": "20231101.tum", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 13429984, "num_examples": 18708}], "download_size": 5459856, "dataset_size": 13429984}, {"config_name": "20231101.tw", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7982767, "num_examples": 3978}], "download_size": 4118530, "dataset_size": 7982767}, {"config_name": "20231101.ty", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 338743, "num_examples": 1355}], "download_size": 150963, "dataset_size": 338743}, {"config_name": "20231101.tyv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14324694, "num_examples": 3491}], "download_size": 6528290, "dataset_size": 14324694}, {"config_name": "20231101.udm", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7036113, "num_examples": 5677}], "download_size": 2982821, "dataset_size": 7036113}, {"config_name": "20231101.ug", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 42254159, "num_examples": 8634}], "download_size": 17741860, "dataset_size": 42254159}, {"config_name": "20231101.uk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 4969483901, "num_examples": 1294720}], "download_size": 2276769383, "dataset_size": 4969483901}, {"config_name": "20231101.ur", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 410511855, "num_examples": 200154}], "download_size": 167627869, "dataset_size": 410511855}, {"config_name": "20231101.uz", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 397176774, "num_examples": 246729}], "download_size": 210262652, "dataset_size": 397176774}, {"config_name": "20231101.ve", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 359542, "num_examples": 840}], "download_size": 163318, "dataset_size": 359542}, {"config_name": "20231101.vec", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 37917528, "num_examples": 69268}], "download_size": 16179506, "dataset_size": 37917528}, {"config_name": "20231101.vep", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11643856, "num_examples": 6960}], "download_size": 6423002, "dataset_size": 11643856}, {"config_name": "20231101.vi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1617830227, "num_examples": 1288680}], "download_size": 729557588, "dataset_size": 1617830227}, {"config_name": "20231101.vls", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 11336278, "num_examples": 7872}], "download_size": 6985406, "dataset_size": 11336278}, {"config_name": "20231101.vo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19521708, "num_examples": 35193}], "download_size": 6582571, "dataset_size": 19521708}, {"config_name": "20231101.wa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12268826, "num_examples": 12038}], "download_size": 7327616, "dataset_size": 12268826}, {"config_name": "20231101.war", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 467647882, "num_examples": 1266394}], "download_size": 104588442, "dataset_size": 467647882}, {"config_name": "20231101.wo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3525303, "num_examples": 1746}], "download_size": 2094574, "dataset_size": 3525303}, {"config_name": "20231101.wuu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 25029545, "num_examples": 43010}], "download_size": 15985963, "dataset_size": 25029545}, {"config_name": "20231101.xal", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1391731, "num_examples": 2295}], "download_size": 507198, "dataset_size": 1391731}, {"config_name": "20231101.xh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3665998, "num_examples": 1883}], "download_size": 2505472, "dataset_size": 3665998}, {"config_name": "20231101.xmf", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 37712629, "num_examples": 18099}], "download_size": 12948576, "dataset_size": 37712629}, {"config_name": "20231101.yi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 36038273, "num_examples": 15179}], "download_size": 16218296, "dataset_size": 36038273}, {"config_name": "20231101.yo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 19081408, "num_examples": 33819}], "download_size": 8861465, "dataset_size": 19081408}, {"config_name": "20231101.za", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1365300, "num_examples": 2993}], "download_size": 666521, "dataset_size": 1365300}, {"config_name": "20231101.zea", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5224563, "num_examples": 6082}], "download_size": 2620396, "dataset_size": 5224563}, {"config_name": "20231101.zh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2790577882, "num_examples": 1384748}], "download_size": 1721150260, "dataset_size": 2790577882}, {"config_name": "20231101.zh-classical", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14869227, "num_examples": 12708}], "download_size": 10098073, "dataset_size": 14869227}, {"config_name": "20231101.zh-min-nan", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 153672031, "num_examples": 432798}], "download_size": 37122048, "dataset_size": 153672031}, {"config_name": "20231101.zh-yue", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 109936351, "num_examples": 134140}], "download_size": 64950815, "dataset_size": 109936351}, {"config_name": "20231101.zu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7088246, "num_examples": 11561}], "download_size": 3792429, "dataset_size": 7088246}], "language_bcp47": ["be-tarask", "en-simple"]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #language-Abkhazian #language-Achinese #language-Adyghe #language-Afrikaans #language-Southern Altai #language-Amharic #language-Amis #language-Aragonese #language-Old English (ca. 450-1100) #language-Angika #language-Arabic #language-Official Aramaic (700-300 BCE) #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Atikamekw #language-Avaric #language-Kotava #language-Awadhi #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Balinese #language-Bavarian #language-Batak Toba #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bislama #language-Banjar #language-Pa'o Karen #language-Bambara #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Buginese #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Chamorro #language-Cherokee #language-Cheyenne #language-Central Kurdish #language-Corsican #language-Cree #language-Crimean Tatar #language-Czech #language-Kashubian #language-Church Slavic #language-Chuvash #language-Welsh #language-Danish #language-Dagbani #language-German #language-Southern Dagaare #language-Dinka #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Dzongkha #language-Ewe #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Fanti #language-Fulah #language-Finnish #language-Fijian #language-Faroese #language-Fon #language-French #language-Arpitan #language-Northern Frisian #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Gan Chinese #language-Guianese Creole French #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Goan Konkani #language-Gorontalo #language-Gothic #language-Ghanaian Pidgin English #language-Swiss German #language-Gujarati #language-Wayuu #language-Farefare #language-Gun #language-Manx #language-Hausa #language-Hakka Chinese #language-Hawaiian #language-Serbo-Croatian #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Western Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Inupiaq #language-Iloko #language-Ingush #language-Ido #language-Icelandic #language-Italian #language-Inuktitut #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kabiyè #language-Tyap #language-Kongo #language-Kikuyu #language-Kazakh #language-Kalaallisut #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Karachay-Balkar #language-Kashmiri #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lak #language-Lezghian #language-Lingua Franca Nova #language-Ganda #language-Limburgan #language-Ligurian #language-Ladin #language-Lombard #language-Lingala #language-Lao #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Madurese #language-Maithili #language-map #language-Moksha #language-Malagasy #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Manipuri #language-Mon #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-nah #language-Min Nan Chinese #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Nias #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Novial #language-N'Ko #language-Jèrriais #language-Pedi #language-Navajo #language-Nyanja #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Nigerian Pidgin #language-Pennsylvania German #language-Pfaelzisch #language-Pali #language-Pitcairn-Norfolk #language-Polish #language-Piemontese #language-Western Panjabi #language-Pontic #language-Pushto #language-Portuguese #language-Paiwan #language-Quechua #language-Romansh #language-Vlax Romani #language-Rundi #language-Romanian #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Santali #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Northern Sami #language-Sango #language-Samogitian #language-Tachelhit #language-Shan #language-Sinhala #language-Slovak #language-Saraiki #language-Slovenian #language-Samoan #language-Inari Sami #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Swati #language-Southern Sotho #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Sakizaya #language-Tamil #language-Atayal #language-Tulu #language-Telugu #language-Tetum #language-Tajik #language-Thai #language-Tigrinya #language-Turkmen #language-Tagalog #language-Talysh #language-Tswana #language-Tonga (Tonga Islands) #language-Tok Pisin #language-Turkish #language-Sediq #language-Tsonga #language-Tatar #language-Tumbuka #language-Twi #language-Tahitian #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venda #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Kalmyk #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Zhuang #language-Zeeuws #language-Standard Moroccan Tamazight #language-Chinese #language-Zulu #license-cc-by-sa-3.0 #license-gfdl #region-us 


# Dataset Card for Wikimedia Wikipedia

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Point of Contact:

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.

The dataset is built from the Wikipedia dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them per date and language this way:


#### Data Visualization
Click the Nomic Atlas map below to visualize the 6.4 million samples in the 'URL' split.

<a href="URL
  <img src="URL alt="Nomic-Atlas Wikipedia Map" width="25%"/>
</a>

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of languages here: URL

## Dataset Structure

### Data Instances

An example looks as follows:


### Data Fields

The data fields are the same among all configurations:
- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article.

### Data Splits

All configurations contain a single 'train' split.

## Dataset Creation

### Curation Rationale



### Source Data

#### Initial Data Collection and Normalization

The dataset is built from the Wikipedia dumps: URL

You can find the full list of languages and dates here: URL

The articles have been parsed using the 'mwparserfromhell' tool.

When uploading the data files for the 20231101 dump, we noticed that the Wikimedia Dumps website does not contain this date dump
for the "bbc", "dga", nor "zgh" Wikipedias. We have reported the issue to the Wikimedia Phabricator: URL

#### Who are the source language producers?



### Annotations

#### Annotation process



#### Who are the annotators?



### Personal and Sensitive Information



## Considerations for Using the Data

### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



## Additional Information

### Dataset Curators



### Licensing Information

Copyright licensing information: URL

All original textual content is licensed under the GNU Free Documentation License (GFDL)
and the Creative Commons Attribution-Share-Alike 3.0 License.
Some text may be available only under the Creative Commons license; see their Terms of Use for details.
Text written by some authors may be released under additional licenses or into the public domain.




# Dataset Card for Wikimedia Wikipedia ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Point of Contact: ### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.

The dataset is built from the Wikipedia dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them per date and language this way: #### Data Visualization
Click the Nomic Atlas map below to visualize the 6.4 million samples in the 'URL' split.

<a href="URL
  <img src="URL alt="Nomic-Atlas Wikipedia Map" width="25%"/>
</a> ### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling. ### Languages

You can find the list of languages here: URL ## Dataset Structure ### Data Instances

An example looks as follows: ### Data Fields

The data fields are the same among all configurations:
- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article. ### Data Splits

All configurations contain a single 'train' split. ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization

The dataset is built from the Wikipedia dumps: URL

You can find the full list of languages and dates here: URL

The articles have been parsed using the 'mwparserfromhell' tool.

When uploading the data files for the 20231101 dump, we noticed that the Wikimedia Dumps website does not contain this date dump
for the "bbc", "dga", nor "zgh" Wikipedias. We have reported the issue to the Wikimedia Phabricator: URL #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Copyright licensing information: URL

All original textual content is licensed under the GNU Free Documentation License (GFDL)
and the Creative Commons Attribution-Share-Alike 3.0 License.
Some text may be available only under the Creative Commons license; see their Terms of Use for details.
Text written by some authors may be released under additional licenses or into the public domain.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #language-Abkhazian #language-Achinese #language-Adyghe #language-Afrikaans #language-Southern Altai #language-Amharic #language-Amis #language-Aragonese #language-Old English (ca. 450-1100) #language-Angika #language-Arabic #language-Official Aramaic (700-300 BCE) #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Atikamekw #language-Avaric #language-Kotava #language-Awadhi #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Balinese #language-Bavarian #language-Batak Toba #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bislama #language-Banjar #language-Pa'o Karen #language-Bambara #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Buginese #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Chamorro #language-Cherokee #language-Cheyenne #language-Central Kurdish #language-Corsican #language-Cree #language-Crimean Tatar #language-Czech #language-Kashubian #language-Church Slavic #language-Chuvash #language-Welsh #language-Danish #language-Dagbani #language-German #language-Southern Dagaare #language-Dinka #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Dzongkha #language-Ewe #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Fanti #language-Fulah #language-Finnish #language-Fijian #language-Faroese #language-Fon #language-French #language-Arpitan #language-Northern Frisian #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Gan Chinese #language-Guianese Creole French #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Goan Konkani #language-Gorontalo #language-Gothic #language-Ghanaian Pidgin English #language-Swiss German #language-Gujarati #language-Wayuu #language-Farefare #language-Gun #language-Manx #language-Hausa #language-Hakka Chinese #language-Hawaiian #language-Serbo-Croatian #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Western Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Inupiaq #language-Iloko #language-Ingush #language-Ido #language-Icelandic #language-Italian #language-Inuktitut #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kabiyè #language-Tyap #language-Kongo #language-Kikuyu #language-Kazakh #language-Kalaallisut #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Karachay-Balkar #language-Kashmiri #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lak #language-Lezghian #language-Lingua Franca Nova #language-Ganda #language-Limburgan #language-Ligurian #language-Ladin #language-Lombard #language-Lingala #language-Lao #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Madurese #language-Maithili #language-map #language-Moksha #language-Malagasy #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Manipuri #language-Mon #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-nah #language-Min Nan Chinese #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Nias #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Novial #language-N'Ko #language-Jèrriais #language-Pedi #language-Navajo #language-Nyanja #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Nigerian Pidgin #language-Pennsylvania German #language-Pfaelzisch #language-Pali #language-Pitcairn-Norfolk #language-Polish #language-Piemontese #language-Western Panjabi #language-Pontic #language-Pushto #language-Portuguese #language-Paiwan #language-Quechua #language-Romansh #language-Vlax Romani #language-Rundi #language-Romanian #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Santali #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Northern Sami #language-Sango #language-Samogitian #language-Tachelhit #language-Shan #language-Sinhala #language-Slovak #language-Saraiki #language-Slovenian #language-Samoan #language-Inari Sami #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Swati #language-Southern Sotho #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Sakizaya #language-Tamil #language-Atayal #language-Tulu #language-Telugu #language-Tetum #language-Tajik #language-Thai #language-Tigrinya #language-Turkmen #language-Tagalog #language-Talysh #language-Tswana #language-Tonga (Tonga Islands) #language-Tok Pisin #language-Turkish #language-Sediq #language-Tsonga #language-Tatar #language-Tumbuka #language-Twi #language-Tahitian #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venda #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Kalmyk #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Zhuang #language-Zeeuws #language-Standard Moroccan Tamazight #language-Chinese #language-Zulu #license-cc-by-sa-3.0 #license-gfdl #region-us 
 # Dataset Card for Wikimedia Wikipedia ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Point of Contact: ### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.

The dataset is built from the Wikipedia dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them per date and language this way: #### Data Visualization
Click the Nomic Atlas map below to visualize the 6.4 million samples in the 'URL' split.

<a href="URL
  <img src="URL alt="Nomic-Atlas Wikipedia Map" width="25%"/>
</a> ### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling. ### Languages

You can find the list of languages here: URL ## Dataset Structure ### Data Instances

An example looks as follows: ### Data Fields

The data fields are the same among all configurations:
- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article. ### Data Splits

All configurations contain a single 'train' split. ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization

The dataset is built from the Wikipedia dumps: URL

You can find the full list of languages and dates here: URL

The articles have been parsed using the 'mwparserfromhell' tool.

When uploading the data files for the 20231101 dump, we noticed that the Wikimedia Dumps website does not contain this date dump
for the "bbc", "dga", nor "zgh" Wikipedias. We have reported the issue to the Wikimedia Phabricator: URL #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Copyright licensing information: URL

All original textual content is licensed under the GNU Free Documentation License (GFDL)
and the Creative Commons Attribution-Share-Alike 3.0 License.
Some text may be available only under the Creative Commons license; see their Terms of Use for details.
Text written by some authors may be released under additional licenses or into the public domain.
passage:  passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #language-Abkhazian #language-Achinese #language-Adyghe #language-Afrikaans #language-Southern Altai #language-Amharic #language-Amis #language-Aragonese #language-Old English (ca. 450-1100) #language-Angika #language-Arabic #language-Official Aramaic (700-300 BCE) #language-Moroccan Arabic #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Atikamekw #language-Avaric #language-Kotava #language-Awadhi #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Balinese #language-Bavarian #language-Batak Toba #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bislama #language-Banjar #language-Pa'o Karen #language-Bambara #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Buginese #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Chamorro #language-Cherokee #language-Cheyenne #language-Central Kurdish #language-Corsican #language-Cree #language-Crimean Tatar #language-Czech #language-Kashubian #language-Church Slavic #language-Chuvash #language-Welsh #language-Danish #language-Dagbani #language-German #language-Southern Dagaare #language-Dinka #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Dzongkha #language-Ewe #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Fanti #language-Fulah #language-Finnish #language-Fijian #language-Faroese #language-Fon #language-French #language-Arpitan #language-Northern Frisian #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Gan Chinese #language-Guianese Creole French #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Goan Konkani #language-Gorontalo #language-Gothic #language-Ghanaian Pidgin English #language-Swiss German #language-Gujarati #language-Wayuu #language-Farefare #language-Gun #language-Manx #language-Hausa #language-Hakka Chinese #language-Hawaiian #language-Serbo-Croatian #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Western Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Inupiaq #language-Iloko #language-Ingush #language-Ido #language-Icelandic #language-Italian #language-Inuktitut #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kabiyè #language-Tyap #language-Kongo #language-Kikuyu #language-Kazakh #language-Kalaallisut #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Karachay-Balkar #language-Kashmiri #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lak #language-Lezghian #language-Lingua Franca Nova #language-Ganda #language-Limburgan #language-Ligurian #language-Ladin #language-Lombard #language-Lingala #language-Lao #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Madurese #language-Maithili #language-map #language-Moksha #language-Malagasy #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Manipuri #language-Mon #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-nah #language-Min Nan Chinese #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Nias #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Novial #language-N'Ko #language-Jèrriais #language-Pedi #language-Navajo #language-Nyanja #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Nigerian Pidgin #language-Pennsylvania German #language-Pfaelzisch #language-Pali #language-Pitcairn-Norfolk #language-Polish #language-Piemontese #language-Western Panjabi #language-Pontic #language-Pushto #language-Portuguese #language-Paiwan #language-Quechua #language-Romansh #language-Vlax Romani #language-Rundi #language-Romanian #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Santali #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Northern Sami #language-Sango #language-Samogitian #language-Tachelhit #language-Shan #language-Sinhala #language-Slovak #language-Saraiki #language-Slovenian #language-Samoan #language-Inari Sami #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Swati #language-Southern Sotho #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Sakizaya #language-Tamil #language-Atayal #language-Tulu #language-Telugu #language-Tetum #language-Tajik #language-Thai #language-Tigrinya #language-Turkmen #language-Tagalog #language-Talysh #language-Tswana #language-Tonga (Tonga Islands) #language-Tok Pisin #language-Turkish #language-Sediq #language-Tsonga #language-Tatar #language-Tumbuka #language-Twi #language-Tahitian #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venda #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Kalmyk #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Zhuang #language-Zeeuws #language-Standard Moroccan Tamazight #language-Chinese #language-Zulu #license-cc-by-sa-3.0 #license-gfdl #region-us 
# Dataset Card for Wikimedia Wikipedia## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Point of Contact:### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.

The dataset is built from the Wikipedia dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them per date and language this way:#### Data Visualization
Click the Nomic Atlas map below to visualize the 6.4 million samples in the 'URL' split.

<a href="URL
  <img src="URL alt="Nomic-Atlas Wikipedia Map" width="25%"/>
</a>### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.### Languages

You can find the list of languages here: URL## Dataset Structure### Data Instances

An example looks as follows:### Data Fields

The data fields are the same among all configurations:
- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article.### Data Splits

All configurations contain a single 'train' split.## Dataset Creation### Curation Rationale### Source Data