
# Dataset Card for Wikipedia

This repo is a fork of the original Hugging Face Wikipedia repo [here](https://huggingface.co/datasets/wikipedia).
The difference is that this fork does away with the need for `apache-beam`, and this fork is very fast if you have a lot of CPUs on your machine.
It will use all CPUs available to create a clean Wikipedia pretraining dataset. It takes less than an hour to process all of English wikipedia on a GCP n1-standard-96.
This fork is also used in the [OLM Project](https://github.com/huggingface/olm-datasets) to pull and process up-to-date wikipedia snapshots.

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [https://dumps.wikimedia.org](https://dumps.wikimedia.org)
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

The articles are parsed using the ``mwparserfromhell`` tool, and we use ``multiprocess`` for parallelization.

To load this dataset you need to install these first:

```
pip install mwparserfromhell==0.6.4 multiprocess==0.70.13
```

Then, you can load any subset of Wikipedia per language and per date this way:

```python
from datasets import load_dataset

load_dataset("olm/wikipedia", language="en", date="20220920")
```

You can find the full list of languages and dates [here](https://dumps.wikimedia.org/backup-index.html).

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of languages [here](https://meta.wikimedia.org/wiki/List_of_Wikipedias).

## Dataset Structure

### Data Instances

An example looks as follows:

```
{'id': '1',
 'url': 'https://simple.wikipedia.org/wiki/April',
 'title': 'April',
 'text': 'April is the fourth month...'
}
```

### Data Fields

The data fields are the same among all configurations:

- `id` (`str`): ID of the article.
- `url` (`str`): URL of the article.
- `title` (`str`): Title of the article.
- `text` (`str`): Text content of the article.

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the
[Creative Commons Attribution-ShareAlike 3.0 Unported License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License)
(CC BY-SA) and the [GNU Free Documentation License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License)
(GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). 

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such
text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes
the text.

### Citation Information

```
@ONLINE{wikidump,
    author = "Wikimedia Foundation",
    title  = "Wikimedia Downloads",
    url    = "https://dumps.wikimedia.org"
}
```

task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:crowdsourced multilinguality:multilingual size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M size_categories:1M<n<10M source_datasets:original language:aa language:ab language:ace language:af language:ak language:als language:am language:an language:ang language:ar language:arc language:arz language:as language:ast language:atj language:av language:ay language:az language:azb language:ba language:bar language:bcl language:be language:bg language:bh language:bi language:bjn language:bm language:bn language:bo language:bpy language:br language:bs language:bug language:bxr language:ca language:cbk language:cdo language:ce language:ceb language:ch language:cho language:chr language:chy language:ckb language:co language:cr language:crh language:cs language:csb language:cu language:cv language:cy language:da language:de language:din language:diq language:dsb language:dty language:dv language:dz language:ee language:el language:eml language:en language:eo language:es language:et language:eu language:ext language:fa language:ff language:fi language:fj language:fo language:fr language:frp language:frr language:fur language:fy language:ga language:gag language:gan language:gd language:gl language:glk language:gn language:gom language:gor language:got language:gu language:gv language:ha language:hak language:haw language:he language:hi language:hif language:ho language:hr language:hsb language:ht language:hu language:hy language:ia language:id language:ie language:ig language:ii language:ik language:ilo language:inh language:io language:is language:it language:iu language:ja language:jam language:jbo language:jv language:ka language:kaa language:kab language:kbd language:kbp language:kg language:ki language:kj language:kk language:kl language:km language:kn language:ko language:koi language:krc language:ks language:ksh language:ku language:kv language:kw language:ky language:la language:lad language:lb language:lbe language:lez language:lfn language:lg language:li language:lij language:lmo language:ln language:lo language:lrc language:lt language:ltg language:lv language:lzh language:mai language:mdf language:mg language:mh language:mhr language:mi language:min language:mk language:ml language:mn language:mr language:mrj language:ms language:mt language:mus language:mwl language:my language:myv language:mzn language:na language:nah language:nan language:nap language:nds language:ne language:new language:ng language:nl language:nn language:no language:nov language:nrf language:nso language:nv language:ny language:oc language:olo language:om language:or language:os language:pa language:pag language:pam language:pap language:pcd language:pdc language:pfl language:pi language:pih language:pl language:pms language:pnb language:pnt language:ps language:pt language:qu language:rm language:rmy language:rn language:ro language:ru language:rue language:rup language:rw language:sa language:sah language:sat language:sc language:scn language:sco language:sd language:se language:sg language:sgs language:sh language:si language:sk language:sl language:sm language:sn language:so language:sq language:sr language:srn language:ss language:st language:stq language:su language:sv language:sw language:szl language:ta language:tcy language:tdt language:te language:tg language:th language:ti language:tk language:tl language:tn language:to language:tpi language:tr language:ts language:tt language:tum language:tw language:ty language:tyv language:udm language:ug language:uk language:ur language:uz language:ve language:vec language:vep language:vi language:vls language:vo language:vro language:wa language:war language:wo language:wuu language:xal language:xh language:xmf language:yi language:yo language:yue language:za language:zea language:zh language:zu license:cc-by-sa-3.0 license:gfdl region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["crowdsourced"], "language": ["aa", "ab", "ace", "af", "ak", "als", "am", "an", "ang", "ar", "arc", "arz", "as", "ast", "atj", "av", "ay", "az", "azb", "ba", "bar", "bcl", "be", "bg", "bh", "bi", "bjn", "bm", "bn", "bo", "bpy", "br", "bs", "bug", "bxr", "ca", "cbk", "cdo", "ce", "ceb", "ch", "cho", "chr", "chy", "ckb", "co", "cr", "crh", "cs", "csb", "cu", "cv", "cy", "da", "de", "din", "diq", "dsb", "dty", "dv", "dz", "ee", "el", "eml", "en", "eo", "es", "et", "eu", "ext", "fa", "ff", "fi", "fj", "fo", "fr", "frp", "frr", "fur", "fy", "ga", "gag", "gan", "gd", "gl", "glk", "gn", "gom", "gor", "got", "gu", "gv", "ha", "hak", "haw", "he", "hi", "hif", "ho", "hr", "hsb", "ht", "hu", "hy", "ia", "id", "ie", "ig", "ii", "ik", "ilo", "inh", "io", "is", "it", "iu", "ja", "jam", "jbo", "jv", "ka", "kaa", "kab", "kbd", "kbp", "kg", "ki", "kj", "kk", "kl", "km", "kn", "ko", "koi", "krc", "ks", "ksh", "ku", "kv", "kw", "ky", "la", "lad", "lb", "lbe", "lez", "lfn", "lg", "li", "lij", "lmo", "ln", "lo", "lrc", "lt", "ltg", "lv", "lzh", "mai", "mdf", "mg", "mh", "mhr", "mi", "min", "mk", "ml", "mn", "mr", "mrj", "ms", "mt", "mus", "mwl", "my", "myv", "mzn", "na", "nah", "nan", "nap", "nds", "ne", "new", "ng", "nl", "nn", "no", "nov", "nrf", "nso", "nv", "ny", "oc", "olo", "om", "or", "os", "pa", "pag", "pam", "pap", "pcd", "pdc", "pfl", "pi", "pih", "pl", "pms", "pnb", "pnt", "ps", "pt", "qu", "rm", "rmy", "rn", "ro", "ru", "rue", "rup", "rw", "sa", "sah", "sat", "sc", "scn", "sco", "sd", "se", "sg", "sgs", "sh", "si", "sk", "sl", "sm", "sn", "so", "sq", "sr", "srn", "ss", "st", "stq", "su", "sv", "sw", "szl", "ta", "tcy", "tdt", "te", "tg", "th", "ti", "tk", "tl", "tn", "to", "tpi", "tr", "ts", "tt", "tum", "tw", "ty", "tyv", "udm", "ug", "uk", "ur", "uz", "ve", "vec", "vep", "vi", "vls", "vo", "vro", "wa", "war", "wo", "wuu", "xal", "xh", "xmf", "yi", "yo", "yue", "za", "zea", "zh", "zu"], "license": ["cc-by-sa-3.0", "gfdl"], "multilinguality": ["multilingual"], "size_categories": ["n<1K", "1K<n<10K", "10K<n<100K", "100K<n<1M", "1M<n<10M"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "pretty_name": "Wikipedia", "config_names": ["20220301.aa", "20220301.ab", "20220301.ace", "20220301.ady", "20220301.af", "20220301.ak", "20220301.als", "20220301.am", "20220301.an", "20220301.ang", "20220301.ar", "20220301.arc", "20220301.arz", "20220301.as", "20220301.ast", "20220301.atj", "20220301.av", "20220301.ay", "20220301.az", "20220301.azb", "20220301.ba", "20220301.bar", "20220301.bat-smg", "20220301.bcl", "20220301.be", "20220301.be-x-old", "20220301.bg", "20220301.bh", "20220301.bi", "20220301.bjn", "20220301.bm", "20220301.bn", "20220301.bo", "20220301.bpy", "20220301.br", "20220301.bs", "20220301.bug", "20220301.bxr", "20220301.ca", "20220301.cbk-zam", "20220301.cdo", "20220301.ce", "20220301.ceb", "20220301.ch", "20220301.cho", "20220301.chr", "20220301.chy", "20220301.ckb", "20220301.co", "20220301.cr", "20220301.crh", "20220301.cs", "20220301.csb", "20220301.cu", "20220301.cv", "20220301.cy", "20220301.da", "20220301.de", "20220301.din", "20220301.diq", "20220301.dsb", "20220301.dty", "20220301.dv", "20220301.dz", "20220301.ee", "20220301.el", "20220301.eml", "20220301.en", "20220301.eo", "20220301.es", "20220301.et", "20220301.eu", "20220301.ext", "20220301.fa", "20220301.ff", "20220301.fi", "20220301.fiu-vro", "20220301.fj", "20220301.fo", "20220301.fr", "20220301.frp", "20220301.frr", "20220301.fur", "20220301.fy", "20220301.ga", "20220301.gag", "20220301.gan", "20220301.gd", "20220301.gl", "20220301.glk", "20220301.gn", "20220301.gom", "20220301.gor", "20220301.got", "20220301.gu", "20220301.gv", "20220301.ha", "20220301.hak", "20220301.haw", "20220301.he", "20220301.hi", "20220301.hif", "20220301.ho", "20220301.hr", "20220301.hsb", "20220301.ht", "20220301.hu", "20220301.hy", "20220301.ia", "20220301.id", "20220301.ie", "20220301.ig", "20220301.ii", "20220301.ik", "20220301.ilo", "20220301.inh", "20220301.io", "20220301.is", "20220301.it", "20220301.iu", "20220301.ja", "20220301.jam", "20220301.jbo", "20220301.jv", "20220301.ka", "20220301.kaa", "20220301.kab", "20220301.kbd", "20220301.kbp", "20220301.kg", "20220301.ki", "20220301.kj", "20220301.kk", "20220301.kl", "20220301.km", "20220301.kn", "20220301.ko", "20220301.koi", "20220301.krc", "20220301.ks", "20220301.ksh", "20220301.ku", "20220301.kv", "20220301.kw", "20220301.ky", "20220301.la", "20220301.lad", "20220301.lb", "20220301.lbe", "20220301.lez", "20220301.lfn", "20220301.lg", "20220301.li", "20220301.lij", "20220301.lmo", "20220301.ln", "20220301.lo", "20220301.lrc", "20220301.lt", "20220301.ltg", "20220301.lv", "20220301.mai", "20220301.map-bms", "20220301.mdf", "20220301.mg", "20220301.mh", "20220301.mhr", "20220301.mi", "20220301.min", "20220301.mk", "20220301.ml", "20220301.mn", "20220301.mr", "20220301.mrj", "20220301.ms", "20220301.mt", "20220301.mus", "20220301.mwl", "20220301.my", "20220301.myv", "20220301.mzn", "20220301.na", "20220301.nah", "20220301.nap", "20220301.nds", "20220301.nds-nl", "20220301.ne", "20220301.new", "20220301.ng", "20220301.nl", "20220301.nn", "20220301.no", "20220301.nov", "20220301.nrm", "20220301.nso", "20220301.nv", "20220301.ny", "20220301.oc", "20220301.olo", "20220301.om", "20220301.or", "20220301.os", "20220301.pa", "20220301.pag", "20220301.pam", "20220301.pap", "20220301.pcd", "20220301.pdc", "20220301.pfl", "20220301.pi", "20220301.pih", "20220301.pl", "20220301.pms", "20220301.pnb", "20220301.pnt", "20220301.ps", "20220301.pt", "20220301.qu", "20220301.rm", "20220301.rmy", "20220301.rn", "20220301.ro", "20220301.roa-rup", "20220301.roa-tara", "20220301.ru", "20220301.rue", "20220301.rw", "20220301.sa", "20220301.sah", "20220301.sat", "20220301.sc", "20220301.scn", "20220301.sco", "20220301.sd", "20220301.se", "20220301.sg", "20220301.sh", "20220301.si", "20220301.simple", "20220301.sk", "20220301.sl", "20220301.sm", "20220301.sn", "20220301.so", "20220301.sq", "20220301.sr", "20220301.srn", "20220301.ss", "20220301.st", "20220301.stq", "20220301.su", "20220301.sv", "20220301.sw", "20220301.szl", "20220301.ta", "20220301.tcy", "20220301.te", "20220301.tet", "20220301.tg", "20220301.th", "20220301.ti", "20220301.tk", "20220301.tl", "20220301.tn", "20220301.to", "20220301.tpi", "20220301.tr", "20220301.ts", "20220301.tt", "20220301.tum", "20220301.tw", "20220301.ty", "20220301.tyv", "20220301.udm", "20220301.ug", "20220301.uk", "20220301.ur", "20220301.uz", "20220301.ve", "20220301.vec", "20220301.vep", "20220301.vi", "20220301.vls", "20220301.vo", "20220301.wa", "20220301.war", "20220301.wo", "20220301.wuu", "20220301.xal", "20220301.xh", "20220301.xmf", "20220301.yi", "20220301.yo", "20220301.za", "20220301.zea", "20220301.zh", "20220301.zh-classical", "20220301.zh-min-nan", "20220301.zh-yue", "20220301.zu"], "language_bcp47": ["nds-nl"]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-crowdsourced #multilinguality-multilingual #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #source_datasets-original #language-Afar #language-Abkhazian #language-Achinese #language-Afrikaans #language-Akan #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Old English (ca. 450-1100) #language-Arabic #language-Official Aramaic (700-300 BCE) #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Atikamekw #language-Avaric #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bislama #language-Banjar #language-Bambara #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Buginese #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Chamorro #language-Choctaw #language-Cherokee #language-Cheyenne #language-Central Kurdish #language-Corsican #language-Cree #language-Crimean Tatar #language-Czech #language-Kashubian #language-Church Slavic #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dinka #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Dzongkha #language-Ewe #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Fulah #language-Finnish #language-Fijian #language-Faroese #language-French #language-Arpitan #language-Northern Frisian #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Gan Chinese #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Goan Konkani #language-Gorontalo #language-Gothic #language-Gujarati #language-Manx #language-Hausa #language-Hakka Chinese #language-Hawaiian #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Hiri Motu #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Sichuan Yi #language-Inupiaq #language-Iloko #language-Ingush #language-Ido #language-Icelandic #language-Italian #language-Inuktitut #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kabiyè #language-Kongo #language-Kikuyu #language-Kuanyama #language-Kazakh #language-Kalaallisut #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Karachay-Balkar #language-Kashmiri #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lak #language-Lezghian #language-Lingua Franca Nova #language-Ganda #language-Limburgan #language-Ligurian #language-Lombard #language-Lingala #language-Lao #language-Northern Luri #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Maithili #language-Moksha #language-Malagasy #language-Marshallese #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Creek #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-Nauru #language-nah #language-Min Nan Chinese #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Ndonga #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Novial #language-Jèrriais #language-Pedi #language-Navajo #language-Nyanja #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Pennsylvania German #language-Pfaelzisch #language-Pali #language-Pitcairn-Norfolk #language-Polish #language-Piemontese #language-Western Panjabi #language-Pontic #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Vlax Romani #language-Rundi #language-Romanian #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Santali #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Northern Sami #language-Sango #language-Samogitian #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Swati #language-Southern Sotho #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Tamil #language-Tulu #language-Tetun Dili #language-Telugu #language-Tajik #language-Thai #language-Tigrinya #language-Turkmen #language-Tagalog #language-Tswana #language-Tonga (Tonga Islands) #language-Tok Pisin #language-Turkish #language-Tsonga #language-Tatar #language-Tumbuka #language-Twi #language-Tahitian #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venda #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Kalmyk #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Zhuang #language-Zeeuws #language-Chinese #language-Zulu #license-cc-by-sa-3.0 #license-gfdl #region-us 


# Dataset Card for Wikipedia

This repo is a fork of the original Hugging Face Wikipedia repo here.
The difference is that this fork does away with the need for 'apache-beam', and this fork is very fast if you have a lot of CPUs on your machine.
It will use all CPUs available to create a clean Wikipedia pretraining dataset. It takes less than an hour to process all of English wikipedia on a GCP n1-standard-96.
This fork is also used in the OLM Project to pull and process up-to-date wikipedia snapshots.

## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: URL
- Repository: 
- Paper: 
- Point of Contact: 

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

The articles are parsed using the ''mwparserfromhell'' tool, and we use ''multiprocess'' for parallelization.

To load this dataset you need to install these first:



Then, you can load any subset of Wikipedia per language and per date this way:



You can find the full list of languages and dates here.

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of languages here.

## Dataset Structure

### Data Instances

An example looks as follows:



### Data Fields

The data fields are the same among all configurations:

- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article.

### Curation Rationale



### Source Data

#### Initial Data Collection and Normalization



#### Who are the source language producers?



### Annotations

#### Annotation process



#### Who are the annotators?



### Personal and Sensitive Information



## Considerations for Using the Data

### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



## Additional Information

### Dataset Curators



### Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the
Creative Commons Attribution-ShareAlike 3.0 Unported License
(CC BY-SA) and the GNU Free Documentation License
(GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). 

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such
text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes
the text.





# Dataset Card for Wikipedia

This repo is a fork of the original Hugging Face Wikipedia repo here.
The difference is that this fork does away with the need for 'apache-beam', and this fork is very fast if you have a lot of CPUs on your machine.
It will use all CPUs available to create a clean Wikipedia pretraining dataset. It takes less than an hour to process all of English wikipedia on a GCP n1-standard-96.
This fork is also used in the OLM Project to pull and process up-to-date wikipedia snapshots. ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: 
- Paper: 
- Point of Contact: ### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

The articles are parsed using the ''mwparserfromhell'' tool, and we use ''multiprocess'' for parallelization.

To load this dataset you need to install these first:



Then, you can load any subset of Wikipedia per language and per date this way:



You can find the full list of languages and dates here. ### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling. ### Languages

You can find the list of languages here. ## Dataset Structure ### Data Instances

An example looks as follows: ### Data Fields

The data fields are the same among all configurations:

- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article. ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the
Creative Commons Attribution-ShareAlike 3.0 Unported License
(CC BY-SA) and the GNU Free Documentation License
(GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). 

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such
text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes
the text.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-crowdsourced #multilinguality-multilingual #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #source_datasets-original #language-Afar #language-Abkhazian #language-Achinese #language-Afrikaans #language-Akan #language-Tosk Albanian #language-Amharic #language-Aragonese #language-Old English (ca. 450-1100) #language-Arabic #language-Official Aramaic (700-300 BCE) #language-Egyptian Arabic #language-Assamese #language-Asturian #language-Atikamekw #language-Avaric #language-Aymara #language-Azerbaijani #language-South Azerbaijani #language-Bashkir #language-Bavarian #language-Central Bikol #language-Belarusian #language-Bulgarian #language-bh #language-Bislama #language-Banjar #language-Bambara #language-Bengali #language-Tibetan #language-Bishnupriya #language-Breton #language-Bosnian #language-Buginese #language-Russia Buriat #language-Catalan #language-Chavacano #language-Min Dong Chinese #language-Chechen #language-Cebuano #language-Chamorro #language-Choctaw #language-Cherokee #language-Cheyenne #language-Central Kurdish #language-Corsican #language-Cree #language-Crimean Tatar #language-Czech #language-Kashubian #language-Church Slavic #language-Chuvash #language-Welsh #language-Danish #language-German #language-Dinka #language-Dimli (individual language) #language-Lower Sorbian #language-Dotyali #language-Dhivehi #language-Dzongkha #language-Ewe #language-Modern Greek (1453-) #language-Emiliano-Romagnolo #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Extremaduran #language-Persian #language-Fulah #language-Finnish #language-Fijian #language-Faroese #language-French #language-Arpitan #language-Northern Frisian #language-Friulian #language-Western Frisian #language-Irish #language-Gagauz #language-Gan Chinese #language-Scottish Gaelic #language-Galician #language-Gilaki #language-Guarani #language-Goan Konkani #language-Gorontalo #language-Gothic #language-Gujarati #language-Manx #language-Hausa #language-Hakka Chinese #language-Hawaiian #language-Hebrew #language-Hindi #language-Fiji Hindi #language-Hiri Motu #language-Croatian #language-Upper Sorbian #language-Haitian #language-Hungarian #language-Armenian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Interlingue #language-Igbo #language-Sichuan Yi #language-Inupiaq #language-Iloko #language-Ingush #language-Ido #language-Icelandic #language-Italian #language-Inuktitut #language-Japanese #language-Jamaican Creole English #language-Lojban #language-Javanese #language-Georgian #language-Kara-Kalpak #language-Kabyle #language-Kabardian #language-Kabiyè #language-Kongo #language-Kikuyu #language-Kuanyama #language-Kazakh #language-Kalaallisut #language-Khmer #language-Kannada #language-Korean #language-Komi-Permyak #language-Karachay-Balkar #language-Kashmiri #language-Kölsch #language-Kurdish #language-Komi #language-Cornish #language-Kirghiz #language-Latin #language-Ladino #language-Luxembourgish #language-Lak #language-Lezghian #language-Lingua Franca Nova #language-Ganda #language-Limburgan #language-Ligurian #language-Lombard #language-Lingala #language-Lao #language-Northern Luri #language-Lithuanian #language-Latgalian #language-Latvian #language-Literary Chinese #language-Maithili #language-Moksha #language-Malagasy #language-Marshallese #language-Eastern Mari #language-Maori #language-Minangkabau #language-Macedonian #language-Malayalam #language-Mongolian #language-Marathi #language-Western Mari #language-Malay (macrolanguage) #language-Maltese #language-Creek #language-Mirandese #language-Burmese #language-Erzya #language-Mazanderani #language-Nauru #language-nah #language-Min Nan Chinese #language-Neapolitan #language-Low German #language-Nepali (macrolanguage) #language-Newari #language-Ndonga #language-Dutch #language-Norwegian Nynorsk #language-Norwegian #language-Novial #language-Jèrriais #language-Pedi #language-Navajo #language-Nyanja #language-Occitan (post 1500) #language-Livvi #language-Oromo #language-Oriya (macrolanguage) #language-Ossetian #language-Panjabi #language-Pangasinan #language-Pampanga #language-Papiamento #language-Picard #language-Pennsylvania German #language-Pfaelzisch #language-Pali #language-Pitcairn-Norfolk #language-Polish #language-Piemontese #language-Western Panjabi #language-Pontic #language-Pushto #language-Portuguese #language-Quechua #language-Romansh #language-Vlax Romani #language-Rundi #language-Romanian #language-Russian #language-Rusyn #language-Macedo-Romanian #language-Kinyarwanda #language-Sanskrit #language-Yakut #language-Santali #language-Sardinian #language-Sicilian #language-Scots #language-Sindhi #language-Northern Sami #language-Sango #language-Samogitian #language-Serbo-Croatian #language-Sinhala #language-Slovak #language-Slovenian #language-Samoan #language-Shona #language-Somali #language-Albanian #language-Serbian #language-Sranan Tongo #language-Swati #language-Southern Sotho #language-Saterfriesisch #language-Sundanese #language-Swedish #language-Swahili (macrolanguage) #language-Silesian #language-Tamil #language-Tulu #language-Tetun Dili #language-Telugu #language-Tajik #language-Thai #language-Tigrinya #language-Turkmen #language-Tagalog #language-Tswana #language-Tonga (Tonga Islands) #language-Tok Pisin #language-Turkish #language-Tsonga #language-Tatar #language-Tumbuka #language-Twi #language-Tahitian #language-Tuvinian #language-Udmurt #language-Uighur #language-Ukrainian #language-Urdu #language-Uzbek #language-Venda #language-Venetian #language-Veps #language-Vietnamese #language-Vlaams #language-Volapük #language-Võro #language-Walloon #language-Waray (Philippines) #language-Wolof #language-Wu Chinese #language-Kalmyk #language-Xhosa #language-Mingrelian #language-Yiddish #language-Yoruba #language-Yue Chinese #language-Zhuang #language-Zeeuws #language-Chinese #language-Zulu #license-cc-by-sa-3.0 #license-gfdl #region-us 
 # Dataset Card for Wikipedia

This repo is a fork of the original Hugging Face Wikipedia repo here.
The difference is that this fork does away with the need for 'apache-beam', and this fork is very fast if you have a lot of CPUs on your machine.
It will use all CPUs available to create a clean Wikipedia pretraining dataset. It takes less than an hour to process all of English wikipedia on a GCP n1-standard-96.
This fork is also used in the OLM Project to pull and process up-to-date wikipedia snapshots. ## Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: 
- Paper: 
- Point of Contact: ### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

The articles are parsed using the ''mwparserfromhell'' tool, and we use ''multiprocess'' for parallelization.

To load this dataset you need to install these first:



Then, you can load any subset of Wikipedia per language and per date this way:



You can find the full list of languages and dates here. ### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling. ### Languages

You can find the list of languages here. ## Dataset Structure ### Data Instances

An example looks as follows: ### Data Fields

The data fields are the same among all configurations:

- 'id' ('str'): ID of the article.
- 'url' ('str'): URL of the article.
- 'title' ('str'): Title of the article.
- 'text' ('str'): Text content of the article. ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the
Creative Commons Attribution-ShareAlike 3.0 Unported License
(CC BY-SA) and the GNU Free Documentation License
(GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). 

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such
text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes
the text.