
# Dataset Card for Wikimedia Wikisource

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://dumps.wikimedia.org
- **Repository:**
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

Wikisource dataset containing cleaned articles of all languages.

The dataset is built from the Wikisource dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikisource text with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them by date and language like this:
```python
from datasets import load_dataset

ds = load_dataset("wikimedia/wikisource", "20231201.en")
```

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of all languages here: https://meta.wikimedia.org/wiki/Wikisource#List_of_Wikisources

Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual
Wikisource here: https://wikisource.org/wiki/Wikisource:Languages

## Dataset Structure

### Data Instances

An example looks as follows:
```
{'id': '36',
 'url': 'https://ca.wikisource.org/wiki/Comunicat%20de%20Berl%C3%ADn',
 'title': 'Comunicat de Berlín',
 'text': "\n\nPreàmbul \nEl 19 de juny de 1999, un any després de la Declaració de la Sorbona,..."
}
```

### Data Fields

The data fields are the same among all language configurations:
- `id` (`str`): ID of the text.
- `url` (`str`): URL of the text.
- `title` (`str`): Title of the text.
- `text` (`str`): Content of the text.

### Data Splits

All language configurations contain a single `train` split.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

The dataset is built from the Wikisource dumps: https://dumps.wikimedia.org

You can find the full list of languages and dates here: https://dumps.wikimedia.org/backup-index.html

The articles have been parsed using the [`mwparserfromhell`](https://mwparserfromhell.readthedocs.io) tool.

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Copyright licensing information: https://dumps.wikimedia.org/legal.html

All original textual content is licensed under the [GNU Free Documentation License](https://www.gnu.org/licenses/fdl-1.3.html) (GFDL)
and the [Creative Commons Attribution-Share-Alike 3.0 License](https://creativecommons.org/licenses/by-sa/3.0/).
Some text may be available only under the Creative Commons license; see their [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) for details.
Text written by some authors may be released under additional licenses or into the public domain.

### Citation Information

```
@ONLINE{wikidump,
    author = "Wikimedia Foundation",
    title  = "Wikimedia Downloads",
    url    = "https://dumps.wikimedia.org"
}
```

### Contributions

Thanks to [@albertvillanova](https://huggingface.co/albertvillanova) for adding this dataset.

task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M language:ar language:as language:az language:ban language:be language:bg language:bn language:br language:bs language:ca language:cs language:cy language:da language:de language:el language:en language:eo language:es language:et language:eu language:fa language:fi language:fo language:fr language:gl language:gu language:he language:hi language:hr language:hu language:hy language:id language:is language:it language:ja language:jv language:kn language:ko language:la language:li language:lij language:lt language:mk language:ml language:mr language:nan language:nap language:nl language:no language:or language:pa language:pl language:pms language:pt language:ro language:ru language:sa language:sah language:sk language:sl language:sr language:su language:sv language:ta language:te language:th language:tr language:uk language:vec language:vi language:wa language:yi language:zh license:cc-by-sa-3.0 license:gfdl region:us
{"language": ["ar", "as", "az", "ban", "be", "bg", "bn", "br", "bs", "ca", "cs", "cy", "da", "de", "el", "en", "eo", "es", "et", "eu", "fa", "fi", "fo", "fr", "gl", "gu", "he", "hi", "hr", "hu", "hy", "id", "is", "it", "ja", "jv", "kn", "ko", "la", "li", "lij", "lt", "mk", "ml", "mr", "nan", "nap", "nl", "no", "or", "pa", "pl", "pms", "pt", "ro", "ru", "sa", "sah", "sk", "sl", "sr", "su", "sv", "ta", "te", "th", "tr", "uk", "vec", "vi", "wa", "yi", "zh"], "license": ["cc-by-sa-3.0", "gfdl"], "size_categories": ["n<1K", "1K<n<10K", "10K<n<100K", "100K<n<1M"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "dataset_info": [{"config_name": "20231201.ar", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1027384499, "num_examples": 38235}], "download_size": 471633595, "dataset_size": 1027384499}, {"config_name": "20231201.as", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 10334689, "num_examples": 1191}], "download_size": 3976908, "dataset_size": 10334689}, {"config_name": "20231201.az", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 37612618, "num_examples": 9706}], "download_size": 20953203, "dataset_size": 37612618}, {"config_name": "20231201.ban", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 468189, "num_examples": 591}], "download_size": 169732, "dataset_size": 468189}, {"config_name": "20231201.be", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 52555230, "num_examples": 4876}], "download_size": 26356864, "dataset_size": 52555230}, {"config_name": "20231201.bg", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 33320786, "num_examples": 2316}], "download_size": 14416495, "dataset_size": 33320786}, {"config_name": "20231201.bn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12256, "num_examples": 5}], "download_size": 11958, "dataset_size": 12256}, {"config_name": "20231201.br", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 179457, "num_examples": 314}], "download_size": 89388, "dataset_size": 179457}, {"config_name": "20231201.bs", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15735639, "num_examples": 1918}], "download_size": 9427044, "dataset_size": 15735639}, {"config_name": "20231201.ca", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9470138, "num_examples": 1229}], "download_size": 5021947, "dataset_size": 9470138}, {"config_name": "20231201.cs", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 190358421, "num_examples": 42735}], "download_size": 124249346, "dataset_size": 190358421}, {"config_name": "20231201.cy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2161046, "num_examples": 1090}], "download_size": 1251259, "dataset_size": 2161046}, {"config_name": "20231201.da", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18564343, "num_examples": 1043}], "download_size": 10957998, "dataset_size": 18564343}, {"config_name": "20231201.de", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 527146739, "num_examples": 141657}], "download_size": 312816088, "dataset_size": 527146739}, {"config_name": "20231201.el", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 215554120, "num_examples": 8024}], "download_size": 103217935, "dataset_size": 215554120}, {"config_name": "20231201.en", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2470274084, "num_examples": 208279}], "download_size": 1382960909, "dataset_size": 2470274084}, {"config_name": "20231201.eo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3207308, "num_examples": 384}], "download_size": 2009128, "dataset_size": 3207308}, {"config_name": "20231201.es", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 381152287, "num_examples": 37831}], "download_size": 224097690, "dataset_size": 381152287}, {"config_name": "20231201.et", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3906488, "num_examples": 722}], "download_size": 2316406, "dataset_size": 3906488}, {"config_name": "20231201.eu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17014224, "num_examples": 923}], "download_size": 9473130, "dataset_size": 17014224}, {"config_name": "20231201.fa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 79812303, "num_examples": 5751}], "download_size": 33916994, "dataset_size": 79812303}, {"config_name": "20231201.fi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 55271379, "num_examples": 13414}], "download_size": 33265827, "dataset_size": 55271379}, {"config_name": "20231201.fo", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 174113, "num_examples": 62}], "download_size": 112092, "dataset_size": 174113}, {"config_name": "20231201.fr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 90126375, "num_examples": 23201}], "download_size": 49429480, "dataset_size": 90126375}, {"config_name": "20231201.gl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6679826, "num_examples": 747}], "download_size": 3712275, "dataset_size": 6679826}, {"config_name": "20231201.gu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2430315, "num_examples": 797}], "download_size": 948872, "dataset_size": 2430315}, {"config_name": "20231201.he", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1166312812, "num_examples": 107248}], "download_size": 519792862, "dataset_size": 1166312812}, {"config_name": "20231201.hi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2203936, "num_examples": 3494}], "download_size": 443194, "dataset_size": 2203936}, {"config_name": "20231201.hr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 61069921, "num_examples": 8278}], "download_size": 38797697, "dataset_size": 61069921}, {"config_name": "20231201.hu", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 94429364, "num_examples": 20846}], "download_size": 62012894, "dataset_size": 94429364}, {"config_name": "20231201.hy", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 39941751, "num_examples": 2248}], "download_size": 18574182, "dataset_size": 39941751}, {"config_name": "20231201.id", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 40100527, "num_examples": 2234}], "download_size": 18175030, "dataset_size": 40100527}, {"config_name": "20231201.is", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 20657687, "num_examples": 4880}], "download_size": 11620112, "dataset_size": 20657687}, {"config_name": "20231201.it", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 211472789, "num_examples": 65047}], "download_size": 115227856, "dataset_size": 211472789}, {"config_name": "20231201.ja", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 202476246, "num_examples": 11879}], "download_size": 90838204, "dataset_size": 202476246}, {"config_name": "20231201.jv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6977954, "num_examples": 534}], "download_size": 3409151, "dataset_size": 6977954}, {"config_name": "20231201.kn", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 202914626, "num_examples": 14980}], "download_size": 73290389, "dataset_size": 202914626}, {"config_name": "20231201.ko", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 661997703, "num_examples": 24858}], "download_size": 302950424, "dataset_size": 661997703}, {"config_name": "20231201.la", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 583348181, "num_examples": 11032}], "download_size": 351767028, "dataset_size": 583348181}, {"config_name": "20231201.li", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2143869, "num_examples": 1857}], "download_size": 1191398, "dataset_size": 2143869}, {"config_name": "20231201.lij", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 1281480, "num_examples": 1185}], "download_size": 651083, "dataset_size": 1281480}, {"config_name": "20231201.lt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 7513991, "num_examples": 1874}], "download_size": 4637316, "dataset_size": 7513991}, {"config_name": "20231201.mk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 12706090, "num_examples": 2166}], "download_size": 5077478, "dataset_size": 12706090}, {"config_name": "20231201.ml", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 81611041, "num_examples": 6052}], "download_size": 29462281, "dataset_size": 81611041}, {"config_name": "20231201.mr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 35302346, "num_examples": 1485}], "download_size": 13300483, "dataset_size": 35302346}, {"config_name": "20231201.nap", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 91852, "num_examples": 155}], "download_size": 53478, "dataset_size": 91852}, {"config_name": "20231201.nl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 48325965, "num_examples": 5260}], "download_size": 27915130, "dataset_size": 48325965}, {"config_name": "20231201.no", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2289098, "num_examples": 379}], "download_size": 1397633, "dataset_size": 2289098}, {"config_name": "20231201.or", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18535382, "num_examples": 693}], "download_size": 7348706, "dataset_size": 18535382}, {"config_name": "20231201.pa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6536266, "num_examples": 107}], "download_size": 2583902, "dataset_size": 6536266}, {"config_name": "20231201.pl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 56457491, "num_examples": 12020}], "download_size": 34312764, "dataset_size": 56457491}, {"config_name": "20231201.pms", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 16256157, "num_examples": 4093}], "download_size": 9703819, "dataset_size": 16256157}, {"config_name": "20231201.pt", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 106619107, "num_examples": 23171}], "download_size": 62791422, "dataset_size": 106619107}, {"config_name": "20231201.ro", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 134273629, "num_examples": 12921}], "download_size": 81375524, "dataset_size": 134273629}, {"config_name": "20231201.ru", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 9393299725, "num_examples": 372768}], "download_size": 4601162148, "dataset_size": 9393299725}, {"config_name": "20231201.sa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 636225055, "num_examples": 22986}], "download_size": 231955608, "dataset_size": 636225055}, {"config_name": "20231201.sah", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17305188, "num_examples": 903}], "download_size": 7654932, "dataset_size": 17305188}, {"config_name": "20231201.sk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3532173, "num_examples": 390}], "download_size": 2217851, "dataset_size": 3532173}, {"config_name": "20231201.sl", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 366151991, "num_examples": 17267}], "download_size": 242655257, "dataset_size": 366151991}, {"config_name": "20231201.sr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 195710992, "num_examples": 38987}], "download_size": 86833442, "dataset_size": 195710992}, {"config_name": "20231201.su", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 356902, "num_examples": 20}], "download_size": 220452, "dataset_size": 356902}, {"config_name": "20231201.sv", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 27912113, "num_examples": 6296}], "download_size": 16513469, "dataset_size": 27912113}, {"config_name": "20231201.ta", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 113836204, "num_examples": 4702}], "download_size": 40070603, "dataset_size": 113836204}, {"config_name": "20231201.te", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 94840451, "num_examples": 9012}], "download_size": 36668092, "dataset_size": 94840451}, {"config_name": "20231201.th", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 73437990, "num_examples": 2383}], "download_size": 23644914, "dataset_size": 73437990}, {"config_name": "20231201.tr", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 64957772, "num_examples": 7220}], "download_size": 34039502, "dataset_size": 64957772}, {"config_name": "20231201.uk", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 46059083, "num_examples": 4171}], "download_size": 21135029, "dataset_size": 46059083}, {"config_name": "20231201.vec", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5700371, "num_examples": 3492}], "download_size": 3097037, "dataset_size": 5700371}, {"config_name": "20231201.vi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 48099940, "num_examples": 5471}], "download_size": 17336608, "dataset_size": 48099940}, {"config_name": "20231201.wa", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 3735624, "num_examples": 897}], "download_size": 2222694, "dataset_size": 3735624}, {"config_name": "20231201.yi", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 24802558, "num_examples": 1669}], "download_size": 10686751, "dataset_size": 24802558}, {"config_name": "20231201.zh", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5070438246, "num_examples": 265669}], "download_size": 3309500049, "dataset_size": 5070438246}, {"config_name": "20231201.zh-min-nan", "features": [{"name": "id", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21109492, "num_examples": 2360}], "download_size": 10288524, "dataset_size": 21109492}], "configs": [{"config_name": "20231201.ar", "data_files": [{"split": "train", "path": "20231201.ar/train-*"}]}, {"config_name": "20231201.as", "data_files": [{"split": "train", "path": "20231201.as/train-*"}]}, {"config_name": "20231201.az", "data_files": [{"split": "train", "path": "20231201.az/train-*"}]}, {"config_name": "20231201.ban", "data_files": [{"split": "train", "path": "20231201.ban/train-*"}]}, {"config_name": "20231201.be", "data_files": [{"split": "train", "path": "20231201.be/train-*"}]}, {"config_name": "20231201.bg", "data_files": [{"split": "train", "path": "20231201.bg/train-*"}]}, {"config_name": "20231201.bn", "data_files": [{"split": "train", "path": "20231201.bn/train-*"}]}, {"config_name": "20231201.br", "data_files": [{"split": "train", "path": "20231201.br/train-*"}]}, {"config_name": "20231201.bs", "data_files": [{"split": "train", "path": "20231201.bs/train-*"}]}, {"config_name": "20231201.ca", "data_files": [{"split": "train", "path": "20231201.ca/train-*"}]}, {"config_name": "20231201.cs", "data_files": [{"split": "train", "path": "20231201.cs/train-*"}]}, {"config_name": "20231201.cy", "data_files": [{"split": "train", "path": "20231201.cy/train-*"}]}, {"config_name": "20231201.da", "data_files": [{"split": "train", "path": "20231201.da/train-*"}]}, {"config_name": "20231201.de", "data_files": [{"split": "train", "path": "20231201.de/train-*"}]}, {"config_name": "20231201.el", "data_files": [{"split": "train", "path": "20231201.el/train-*"}]}, {"config_name": "20231201.en", "data_files": [{"split": "train", "path": "20231201.en/train-*"}]}, {"config_name": "20231201.eo", "data_files": [{"split": "train", "path": "20231201.eo/train-*"}]}, {"config_name": "20231201.es", "data_files": [{"split": "train", "path": "20231201.es/train-*"}]}, {"config_name": "20231201.et", "data_files": [{"split": "train", "path": "20231201.et/train-*"}]}, {"config_name": "20231201.eu", "data_files": [{"split": "train", "path": "20231201.eu/train-*"}]}, {"config_name": "20231201.fa", "data_files": [{"split": "train", "path": "20231201.fa/train-*"}]}, {"config_name": "20231201.fi", "data_files": [{"split": "train", "path": "20231201.fi/train-*"}]}, {"config_name": "20231201.fo", "data_files": [{"split": "train", "path": "20231201.fo/train-*"}]}, {"config_name": "20231201.fr", "data_files": [{"split": "train", "path": "20231201.fr/train-*"}]}, {"config_name": "20231201.gl", "data_files": [{"split": "train", "path": "20231201.gl/train-*"}]}, {"config_name": "20231201.gu", "data_files": [{"split": "train", "path": "20231201.gu/train-*"}]}, {"config_name": "20231201.he", "data_files": [{"split": "train", "path": "20231201.he/train-*"}]}, {"config_name": "20231201.hi", "data_files": [{"split": "train", "path": "20231201.hi/train-*"}]}, {"config_name": "20231201.hr", "data_files": [{"split": "train", "path": "20231201.hr/train-*"}]}, {"config_name": "20231201.hu", "data_files": [{"split": "train", "path": "20231201.hu/train-*"}]}, {"config_name": "20231201.hy", "data_files": [{"split": "train", "path": "20231201.hy/train-*"}]}, {"config_name": "20231201.id", "data_files": [{"split": "train", "path": "20231201.id/train-*"}]}, {"config_name": "20231201.is", "data_files": [{"split": "train", "path": "20231201.is/train-*"}]}, {"config_name": "20231201.it", "data_files": [{"split": "train", "path": "20231201.it/train-*"}]}, {"config_name": "20231201.ja", "data_files": [{"split": "train", "path": "20231201.ja/train-*"}]}, {"config_name": "20231201.jv", "data_files": [{"split": "train", "path": "20231201.jv/train-*"}]}, {"config_name": "20231201.kn", "data_files": [{"split": "train", "path": "20231201.kn/train-*"}]}, {"config_name": "20231201.ko", "data_files": [{"split": "train", "path": "20231201.ko/train-*"}]}, {"config_name": "20231201.la", "data_files": [{"split": "train", "path": "20231201.la/train-*"}]}, {"config_name": "20231201.li", "data_files": [{"split": "train", "path": "20231201.li/train-*"}]}, {"config_name": "20231201.lij", "data_files": [{"split": "train", "path": "20231201.lij/train-*"}]}, {"config_name": "20231201.lt", "data_files": [{"split": "train", "path": "20231201.lt/train-*"}]}, {"config_name": "20231201.mk", "data_files": [{"split": "train", "path": "20231201.mk/train-*"}]}, {"config_name": "20231201.ml", "data_files": [{"split": "train", "path": "20231201.ml/train-*"}]}, {"config_name": "20231201.mr", "data_files": [{"split": "train", "path": "20231201.mr/train-*"}]}, {"config_name": "20231201.nap", "data_files": [{"split": "train", "path": "20231201.nap/train-*"}]}, {"config_name": "20231201.nl", "data_files": [{"split": "train", "path": "20231201.nl/train-*"}]}, {"config_name": "20231201.no", "data_files": [{"split": "train", "path": "20231201.no/train-*"}]}, {"config_name": "20231201.or", "data_files": [{"split": "train", "path": "20231201.or/train-*"}]}, {"config_name": "20231201.pa", "data_files": [{"split": "train", "path": "20231201.pa/train-*"}]}, {"config_name": "20231201.pl", "data_files": [{"split": "train", "path": "20231201.pl/train-*"}]}, {"config_name": "20231201.pms", "data_files": [{"split": "train", "path": "20231201.pms/train-*"}]}, {"config_name": "20231201.pt", "data_files": [{"split": "train", "path": "20231201.pt/train-*"}]}, {"config_name": "20231201.ro", "data_files": [{"split": "train", "path": "20231201.ro/train-*"}]}, {"config_name": "20231201.ru", "data_files": [{"split": "train", "path": "20231201.ru/train-*"}]}, {"config_name": "20231201.sa", "data_files": [{"split": "train", "path": "20231201.sa/train-*"}]}, {"config_name": "20231201.sah", "data_files": [{"split": "train", "path": "20231201.sah/train-*"}]}, {"config_name": "20231201.sk", "data_files": [{"split": "train", "path": "20231201.sk/train-*"}]}, {"config_name": "20231201.sl", "data_files": [{"split": "train", "path": "20231201.sl/train-*"}]}, {"config_name": "20231201.sr", "data_files": [{"split": "train", "path": "20231201.sr/train-*"}]}, {"config_name": "20231201.su", "data_files": [{"split": "train", "path": "20231201.su/train-*"}]}, {"config_name": "20231201.sv", "data_files": [{"split": "train", "path": "20231201.sv/train-*"}]}, {"config_name": "20231201.ta", "data_files": [{"split": "train", "path": "20231201.ta/train-*"}]}, {"config_name": "20231201.te", "data_files": [{"split": "train", "path": "20231201.te/train-*"}]}, {"config_name": "20231201.th", "data_files": [{"split": "train", "path": "20231201.th/train-*"}]}, {"config_name": "20231201.tr", "data_files": [{"split": "train", "path": "20231201.tr/train-*"}]}, {"config_name": "20231201.uk", "data_files": [{"split": "train", "path": "20231201.uk/train-*"}]}, {"config_name": "20231201.vec", "data_files": [{"split": "train", "path": "20231201.vec/train-*"}]}, {"config_name": "20231201.vi", "data_files": [{"split": "train", "path": "20231201.vi/train-*"}]}, {"config_name": "20231201.wa", "data_files": [{"split": "train", "path": "20231201.wa/train-*"}]}, {"config_name": "20231201.yi", "data_files": [{"split": "train", "path": "20231201.yi/train-*"}]}, {"config_name": "20231201.zh", "data_files": [{"split": "train", "path": "20231201.zh/train-*"}]}, {"config_name": "20231201.zh-min-nan", "data_files": [{"split": "train", "path": "20231201.zh-min-nan/train-*"}]}]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #language-Arabic #language-Assamese #language-Azerbaijani #language-Balinese #language-Belarusian #language-Bulgarian #language-Bengali #language-Breton #language-Bosnian #language-Catalan #language-Czech #language-Welsh #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-Faroese #language-French #language-Galician #language-Gujarati #language-Hebrew #language-Hindi #language-Croatian #language-Hungarian #language-Armenian #language-Indonesian #language-Icelandic #language-Italian #language-Japanese #language-Javanese #language-Kannada #language-Korean #language-Latin #language-Limburgan #language-Ligurian #language-Lithuanian #language-Macedonian #language-Malayalam #language-Marathi #language-Min Nan Chinese #language-Neapolitan #language-Dutch #language-Norwegian #language-Oriya (macrolanguage) #language-Panjabi #language-Polish #language-Piemontese #language-Portuguese #language-Romanian #language-Russian #language-Sanskrit #language-Yakut #language-Slovak #language-Slovenian #language-Serbian #language-Sundanese #language-Swedish #language-Tamil #language-Telugu #language-Thai #language-Turkish #language-Ukrainian #language-Venetian #language-Vietnamese #language-Walloon #language-Yiddish #language-Chinese #license-cc-by-sa-3.0 #license-gfdl #region-us 


# Dataset Card for Wikimedia Wikisource

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact:

### Dataset Summary

Wikisource dataset containing cleaned articles of all languages.

The dataset is built from the Wikisource dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikisource text with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them by date and language like this:


### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

You can find the list of all languages here: URL

Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual
Wikisource here: URL

## Dataset Structure

### Data Instances

An example looks as follows:


### Data Fields

The data fields are the same among all language configurations:
- 'id' ('str'): ID of the text.
- 'url' ('str'): URL of the text.
- 'title' ('str'): Title of the text.
- 'text' ('str'): Content of the text.

### Data Splits

All language configurations contain a single 'train' split.

## Dataset Creation

### Curation Rationale



### Source Data

#### Initial Data Collection and Normalization

The dataset is built from the Wikisource dumps: URL

You can find the full list of languages and dates here: URL

The articles have been parsed using the 'mwparserfromhell' tool.

#### Who are the source language producers?



### Annotations

#### Annotation process



#### Who are the annotators?



### Personal and Sensitive Information



## Considerations for Using the Data

### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



## Additional Information

### Dataset Curators



### Licensing Information

Copyright licensing information: URL

All original textual content is licensed under the GNU Free Documentation License (GFDL)
and the Creative Commons Attribution-Share-Alike 3.0 License.
Some text may be available only under the Creative Commons license; see their Terms of Use for details.
Text written by some authors may be released under additional licenses or into the public domain.





### Contributions

Thanks to @albertvillanova for adding this dataset.

# Dataset Card for Wikimedia Wikisource ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: ### Dataset Summary

Wikisource dataset containing cleaned articles of all languages.

The dataset is built from the Wikisource dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikisource text with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them by date and language like this: ### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling. ### Languages

You can find the list of all languages here: URL

Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual
Wikisource here: URL ## Dataset Structure ### Data Instances

An example looks as follows: ### Data Fields

The data fields are the same among all language configurations:
- 'id' ('str'): ID of the text.
- 'url' ('str'): URL of the text.
- 'title' ('str'): Title of the text.
- 'text' ('str'): Content of the text. ### Data Splits

All language configurations contain a single 'train' split. ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization

The dataset is built from the Wikisource dumps: URL

You can find the full list of languages and dates here: URL

The articles have been parsed using the 'mwparserfromhell' tool. #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Copyright licensing information: URL

All original textual content is licensed under the GNU Free Documentation License (GFDL)
and the Creative Commons Attribution-Share-Alike 3.0 License.
Some text may be available only under the Creative Commons license; see their Terms of Use for details.
Text written by some authors may be released under additional licenses or into the public domain. ### Contributions

Thanks to @albertvillanova for adding this dataset.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #language-Arabic #language-Assamese #language-Azerbaijani #language-Balinese #language-Belarusian #language-Bulgarian #language-Bengali #language-Breton #language-Bosnian #language-Catalan #language-Czech #language-Welsh #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-Faroese #language-French #language-Galician #language-Gujarati #language-Hebrew #language-Hindi #language-Croatian #language-Hungarian #language-Armenian #language-Indonesian #language-Icelandic #language-Italian #language-Japanese #language-Javanese #language-Kannada #language-Korean #language-Latin #language-Limburgan #language-Ligurian #language-Lithuanian #language-Macedonian #language-Malayalam #language-Marathi #language-Min Nan Chinese #language-Neapolitan #language-Dutch #language-Norwegian #language-Oriya (macrolanguage) #language-Panjabi #language-Polish #language-Piemontese #language-Portuguese #language-Romanian #language-Russian #language-Sanskrit #language-Yakut #language-Slovak #language-Slovenian #language-Serbian #language-Sundanese #language-Swedish #language-Tamil #language-Telugu #language-Thai #language-Turkish #language-Ukrainian #language-Venetian #language-Vietnamese #language-Walloon #language-Yiddish #language-Chinese #license-cc-by-sa-3.0 #license-gfdl #region-us 
 # Dataset Card for Wikimedia Wikisource ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: ### Dataset Summary

Wikisource dataset containing cleaned articles of all languages.

The dataset is built from the Wikisource dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikisource text with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them by date and language like this: ### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling. ### Languages

You can find the list of all languages here: URL

Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual
Wikisource here: URL ## Dataset Structure ### Data Instances

An example looks as follows: ### Data Fields

The data fields are the same among all language configurations:
- 'id' ('str'): ID of the text.
- 'url' ('str'): URL of the text.
- 'title' ('str'): Title of the text.
- 'text' ('str'): Content of the text. ### Data Splits

All language configurations contain a single 'train' split. ## Dataset Creation ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization

The dataset is built from the Wikisource dumps: URL

You can find the full list of languages and dates here: URL

The articles have been parsed using the 'mwparserfromhell' tool. #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ## Additional Information ### Dataset Curators ### Licensing Information

Copyright licensing information: URL

All original textual content is licensed under the GNU Free Documentation License (GFDL)
and the Creative Commons Attribution-Share-Alike 3.0 License.
Some text may be available only under the Creative Commons license; see their Terms of Use for details.
Text written by some authors may be released under additional licenses or into the public domain. ### Contributions

Thanks to @albertvillanova for adding this dataset.
passage:  passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #language-Arabic #language-Assamese #language-Azerbaijani #language-Balinese #language-Belarusian #language-Bulgarian #language-Bengali #language-Breton #language-Bosnian #language-Catalan #language-Czech #language-Welsh #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-Finnish #language-Faroese #language-French #language-Galician #language-Gujarati #language-Hebrew #language-Hindi #language-Croatian #language-Hungarian #language-Armenian #language-Indonesian #language-Icelandic #language-Italian #language-Japanese #language-Javanese #language-Kannada #language-Korean #language-Latin #language-Limburgan #language-Ligurian #language-Lithuanian #language-Macedonian #language-Malayalam #language-Marathi #language-Min Nan Chinese #language-Neapolitan #language-Dutch #language-Norwegian #language-Oriya (macrolanguage) #language-Panjabi #language-Polish #language-Piemontese #language-Portuguese #language-Romanian #language-Russian #language-Sanskrit #language-Yakut #language-Slovak #language-Slovenian #language-Serbian #language-Sundanese #language-Swedish #language-Tamil #language-Telugu #language-Thai #language-Turkish #language-Ukrainian #language-Venetian #language-Vietnamese #language-Walloon #language-Yiddish #language-Chinese #license-cc-by-sa-3.0 #license-gfdl #region-us 
# Dataset Card for Wikimedia Wikisource## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions## Dataset Description

- Homepage: URL
- Repository:
- Paper:
- Leaderboard:
- Point of Contact:### Dataset Summary

Wikisource dataset containing cleaned articles of all languages.

The dataset is built from the Wikisource dumps (URL
with one subset per language, each containing a single train split.

Each example contains the content of one full Wikisource text with cleaning to strip
markdown and unwanted sections (references, etc.).


All language subsets have already been processed for recent dump, and you can load them by date and language like this:### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.### Languages

You can find the list of all languages here: URL

Note that the wiki code "www" contains multilingual texts. You can find the list of languages at the "www" Multilingual
Wikisource here: URL## Dataset Structure### Data Instances

An example looks as follows:### Data Fields

The data fields are the same among all language configurations:
- 'id' ('str'): ID of the text.
- 'url' ('str'): URL of the text.
- 'title' ('str'): Title of the text.
- 'text' ('str'): Content of the text.### Data Splits

All language configurations contain a single 'train' split.## Dataset Creation### Curation Rationale### Source Data