
# Dataset Card for EUWikipedias: A dataset of Wikipedias in the EU languages

## Table of Contents

- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
    - [Annotations](#annotations)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:** 
- **Paper:** 
- **Leaderboard:**
- **Point of Contact:** [Joel Niklaus](mailto:joel.niklaus.2@bfh.ch)

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

### Supported Tasks and Leaderboards

The dataset supports the tasks of fill-mask.

### Languages

The following languages are supported: 
bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

## Dataset Structure

It is structured in the following format: {date}/{language}_{shard}.jsonl.xz
At the moment only the date '20221120' is supported.

Use the dataset like this:
```python
from datasets import load_dataset

dataset = load_dataset('joelito/EU_Wikipedias', date="20221120", language="de", split='train', streaming=True)
```


### Data Instances

The file format is jsonl.xz and there is one split available (`train`). 

| Source       |   Size (MB) |      Words |   Documents |   Words/Document |
|:-------------|------------:|-----------:|------------:|-----------------:|
| 20221120.all |       86034 | 9506846949 |    26481379 |              359 |
| 20221120.bg  |        1261 |   88138772 |      285876 |              308 |
| 20221120.cs  |        1904 |  189580185 |      513851 |              368 |
| 20221120.da  |         679 |   74546410 |      286864 |              259 |
| 20221120.de  |       11761 | 1191919523 |     2740891 |              434 |
| 20221120.el  |        1531 |  103504078 |      215046 |              481 |
| 20221120.en  |       26685 | 3192209334 |     6575634 |              485 |
| 20221120.es  |        6636 |  801322400 |     1583597 |              506 |
| 20221120.et  |         538 |   48618507 |      231609 |              209 |
| 20221120.fi  |        1391 |  115779646 |      542134 |              213 |
| 20221120.fr  |        9703 | 1140823165 |     2472002 |              461 |
| 20221120.ga  |          72 |    8025297 |       57808 |              138 |
| 20221120.hr  |         555 |   58853753 |      198746 |              296 |
| 20221120.hu  |        1855 |  167732810 |      515777 |              325 |
| 20221120.it  |        5999 |  687745355 |     1782242 |              385 |
| 20221120.lt  |         409 |   37572513 |      203233 |              184 |
| 20221120.lv  |         269 |   25091547 |      116740 |              214 |
| 20221120.mt  |          29 |    2867779 |        5030 |              570 |
| 20221120.nl  |        3208 |  355031186 |     2107071 |              168 |
| 20221120.pl  |        3608 |  349900622 |     1543442 |              226 |
| 20221120.pt  |        3315 |  389786026 |     1095808 |              355 |
| 20221120.ro  |        1017 |  111455336 |      434935 |              256 |
| 20221120.sk  |         506 |   49612232 |      238439 |              208 |
| 20221120.sl  |         543 |   58858041 |      178472 |              329 |
| 20221120.sv  |        2560 |  257872432 |     2556132 |              100 |


### Data Fields

[More Information Needed]

### Data Splits

[More Information Needed]

## Dataset Creation

This dataset has been created by downloading the wikipedias using [olm/wikipedia](https://huggingface.co/datasets/olm/wikipedia) for the 24 EU languages.
For more information about the creation of the dataset please refer to prepare_wikipedias.py

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]


### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

```
TODO add citation
```

### Contributions

Thanks to [@JoelNiklaus](https://github.com/joelniklaus) for adding this dataset.

task_categories:fill-mask annotations_creators:other language_creators:found multilinguality:multilingual size_categories:10M<n<100M source_datasets:original language:bg language:cs language:da language:de language:el language:en language:es language:et language:fi language:fr language:ga language:hr language:hu language:it language:lt language:lv language:mt language:nl language:pl language:pt language:ro language:sk language:sl language:sv license:cc-by-4.0 region:us
{"annotations_creators": ["other"], "language_creators": ["found"], "language": ["bg", "cs", "da", "de", "el", "en", "es", "et", "fi", "fr", "ga", "hr", "hu", "it", "lt", "lv", "mt", "nl", "pl", "pt", "ro", "sk", "sl", "sv"], "license": ["cc-by-4.0"], "multilinguality": ["multilingual"], "size_categories": ["10M<n<100M"], "source_datasets": ["original"], "task_categories": ["fill-mask"], "pretty_name": "EUWikipedias: A dataset of Wikipedias in the EU languages"}
TAGS
#task_categories-fill-mask #annotations_creators-other #language_creators-found #multilinguality-multilingual #size_categories-10M<n<100M #source_datasets-original #language-Bulgarian #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Spanish #language-Estonian #language-Finnish #language-French #language-Irish #language-Croatian #language-Hungarian #language-Italian #language-Lithuanian #language-Latvian #language-Maltese #language-Dutch #language-Polish #language-Portuguese #language-Romanian #language-Slovak #language-Slovenian #language-Swedish #license-cc-by-4.0 #region-us 

Dataset Card for EUWikipedias: A dataset of Wikipedias in the EU languages
==========================================================================


Table of Contents
-----------------


* Table of Contents
* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage:
* Repository:
* Paper:
* Leaderboard:
* Point of Contact: Joel Niklaus


### Dataset Summary


Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).


### Supported Tasks and Leaderboards


The dataset supports the tasks of fill-mask.


### Languages


The following languages are supported:
bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv


Dataset Structure
-----------------


It is structured in the following format: {date}/{language}\_{shard}.URL
At the moment only the date '20221120' is supported.


Use the dataset like this:


### Data Instances


The file format is URL and there is one split available ('train').



### Data Fields


### Data Splits


Dataset Creation
----------------


This dataset has been created by downloading the wikipedias using olm/wikipedia for the 24 EU languages.
For more information about the creation of the dataset please refer to prepare\_wikipedias.py


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


### Contributions


Thanks to @JoelNiklaus for adding this dataset.



### Dataset Summary


Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.). ### Supported Tasks and Leaderboards


The dataset supports the tasks of fill-mask. ### Languages


The following languages are supported:
bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv


Dataset Structure
-----------------


It is structured in the following format: {date}/{language}\_{shard}.URL
At the moment only the date '20221120' is supported.


Use the dataset like this: ### Data Instances


The file format is URL and there is one split available ('train'). ### Data Fields ### Data Splits


Dataset Creation
----------------


This dataset has been created by downloading the wikipedias using olm/wikipedia for the 24 EU languages.
For more information about the creation of the dataset please refer to prepare\_wikipedias.py ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions


Thanks to @JoelNiklaus for adding this dataset.
TAGS
#task_categories-fill-mask #annotations_creators-other #language_creators-found #multilinguality-multilingual #size_categories-10M<n<100M #source_datasets-original #language-Bulgarian #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Spanish #language-Estonian #language-Finnish #language-French #language-Irish #language-Croatian #language-Hungarian #language-Italian #language-Lithuanian #language-Latvian #language-Maltese #language-Dutch #language-Polish #language-Portuguese #language-Romanian #language-Slovak #language-Slovenian #language-Swedish #license-cc-by-4.0 #region-us 
 ### Dataset Summary


Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.). ### Supported Tasks and Leaderboards


The dataset supports the tasks of fill-mask. ### Languages


The following languages are supported:
bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv


Dataset Structure
-----------------


It is structured in the following format: {date}/{language}\_{shard}.URL
At the moment only the date '20221120' is supported.


Use the dataset like this: ### Data Instances


The file format is URL and there is one split available ('train'). ### Data Fields ### Data Splits


Dataset Creation
----------------


This dataset has been created by downloading the wikipedias using olm/wikipedia for the 24 EU languages.
For more information about the creation of the dataset please refer to prepare\_wikipedias.py ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions


Thanks to @JoelNiklaus for adding this dataset.
passage: TAGS
#task_categories-fill-mask #annotations_creators-other #language_creators-found #multilinguality-multilingual #size_categories-10M<n<100M #source_datasets-original #language-Bulgarian #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Spanish #language-Estonian #language-Finnish #language-French #language-Irish #language-Croatian #language-Hungarian #language-Italian #language-Lithuanian #language-Latvian #language-Maltese #language-Dutch #language-Polish #language-Portuguese #language-Romanian #language-Slovak #language-Slovenian #language-Swedish #license-cc-by-4.0 #region-us 
### Dataset Summary


Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(URL with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).### Supported Tasks and Leaderboards


The dataset supports the tasks of fill-mask.### Languages


The following languages are supported:
bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv


Dataset Structure
-----------------


It is structured in the following format: {date}/{language}\_{shard}.URL
At the moment only the date '20221120' is supported.


Use the dataset like this:### Data Instances


The file format is URL and there is one split available ('train').### Data Fields### Data Splits


Dataset Creation
----------------


This dataset has been created by downloading the wikipedias using olm/wikipedia for the 24 EU languages.
For more information about the creation of the dataset please refer to prepare\_wikipedias.py