
# Arabic Wiki Dataset

## Dataset Summary
This dataset is extracted using [`wikiextractor`](https://github.com/attardi/wikiextractor) tool, from [Wikipedia Arabic pages](https://dumps.wikimedia.org/arwiki/).

## Supported Tasks and Leaderboards
Intended to train **Arabic** language models on MSA (Modern Standard Arabic).

## Dataset Structure
The dataset is structured into 2 folders:
- `arwiki_20211213_txt`: dataset is divided into subfolders each of which contains no more than 100 documents.
- `arwiki_20211213_txt_single`: all documents merged together in a single txt file.

## Dataset Statistics

#### Extracts from **December 13, 2021**:

| documents | vocabulary | words |
| --- | --- | --- |
| 1,136,455 | 5,446,560 | 175,566,016 |

## Usage
Load all dataset from the single txt file:
```python
load_dataset('CALM/arwiki', 
              data_files='arwiki_2021_txt_single/arwiki_20211213.txt')

# OR with stream

load_dataset('CALM/arwiki', 
              data_files='arwiki_2021_txt_single/arwiki_20211213.txt', 
              streaming=True)
```
Load a smaller subset from the individual txt files:
```python
load_dataset('CALM/arwiki', 
              data_files='arwiki_2021_txt/AA/arwiki_20211213_1208.txt')

# OR with stream

load_dataset('CALM/arwiki', 
              data_files='arwiki_2021_txt/AA/arwiki_20211213_1208.txt', 
              streaming=True)
```
multilinguality:monolingual language:ar license:unknown region:us
{"language": ["ar"], "license": ["unknown"], "multilinguality": ["monolingual"], "pretty_name": "Wikipedia Arabic dumps dataset."}
TAGS
#multilinguality-monolingual #language-Arabic #license-unknown #region-us 

Arabic Wiki Dataset
===================


Dataset Summary
---------------


This dataset is extracted using 'wikiextractor' tool, from Wikipedia Arabic pages.


Supported Tasks and Leaderboards
--------------------------------


Intended to train Arabic language models on MSA (Modern Standard Arabic).


Dataset Structure
-----------------


The dataset is structured into 2 folders:


* 'arwiki\_20211213\_txt': dataset is divided into subfolders each of which contains no more than 100 documents.
* 'arwiki\_20211213\_txt\_single': all documents merged together in a single txt file.


Dataset Statistics
------------------


#### Extracts from December 13, 2021:


documents: 1,136,455, vocabulary: 5,446,560, words: 175,566,016


Usage
-----


Load all dataset from the single txt file:


Load a smaller subset from the individual txt files:



#### Extracts from December 13, 2021:


documents: 1,136,455, vocabulary: 5,446,560, words: 175,566,016


Usage
-----


Load all dataset from the single txt file:


Load a smaller subset from the individual txt files:
TAGS
#multilinguality-monolingual #language-Arabic #license-unknown #region-us 
 #### Extracts from December 13, 2021:


documents: 1,136,455, vocabulary: 5,446,560, words: 175,566,016


Usage
-----


Load all dataset from the single txt file:


Load a smaller subset from the individual txt files:
passage: TAGS
#multilinguality-monolingual #language-Arabic #license-unknown #region-us 
#### Extracts from December 13, 2021:


documents: 1,136,455, vocabulary: 5,446,560, words: 175,566,016


Usage
-----


Load all dataset from the single txt file:


Load a smaller subset from the individual txt files: