# Dataset Card for yaakov/wikipedia-de-splits

## Dataset Description
The only goal of this dataset is to have random German Wikipedia articles at
various dataset sizes:  Small datasets for fast development and large datasets for statistically relevant measurements.

For this purpose, I loaded the 2665357 articles in the `test` set of the pre-processed German Wikipedia dump from 2022-03-01, randomly permuted the articles and created splits of sizes `2**n`: `1, 2, 4, 8, ...`.  The split names are strings.  The split `'all'` contains all 2665357 available articles.

## Dataset creation
This dataset has been created with the following script:

    !apt install git-lfs
    !pip install -q transformers datasets
    
    from huggingface_hub import notebook_login
    notebook_login()
    
    from datasets import load_dataset
    wikipedia_de = load_dataset("wikipedia", "20220301.de")['train']
    
    shuffled = wikipedia_de.shuffle(seed=42)
    
    from datasets import DatasetDict
    res = DatasetDict()
    
    k, n = 0, 1
    while n <= shuffled.num_rows:
        res[str(k)] = shuffled.select(range(n))
        k += 1; n *= 2
    res['all'] = shuffled
    
    res.push_to_hub('yaakov/wikipedia-de-splits')
task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:crowdsourced size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M size_categories:1M<n<10M source_datasets:wikipedia language:de license:cc-by-sa-3.0 license:gfdl region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["crowdsourced"], "language": ["de"], "license": ["cc-by-sa-3.0", "gfdl"], "size_categories": ["n<1K", "1K<n<10K", "10K<n<100K", "100K<n<1M", "1M<n<10M"], "source_datasets": ["wikipedia"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "pretty_name": "wikipedia-de-splits", "configs": ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "all"]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-crowdsourced #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #source_datasets-wikipedia #language-German #license-cc-by-sa-3.0 #license-gfdl #region-us 

# Dataset Card for yaakov/wikipedia-de-splits

## Dataset Description
The only goal of this dataset is to have random German Wikipedia articles at
various dataset sizes:  Small datasets for fast development and large datasets for statistically relevant measurements.

For this purpose, I loaded the 2665357 articles in the 'test' set of the pre-processed German Wikipedia dump from 2022-03-01, randomly permuted the articles and created splits of sizes '2n': '1, 2, 4, 8, ...'.  The split names are strings.  The split ''all'' contains all 2665357 available articles.

## Dataset creation
This dataset has been created with the following script:

    !apt install git-lfs
    !pip install -q transformers datasets
    
    from huggingface_hub import notebook_login
    notebook_login()
    
    from datasets import load_dataset
    wikipedia_de = load_dataset("wikipedia", "URL")['train']
    
    shuffled = wikipedia_de.shuffle(seed=42)
    
    from datasets import DatasetDict
    res = DatasetDict()
    
    k, n = 0, 1
    while n <= shuffled.num_rows:
        res[str(k)] = URL(range(n))
        k += 1; n *= 2
    res['all'] = shuffled
    
    res.push_to_hub('yaakov/wikipedia-de-splits')
# Dataset Card for yaakov/wikipedia-de-splits ## Dataset Description
The only goal of this dataset is to have random German Wikipedia articles at
various dataset sizes:  Small datasets for fast development and large datasets for statistically relevant measurements.

For this purpose, I loaded the 2665357 articles in the 'test' set of the pre-processed German Wikipedia dump from 2022-03-01, randomly permuted the articles and created splits of sizes '2n': '1, 2, 4, 8, ...'.  The split names are strings.  The split ''all'' contains all 2665357 available articles. ## Dataset creation
This dataset has been created with the following script:

    !apt install git-lfs
    !pip install -q transformers datasets
    
    from huggingface_hub import notebook_login
    notebook_login()
    
    from datasets import load_dataset
    wikipedia_de = load_dataset("wikipedia", "URL")['train']
    
    shuffled = wikipedia_de.shuffle(seed=42)
    
    from datasets import DatasetDict
    res = DatasetDict()
    
    k, n = 0, 1
    while n <= shuffled.num_rows:
        res[str(k)] = URL(range(n))
        k += 1; n *= 2
    res['all'] = shuffled
    
    res.push_to_hub('yaakov/wikipedia-de-splits')
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-crowdsourced #size_categories-n<1K #size_categories-1K<n<10K #size_categories-10K<n<100K #size_categories-100K<n<1M #size_categories-1M<n<10M #source_datasets-wikipedia #language-German #license-cc-by-sa-3.0 #license-gfdl #region-us 
 # Dataset Card for yaakov/wikipedia-de-splits ## Dataset Description
The only goal of this dataset is to have random German Wikipedia articles at
various dataset sizes:  Small datasets for fast development and large datasets for statistically relevant measurements.

For this purpose, I loaded the 2665357 articles in the 'test' set of the pre-processed German Wikipedia dump from 2022-03-01, randomly permuted the articles and created splits of sizes '2n': '1, 2, 4, 8, ...'.  The split names are strings.  The split ''all'' contains all 2665357 available articles. ## Dataset creation
This dataset has been created with the following script:

    !apt install git-lfs
    !pip install -q transformers datasets
    
    from huggingface_hub import notebook_login
    notebook_login()
    
    from datasets import load_dataset
    wikipedia_de = load_dataset("wikipedia", "URL")['train']
    
    shuffled = wikipedia_de.shuffle(seed=42)
    
    from datasets import DatasetDict
    res = DatasetDict()
    
    k, n = 0, 1
    while n <= shuffled.num_rows:
        res[str(k)] = URL(range(n))
        k += 1; n *= 2
    res['all'] = shuffled
    
    res.push_to_hub('yaakov/wikipedia-de-splits')