
# Dataset Card for MiniPile

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Languages](#languages)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)


## Dataset Description

[The MiniPile Challenge for Data-Efficient Language Models](https://arxiv.org/abs/2304.08442)

### Dataset Summary

MiniPile is a 6GB subset of the [deduplicated The Pile corpus](https://huggingface.co/datasets/EleutherAI/the_pile_deduplicated). To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters.

The primary motivation for curating MiniPile is that (i) diverse pre-training datasets (like the Pile) are often too large for academic budgets and (ii) most smaller-scale datasets are fairly homogeneous and thereby unrepresentative of contemporary general-purpose language models. MiniPile aims to fill this gap and thereby facilitate data-efficient research on model architectures, training procedures, optimizers, etc.

More details on the MiniPile curation procedure and some pre-training results be found in the [MiniPile paper](https://arxiv.org/abs/2304.08442).

For more details on the Pile corpus, we refer the reader to [the Pile datasheet](https://arxiv.org/abs/2201.07311).

### Languages

English (`EN`)

## Additional Information

### Dataset Curators

MiniPile is a subset of the Pile, curated by Jean Kaddour. The Pile was created by Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy.

### Licensing Information

Since MiniPile is a subset of the Pile, the same MIT License holds.

### Citation Information

```
@article{kaddour2023minipile,
  title={The MiniPile Challenge for Data-Efficient Language Models},
  author={Kaddour, Jean},
  journal={arXiv preprint arXiv:2304.08442},
  year={2023}
}

@article{gao2020pile,
  title={The {P}ile: An 800{GB} dataset of diverse text for language modeling},
  author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and others},
  journal={arXiv preprint arXiv:2101.00027},
  year={2020}
}
```

task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:found multilinguality:monolingual size_categories:1M<n<10M source_datasets:original language:en license:other arxiv:2304.08442 arxiv:2201.07311 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["en"], "license": "other", "multilinguality": ["monolingual"], "size_categories": ["1M<n<10M"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask"], "task_ids": ["language-modeling", "masked-language-modeling"], "paperswithcode_id": "minipile", "pretty_name": "MiniPile", "dataset_info": {"features": [{"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 5906108510, "num_examples": 1000000}, {"name": "validation", "num_bytes": 2779386, "num_examples": 500}, {"name": "test", "num_bytes": 58558191, "num_examples": 10000}], "download_size": 3177432813, "dataset_size": 5967446087}}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-monolingual #size_categories-1M<n<10M #source_datasets-original #language-English #license-other #arxiv-2304.08442 #arxiv-2201.07311 #region-us 


# Dataset Card for MiniPile

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Languages
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information


## Dataset Description

The MiniPile Challenge for Data-Efficient Language Models

### Dataset Summary

MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters.

The primary motivation for curating MiniPile is that (i) diverse pre-training datasets (like the Pile) are often too large for academic budgets and (ii) most smaller-scale datasets are fairly homogeneous and thereby unrepresentative of contemporary general-purpose language models. MiniPile aims to fill this gap and thereby facilitate data-efficient research on model architectures, training procedures, optimizers, etc.

More details on the MiniPile curation procedure and some pre-training results be found in the MiniPile paper.

For more details on the Pile corpus, we refer the reader to the Pile datasheet.

### Languages

English ('EN')

## Additional Information

### Dataset Curators

MiniPile is a subset of the Pile, curated by Jean Kaddour. The Pile was created by Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy.

### Licensing Information

Since MiniPile is a subset of the Pile, the same MIT License holds.





# Dataset Card for MiniPile ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Languages
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information ## Dataset Description

The MiniPile Challenge for Data-Efficient Language Models ### Dataset Summary

MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters.

The primary motivation for curating MiniPile is that (i) diverse pre-training datasets (like the Pile) are often too large for academic budgets and (ii) most smaller-scale datasets are fairly homogeneous and thereby unrepresentative of contemporary general-purpose language models. MiniPile aims to fill this gap and thereby facilitate data-efficient research on model architectures, training procedures, optimizers, etc.

More details on the MiniPile curation procedure and some pre-training results be found in the MiniPile paper.

For more details on the Pile corpus, we refer the reader to the Pile datasheet. ### Languages

English ('EN') ## Additional Information ### Dataset Curators

MiniPile is a subset of the Pile, curated by Jean Kaddour. The Pile was created by Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy. ### Licensing Information

Since MiniPile is a subset of the Pile, the same MIT License holds.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-monolingual #size_categories-1M<n<10M #source_datasets-original #language-English #license-other #arxiv-2304.08442 #arxiv-2201.07311 #region-us 
 # Dataset Card for MiniPile ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Languages
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information ## Dataset Description

The MiniPile Challenge for Data-Efficient Language Models ### Dataset Summary

MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters.

The primary motivation for curating MiniPile is that (i) diverse pre-training datasets (like the Pile) are often too large for academic budgets and (ii) most smaller-scale datasets are fairly homogeneous and thereby unrepresentative of contemporary general-purpose language models. MiniPile aims to fill this gap and thereby facilitate data-efficient research on model architectures, training procedures, optimizers, etc.

More details on the MiniPile curation procedure and some pre-training results be found in the MiniPile paper.

For more details on the Pile corpus, we refer the reader to the Pile datasheet. ### Languages

English ('EN') ## Additional Information ### Dataset Curators

MiniPile is a subset of the Pile, curated by Jean Kaddour. The Pile was created by Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy. ### Licensing Information

Since MiniPile is a subset of the Pile, the same MIT License holds.
passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-monolingual #size_categories-1M<n<10M #source_datasets-original #language-English #license-other #arxiv-2304.08442 #arxiv-2201.07311 #region-us 
# Dataset Card for MiniPile## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Languages
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information## Dataset Description

The MiniPile Challenge for Data-Efficient Language Models### Dataset Summary

MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters.

The primary motivation for curating MiniPile is that (i) diverse pre-training datasets (like the Pile) are often too large for academic budgets and (ii) most smaller-scale datasets are fairly homogeneous and thereby unrepresentative of contemporary general-purpose language models. MiniPile aims to fill this gap and thereby facilitate data-efficient research on model architectures, training procedures, optimizers, etc.

More details on the MiniPile curation procedure and some pre-training results be found in the MiniPile paper.

For more details on the Pile corpus, we refer the reader to the Pile datasheet.### Languages

English ('EN')## Additional Information