
# Dataset Card for MMLU

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Repository**: https://github.com/hendrycks/test
- **Paper**: https://arxiv.org/abs/2009.03300

### Dataset Summary

[Measuring Massive Multitask Language Understanding](https://arxiv.org/pdf/2009.03300) by [Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/), [Collin Burns](http://collinpburns.com), [Steven Basart](https://stevenbas.art), Andy Zou, Mantas Mazeika, [Dawn Song](https://people.eecs.berkeley.edu/~dawnsong/), and [Jacob Steinhardt](https://www.stat.berkeley.edu/~jsteinhardt/) (ICLR 2021).

This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.

A complete list of tasks: ['abstract_algebra', 'anatomy', 'astronomy', 'business_ethics', 'clinical_knowledge', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_medicine', 'college_physics', 'computer_security', 'conceptual_physics', 'econometrics', 'electrical_engineering', 'elementary_mathematics', 'formal_logic', 'global_facts', 'high_school_biology', 'high_school_chemistry', 'high_school_computer_science', 'high_school_european_history', 'high_school_geography', 'high_school_government_and_politics', 'high_school_macroeconomics', 'high_school_mathematics', 'high_school_microeconomics', 'high_school_physics', 'high_school_psychology', 'high_school_statistics', 'high_school_us_history', 'high_school_world_history', 'human_aging', 'human_sexuality', 'international_law', 'jurisprudence', 'logical_fallacies', 'machine_learning', 'management', 'marketing', 'medical_genetics', 'miscellaneous', 'moral_disputes', 'moral_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional_accounting', 'professional_law', 'professional_medicine', 'professional_psychology', 'public_relations', 'security_studies', 'sociology', 'us_foreign_policy', 'virology', 'world_religions']

### Supported Tasks and Leaderboards

|                Model               | Authors |  Humanities |  Social Science  | STEM | Other | Average |
|------------------------------------|----------|:-------:|:-------:|:-------:|:-------:|:-------:|
| [UnifiedQA](https://arxiv.org/abs/2005.00700) | Khashabi et al., 2020 | 45.6 | 56.6 | 40.2 | 54.6 | 48.9
| [GPT-3](https://arxiv.org/abs/2005.14165) (few-shot) | Brown et al., 2020 | 40.8 | 50.4 | 36.7 | 48.8 | 43.9
| [GPT-2](https://arxiv.org/abs/2005.14165) | Radford et al., 2019 | 32.8 | 33.3 | 30.2 | 33.1 | 32.4
| Random Baseline           | N/A | 25.0 | 25.0 | 25.0 | 25.0 | 25.0 | 25.0

### Languages

English

## Dataset Structure

### Data Instances

An example from anatomy subtask looks as follows:
```
{
  "question": "What is the embryological origin of the hyoid bone?",
  "choices": ["The first pharyngeal arch", "The first and second pharyngeal arches", "The second pharyngeal arch", "The second and third pharyngeal arches"],
  "answer": "D"
}
```

### Data Fields

- `question`: a string feature
- `choices`: a list of 4 string features
- `answer`: a ClassLabel feature

### Data Splits

- `auxiliary_train`: auxiliary multiple-choice training questions from ARC, MC_TEST, OBQA, RACE, etc.
- `dev`: 5 examples per subtask, meant for few-shot setting
- `test`: there are at least 100 examples per subtask

|       | auxiliary_train   | dev | val | test |
| ----- | :------: | :-----: | :-----: | :-----: |
| TOTAL | 99842 | 285 | 1531 | 14042

## Dataset Creation

### Curation Rationale

Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn.

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[MIT License](https://github.com/hendrycks/test/blob/master/LICENSE)

### Citation Information

If you find this useful in your research, please consider citing the test and also the [ETHICS](https://arxiv.org/abs/2008.02275) dataset it draws from:
```
    @article{hendryckstest2021,
      title={Measuring Massive Multitask Language Understanding},
      author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
      journal={Proceedings of the International Conference on Learning Representations (ICLR)},
      year={2021}
    }

    @article{hendrycks2021ethics,
      title={Aligning AI With Shared Human Values},
      author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},
      journal={Proceedings of the International Conference on Learning Representations (ICLR)},
      year={2021}
    }
```
### Contributions

Thanks to [@andyzoujm](https://github.com/andyzoujm) for adding this dataset.

task_categories:question-answering task_ids:multiple-choice-qa annotations_creators:no-annotation language_creators:expert-generated multilinguality:monolingual size_categories:10K<n<100K source_datasets:original language:en license:mit arxiv:2009.03300 arxiv:2005.00700 arxiv:2005.14165 arxiv:2008.02275 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["expert-generated"], "language": ["en"], "license": ["mit"], "multilinguality": ["monolingual"], "size_categories": ["10K<n<100K"], "source_datasets": ["original"], "task_categories": ["question-answering"], "task_ids": ["multiple-choice-qa"], "paperswithcode_id": "mmlu", "pretty_name": "Measuring Massive Multitask Language Understanding", "language_bcp47": ["en-US"], "dataset_info": [{"config_name": "abstract_algebra", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 19328, "num_examples": 100}, {"name": "validation", "num_bytes": 2024, "num_examples": 11}, {"name": "dev", "num_bytes": 830, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160623559}, {"config_name": "anatomy", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 33121, "num_examples": 135}, {"name": "validation", "num_bytes": 3140, "num_examples": 14}, {"name": "dev", "num_bytes": 967, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160638605}, {"config_name": "astronomy", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 46771, "num_examples": 152}, {"name": "validation", "num_bytes": 5027, "num_examples": 16}, {"name": "dev", "num_bytes": 2076, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160655251}, {"config_name": "business_ethics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 33252, "num_examples": 100}, {"name": "validation", "num_bytes": 3038, "num_examples": 11}, {"name": "dev", "num_bytes": 2190, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160639857}, {"config_name": "clinical_knowledge", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 62754, "num_examples": 265}, {"name": "validation", "num_bytes": 6664, "num_examples": 29}, {"name": "dev", "num_bytes": 1210, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160672005}, {"config_name": "college_biology", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 48797, "num_examples": 144}, {"name": "validation", "num_bytes": 4819, "num_examples": 16}, {"name": "dev", "num_bytes": 1532, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160656525}, {"config_name": "college_chemistry", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 24708, "num_examples": 100}, {"name": "validation", "num_bytes": 2328, "num_examples": 8}, {"name": "dev", "num_bytes": 1331, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160629744}, {"config_name": "college_computer_science", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 42641, "num_examples": 100}, {"name": "validation", "num_bytes": 4663, "num_examples": 11}, {"name": "dev", "num_bytes": 2765, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160651446}, {"config_name": "college_mathematics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 24711, "num_examples": 100}, {"name": "validation", "num_bytes": 2668, "num_examples": 11}, {"name": "dev", "num_bytes": 1493, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160630249}, {"config_name": "college_medicine", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 82397, "num_examples": 173}, {"name": "validation", "num_bytes": 7909, "num_examples": 22}, {"name": "dev", "num_bytes": 1670, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160693353}, {"config_name": "college_physics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 30181, "num_examples": 102}, {"name": "validation", "num_bytes": 3490, "num_examples": 11}, {"name": "dev", "num_bytes": 1412, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160636460}, {"config_name": "computer_security", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 27124, "num_examples": 100}, {"name": "validation", "num_bytes": 4549, "num_examples": 11}, {"name": "dev", "num_bytes": 1101, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160634151}, {"config_name": "conceptual_physics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 40709, "num_examples": 235}, {"name": "validation", "num_bytes": 4474, "num_examples": 26}, {"name": "dev", "num_bytes": 934, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160647494}, {"config_name": "econometrics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 46547, "num_examples": 114}, {"name": "validation", "num_bytes": 4967, "num_examples": 12}, {"name": "dev", "num_bytes": 1644, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160654535}, {"config_name": "electrical_engineering", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 25142, "num_examples": 145}, {"name": "validation", "num_bytes": 2903, "num_examples": 16}, {"name": "dev", "num_bytes": 972, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160630394}, {"config_name": "elementary_mathematics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 70108, "num_examples": 378}, {"name": "validation", "num_bytes": 8988, "num_examples": 41}, {"name": "dev", "num_bytes": 1440, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160681913}, {"config_name": "formal_logic", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 49785, "num_examples": 126}, {"name": "validation", "num_bytes": 6252, "num_examples": 14}, {"name": "dev", "num_bytes": 1757, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160659171}, {"config_name": "global_facts", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 18403, "num_examples": 100}, {"name": "validation", "num_bytes": 1865, "num_examples": 10}, {"name": "dev", "num_bytes": 1229, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160622874}, {"config_name": "high_school_biology", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 109732, "num_examples": 310}, {"name": "validation", "num_bytes": 11022, "num_examples": 32}, {"name": "dev", "num_bytes": 1673, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160723804}, {"config_name": "high_school_chemistry", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 58464, "num_examples": 203}, {"name": "validation", "num_bytes": 7092, "num_examples": 22}, {"name": "dev", "num_bytes": 1220, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160668153}, {"config_name": "high_school_computer_science", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 44476, "num_examples": 100}, {"name": "validation", "num_bytes": 3343, "num_examples": 9}, {"name": "dev", "num_bytes": 2918, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160652114}, {"config_name": "high_school_european_history", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 270300, "num_examples": 165}, {"name": "validation", "num_bytes": 29632, "num_examples": 18}, {"name": "dev", "num_bytes": 11564, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160912873}, {"config_name": "high_school_geography", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 42034, "num_examples": 198}, {"name": "validation", "num_bytes": 4332, "num_examples": 22}, {"name": "dev", "num_bytes": 1403, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160649146}, {"config_name": "high_school_government_and_politics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 66074, "num_examples": 193}, {"name": "validation", "num_bytes": 7063, "num_examples": 21}, {"name": "dev", "num_bytes": 1779, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160676293}, {"config_name": "high_school_macroeconomics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 117687, "num_examples": 390}, {"name": "validation", "num_bytes": 13020, "num_examples": 43}, {"name": "dev", "num_bytes": 1328, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160733412}, {"config_name": "high_school_mathematics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 54854, "num_examples": 270}, {"name": "validation", "num_bytes": 5765, "num_examples": 29}, {"name": "dev", "num_bytes": 1297, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160663293}, {"config_name": "high_school_microeconomics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 75703, "num_examples": 238}, {"name": "validation", "num_bytes": 7553, "num_examples": 26}, {"name": "dev", "num_bytes": 1298, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160685931}, {"config_name": "high_school_physics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 59538, "num_examples": 151}, {"name": "validation", "num_bytes": 6771, "num_examples": 17}, {"name": "dev", "num_bytes": 1489, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160669175}, {"config_name": "high_school_psychology", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 159407, "num_examples": 545}, {"name": "validation", "num_bytes": 17269, "num_examples": 60}, {"name": "dev", "num_bytes": 1905, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160779958}, {"config_name": "high_school_statistics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 110702, "num_examples": 216}, {"name": "validation", "num_bytes": 9997, "num_examples": 23}, {"name": "dev", "num_bytes": 2528, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160724604}, {"config_name": "high_school_us_history", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 296734, "num_examples": 204}, {"name": "validation", "num_bytes": 31706, "num_examples": 22}, {"name": "dev", "num_bytes": 8864, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160938681}, {"config_name": "high_school_world_history", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 378617, "num_examples": 237}, {"name": "validation", "num_bytes": 45501, "num_examples": 26}, {"name": "dev", "num_bytes": 4882, "num_examples": 5}], "download_size": 166184960, "dataset_size": 161030377}, {"config_name": "human_aging", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 46098, "num_examples": 223}, {"name": "validation", "num_bytes": 4707, "num_examples": 23}, {"name": "dev", "num_bytes": 1008, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160653190}, {"config_name": "human_sexuality", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 32110, "num_examples": 131}, {"name": "validation", "num_bytes": 2421, "num_examples": 12}, {"name": "dev", "num_bytes": 1077, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160636985}, {"config_name": "international_law", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 53531, "num_examples": 121}, {"name": "validation", "num_bytes": 6473, "num_examples": 13}, {"name": "dev", "num_bytes": 2418, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160663799}, {"config_name": "jurisprudence", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 33986, "num_examples": 108}, {"name": "validation", "num_bytes": 3729, "num_examples": 11}, {"name": "dev", "num_bytes": 1303, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160640395}, {"config_name": "logical_fallacies", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 50117, "num_examples": 163}, {"name": "validation", "num_bytes": 5103, "num_examples": 18}, {"name": "dev", "num_bytes": 1573, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160658170}, {"config_name": "machine_learning", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 33880, "num_examples": 112}, {"name": "validation", "num_bytes": 3232, "num_examples": 11}, {"name": "dev", "num_bytes": 2323, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160640812}, {"config_name": "management", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 20002, "num_examples": 103}, {"name": "validation", "num_bytes": 1820, "num_examples": 11}, {"name": "dev", "num_bytes": 898, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160624097}, {"config_name": "marketing", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 63025, "num_examples": 234}, {"name": "validation", "num_bytes": 7394, "num_examples": 25}, {"name": "dev", "num_bytes": 1481, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160673277}, {"config_name": "medical_genetics", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 20864, "num_examples": 100}, {"name": "validation", "num_bytes": 3005, "num_examples": 11}, {"name": "dev", "num_bytes": 1089, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160626335}, {"config_name": "miscellaneous", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 147704, "num_examples": 783}, {"name": "validation", "num_bytes": 14330, "num_examples": 86}, {"name": "dev", "num_bytes": 699, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160764110}, {"config_name": "moral_disputes", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 107818, "num_examples": 346}, {"name": "validation", "num_bytes": 12420, "num_examples": 38}, {"name": "dev", "num_bytes": 1755, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160723370}, {"config_name": "moral_scenarios", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 374026, "num_examples": 895}, {"name": "validation", "num_bytes": 42338, "num_examples": 100}, {"name": "dev", "num_bytes": 2058, "num_examples": 5}], "download_size": 166184960, "dataset_size": 161019799}, {"config_name": "nutrition", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 92410, "num_examples": 306}, {"name": "validation", "num_bytes": 8436, "num_examples": 33}, {"name": "dev", "num_bytes": 2085, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160704308}, {"config_name": "philosophy", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 80073, "num_examples": 311}, {"name": "validation", "num_bytes": 9184, "num_examples": 34}, {"name": "dev", "num_bytes": 988, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160691622}, {"config_name": "prehistory", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 89594, "num_examples": 324}, {"name": "validation", "num_bytes": 10285, "num_examples": 35}, {"name": "dev", "num_bytes": 1878, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160703134}, {"config_name": "professional_accounting", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 124550, "num_examples": 282}, {"name": "validation", "num_bytes": 14372, "num_examples": 31}, {"name": "dev", "num_bytes": 2148, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160742447}, {"config_name": "professional_law", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 1891762, "num_examples": 1534}, {"name": "validation", "num_bytes": 203519, "num_examples": 170}, {"name": "dev", "num_bytes": 6610, "num_examples": 5}], "download_size": 166184960, "dataset_size": 162703268}, {"config_name": "professional_medicine", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 217561, "num_examples": 272}, {"name": "validation", "num_bytes": 23847, "num_examples": 31}, {"name": "dev", "num_bytes": 3807, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160846592}, {"config_name": "professional_psychology", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 225899, "num_examples": 612}, {"name": "validation", "num_bytes": 29101, "num_examples": 69}, {"name": "dev", "num_bytes": 2267, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160858644}, {"config_name": "public_relations", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 28760, "num_examples": 110}, {"name": "validation", "num_bytes": 4566, "num_examples": 12}, {"name": "dev", "num_bytes": 1496, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160636199}, {"config_name": "security_studies", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 204844, "num_examples": 245}, {"name": "validation", "num_bytes": 22637, "num_examples": 27}, {"name": "dev", "num_bytes": 5335, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160834193}, {"config_name": "sociology", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 66243, "num_examples": 201}, {"name": "validation", "num_bytes": 7184, "num_examples": 22}, {"name": "dev", "num_bytes": 1613, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160676417}, {"config_name": "us_foreign_policy", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 28443, "num_examples": 100}, {"name": "validation", "num_bytes": 3264, "num_examples": 11}, {"name": "dev", "num_bytes": 1611, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160634695}, {"config_name": "virology", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 38759, "num_examples": 166}, {"name": "validation", "num_bytes": 5463, "num_examples": 18}, {"name": "dev", "num_bytes": 1096, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160646695}, {"config_name": "world_religions", "features": [{"name": "question", "dtype": "string"}, {"name": "choices", "sequence": "string"}, {"name": "answer", "dtype": {"class_label": {"names": {"0": "A", "1": "B", "2": "C", "3": "D"}}}}], "splits": [{"name": "auxiliary_train", "num_bytes": 160601377, "num_examples": 99842}, {"name": "test", "num_bytes": 25274, "num_examples": 171}, {"name": "validation", "num_bytes": 2765, "num_examples": 19}, {"name": "dev", "num_bytes": 670, "num_examples": 5}], "download_size": 166184960, "dataset_size": 160630086}]}
TAGS
#task_categories-question-answering #task_ids-multiple-choice-qa #annotations_creators-no-annotation #language_creators-expert-generated #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-original #language-English #license-mit #arxiv-2009.03300 #arxiv-2005.00700 #arxiv-2005.14165 #arxiv-2008.02275 #region-us 

Dataset Card for MMLU
=====================


Table of Contents
-----------------


* Table of Contents
* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Repository: URL
* Paper: URL


### Dataset Summary


Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).


This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.


A complete list of tasks: ['abstract\_algebra', 'anatomy', 'astronomy', 'business\_ethics', 'clinical\_knowledge', 'college\_biology', 'college\_chemistry', 'college\_computer\_science', 'college\_mathematics', 'college\_medicine', 'college\_physics', 'computer\_security', 'conceptual\_physics', 'econometrics', 'electrical\_engineering', 'elementary\_mathematics', 'formal\_logic', 'global\_facts', 'high\_school\_biology', 'high\_school\_chemistry', 'high\_school\_computer\_science', 'high\_school\_european\_history', 'high\_school\_geography', 'high\_school\_government\_and\_politics', 'high\_school\_macroeconomics', 'high\_school\_mathematics', 'high\_school\_microeconomics', 'high\_school\_physics', 'high\_school\_psychology', 'high\_school\_statistics', 'high\_school\_us\_history', 'high\_school\_world\_history', 'human\_aging', 'human\_sexuality', 'international\_law', 'jurisprudence', 'logical\_fallacies', 'machine\_learning', 'management', 'marketing', 'medical\_genetics', 'miscellaneous', 'moral\_disputes', 'moral\_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional\_accounting', 'professional\_law', 'professional\_medicine', 'professional\_psychology', 'public\_relations', 'security\_studies', 'sociology', 'us\_foreign\_policy', 'virology', 'world\_religions']


### Supported Tasks and Leaderboards



### Languages


English


Dataset Structure
-----------------


### Data Instances


An example from anatomy subtask looks as follows:


### Data Fields


* 'question': a string feature
* 'choices': a list of 4 string features
* 'answer': a ClassLabel feature


### Data Splits


* 'auxiliary\_train': auxiliary multiple-choice training questions from ARC, MC\_TEST, OBQA, RACE, etc.
* 'dev': 5 examples per subtask, meant for few-shot setting
* 'test': there are at least 100 examples per subtask



Dataset Creation
----------------


### Curation Rationale


Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn.


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


MIT License


If you find this useful in your research, please consider citing the test and also the ETHICS dataset it draws from:


### Contributions


Thanks to @andyzoujm for adding this dataset.



### Dataset Summary


Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).


This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.


A complete list of tasks: ['abstract\_algebra', 'anatomy', 'astronomy', 'business\_ethics', 'clinical\_knowledge', 'college\_biology', 'college\_chemistry', 'college\_computer\_science', 'college\_mathematics', 'college\_medicine', 'college\_physics', 'computer\_security', 'conceptual\_physics', 'econometrics', 'electrical\_engineering', 'elementary\_mathematics', 'formal\_logic', 'global\_facts', 'high\_school\_biology', 'high\_school\_chemistry', 'high\_school\_computer\_science', 'high\_school\_european\_history', 'high\_school\_geography', 'high\_school\_government\_and\_politics', 'high\_school\_macroeconomics', 'high\_school\_mathematics', 'high\_school\_microeconomics', 'high\_school\_physics', 'high\_school\_psychology', 'high\_school\_statistics', 'high\_school\_us\_history', 'high\_school\_world\_history', 'human\_aging', 'human\_sexuality', 'international\_law', 'jurisprudence', 'logical\_fallacies', 'machine\_learning', 'management', 'marketing', 'medical\_genetics', 'miscellaneous', 'moral\_disputes', 'moral\_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional\_accounting', 'professional\_law', 'professional\_medicine', 'professional\_psychology', 'public\_relations', 'security\_studies', 'sociology', 'us\_foreign\_policy', 'virology', 'world\_religions'] ### Supported Tasks and Leaderboards ### Languages


English


Dataset Structure
----------------- ### Data Instances


An example from anatomy subtask looks as follows: ### Data Fields


* 'question': a string feature
* 'choices': a list of 4 string features
* 'answer': a ClassLabel feature ### Data Splits


* 'auxiliary\_train': auxiliary multiple-choice training questions from ARC, MC\_TEST, OBQA, RACE, etc.
* 'dev': 5 examples per subtask, meant for few-shot setting
* 'test': there are at least 100 examples per subtask



Dataset Creation
---------------- ### Curation Rationale


Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn. ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


MIT License


If you find this useful in your research, please consider citing the test and also the ETHICS dataset it draws from: ### Contributions


Thanks to @andyzoujm for adding this dataset.
TAGS
#task_categories-question-answering #task_ids-multiple-choice-qa #annotations_creators-no-annotation #language_creators-expert-generated #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-original #language-English #license-mit #arxiv-2009.03300 #arxiv-2005.00700 #arxiv-2005.14165 #arxiv-2008.02275 #region-us 
 ### Dataset Summary


Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).


This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.


A complete list of tasks: ['abstract\_algebra', 'anatomy', 'astronomy', 'business\_ethics', 'clinical\_knowledge', 'college\_biology', 'college\_chemistry', 'college\_computer\_science', 'college\_mathematics', 'college\_medicine', 'college\_physics', 'computer\_security', 'conceptual\_physics', 'econometrics', 'electrical\_engineering', 'elementary\_mathematics', 'formal\_logic', 'global\_facts', 'high\_school\_biology', 'high\_school\_chemistry', 'high\_school\_computer\_science', 'high\_school\_european\_history', 'high\_school\_geography', 'high\_school\_government\_and\_politics', 'high\_school\_macroeconomics', 'high\_school\_mathematics', 'high\_school\_microeconomics', 'high\_school\_physics', 'high\_school\_psychology', 'high\_school\_statistics', 'high\_school\_us\_history', 'high\_school\_world\_history', 'human\_aging', 'human\_sexuality', 'international\_law', 'jurisprudence', 'logical\_fallacies', 'machine\_learning', 'management', 'marketing', 'medical\_genetics', 'miscellaneous', 'moral\_disputes', 'moral\_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional\_accounting', 'professional\_law', 'professional\_medicine', 'professional\_psychology', 'public\_relations', 'security\_studies', 'sociology', 'us\_foreign\_policy', 'virology', 'world\_religions'] ### Supported Tasks and Leaderboards ### Languages


English


Dataset Structure
----------------- ### Data Instances


An example from anatomy subtask looks as follows: ### Data Fields


* 'question': a string feature
* 'choices': a list of 4 string features
* 'answer': a ClassLabel feature ### Data Splits


* 'auxiliary\_train': auxiliary multiple-choice training questions from ARC, MC\_TEST, OBQA, RACE, etc.
* 'dev': 5 examples per subtask, meant for few-shot setting
* 'test': there are at least 100 examples per subtask



Dataset Creation
---------------- ### Curation Rationale


Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn. ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


MIT License


If you find this useful in your research, please consider citing the test and also the ETHICS dataset it draws from: ### Contributions


Thanks to @andyzoujm for adding this dataset.