
# Dataset Card for Mostly Basic Python Problems (mbpp)

## Table of Contents
- [Dataset Card for Mostly Basic Python Problems (mbpp)](#dataset-card-for-mostly-basic-python-problems-(mbpp))
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
      - [Who are the source language producers?](#who-are-the-source-language-producers)
    - [Annotations](#annotations)
      - [Annotation process](#annotation-process)
      - [Who are the annotators?](#who-are-the-annotators)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description
- **Repository:** https://github.com/google-research/google-research/tree/master/mbpp
- **Paper:** [Program Synthesis with Large Language Models](https://arxiv.org/abs/2108.07732)

### Dataset Summary
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. 

Released [here](https://github.com/google-research/google-research/tree/master/mbpp) as part of [Program Synthesis with Large Language Models, Austin et. al., 2021](https://arxiv.org/abs/2108.07732).

### Supported Tasks and Leaderboards
This dataset is used to evaluate code generations.

### Languages
English - Python code

## Dataset Structure

```python
dataset_full = load_dataset("mbpp")
DatasetDict({
    test: Dataset({
        features: ['task_id', 'text', 'code', 'test_list', 'test_setup_code', 'challenge_test_list'],
        num_rows: 974
    })
})

dataset_sanitized = load_dataset("mbpp", "sanitized")
DatasetDict({
    test: Dataset({
        features: ['source_file', 'task_id', 'prompt', 'code', 'test_imports', 'test_list'],
        num_rows: 427
    })
})
```

### Data Instances

#### mbpp - full
```
{
    'task_id': 1,
    'text': 'Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix cost[][] and a position (m, n) in cost[][].',
    'code': 'R = 3\r\nC = 3\r\ndef min_cost(cost, m, n): \r\n\ttc = [[0 for x in range(C)] for x in range(R)] \r\n\ttc[0][0] = cost[0][0] \r\n\tfor i in range(1, m+1): \r\n\t\ttc[i][0] = tc[i-1][0] + cost[i][0] \r\n\tfor j in range(1, n+1): \r\n\t\ttc[0][j] = tc[0][j-1] + cost[0][j] \r\n\tfor i in range(1, m+1): \r\n\t\tfor j in range(1, n+1): \r\n\t\t\ttc[i][j] = min(tc[i-1][j-1], tc[i-1][j], tc[i][j-1]) + cost[i][j] \r\n\treturn tc[m][n]',
    'test_list': [
        'assert min_cost([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2) == 8',
        'assert min_cost([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2) == 12',
        'assert min_cost([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2) == 16'],
    'test_setup_code': '',
    'challenge_test_list': []
}
```
#### mbpp - sanitized
```
{
    'source_file': 'Benchmark Questions Verification V2.ipynb',
    'task_id': 2,
    'prompt': 'Write a function to find the shared elements from the given two lists.',
    'code': 'def similar_elements(test_tup1, test_tup2):\n  res = tuple(set(test_tup1) & set(test_tup2))\n  return (res) ',
    'test_imports': [],
    'test_list': [
        'assert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))',
        'assert set(similar_elements((1, 2, 3, 4),(5, 4, 3, 7))) == set((3, 4))',
        'assert set(similar_elements((11, 12, 14, 13),(17, 15, 14, 13))) == set((13, 14))'
        ]
}
```
### Data Fields

- `source_file`: unknown
- `text`/`prompt`: description of programming task
- `code`: solution for programming task
- `test_setup_code`/`test_imports`: necessary code imports to execute tests
- `test_list`: list of tests to verify solution
- `challenge_test_list`: list of more challenging test to further probe solution

### Data Splits
There are two version of the dataset (full and sanitized), each with four splits:
- train
- evaluation
- test
- prompt

The `prompt` split corresponds to samples used for few-shot prompting and not for training.

## Dataset Creation
See section 2.1 of original [paper](https://arxiv.org/abs/2108.07732).

### Curation Rationale
In order to evaluate code generation functions a set of simple programming tasks as well as solutions is necessary which this dataset provides.

### Source Data

#### Initial Data Collection and Normalization
The dataset was manually created from scratch.

#### Who are the source language producers?
The dataset was created with an internal crowdsourcing effort at Google.

### Annotations

#### Annotation process
The full dataset was created first and a subset then underwent a second round to improve the task descriptions.

#### Who are the annotators?
The dataset was created with an internal crowdsourcing effort at Google.

### Personal and Sensitive Information
None.

## Considerations for Using the Data
Make sure you execute generated Python code in a safe environment when evauating against this dataset as generated code could be harmful.

### Social Impact of Dataset
With this dataset code generating models can be better evaluated which leads to fewer issues introduced when using such models.

### Discussion of Biases

### Other Known Limitations
Since the task descriptions might not be expressive enough to solve the task. The `sanitized` split aims at addressing this issue by having a second round of annotators improve the dataset.

## Additional Information

### Dataset Curators
Google Research

### Licensing Information
CC-BY-4.0

### Citation Information
```
@article{austin2021program,
  title={Program Synthesis with Large Language Models},
  author={Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others},
  journal={arXiv preprint arXiv:2108.07732},
  year={2021}
```
### Contributions
Thanks to [@lvwerra](https://github.com/lvwerra) for adding this dataset.
task_categories:text2text-generation annotations_creators:crowdsourced annotations_creators:expert-generated language_creators:crowdsourced language_creators:expert-generated multilinguality:monolingual size_categories:n<1K source_datasets:original language:en license:cc-by-4.0 code-generation arxiv:2108.07732 region:us
{"annotations_creators": ["crowdsourced", "expert-generated"], "language_creators": ["crowdsourced", "expert-generated"], "language": ["en"], "license": ["cc-by-4.0"], "multilinguality": ["monolingual"], "size_categories": ["n<1K"], "source_datasets": ["original"], "task_categories": ["text2text-generation"], "task_ids": [], "pretty_name": "Mostly Basic Python Problems", "tags": ["code-generation"], "dataset_info": [{"config_name": "full", "features": [{"name": "task_id", "dtype": "int32"}, {"name": "text", "dtype": "string"}, {"name": "code", "dtype": "string"}, {"name": "test_list", "sequence": "string"}, {"name": "test_setup_code", "dtype": "string"}, {"name": "challenge_test_list", "sequence": "string"}], "splits": [{"name": "train", "num_bytes": 176879, "num_examples": 374}, {"name": "test", "num_bytes": 244104, "num_examples": 500}, {"name": "validation", "num_bytes": 42405, "num_examples": 90}, {"name": "prompt", "num_bytes": 4550, "num_examples": 10}], "download_size": 236069, "dataset_size": 467938}, {"config_name": "sanitized", "features": [{"name": "source_file", "dtype": "string"}, {"name": "task_id", "dtype": "int32"}, {"name": "prompt", "dtype": "string"}, {"name": "code", "dtype": "string"}, {"name": "test_imports", "sequence": "string"}, {"name": "test_list", "sequence": "string"}], "splits": [{"name": "train", "num_bytes": 63453, "num_examples": 120}, {"name": "test", "num_bytes": 132720, "num_examples": 257}, {"name": "validation", "num_bytes": 20050, "num_examples": 43}, {"name": "prompt", "num_bytes": 3407, "num_examples": 7}], "download_size": 115422, "dataset_size": 219630}], "configs": [{"config_name": "full", "data_files": [{"split": "train", "path": "full/train-*"}, {"split": "test", "path": "full/test-*"}, {"split": "validation", "path": "full/validation-*"}, {"split": "prompt", "path": "full/prompt-*"}], "default": true}, {"config_name": "sanitized", "data_files": [{"split": "train", "path": "sanitized/train-*"}, {"split": "test", "path": "sanitized/test-*"}, {"split": "validation", "path": "sanitized/validation-*"}, {"split": "prompt", "path": "sanitized/prompt-*"}]}]}
TAGS
#task_categories-text2text-generation #annotations_creators-crowdsourced #annotations_creators-expert-generated #language_creators-crowdsourced #language_creators-expert-generated #multilinguality-monolingual #size_categories-n<1K #source_datasets-original #language-English #license-cc-by-4.0 #code-generation #arxiv-2108.07732 #region-us 


# Dataset Card for Mostly Basic Python Problems (mbpp)

## Table of Contents
- Dataset Card for Mostly Basic Python Problems (mbpp))
  - Table of Contents
  - Dataset Description
    - Dataset Summary
    - Supported Tasks and Leaderboards
    - Languages
  - Dataset Structure
    - Data Instances
    - Data Fields
    - Data Splits
  - Dataset Creation
    - Curation Rationale
    - Source Data
      - Initial Data Collection and Normalization
      - Who are the source language producers?
    - Annotations
      - Annotation process
      - Who are the annotators?
    - Personal and Sensitive Information
  - Considerations for Using the Data
    - Social Impact of Dataset
    - Discussion of Biases
    - Other Known Limitations
  - Additional Information
    - Dataset Curators
    - Licensing Information
    - Citation Information
    - Contributions

## Dataset Description
- Repository: URL
- Paper: Program Synthesis with Large Language Models

### Dataset Summary
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. 

Released here as part of Program Synthesis with Large Language Models, Austin et. al., 2021.

### Supported Tasks and Leaderboards
This dataset is used to evaluate code generations.

### Languages
English - Python code

## Dataset Structure



### Data Instances

#### mbpp - full

#### mbpp - sanitized

### Data Fields

- 'source_file': unknown
- 'text'/'prompt': description of programming task
- 'code': solution for programming task
- 'test_setup_code'/'test_imports': necessary code imports to execute tests
- 'test_list': list of tests to verify solution
- 'challenge_test_list': list of more challenging test to further probe solution

### Data Splits
There are two version of the dataset (full and sanitized), each with four splits:
- train
- evaluation
- test
- prompt

The 'prompt' split corresponds to samples used for few-shot prompting and not for training.

## Dataset Creation
See section 2.1 of original paper.

### Curation Rationale
In order to evaluate code generation functions a set of simple programming tasks as well as solutions is necessary which this dataset provides.

### Source Data

#### Initial Data Collection and Normalization
The dataset was manually created from scratch.

#### Who are the source language producers?
The dataset was created with an internal crowdsourcing effort at Google.

### Annotations

#### Annotation process
The full dataset was created first and a subset then underwent a second round to improve the task descriptions.

#### Who are the annotators?
The dataset was created with an internal crowdsourcing effort at Google.

### Personal and Sensitive Information
None.

## Considerations for Using the Data
Make sure you execute generated Python code in a safe environment when evauating against this dataset as generated code could be harmful.

### Social Impact of Dataset
With this dataset code generating models can be better evaluated which leads to fewer issues introduced when using such models.

### Discussion of Biases

### Other Known Limitations
Since the task descriptions might not be expressive enough to solve the task. The 'sanitized' split aims at addressing this issue by having a second round of annotators improve the dataset.

## Additional Information

### Dataset Curators
Google Research

### Licensing Information
CC-BY-4.0



### Contributions
Thanks to @lvwerra for adding this dataset.
# Dataset Card for Mostly Basic Python Problems (mbpp) ## Table of Contents
- Dataset Card for Mostly Basic Python Problems (mbpp))
  - Table of Contents
  - Dataset Description
    - Dataset Summary
    - Supported Tasks and Leaderboards
    - Languages
  - Dataset Structure
    - Data Instances
    - Data Fields
    - Data Splits
  - Dataset Creation
    - Curation Rationale
    - Source Data
      - Initial Data Collection and Normalization
      - Who are the source language producers?
    - Annotations
      - Annotation process
      - Who are the annotators?
    - Personal and Sensitive Information
  - Considerations for Using the Data
    - Social Impact of Dataset
    - Discussion of Biases
    - Other Known Limitations
  - Additional Information
    - Dataset Curators
    - Licensing Information
    - Citation Information
    - Contributions ## Dataset Description
- Repository: URL
- Paper: Program Synthesis with Large Language Models ### Dataset Summary
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. 

Released here as part of Program Synthesis with Large Language Models, Austin et. al., 2021. ### Supported Tasks and Leaderboards
This dataset is used to evaluate code generations. ### Languages
English - Python code ## Dataset Structure ### Data Instances #### mbpp - full #### mbpp - sanitized ### Data Fields

- 'source_file': unknown
- 'text'/'prompt': description of programming task
- 'code': solution for programming task
- 'test_setup_code'/'test_imports': necessary code imports to execute tests
- 'test_list': list of tests to verify solution
- 'challenge_test_list': list of more challenging test to further probe solution ### Data Splits
There are two version of the dataset (full and sanitized), each with four splits:
- train
- evaluation
- test
- prompt

The 'prompt' split corresponds to samples used for few-shot prompting and not for training. ## Dataset Creation
See section 2.1 of original paper. ### Curation Rationale
In order to evaluate code generation functions a set of simple programming tasks as well as solutions is necessary which this dataset provides. ### Source Data #### Initial Data Collection and Normalization
The dataset was manually created from scratch. #### Who are the source language producers?
The dataset was created with an internal crowdsourcing effort at Google. ### Annotations #### Annotation process
The full dataset was created first and a subset then underwent a second round to improve the task descriptions. #### Who are the annotators?
The dataset was created with an internal crowdsourcing effort at Google. ### Personal and Sensitive Information
None. ## Considerations for Using the Data
Make sure you execute generated Python code in a safe environment when evauating against this dataset as generated code could be harmful. ### Social Impact of Dataset
With this dataset code generating models can be better evaluated which leads to fewer issues introduced when using such models. ### Discussion of Biases ### Other Known Limitations
Since the task descriptions might not be expressive enough to solve the task. The 'sanitized' split aims at addressing this issue by having a second round of annotators improve the dataset. ## Additional Information ### Dataset Curators
Google Research ### Licensing Information
CC-BY-4.0 ### Contributions
Thanks to @lvwerra for adding this dataset.
TAGS
#task_categories-text2text-generation #annotations_creators-crowdsourced #annotations_creators-expert-generated #language_creators-crowdsourced #language_creators-expert-generated #multilinguality-monolingual #size_categories-n<1K #source_datasets-original #language-English #license-cc-by-4.0 #code-generation #arxiv-2108.07732 #region-us 
 # Dataset Card for Mostly Basic Python Problems (mbpp) ## Table of Contents
- Dataset Card for Mostly Basic Python Problems (mbpp))
  - Table of Contents
  - Dataset Description
    - Dataset Summary
    - Supported Tasks and Leaderboards
    - Languages
  - Dataset Structure
    - Data Instances
    - Data Fields
    - Data Splits
  - Dataset Creation
    - Curation Rationale
    - Source Data
      - Initial Data Collection and Normalization
      - Who are the source language producers?
    - Annotations
      - Annotation process
      - Who are the annotators?
    - Personal and Sensitive Information
  - Considerations for Using the Data
    - Social Impact of Dataset
    - Discussion of Biases
    - Other Known Limitations
  - Additional Information
    - Dataset Curators
    - Licensing Information
    - Citation Information
    - Contributions ## Dataset Description
- Repository: URL
- Paper: Program Synthesis with Large Language Models ### Dataset Summary
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. 

Released here as part of Program Synthesis with Large Language Models, Austin et. al., 2021. ### Supported Tasks and Leaderboards
This dataset is used to evaluate code generations. ### Languages
English - Python code ## Dataset Structure ### Data Instances #### mbpp - full #### mbpp - sanitized ### Data Fields

- 'source_file': unknown
- 'text'/'prompt': description of programming task
- 'code': solution for programming task
- 'test_setup_code'/'test_imports': necessary code imports to execute tests
- 'test_list': list of tests to verify solution
- 'challenge_test_list': list of more challenging test to further probe solution ### Data Splits
There are two version of the dataset (full and sanitized), each with four splits:
- train
- evaluation
- test
- prompt

The 'prompt' split corresponds to samples used for few-shot prompting and not for training. ## Dataset Creation
See section 2.1 of original paper. ### Curation Rationale
In order to evaluate code generation functions a set of simple programming tasks as well as solutions is necessary which this dataset provides. ### Source Data #### Initial Data Collection and Normalization
The dataset was manually created from scratch. #### Who are the source language producers?
The dataset was created with an internal crowdsourcing effort at Google. ### Annotations #### Annotation process
The full dataset was created first and a subset then underwent a second round to improve the task descriptions. #### Who are the annotators?
The dataset was created with an internal crowdsourcing effort at Google. ### Personal and Sensitive Information
None. ## Considerations for Using the Data
Make sure you execute generated Python code in a safe environment when evauating against this dataset as generated code could be harmful. ### Social Impact of Dataset
With this dataset code generating models can be better evaluated which leads to fewer issues introduced when using such models. ### Discussion of Biases ### Other Known Limitations
Since the task descriptions might not be expressive enough to solve the task. The 'sanitized' split aims at addressing this issue by having a second round of annotators improve the dataset. ## Additional Information ### Dataset Curators
Google Research ### Licensing Information
CC-BY-4.0 ### Contributions
Thanks to @lvwerra for adding this dataset.