arxiv:2310.06770 region:us
---
dataset_info:
  features:
  - name: instance_id
    dtype: string
  - name: base_commit
    dtype: string
  - name: hints_text
    dtype: string
  - name: created_at
    dtype: string
  - name: test_patch
    dtype: string
  - name: repo
    dtype: string
  - name: problem_statement
    dtype: string
  - name: version
    dtype: string
  - name: FAIL_TO_PASS
    dtype: string
  - name: PASS_TO_PASS
    dtype: string
  - name: environment_setup_commit
    dtype: string
  - name: patch
    dtype: string
  splits:
  - name: train
    num_bytes: 399498956
    num_examples: 19008
  - name: test
    num_bytes: 41860075
    num_examples: 2294
  download_size: 125366079
  dataset_size: 441359031
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: test
    path: data/test-*
---


### Dataset Summary
SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.

The dataset was released as part of [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770)

### Supported Tasks and Leaderboards
SWE-bench proposes a new task: issue resolution provided a full repository and GitHub issue. The leaderboard can be found at www.swebench.com

### Languages

The text of the dataset is primarily English, but we make no effort to filter or otherwise clean based on language type.

## Dataset Structure

### Data Instances
An example of a SWE-bench datum is as follows:

```
instance_id: (str) - A formatted instance identifier, usually as repo_owner__repo_name-PR-number.
patch: (str) - The gold patch, the patch generated by the PR (minus test-related code), that resolved the issue.
repo: (str) - The repository owner/name identifier from GitHub.
base_commit: (str) - The commit hash of the repository representing the HEAD of the repository before the solution PR is applied.
hints_text: (str) - Comments made on the issue prior to the creation of the solution PR’s first commit creation date.
created_at: (str) - The creation date of the pull request.
test_patch: (str) - A test-file patch that was contributed by the solution PR.
Problem_statement: (str) - The issue title and body.
Version: (str) - Installation version to use for running evaluation.
environment_setup_commit: (str) - commit hash to use for environment setup and installation.
FAIL_TO_PASS: (str) - A json list of strings that represent the set of tests resolved by the PR and tied to the issue resolution.
PASS_TO_PASS: (str) - A json list of strings that represent tests that should pass before and after the PR application.
```

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)