
# Dataset Card for StaQC (A Systematically Mined Question-Code Dataset from Stack Overflow)


## Dataset Description

- **Homepage: [GitHub](https://github.com/LittleYUYU/StackOverflow-Question-Code-Dataset)** 
- **Paper: [StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow](https://arxiv.org/abs/1803.09371)** 

### Dataset Summary

StaQC (Stack Overflow Question-Code pairs) is a large dataset of around 148K Python and 120K SQL domain question-code pairs, 
which are automatically mined from Stack Overflow using a Bi-View Hierarchical Neural Network. StaQC is collected from three sources: multi-code answer posts, single-code answer posts, and manual annotations on multi-code answer posts. 

The dataset was originally released by the main authors on [GitHub](https://github.com/LittleYUYU/StackOverflow-Question-Code-Dataset). This version is a *non-modified* redistributed copy (under the [license](#licensing-information) permission) made available on the hub for easier access. 

#### Standalone solutions

As noted in the paper, the authors *define a code snippet as a code solution when the
questioner can solve the problem solely based on it (also named as
“standalone” solution).* 

#### Manual annotations

The manual annotations are the collection of multi-code answer posts for which each code snippet was annotated with a boolean indicating whether or not the snippet is a *standalone solution* to the question.


#### Multi-code answer posts
A *Multi-code answer post* is an (accepted) answer post that contains multiple code snippets, 
some of which may not be a *standalone* code solution to the question (see Section 1 in [paper](http://web.cse.ohio-state.edu/~sun.397/docs/StaQC-www18.pdf)). 
For example, in [this multi-code answer post](https://stackoverflow.com/a/5996949), 
the third code snippet is not a code solution to the question "How to limit a number to be within a specified range? (Python)".

Note: the multi-code answer posts contain also the manual annotations. 

#### Single-code answer posts
A *Single-code answer post* is an (accepted) answer post that contains only one code snippet. 
We pair such code snippets with the question title as a question-code pair.



### Supported Tasks and Leaderboards

This dataset can be used for Natural Language to Code Generation tasks. 

### Languages

Python, SQL, English

## Dataset Structure

### Data Instances

Each configuration correspond to one of the three parts, in a given programming language.

There are three parts for the dataset:

- mca (Multi-code answer posts)
- sca (Single-code answer posts)
- man (Manual annotations)

And two programming/query languages:

- python
- sql

One can obtain obtain a configuration as a combination of a part in a programing language. For instance, one can obtain the automatically mined multi-code answers in python using:

```python
dataset = load_dataset("koutch/staqc", 'mca_python')
DatasetDict({
    train: Dataset({
        features: ['id', 'question_id', 'question', 'snippet'],
        num_rows: 40391
    })
})
```
or the manual annotations using:

```python
dataset = load_dataset("koutch/staqc", 'man_sql')
DatasetDict({
    train: Dataset({
        features: ['id', 'question_id', 'question', 'snippet'],
        num_rows: 1587
    })
})
```


#### Manual annotations

The manual annotations contain, for a given stackoverflow questions, for each individual code block in the accepted answer of that post, information on whether or not the given code block is a *standalone* solution to the question asked (the question title).

```
{
  'question_id': 5947137,
  'question': 'How can I use a list comprehension to extend a list in python?',
  'snippet': {'text': ['import itertools as it\n\nreturn sum(it.imap(doSomething, originalList), [])\n',
   'return sum(map(doSomething, originalList), [])\n',
   'return sum((doSomething(x) for x in originalList), [])\n',
   'accumulationList = []\nfor x in originalList:\n    accumulationList.extend(doSomething(x))\nreturn accumulationList\n'],
  'is_sda': [True, True, True, True]}
}
```

#### Multi-code answer posts


```
{
  'question_id': 35349290,
  'question': 'Python: Generating YYMM string between two dates',
  'snippet': ['start_year = 2005\nend_year = 2007\nstart_month = 3\nend_month = 2\nyymm = [(yy, mm) for yy in range(start_year, end_year + 1) for mm in range(1, 13)\n        if (start_year, start_month) <= (yy, mm) <= (end_year, end_month)]\n',
  "formatted_yymm = ['{:>02}{:>02}.mat'.format(yy % 100, mm) for yy, mm in yymm]\n"]
  }
```

#### Single-code answer posts

```
{
 'question_id': 19387200,
 'question': 'Python: get OS language',
 'snippet': "import locale\nloc = locale.getlocale() # get current locale\nlocale.getdefaultlocale() # Tries to determine the default locale settings and returns them as a tuple of the form (language code, encoding); e.g, ('en_US', 'UTF-8').\n"
}
```

### Data Fields

- `question_id`: id of the stackoverflow question 
- `question`: title of the stackoverflow question repurposed as the natural language intent
- `snippet`: mined or annotated standalone solution(s) (potentially) answerring the question
- `is_sda`: for the manual annotations, whether or not the given code snippet is a standalone solution to the question. 

### Data Splits

Each configuration of the dataset contains only a training split. 

## Dataset Creation

### Source Data

StackOverflow data dump.

### Annotations

See section 2.3 "Annotating QC Pairs for Model Training" of the [paper](https://arxiv.org/abs/1803.09371)

## Additional Information

### Licensing Information

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.

### Citation Information

If you use the dataset or the code in your research, please cite the following paper:

```
@inproceedings{yao2018staqc,
  title={StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow},
  author={Yao, Ziyu and Weld, Daniel S and Chen, Wei-Peng and Sun, Huan},
  booktitle={Proceedings of the 2018 World Wide Web Conference on World Wide Web},
  pages={1693--1703},
  year={2018},
  organization={International World Wide Web Conferences Steering Committee}
}
```

### Contributions information

I did *not* contribute to the *creation* of this dataset, only to the redistribution. All credits should be attributed to the original authors.
task_categories:question-answering size_categories:10K<n<100K language:en license:cc-by-4.0 code arxiv:1803.09371 region:us
{"language": ["en"], "license": "cc-by-4.0", "size_categories": ["10K<n<100K"], "task_categories": ["question-answering"], "pretty_name": "staqc", "dataset_info": [{"config_name": "mca_python", "features": [{"name": "id", "dtype": "int32"}, {"name": "question_id", "dtype": "int32"}, {"name": "question", "dtype": "string"}, {"name": "snippet", "sequence": "string"}], "splits": [{"name": "train", "num_bytes": 23286786, "num_examples": 40391}], "download_size": 72054260, "dataset_size": 23286786}, {"config_name": "mca_sql", "features": [{"name": "id", "dtype": "int32"}, {"name": "question_id", "dtype": "int32"}, {"name": "question", "dtype": "string"}, {"name": "snippet", "sequence": "string"}], "splits": [{"name": "train", "num_bytes": 15164206, "num_examples": 26052}], "download_size": 50304531, "dataset_size": 15164206}, {"config_name": "sca_python", "features": [{"name": "id", "dtype": "int32"}, {"name": "question_id", "dtype": "int32"}, {"name": "question", "dtype": "string"}, {"name": "snippet", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 39678168, "num_examples": 85294}], "download_size": 47378850, "dataset_size": 39678168}, {"config_name": "sca_sql", "features": [{"name": "id", "dtype": "int32"}, {"name": "question_id", "dtype": "int32"}, {"name": "question", "dtype": "string"}, {"name": "snippet", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 28656467, "num_examples": 75637}], "download_size": 34194025, "dataset_size": 28656467}, {"config_name": "man_python", "features": [{"name": "id", "dtype": "int32"}, {"name": "question_id", "dtype": "int32"}, {"name": "question", "dtype": "string"}, {"name": "snippet", "sequence": [{"name": "text", "dtype": "string"}, {"name": "is_sda", "dtype": "bool"}]}], "splits": [{"name": "train", "num_bytes": 1445103, "num_examples": 2052}], "download_size": 71250225, "dataset_size": 1445103}, {"config_name": "man_sql", "features": [{"name": "id", "dtype": "int32"}, {"name": "question_id", "dtype": "int32"}, {"name": "question", "dtype": "string"}, {"name": "snippet", "sequence": [{"name": "text", "dtype": "string"}, {"name": "is_sda", "dtype": "bool"}]}], "splits": [{"name": "train", "num_bytes": 1123721, "num_examples": 1587}], "download_size": 49745860, "dataset_size": 1123721}], "tags": ["code"]}
TAGS
#task_categories-question-answering #size_categories-10K<n<100K #language-English #license-cc-by-4.0 #code #arxiv-1803.09371 #region-us 


# Dataset Card for StaQC (A Systematically Mined Question-Code Dataset from Stack Overflow)


## Dataset Description

- Homepage: GitHub 
- Paper: StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow 

### Dataset Summary

StaQC (Stack Overflow Question-Code pairs) is a large dataset of around 148K Python and 120K SQL domain question-code pairs, 
which are automatically mined from Stack Overflow using a Bi-View Hierarchical Neural Network. StaQC is collected from three sources: multi-code answer posts, single-code answer posts, and manual annotations on multi-code answer posts. 

The dataset was originally released by the main authors on GitHub. This version is a *non-modified* redistributed copy (under the license permission) made available on the hub for easier access. 

#### Standalone solutions

As noted in the paper, the authors *define a code snippet as a code solution when the
questioner can solve the problem solely based on it (also named as
“standalone” solution).* 

#### Manual annotations

The manual annotations are the collection of multi-code answer posts for which each code snippet was annotated with a boolean indicating whether or not the snippet is a *standalone solution* to the question.


#### Multi-code answer posts
A *Multi-code answer post* is an (accepted) answer post that contains multiple code snippets, 
some of which may not be a *standalone* code solution to the question (see Section 1 in paper). 
For example, in this multi-code answer post, 
the third code snippet is not a code solution to the question "How to limit a number to be within a specified range? (Python)".

Note: the multi-code answer posts contain also the manual annotations. 

#### Single-code answer posts
A *Single-code answer post* is an (accepted) answer post that contains only one code snippet. 
We pair such code snippets with the question title as a question-code pair.



### Supported Tasks and Leaderboards

This dataset can be used for Natural Language to Code Generation tasks. 

### Languages

Python, SQL, English

## Dataset Structure

### Data Instances

Each configuration correspond to one of the three parts, in a given programming language.

There are three parts for the dataset:

- mca (Multi-code answer posts)
- sca (Single-code answer posts)
- man (Manual annotations)

And two programming/query languages:

- python
- sql

One can obtain obtain a configuration as a combination of a part in a programing language. For instance, one can obtain the automatically mined multi-code answers in python using:


or the manual annotations using:




#### Manual annotations

The manual annotations contain, for a given stackoverflow questions, for each individual code block in the accepted answer of that post, information on whether or not the given code block is a *standalone* solution to the question asked (the question title).



#### Multi-code answer posts




#### Single-code answer posts



### Data Fields

- 'question_id': id of the stackoverflow question 
- 'question': title of the stackoverflow question repurposed as the natural language intent
- 'snippet': mined or annotated standalone solution(s) (potentially) answerring the question
- 'is_sda': for the manual annotations, whether or not the given code snippet is a standalone solution to the question. 

### Data Splits

Each configuration of the dataset contains only a training split. 

## Dataset Creation

### Source Data

StackOverflow data dump.

### Annotations

See section 2.3 "Annotating QC Pairs for Model Training" of the paper

## Additional Information

### Licensing Information

<a rel="license" href="URL alt="Creative Commons License" style="border-width:0" src="https://i.URL /></a><br />This work is licensed under a <a rel="license" href="URL Commons Attribution 4.0 International License</a>.



If you use the dataset or the code in your research, please cite the following paper:



### Contributions information

I did *not* contribute to the *creation* of this dataset, only to the redistribution. All credits should be attributed to the original authors.
# Dataset Card for StaQC (A Systematically Mined Question-Code Dataset from Stack Overflow) ## Dataset Description

- Homepage: GitHub 
- Paper: StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow ### Dataset Summary

StaQC (Stack Overflow Question-Code pairs) is a large dataset of around 148K Python and 120K SQL domain question-code pairs, 
which are automatically mined from Stack Overflow using a Bi-View Hierarchical Neural Network. StaQC is collected from three sources: multi-code answer posts, single-code answer posts, and manual annotations on multi-code answer posts. 

The dataset was originally released by the main authors on GitHub. This version is a *non-modified* redistributed copy (under the license permission) made available on the hub for easier access. #### Standalone solutions

As noted in the paper, the authors *define a code snippet as a code solution when the
questioner can solve the problem solely based on it (also named as
“standalone” solution).* #### Manual annotations

The manual annotations are the collection of multi-code answer posts for which each code snippet was annotated with a boolean indicating whether or not the snippet is a *standalone solution* to the question. #### Multi-code answer posts
A *Multi-code answer post* is an (accepted) answer post that contains multiple code snippets, 
some of which may not be a *standalone* code solution to the question (see Section 1 in paper). 
For example, in this multi-code answer post, 
the third code snippet is not a code solution to the question "How to limit a number to be within a specified range? (Python)".

Note: the multi-code answer posts contain also the manual annotations. #### Single-code answer posts
A *Single-code answer post* is an (accepted) answer post that contains only one code snippet. 
We pair such code snippets with the question title as a question-code pair. ### Supported Tasks and Leaderboards

This dataset can be used for Natural Language to Code Generation tasks. ### Languages

Python, SQL, English ## Dataset Structure ### Data Instances

Each configuration correspond to one of the three parts, in a given programming language.

There are three parts for the dataset:

- mca (Multi-code answer posts)
- sca (Single-code answer posts)
- man (Manual annotations)

And two programming/query languages:

- python
- sql

One can obtain obtain a configuration as a combination of a part in a programing language. For instance, one can obtain the automatically mined multi-code answers in python using:


or the manual annotations using: #### Manual annotations

The manual annotations contain, for a given stackoverflow questions, for each individual code block in the accepted answer of that post, information on whether or not the given code block is a *standalone* solution to the question asked (the question title). #### Multi-code answer posts #### Single-code answer posts ### Data Fields

- 'question_id': id of the stackoverflow question 
- 'question': title of the stackoverflow question repurposed as the natural language intent
- 'snippet': mined or annotated standalone solution(s) (potentially) answerring the question
- 'is_sda': for the manual annotations, whether or not the given code snippet is a standalone solution to the question. ### Data Splits

Each configuration of the dataset contains only a training split. ## Dataset Creation ### Source Data

StackOverflow data dump. ### Annotations

See section 2.3 "Annotating QC Pairs for Model Training" of the paper ## Additional Information ### Licensing Information

<a rel="license" href="URL alt="Creative Commons License" style="border-width:0" src="https://i.URL /></a><br />This work is licensed under a <a rel="license" href="URL Commons Attribution 4.0 International License</a>.



If you use the dataset or the code in your research, please cite the following paper: ### Contributions information

I did *not* contribute to the *creation* of this dataset, only to the redistribution. All credits should be attributed to the original authors.
TAGS
#task_categories-question-answering #size_categories-10K<n<100K #language-English #license-cc-by-4.0 #code #arxiv-1803.09371 #region-us 
 # Dataset Card for StaQC (A Systematically Mined Question-Code Dataset from Stack Overflow) ## Dataset Description

- Homepage: GitHub 
- Paper: StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow ### Dataset Summary

StaQC (Stack Overflow Question-Code pairs) is a large dataset of around 148K Python and 120K SQL domain question-code pairs, 
which are automatically mined from Stack Overflow using a Bi-View Hierarchical Neural Network. StaQC is collected from three sources: multi-code answer posts, single-code answer posts, and manual annotations on multi-code answer posts. 

The dataset was originally released by the main authors on GitHub. This version is a *non-modified* redistributed copy (under the license permission) made available on the hub for easier access. #### Standalone solutions

As noted in the paper, the authors *define a code snippet as a code solution when the
questioner can solve the problem solely based on it (also named as
“standalone” solution).* #### Manual annotations

The manual annotations are the collection of multi-code answer posts for which each code snippet was annotated with a boolean indicating whether or not the snippet is a *standalone solution* to the question. #### Multi-code answer posts
A *Multi-code answer post* is an (accepted) answer post that contains multiple code snippets, 
some of which may not be a *standalone* code solution to the question (see Section 1 in paper). 
For example, in this multi-code answer post, 
the third code snippet is not a code solution to the question "How to limit a number to be within a specified range? (Python)".

Note: the multi-code answer posts contain also the manual annotations. #### Single-code answer posts
A *Single-code answer post* is an (accepted) answer post that contains only one code snippet. 
We pair such code snippets with the question title as a question-code pair. ### Supported Tasks and Leaderboards

This dataset can be used for Natural Language to Code Generation tasks. ### Languages

Python, SQL, English ## Dataset Structure ### Data Instances

Each configuration correspond to one of the three parts, in a given programming language.

There are three parts for the dataset:

- mca (Multi-code answer posts)
- sca (Single-code answer posts)
- man (Manual annotations)

And two programming/query languages:

- python
- sql

One can obtain obtain a configuration as a combination of a part in a programing language. For instance, one can obtain the automatically mined multi-code answers in python using:


or the manual annotations using: #### Manual annotations

The manual annotations contain, for a given stackoverflow questions, for each individual code block in the accepted answer of that post, information on whether or not the given code block is a *standalone* solution to the question asked (the question title). #### Multi-code answer posts #### Single-code answer posts ### Data Fields

- 'question_id': id of the stackoverflow question 
- 'question': title of the stackoverflow question repurposed as the natural language intent
- 'snippet': mined or annotated standalone solution(s) (potentially) answerring the question
- 'is_sda': for the manual annotations, whether or not the given code snippet is a standalone solution to the question. ### Data Splits

Each configuration of the dataset contains only a training split. ## Dataset Creation ### Source Data

StackOverflow data dump. ### Annotations

See section 2.3 "Annotating QC Pairs for Model Training" of the paper ## Additional Information ### Licensing Information

<a rel="license" href="URL alt="Creative Commons License" style="border-width:0" src="https://i.URL /></a><br />This work is licensed under a <a rel="license" href="URL Commons Attribution 4.0 International License</a>.



If you use the dataset or the code in your research, please cite the following paper: ### Contributions information

I did *not* contribute to the *creation* of this dataset, only to the redistribution. All credits should be attributed to the original authors.