
### Dataset is imported from CodeXGLUE and pre-processed using their script.

# Where to find in Semeru:
The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru


# CodeXGLUE -- Code Completion (token level)

**Update 2021.07.30:** We update the code completion dataset with literals normalized to avoid sensitive information. 

Here is the introduction and pipeline for token level code completion task.

## Task Definition

Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.

Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software developers' productivity. We provide code completion evaluation tasks in two granularities -- token level and line level. Here we introduce token level code completion. Token level task is analogous to language modeling. Models should have be able to predict the next token in arbitary types.


## Dataset

The dataset is in python.


### Dependency

- python 3.7




### Github Java Corpus

We use java corpus dataset mined by Allamanis and Sutton, in their MSR 2013 paper [Mining Source Code Repositories at Massive Scale using Language Modeling](https://homepages.inf.ed.ac.uk/csutton/publications/msr2013.pdf). We follow the same split and preprocessing in Karampatsis's ICSE 2020 paper [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](http://homepages.inf.ed.ac.uk/s1467463/documents/icse20-main-1325.pdf).



### Data Format

Code corpus are saved in txt format files. one line is a tokenized code snippets:
```
<s> from __future__ import unicode_literals <EOL> from django . db import models , migrations <EOL> class Migration ( migrations . Migration ) : <EOL> dependencies = [ <EOL> ] <EOL> operations = [ <EOL> migrations . CreateModel ( <EOL> name = '<STR_LIT>' , <EOL> fields = [ <EOL> ( '<STR_LIT:id>' , models . AutoField ( verbose_name = '<STR_LIT>' , serialize = False , auto_created = True , primary_key = True ) ) , <EOL> ( '<STR_LIT:name>' , models . CharField ( help_text = b'<STR_LIT>' , max_length = <NUM_LIT> ) ) , <EOL> ( '<STR_LIT:image>' , models . ImageField ( help_text = b'<STR_LIT>' , null = True , upload_to = b'<STR_LIT>' , blank = True ) ) , <EOL> ] , <EOL> options = { <EOL> '<STR_LIT>' : ( '<STR_LIT:name>' , ) , <EOL> '<STR_LIT>' : '<STR_LIT>' , <EOL> } , <EOL> bases = ( models . Model , ) , <EOL> ) , <EOL> ] </s>
```


### Data Statistics


Data statistics of py150 dataset are shown in the below table, note that there doesn't exist dev set in the origin py150 dataset, we select 5,000 files in the original train set as dev set.

| Data Split  |   #Files    |   #Tokens   |
| ----------- | :---------: | :---------: |
|    Train    |    95,000   |    72.1M    |
|     Dev     |    5,000    |     4.4M    |
|    Test     |    50,000   |    37.3M    |




license:mit region:us
{"license": "mit", "Programminglanguage": "python", "version": "python3", "Date": "From paper [Probabilistic for Code with Decision trees](https://files.sri.inf.ethz.ch/website/papers/oopsla16-dt.pdf)(2016- paper release date)", "Contaminated": "Very Likely", "Size": "Standard Tokenizer (TreeSitter)"}
TAGS
#license-mit #region-us 

### Dataset is imported from CodeXGLUE and pre-processed using their script.


Where to find in Semeru:
========================


The dataset can be found at /nfs/semeru/semeru\_datasets/code\_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru


CodeXGLUE -- Code Completion (token level)
==========================================


Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.


Here is the introduction and pipeline for token level code completion task.


Task Definition
---------------


Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.


Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software developers' productivity. We provide code completion evaluation tasks in two granularities -- token level and line level. Here we introduce token level code completion. Token level task is analogous to language modeling. Models should have be able to predict the next token in arbitary types.


Dataset
-------


The dataset is in python.


### Dependency


* python 3.7


### Github Java Corpus


We use java corpus dataset mined by Allamanis and Sutton, in their MSR 2013 paper Mining Source Code Repositories at Massive Scale using Language Modeling. We follow the same split and preprocessing in Karampatsis's ICSE 2020 paper Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code.


### Data Format


Code corpus are saved in txt format files. one line is a tokenized code snippets:


### Data Statistics


Data statistics of py150 dataset are shown in the below table, note that there doesn't exist dev set in the origin py150 dataset, we select 5,000 files in the original train set as dev set.




### Dataset is imported from CodeXGLUE and pre-processed using their script.


Where to find in Semeru:
========================


The dataset can be found at /nfs/semeru/semeru\_datasets/code\_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru


CodeXGLUE -- Code Completion (token level)
==========================================


Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.


Here is the introduction and pipeline for token level code completion task.


Task Definition
---------------


Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.


Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software developers' productivity. We provide code completion evaluation tasks in two granularities -- token level and line level. Here we introduce token level code completion. Token level task is analogous to language modeling. Models should have be able to predict the next token in arbitary types.


Dataset
-------


The dataset is in python. ### Dependency


* python 3.7 ### Github Java Corpus


We use java corpus dataset mined by Allamanis and Sutton, in their MSR 2013 paper Mining Source Code Repositories at Massive Scale using Language Modeling. We follow the same split and preprocessing in Karampatsis's ICSE 2020 paper Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code. ### Data Format


Code corpus are saved in txt format files. one line is a tokenized code snippets: ### Data Statistics


Data statistics of py150 dataset are shown in the below table, note that there doesn't exist dev set in the origin py150 dataset, we select 5,000 files in the original train set as dev set.
TAGS
#license-mit #region-us 
 ### Dataset is imported from CodeXGLUE and pre-processed using their script.


Where to find in Semeru:
========================


The dataset can be found at /nfs/semeru/semeru\_datasets/code\_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru


CodeXGLUE -- Code Completion (token level)
==========================================


Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.


Here is the introduction and pipeline for token level code completion task.


Task Definition
---------------


Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.


Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software developers' productivity. We provide code completion evaluation tasks in two granularities -- token level and line level. Here we introduce token level code completion. Token level task is analogous to language modeling. Models should have be able to predict the next token in arbitary types.


Dataset
-------


The dataset is in python. ### Dependency


* python 3.7 ### Github Java Corpus


We use java corpus dataset mined by Allamanis and Sutton, in their MSR 2013 paper Mining Source Code Repositories at Massive Scale using Language Modeling. We follow the same split and preprocessing in Karampatsis's ICSE 2020 paper Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code. ### Data Format


Code corpus are saved in txt format files. one line is a tokenized code snippets: ### Data Statistics


Data statistics of py150 dataset are shown in the below table, note that there doesn't exist dev set in the origin py150 dataset, we select 5,000 files in the original train set as dev set.
passage: TAGS
#license-mit #region-us 
### Dataset is imported from CodeXGLUE and pre-processed using their script.


Where to find in Semeru:
========================


The dataset can be found at /nfs/semeru/semeru\_datasets/code\_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru


CodeXGLUE -- Code Completion (token level)
==========================================


Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.


Here is the introduction and pipeline for token level code completion task.


Task Definition
---------------


Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.


Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software developers' productivity. We provide code completion evaluation tasks in two granularities -- token level and line level. Here we introduce token level code completion. Token level task is analogous to language modeling. Models should have be able to predict the next token in arbitary types.


Dataset
-------


The dataset is in python.### Dependency


* python 3.7### Github Java Corpus


We use java corpus dataset mined by Allamanis and Sutton, in their MSR 2013 paper Mining Source Code Repositories at Massive Scale using Language Modeling. We follow the same split and preprocessing in Karampatsis's ICSE 2020 paper Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code.### Data Format


Code corpus are saved in txt format files. one line is a tokenized code snippets:### Data Statistics


Data statistics of py150 dataset are shown in the below table, note that there doesn't exist dev set in the origin py150 dataset, we select 5,000 files in the original train set as dev set.