

[github](https://github.com/ntunlp/xCodeEval)

# xCodeEval
[xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://arxiv.org/abs/2303.03004)

We introduce **xCodeEval**, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding examples from about 7.5 K unique problems covering up to 17 programming languages with execution-level parallelism. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution-based evaluation. We develop a test-case based multilingual code execution engine, [**ExecEval**](https://github.com/ntunlp/ExecEval) that supports all the programming languages in **xCodeEval**. We also propose a novel data splitting and a data selection schema for balancing data distributions over multiple attributes based on geometric mean and graph-theoretic principle. 

This repository contains the sample code and data link for xCodeEval [paper](https://arxiv.org/abs/2303.03004).

# Data Download

Currently this repository supports huggingface [`load_dataset()`](https://huggingface.co/docs/datasets/v1.11.0/package_reference/loading_methods.html#datasets.load_dataset) api. Follow the following example to load dataset for individual examples. 

```
import datasets

prog_synthesis_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "program_synthesis")
code_translation_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "code_translation")
tag_classification_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "tag_classification")
apr_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "apr")
pcode_compilation_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "code_compilation")
retrieval_code_code_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "retrieval_code_code")
retrieval_nl_code_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "retrieval_nl_code")
retrieval_corpus_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "retrieval_corpus")

```

## Hf large data download tricks.

If you are facing long delay with data processing, add a `ignore_verifications=True`.

```
prog_synthesis_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "program_synthesis", ignore_verifications=True)
```

If you are facing long delay with data downloading, use huggingface streaming mode.

```
prog_synthesis_dataset = datasets.load_dataset("NTU-NLP-sg/xCodeEval", "program_synthesis", streaming=True)
```

## Just Give me the raw data (😠)

Data can be also downloaded as a git LFS repo from huggingface. 

![xCodeEval_hf](https://github.com/ntunlp/xCodeEval/blob/main/xcodeeval-hf.png?raw=true)

You can download the full data using the following command.

```
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/NTU-NLP-sg/xCodeEval
cd xCodeEval
git lfs pull
```

To download a specific part of the dataset, 

```
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/NTU-NLP-sg/xCodeEval
cd xCodeEval
git lfs pull --include "apr/test/*"
```


We propose 7 Tasks.

1. [Tag Classification](https://github.com/ntunlp/xCodeEval/blob/main/apr.md)
2. [Code Compilation](https://github.com/ntunlp/xCodeEval/blob/main/code_compilation.md)
3. [Program Synthesis](https://github.com/ntunlp/xCodeEval/blob/main/program_synthesis.md)
4. [Code Translation](https://github.com/ntunlp/xCodeEval/blob/main/code_translation.md)
5. [Automatic Program Repair](https://github.com/ntunlp/xCodeEval/blob/main/apr.md)
6. [Code-Code Retrieval](https://github.com/ntunlp/xCodeEval/blob/main/retrieval.md)
7. [NL-Code Retrieval](https://github.com/ntunlp/xCodeEval/blob/main/retrieval.md)


# Common Data for different tasks

If you are not using huggingface [`load_dataset()`](https://huggingface.co/docs/datasets/v1.11.0/package_reference/loading_methods.html#datasets.load_dataset) api, you may need to link some data with different tasks.

![xCodeEval_fig_1](https://github.com/ntunlp/xCodeEval/blob/main/xcodeeval_fig_1.png?raw=true)

We have two data files that are required for multiple tasks.

1. `problem_descriptions.jsonl`
2. `unittest_db.json`

You can find these two files in the root directory of the [main](https://huggingface.co/datasets/NTU-NLP-sg/xCodeEval/tree/main) branch of huggingface dataset repository. To avoid data redundancy we didn't include these data with the relevant tasks, rather we add a unique id `src_uid` to retrieve these data. 

## Structure of `problem_descriptions.jsonl`

A sample, 

```json
{
    "description": "There are $$$n$$$ positive integers $$$a_1, a_2, \\dots, a_n$$$. For the one move you can choose any even value $$$c$$$ and divide by two all elements that equal $$$c$$$.For example, if $$$a=[6,8,12,6,3,12]$$$ and you choose $$$c=6$$$, and $$$a$$$ is transformed into $$$a=[3,8,12,3,3,12]$$$ after the move.You need to find the minimal number of moves for transforming $$$a$$$ to an array of only odd integers (each element shouldn't be divisible by $$$2$$$).",
    "input_from": "standard input",
    "output_to": "standard output",
    "time_limit": "3 seconds",
    "memory_limit": "256 megabytes",
    "input_spec": "The first line of the input contains one integer $$$t$$$ ($$$1 \\le t \\le 10^4$$$) \u2014 the number of test cases in the input. Then $$$t$$$ test cases follow. The first line of a test case contains $$$n$$$ ($$$1 \\le n \\le 2\\cdot10^5$$$) \u2014 the number of integers in the sequence $$$a$$$. The second line contains positive integers $$$a_1, a_2, \\dots, a_n$$$ ($$$1 \\le a_i \\le 10^9$$$). The sum of $$$n$$$ for all test cases in the input doesn't exceed $$$2\\cdot10^5$$$.",
    "output_spec": "For $$$t$$$ test cases print the answers in the order of test cases in the input. The answer for the test case is the minimal number of moves needed to make all numbers in the test case odd (i.e. not divisible by $$$2$$$).",
    "notes": "NoteIn the first test case of the example, the optimal sequence of moves can be as follows:  before making moves $$$a=[40, 6, 40, 3, 20, 1]$$$;  choose $$$c=6$$$;  now $$$a=[40, 3, 40, 3, 20, 1]$$$;  choose $$$c=40$$$;  now $$$a=[20, 3, 20, 3, 20, 1]$$$;  choose $$$c=20$$$;  now $$$a=[10, 3, 10, 3, 10, 1]$$$;  choose $$$c=10$$$;  now $$$a=[5, 3, 5, 3, 5, 1]$$$ \u2014 all numbers are odd. Thus, all numbers became odd after $$$4$$$ moves. In $$$3$$$ or fewer moves, you cannot make them all odd.",
    "sample_inputs": [
        "4\n6\n40 6 40 3 20 1\n1\n1024\n4\n2 4 8 16\n3\n3 1 7"
    ],
    "sample_outputs": [
        "4\n10\n4\n0"
    ],
    "tags": [
        "number theory",
        "greedy"
    ],
    "src_uid": "afcd41492158e68095b01ff1e88c3dd4",
    "difficulty": 1200,
    "created_at": 1576321500
}
```

### Key Definitions

1. `description`: Problem description in textual format, math operations are written in latex.
2. `input_from`: How the program should take the unit test.
3. `output_to`: Where the program should output the result of the unit test.
4. `time_limit`: Time limit to solve the problem. 
5. `memory_limit`: Memory limit to solve the problem.
6. `input_spec`: How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
7. `output_spec`: How the outputs should be printed. Most of the time the unit test results are matched with an *exact string match* or *floating point comparison* with a precision boundary. 
8. `sample_inputs`: A sample input for the code that is expected to solve the problem described in `description`.
9. `sample_outputs`: The expected output for the `sample_input` that is expected to solve the problem described in `description`.
10. `notes`: Explanation of `sample_inputs` & `sample_outputs`.
11. `tags`: The problem categories.
12. `src_uid`: The unique id of the problem. This ID is referred to in the task data samples instead of putting all this information.
13. `difficulty`: How difficult is it to solve the problem for a human (annotated by an expert human)? 
14. `created_at`: The Unix timestamp when the problem was released. Use `datetime` lib in Python to parse it to a human-readable format.  

## Structure of `unittest_db.json`

The structure of the `json` file, 

```python
unittest_db = {
	"db884d679d9cfb1dc4bc511f83beedda" : [
		{
			"input": "4\r\n3 2 3 2\r\n",
			"output": [
				"1"
			],
		},
		{
			...
		},
		...
	]
	"3bc096d8cd3418948d5be6bf297aa9b5":[
		...
	],
	...
}
```

### Key Definitions

1. `unittest_db.json` dict keys i.e., `db884d679d9cfb1dc4bc511f83beedda` are the `src_uid` from `problem_descriptions.jsonl`.
2. `input`: Input of the unit test.
3. `output`: List of expected outputs for the unit test. 

# Citation

```
@misc{khan2023xcodeeval,
      title={xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval}, 
      author={Mohammad Abdullah Matin Khan and M Saiful Bari and Xuan Long Do and Weishi Wang and Md Rizwan Parvez and Shafiq Joty},
      year={2023},
      eprint={2303.03004},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
Part of this work was submitted as a requirement for the Master of Science degree in Computer Science and Applications at the Islamic University of Technology by Muhammad Abdullah Matin Khan Zarzis. (The thesis or project report will be added upon publication).

```
@misc{khan2024xcodeeval,
      title={Development of a Code Search Engine Using Natural Language Processing Techniques}, 
      author={Mohammad Abdullah Matin Khan},
      year={2024},
      publication={Journal of Engineering and Technology (JET)}
      url=TBA
}
```

 

task_categories:translation task_categories:token-classification task_categories:text2text-generation task_categories:text-retrieval task_categories:text-generation task_categories:text-classification task_categories:feature-extraction task_categories:question-answering annotations_creators:expert-generated language_creators:found language_creators:expert-generated multilinguality:multilingual size_categories:1M<n<10M size_categories:10M<n<100M source_datasets:original language:code language:en license:cc-by-nc-4.0 programming-language code program-synthesis automatic-code-repair code-retrieval code-translation code-classification arxiv:2303.03004 region:us
{"annotations_creators": ["expert-generated"], "language_creators": ["found", "expert-generated"], "language": ["code", "en"], "license": ["cc-by-nc-4.0"], "multilinguality": ["multilingual"], "size_categories": ["1M<n<10M", "10M<n<100M"], "source_datasets": ["original"], "task_categories": ["translation", "token-classification", "text2text-generation", "text-retrieval", "text-generation", "text-classification", "feature-extraction", "question-answering"], "pretty_name": "xCodeEval", "tags": ["programming-language", "code", "program-synthesis", "automatic-code-repair", "code-retrieval", "code-translation", "code-classification"]}
TAGS
#task_categories-translation #task_categories-token-classification #task_categories-text2text-generation #task_categories-text-retrieval #task_categories-text-generation #task_categories-text-classification #task_categories-feature-extraction #task_categories-question-answering #annotations_creators-expert-generated #language_creators-found #language_creators-expert-generated #multilinguality-multilingual #size_categories-1M<n<10M #size_categories-10M<n<100M #source_datasets-original #language-code #language-English #license-cc-by-nc-4.0 #programming-language #code #program-synthesis #automatic-code-repair #code-retrieval #code-translation #code-classification #arxiv-2303.03004 #region-us 



github

# xCodeEval
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval

We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding examples from about 7.5 K unique problems covering up to 17 programming languages with execution-level parallelism. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution-based evaluation. We develop a test-case based multilingual code execution engine, ExecEval that supports all the programming languages in xCodeEval. We also propose a novel data splitting and a data selection schema for balancing data distributions over multiple attributes based on geometric mean and graph-theoretic principle. 

This repository contains the sample code and data link for xCodeEval paper.

# Data Download

Currently this repository supports huggingface 'load_dataset()' api. Follow the following example to load dataset for individual examples. 



## Hf large data download tricks.

If you are facing long delay with data processing, add a 'ignore_verifications=True'.



If you are facing long delay with data downloading, use huggingface streaming mode.



## Just Give me the raw data ()

Data can be also downloaded as a git LFS repo from huggingface. 

!xCodeEval_hf

You can download the full data using the following command.



To download a specific part of the dataset, 




We propose 7 Tasks.

1. Tag Classification
2. Code Compilation
3. Program Synthesis
4. Code Translation
5. Automatic Program Repair
6. Code-Code Retrieval
7. NL-Code Retrieval


# Common Data for different tasks

If you are not using huggingface 'load_dataset()' api, you may need to link some data with different tasks.

!xCodeEval_fig_1

We have two data files that are required for multiple tasks.

1. 'problem_descriptions.jsonl'
2. 'unittest_db.json'

You can find these two files in the root directory of the main branch of huggingface dataset repository. To avoid data redundancy we didn't include these data with the relevant tasks, rather we add a unique id 'src_uid' to retrieve these data. 

## Structure of 'problem_descriptions.jsonl'

A sample, 



### Key Definitions

1. 'description': Problem description in textual format, math operations are written in latex.
2. 'input_from': How the program should take the unit test.
3. 'output_to': Where the program should output the result of the unit test.
4. 'time_limit': Time limit to solve the problem. 
5. 'memory_limit': Memory limit to solve the problem.
6. 'input_spec': How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
7. 'output_spec': How the outputs should be printed. Most of the time the unit test results are matched with an *exact string match* or *floating point comparison* with a precision boundary. 
8. 'sample_inputs': A sample input for the code that is expected to solve the problem described in 'description'.
9. 'sample_outputs': The expected output for the 'sample_input' that is expected to solve the problem described in 'description'.
10. 'notes': Explanation of 'sample_inputs' & 'sample_outputs'.
11. 'tags': The problem categories.
12. 'src_uid': The unique id of the problem. This ID is referred to in the task data samples instead of putting all this information.
13. 'difficulty': How difficult is it to solve the problem for a human (annotated by an expert human)? 
14. 'created_at': The Unix timestamp when the problem was released. Use 'datetime' lib in Python to parse it to a human-readable format.  

## Structure of 'unittest_db.json'

The structure of the 'json' file, 



### Key Definitions

1. 'unittest_db.json' dict keys i.e., 'db884d679d9cfb1dc4bc511f83beedda' are the 'src_uid' from 'problem_descriptions.jsonl'.
2. 'input': Input of the unit test.
3. 'output': List of expected outputs for the unit test. 

Part of this work was submitted as a requirement for the Master of Science degree in Computer Science and Applications at the Islamic University of Technology by Muhammad Abdullah Matin Khan Zarzis. (The thesis or project report will be added upon publication).



 

# xCodeEval
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval

We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding examples from about 7.5 K unique problems covering up to 17 programming languages with execution-level parallelism. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution-based evaluation. We develop a test-case based multilingual code execution engine, ExecEval that supports all the programming languages in xCodeEval. We also propose a novel data splitting and a data selection schema for balancing data distributions over multiple attributes based on geometric mean and graph-theoretic principle. 

This repository contains the sample code and data link for xCodeEval paper. # Data Download

Currently this repository supports huggingface 'load_dataset()' api. Follow the following example to load dataset for individual examples. ## Hf large data download tricks.

If you are facing long delay with data processing, add a 'ignore_verifications=True'.



If you are facing long delay with data downloading, use huggingface streaming mode. ## Just Give me the raw data ()

Data can be also downloaded as a git LFS repo from huggingface. 

!xCodeEval_hf

You can download the full data using the following command.



To download a specific part of the dataset, 




We propose 7 Tasks.

1. Tag Classification
2. Code Compilation
3. Program Synthesis
4. Code Translation
5. Automatic Program Repair
6. Code-Code Retrieval
7. NL-Code Retrieval # Common Data for different tasks

If you are not using huggingface 'load_dataset()' api, you may need to link some data with different tasks.

!xCodeEval_fig_1

We have two data files that are required for multiple tasks.

1. 'problem_descriptions.jsonl'
2. 'unittest_db.json'

You can find these two files in the root directory of the main branch of huggingface dataset repository. To avoid data redundancy we didn't include these data with the relevant tasks, rather we add a unique id 'src_uid' to retrieve these data. ## Structure of 'problem_descriptions.jsonl'

A sample, ### Key Definitions

1. 'description': Problem description in textual format, math operations are written in latex.
2. 'input_from': How the program should take the unit test.
3. 'output_to': Where the program should output the result of the unit test.
4. 'time_limit': Time limit to solve the problem. 
5. 'memory_limit': Memory limit to solve the problem.
6. 'input_spec': How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
7. 'output_spec': How the outputs should be printed. Most of the time the unit test results are matched with an *exact string match* or *floating point comparison* with a precision boundary. 
8. 'sample_inputs': A sample input for the code that is expected to solve the problem described in 'description'.
9. 'sample_outputs': The expected output for the 'sample_input' that is expected to solve the problem described in 'description'.
10. 'notes': Explanation of 'sample_inputs' & 'sample_outputs'.
11. 'tags': The problem categories.
12. 'src_uid': The unique id of the problem. This ID is referred to in the task data samples instead of putting all this information.
13. 'difficulty': How difficult is it to solve the problem for a human (annotated by an expert human)? 
14. 'created_at': The Unix timestamp when the problem was released. Use 'datetime' lib in Python to parse it to a human-readable format. ## Structure of 'unittest_db.json'

The structure of the 'json' file, ### Key Definitions

1. 'unittest_db.json' dict keys i.e., 'db884d679d9cfb1dc4bc511f83beedda' are the 'src_uid' from 'problem_descriptions.jsonl'.
2. 'input': Input of the unit test.
3. 'output': List of expected outputs for the unit test. 

Part of this work was submitted as a requirement for the Master of Science degree in Computer Science and Applications at the Islamic University of Technology by Muhammad Abdullah Matin Khan Zarzis. (The thesis or project report will be added upon publication).
TAGS
#task_categories-translation #task_categories-token-classification #task_categories-text2text-generation #task_categories-text-retrieval #task_categories-text-generation #task_categories-text-classification #task_categories-feature-extraction #task_categories-question-answering #annotations_creators-expert-generated #language_creators-found #language_creators-expert-generated #multilinguality-multilingual #size_categories-1M<n<10M #size_categories-10M<n<100M #source_datasets-original #language-code #language-English #license-cc-by-nc-4.0 #programming-language #code #program-synthesis #automatic-code-repair #code-retrieval #code-translation #code-classification #arxiv-2303.03004 #region-us 
 # xCodeEval
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval

We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding examples from about 7.5 K unique problems covering up to 17 programming languages with execution-level parallelism. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution-based evaluation. We develop a test-case based multilingual code execution engine, ExecEval that supports all the programming languages in xCodeEval. We also propose a novel data splitting and a data selection schema for balancing data distributions over multiple attributes based on geometric mean and graph-theoretic principle. 

This repository contains the sample code and data link for xCodeEval paper. # Data Download

Currently this repository supports huggingface 'load_dataset()' api. Follow the following example to load dataset for individual examples. ## Hf large data download tricks.

If you are facing long delay with data processing, add a 'ignore_verifications=True'.



If you are facing long delay with data downloading, use huggingface streaming mode. ## Just Give me the raw data ()

Data can be also downloaded as a git LFS repo from huggingface. 

!xCodeEval_hf

You can download the full data using the following command.



To download a specific part of the dataset, 




We propose 7 Tasks.

1. Tag Classification
2. Code Compilation
3. Program Synthesis
4. Code Translation
5. Automatic Program Repair
6. Code-Code Retrieval
7. NL-Code Retrieval # Common Data for different tasks

If you are not using huggingface 'load_dataset()' api, you may need to link some data with different tasks.

!xCodeEval_fig_1

We have two data files that are required for multiple tasks.

1. 'problem_descriptions.jsonl'
2. 'unittest_db.json'

You can find these two files in the root directory of the main branch of huggingface dataset repository. To avoid data redundancy we didn't include these data with the relevant tasks, rather we add a unique id 'src_uid' to retrieve these data. ## Structure of 'problem_descriptions.jsonl'

A sample, ### Key Definitions

1. 'description': Problem description in textual format, math operations are written in latex.
2. 'input_from': How the program should take the unit test.
3. 'output_to': Where the program should output the result of the unit test.
4. 'time_limit': Time limit to solve the problem. 
5. 'memory_limit': Memory limit to solve the problem.
6. 'input_spec': How and in what order the input will be given to the program? It also includes the date range, types, and sizes.
7. 'output_spec': How the outputs should be printed. Most of the time the unit test results are matched with an *exact string match* or *floating point comparison* with a precision boundary. 
8. 'sample_inputs': A sample input for the code that is expected to solve the problem described in 'description'.
9. 'sample_outputs': The expected output for the 'sample_input' that is expected to solve the problem described in 'description'.
10. 'notes': Explanation of 'sample_inputs' & 'sample_outputs'.
11. 'tags': The problem categories.
12. 'src_uid': The unique id of the problem. This ID is referred to in the task data samples instead of putting all this information.
13. 'difficulty': How difficult is it to solve the problem for a human (annotated by an expert human)? 
14. 'created_at': The Unix timestamp when the problem was released. Use 'datetime' lib in Python to parse it to a human-readable format. ## Structure of 'unittest_db.json'

The structure of the 'json' file, ### Key Definitions

1. 'unittest_db.json' dict keys i.e., 'db884d679d9cfb1dc4bc511f83beedda' are the 'src_uid' from 'problem_descriptions.jsonl'.
2. 'input': Input of the unit test.
3. 'output': List of expected outputs for the unit test. 

Part of this work was submitted as a requirement for the Master of Science degree in Computer Science and Applications at the Islamic University of Technology by Muhammad Abdullah Matin Khan Zarzis. (The thesis or project report will be added upon publication).
passage: TAGS
#task_categories-translation #task_categories-token-classification #task_categories-text2text-generation #task_categories-text-retrieval #task_categories-text-generation #task_categories-text-classification #task_categories-feature-extraction #task_categories-question-answering #annotations_creators-expert-generated #language_creators-found #language_creators-expert-generated #multilinguality-multilingual #size_categories-1M<n<10M #size_categories-10M<n<100M #source_datasets-original #language-code #language-English #license-cc-by-nc-4.0 #programming-language #code #program-synthesis #automatic-code-repair #code-retrieval #code-translation #code-classification #arxiv-2303.03004 #region-us 
# xCodeEval
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval

We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding examples from about 7.5 K unique problems covering up to 17 programming languages with execution-level parallelism. It features a total of seven tasks involving code understanding, generation, translation and retrieval, and it employs an execution-based evaluation. We develop a test-case based multilingual code execution engine, ExecEval that supports all the programming languages in xCodeEval. We also propose a novel data splitting and a data selection schema for balancing data distributions over multiple attributes based on geometric mean and graph-theoretic principle. 

This repository contains the sample code and data link for xCodeEval paper.# Data Download

Currently this repository supports huggingface 'load_dataset()' api. Follow the following example to load dataset for individual examples. passage: ## Hf large data download tricks.

If you are facing long delay with data processing, add a 'ignore_verifications=True'.



If you are facing long delay with data downloading, use huggingface streaming mode.## Just Give me the raw data ()

Data can be also downloaded as a git LFS repo from huggingface. 

!xCodeEval_hf

You can download the full data using the following command.



To download a specific part of the dataset, 




We propose 7 Tasks.

1. Tag Classification
2. Code Compilation
3. Program Synthesis
4. Code Translation
5. Automatic Program Repair
6. Code-Code Retrieval
7. NL-Code Retrieval# Common Data for different tasks

If you are not using huggingface 'load_dataset()' api, you may need to link some data with different tasks.

!xCodeEval_fig_1

We have two data files that are required for multiple tasks.

1. 'problem_descriptions.jsonl'
2. 'unittest_db.json'

You can find these two files in the root directory of the main branch of huggingface dataset repository. To avoid data redundancy we didn't include these data with the relevant tasks, rather we add a unique id 'src_uid' to retrieve these data.## Structure of 'problem_descriptions.jsonl'

A sample,