
# CodeParrot 🦜 Dataset Cleaned

## What is it?

A dataset of Python files from Github. This is the deduplicated version of the [codeparrot](https://huggingface.co/datasets/transformersbook/codeparrot).

## Processing
The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps:
- Deduplication 
  - Remove exact matches
- Filtering
  - Average line length < 100
  - Maximum line length < 1000
  - Alpha numeric characters fraction > 0.25
  - Remove auto-generated files (keyword search)
  
For more details see the preprocessing script in the transformers repository [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot).

## Splits
The dataset is split in a [train](https://huggingface.co/datasets/codeparrot/codeparrot-clean-train) and [validation](https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid) split used for training and evaluation.

## Structure
This dataset has ~50GB of code and 5361373 files.
```python
DatasetDict({
    train: Dataset({
        features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
        num_rows: 5361373
    })
})
```
python code region:us
{"tags": ["python", "code"]}
TAGS
#python #code #region-us 


# CodeParrot  Dataset Cleaned

## What is it?

A dataset of Python files from Github. This is the deduplicated version of the codeparrot.

## Processing
The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps:
- Deduplication 
  - Remove exact matches
- Filtering
  - Average line length < 100
  - Maximum line length < 1000
  - Alpha numeric characters fraction > 0.25
  - Remove auto-generated files (keyword search)
  
For more details see the preprocessing script in the transformers repository here.

## Splits
The dataset is split in a train and validation split used for training and evaluation.

## Structure
This dataset has ~50GB of code and 5361373 files.

# CodeParrot  Dataset Cleaned ## What is it?

A dataset of Python files from Github. This is the deduplicated version of the codeparrot. ## Processing
The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps:
- Deduplication 
  - Remove exact matches
- Filtering
  - Average line length < 100
  - Maximum line length < 1000
  - Alpha numeric characters fraction > 0.25
  - Remove auto-generated files (keyword search)
  
For more details see the preprocessing script in the transformers repository here. ## Splits
The dataset is split in a train and validation split used for training and evaluation. ## Structure
This dataset has ~50GB of code and 5361373 files.
TAGS
#python #code #region-us 
 # CodeParrot  Dataset Cleaned ## What is it?

A dataset of Python files from Github. This is the deduplicated version of the codeparrot. ## Processing
The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps:
- Deduplication 
  - Remove exact matches
- Filtering
  - Average line length < 100
  - Maximum line length < 1000
  - Alpha numeric characters fraction > 0.25
  - Remove auto-generated files (keyword search)
  
For more details see the preprocessing script in the transformers repository here. ## Splits
The dataset is split in a train and validation split used for training and evaluation. ## Structure
This dataset has ~50GB of code and 5361373 files.
passage: TAGS
#python #code #region-us 
# CodeParrot  Dataset Cleaned## What is it?

A dataset of Python files from Github. This is the deduplicated version of the codeparrot.## Processing
The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps:
- Deduplication 
  - Remove exact matches
- Filtering
  - Average line length < 100
  - Maximum line length < 1000
  - Alpha numeric characters fraction > 0.25
  - Remove auto-generated files (keyword search)
  
For more details see the preprocessing script in the transformers repository here.## Splits
The dataset is split in a train and validation split used for training and evaluation.## Structure
This dataset has ~50GB of code and 5361373 files.