
# IVA Kotlin GitHub Code Dataset

## Dataset Description

This is the raw IVA Kotlin dataset extracted from GitHub. 
It contains uncurated Kotlin files gathered with the purpose to train a code generation model.

The dataset consists of 464215 kotlin code files from GitHub totaling ~361 MB of data. 
The dataset was created from the public GitHub dataset on Google BiqQuery.

### How to use it

To download the full dataset:

```python
from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-kotlin-codeint', split='train')
```

```python

from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-kotlin-codeint', split='train')
print(dataset[723])

#OUTPUT:
{
   "repo_name":"nemerosa/ontrack",
   "path":"ontrack-extension-notifications/src/main/java/net/nemerosa/ontrack/extension/notifications/webhooks/WebhookController.kt",
   "copies":"1",
   "size":"3248",
   "content":"...@RestController\n@RequestMapping(\"/extension/notifications/webhook\")\nclass WebhookController(\n    private val webhookAdminService: WebhookAdminService,\n    private val webhookExecutionService: ",
   "license":"mit"
}
```

## Data Structure

### Data Fields

|Field|Type|Description|
|---|---|---|
|repo_name|string|name of the GitHub repository|
|path|string|path of the file in GitHub repository|
|copies|string|number of occurrences in dataset|
|code|string|content of source file|
|size|string|size of the source file in bytes|
|license|string|license of GitHub repository|

### Instance

```json
{
   "repo_name":"nemerosa/ontrack",
   "path":"ontrack-extension-notifications/src/main/java/net/nemerosa/ontrack/extension/notifications/webhooks/WebhookController.kt",
   "copies":"1",
   "size":"3248",
   "content":"...@RestController\n@RequestMapping(\"/extension/notifications/webhook\")\nclass WebhookController(\n    private val webhookAdminService: WebhookAdminService,\n    private val webhookExecutionService: ",
   "license":"mit"
}
```

## Languages

The dataset contains only Kotlin files.

```json
{
    "Kotlin": [".kt"]
}
```

## Licenses

Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.

```json
{
    "agpl-3.0": 9146,
    "apache-2.0": 272388,
    "artistic-2.0": 219,
    "bsd-2-clause": 896,
    "bsd-3-clause": 12328,
    "cc0-1.0": 411,
    "epl-1.0": 2111,
    "gpl-2.0": 11080,
    "gpl-3.0": 48911,
    "isc": 997,
    "lgpl-2.1": 297,
    "lgpl-3.0": 7749,
    "mit": 92540,
    "mpl-2.0": 3386,
    "unlicense": 1756
}
```

## Dataset Statistics

```json
{
    "Total size": "~361 MB",
    "Number of files": 464215,
    "Number of files under 500 bytes": 99845,
    "Average file size in bytes": 3252,
}
```

## Dataset Creation

The dataset was created using Google Query for Github: 
https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code

The following steps were pursued for data
gathering:
1. Creation of a dataset and a table in Google Big Query Project.
2. Creation of a bucket in Google Cloud Storage.
3. Creation of a query in Google Big Query Project.
4. Running the query with the setting to output the results in the dataset and table
created at step one.
5. Exporting the resulting dataset into the bucket created in step 2. Export format of JSON with gzip compression.

The result of these steps leads to the following results:
* 2.7 TB Processed, 
* number of extracted rows/files was 464,215 
* total logical bytes  1.46 GB. 
* the result amounts to 7 json.gz files in a total of 361 MB.

The SQL Query used is:
```sql
SELECT
  f.repo_name, f.path, c.copies, c.size, c.content, l.license
FROM
 (select f.*, row_number() over (partition by id order by path desc) as seqnum from `bigquery-public-data.github_repos.files` AS f) f
JOIN
  `bigquery-public-data.github_repos.contents` AS c
ON
  f.id = c.id AND seqnum=1
JOIN
  `bigquery-public-data.github_repos.licenses` AS l
ON
  f.repo_name = l.repo_name
WHERE
  NOT c.binary AND ((f.path LIKE '%.kt') AND (c.size BETWEEN 0 AND 1048575))
```

## Data Splits
The dataset only contains a train split.

Using the curated version of this dataset, a split was made into multiple repositories:
* Clean Version: https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint-clean
* Clean Version Train: https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint-clean-train
* Clean Version Valid: https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint-clean-valid

# Considerations for Using the Data

The dataset comprises source code from various repositories, potentially containing harmful or biased code, 
along with sensitive information such as passwords or usernames.

# Additional Information

## Dataset Curators

[mircea.dev@icloud.com](mircea.dev@icloud.com)

## Licensing Information

* The license of this open-source dataset is: other.
* The dataset is gathered from open-source repositories on [GitHub using BigQuery](https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code). 
* Find the license of each entry in the dataset in the corresponding license column.

## Citation Information

```json
@misc {mircea_vasiliniuc_2023,
	author       = { {Mircea Vasiliniuc} },
	title        = { iva-kotlin-codeint (Revision 1af5124) },
	year         = 2023,
	url          = { https://huggingface.co/datasets/mvasiliniuc/iva-kotlin-codeint },
	doi          = { 10.57967/hf/0779 },
	publisher    = { Hugging Face }
}
```
task_categories:text-generation task_ids:language-modeling annotations_creators:crowdsourced language_creators:crowdsourced size_categories:100K<n<1M language:code license:other code, kotlin, native Android development doi:10.57967/hf/0779 region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["crowdsourced"], "language": ["code"], "license": "other", "size_categories": ["100K<n<1M"], "source_datasets": [], "task_categories": ["text-generation"], "task_ids": ["language-modeling"], "pretty_name": "iva-kotlin-codeint-raw", "tags": ["code, kotlin, native Android development"]}
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-crowdsourced #language_creators-crowdsourced #size_categories-100K<n<1M #language-code #license-other #code, kotlin, native Android development #doi-10.57967/hf/0779 #region-us 

IVA Kotlin GitHub Code Dataset
==============================


Dataset Description
-------------------


This is the raw IVA Kotlin dataset extracted from GitHub.
It contains uncurated Kotlin files gathered with the purpose to train a code generation model.


The dataset consists of 464215 kotlin code files from GitHub totaling ~361 MB of data.
The dataset was created from the public GitHub dataset on Google BiqQuery.


### How to use it


To download the full dataset:


Data Structure
--------------


### Data Fields


Field: repo\_name, Type: string, Description: name of the GitHub repository
Field: path, Type: string, Description: path of the file in GitHub repository
Field: copies, Type: string, Description: number of occurrences in dataset
Field: code, Type: string, Description: content of source file
Field: size, Type: string, Description: size of the source file in bytes
Field: license, Type: string, Description: license of GitHub repository


### Instance


Languages
---------


The dataset contains only Kotlin files.


Licenses
--------


Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.


Dataset Statistics
------------------


Dataset Creation
----------------


The dataset was created using Google Query for Github:
URL


The following steps were pursued for data
gathering:


1. Creation of a dataset and a table in Google Big Query Project.
2. Creation of a bucket in Google Cloud Storage.
3. Creation of a query in Google Big Query Project.
4. Running the query with the setting to output the results in the dataset and table
created at step one.
5. Exporting the resulting dataset into the bucket created in step 2. Export format of JSON with gzip compression.


The result of these steps leads to the following results:


* 2.7 TB Processed,
* number of extracted rows/files was 464,215
* total logical bytes 1.46 GB.
* the result amounts to 7 URL files in a total of 361 MB.


The SQL Query used is:


Data Splits
-----------


The dataset only contains a train split.


Using the curated version of this dataset, a split was made into multiple repositories:


* Clean Version: URL
* Clean Version Train: URL
* Clean Version Valid: URL


Considerations for Using the Data
=================================


The dataset comprises source code from various repositories, potentially containing harmful or biased code,
along with sensitive information such as passwords or usernames.


Additional Information
======================


Dataset Curators
----------------


URL@URL


Licensing Information
---------------------


* The license of this open-source dataset is: other.
* The dataset is gathered from open-source repositories on GitHub using BigQuery.
* Find the license of each entry in the dataset in the corresponding license column.



### How to use it


To download the full dataset:


Data Structure
-------------- ### Data Fields


Field: repo\_name, Type: string, Description: name of the GitHub repository
Field: path, Type: string, Description: path of the file in GitHub repository
Field: copies, Type: string, Description: number of occurrences in dataset
Field: code, Type: string, Description: content of source file
Field: size, Type: string, Description: size of the source file in bytes
Field: license, Type: string, Description: license of GitHub repository ### Instance


Languages
---------


The dataset contains only Kotlin files.


Licenses
--------


Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.


Dataset Statistics
------------------


Dataset Creation
----------------


The dataset was created using Google Query for Github:
URL


The following steps were pursued for data
gathering:


1. Creation of a dataset and a table in Google Big Query Project.
2. Creation of a bucket in Google Cloud Storage.
3. Creation of a query in Google Big Query Project.
4. Running the query with the setting to output the results in the dataset and table
created at step one.
5. Exporting the resulting dataset into the bucket created in step 2. Export format of JSON with gzip compression.


The result of these steps leads to the following results:


* 2.7 TB Processed,
* number of extracted rows/files was 464,215
* total logical bytes 1.46 GB.
* the result amounts to 7 URL files in a total of 361 MB.


The SQL Query used is:


Data Splits
-----------


The dataset only contains a train split.


Using the curated version of this dataset, a split was made into multiple repositories:


* Clean Version: URL
* Clean Version Train: URL
* Clean Version Valid: URL


Considerations for Using the Data
=================================


The dataset comprises source code from various repositories, potentially containing harmful or biased code,
along with sensitive information such as passwords or usernames.


Additional Information
======================


Dataset Curators
----------------


URL@URL


Licensing Information
---------------------


* The license of this open-source dataset is: other.
* The dataset is gathered from open-source repositories on GitHub using BigQuery.
* Find the license of each entry in the dataset in the corresponding license column.
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-crowdsourced #language_creators-crowdsourced #size_categories-100K<n<1M #language-code #license-other #code, kotlin, native Android development #doi-10.57967/hf/0779 #region-us 
 ### How to use it


To download the full dataset:


Data Structure
-------------- ### Data Fields


Field: repo\_name, Type: string, Description: name of the GitHub repository
Field: path, Type: string, Description: path of the file in GitHub repository
Field: copies, Type: string, Description: number of occurrences in dataset
Field: code, Type: string, Description: content of source file
Field: size, Type: string, Description: size of the source file in bytes
Field: license, Type: string, Description: license of GitHub repository ### Instance


Languages
---------


The dataset contains only Kotlin files.


Licenses
--------


Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.


Dataset Statistics
------------------


Dataset Creation
----------------


The dataset was created using Google Query for Github:
URL


The following steps were pursued for data
gathering:


1. Creation of a dataset and a table in Google Big Query Project.
2. Creation of a bucket in Google Cloud Storage.
3. Creation of a query in Google Big Query Project.
4. Running the query with the setting to output the results in the dataset and table
created at step one.
5. Exporting the resulting dataset into the bucket created in step 2. Export format of JSON with gzip compression.


The result of these steps leads to the following results:


* 2.7 TB Processed,
* number of extracted rows/files was 464,215
* total logical bytes 1.46 GB.
* the result amounts to 7 URL files in a total of 361 MB.


The SQL Query used is:


Data Splits
-----------


The dataset only contains a train split.


Using the curated version of this dataset, a split was made into multiple repositories:


* Clean Version: URL
* Clean Version Train: URL
* Clean Version Valid: URL


Considerations for Using the Data
=================================


The dataset comprises source code from various repositories, potentially containing harmful or biased code,
along with sensitive information such as passwords or usernames.


Additional Information
======================


Dataset Curators
----------------


URL@URL


Licensing Information
---------------------


* The license of this open-source dataset is: other.
* The dataset is gathered from open-source repositories on GitHub using BigQuery.
* Find the license of each entry in the dataset in the corresponding license column.
passage: TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-crowdsourced #language_creators-crowdsourced #size_categories-100K<n<1M #language-code #license-other #code, kotlin, native Android development #doi-10.57967/hf/0779 #region-us 
### How to use it


To download the full dataset:


Data Structure
--------------### Data Fields


Field: repo\_name, Type: string, Description: name of the GitHub repository
Field: path, Type: string, Description: path of the file in GitHub repository
Field: copies, Type: string, Description: number of occurrences in dataset
Field: code, Type: string, Description: content of source file
Field: size, Type: string, Description: size of the source file in bytes
Field: license, Type: string, Description: license of GitHub repository