task_categories:text-generation task_ids:language-modeling annotations_creators:crowdsourced language_creators:crowdsourced size_categories:100K<n<1M language:code license:other code, swift, native iOS development doi:10.57967/hf/0778 region:us
---
annotations_creators: 
- crowdsourced
license: other
language_creators:
- crowdsourced
language:
- code
task_categories:
- text-generation
tags:
- code, swift, native iOS development
size_categories:
- 100K<n<1M
source_datasets: []
pretty_name: iva-swift-codeint-raw
task_ids:
- language-modeling
---

# IVA Swift GitHub Code Dataset

## Dataset Description

This is the raw IVA Swift dataset extracted from GitHub. 
It contains uncurated Swift files gathered with the purpose to train a code generation model.

The dataset consists of 753693 swift code files from GitHub totaling ~700MB of data. 
The dataset was created from the public GitHub dataset on Google BiqQuery.

### How to use it

To download the full dataset:

```python
from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-swift-codeint', split='train')
```

```python

from datasets import load_dataset
dataset = load_dataset('mvasiliniuc/iva-swift-codeint', split='train')
print(dataset[77723])

#OUTPUT:
{
   "repo_name":"simpleandpretty/decider-ios",
   "path":"MessagesExtension/MediaResources.swift",
   "copies":"1",
   "size":"1232",
   "content":"import Foundation\nimport UIKit\n\nclass MediaResources {\n\n    static func mediaURL(forGameOption option:FightMove) -> URL {\n        let bundle = Bundle.main\n        guard\n            let mediaURL = bundle.url(forResource: option.rawValue, withExtension: \"mp4\")\n ...",
   "license":"gpl-3.0"
}
```

## Data Structure

### Data Fields

|Field|Type|Description|
|---|---|---|
|repo_name|string|name of the GitHub repository|
|path|string|path of the file in GitHub repository|
|copies|string|number of occurrences in dataset|
|code|string|content of source file|
|size|string|size of the source file in bytes|
|license|string|license of GitHub repository|

### Instance

```json
{
   "repo_name":"simpleandpretty/decider-ios",
   "path":"MessagesExtension/MediaResources.swift",
   "copies":"1",
   "size":"1232",
   "content":"import Foundation\nimport UIKit\n\nclass MediaResources {\n\n    static func mediaURL(forGameOption option:FightMove) -> URL {\n        let bundle = Bundle.main\n        guard\n            let mediaURL = bundle.url(forResource: option.rawValue, withExtension: \"mp4\")\n ...",
   "license":"gpl-3.0"
}
```

## Languages

The dataset contains only Swift files.

```json
{
    "Swift": [".swift"]
}
```

## Licenses

Each entry in the dataset contains the associated license. The following is a list of licenses involved and their occurrences.

```json
{
    "agpl-3.0": 2775,
    "apache-2.0": 180178,
    "artistic-2.0": 314,
    "bsd-2-clause": 5342,
    "bsd-3-clause": 11429,
    "cc0-1.0": 2718,
    "epl-1.0": 980,
    "gpl-2.0": 15751,
    "gpl-3.0": 33074,
    "isc": 1647,
    "lgpl-2.1": 1741,
    "lgpl-3.0": 6150,
    "mit": 476518,
    "mpl-2.0": 11799,
    "unlicense": 3277
}
```

## Dataset Statistics

```json
{
    "Total size": "~712 MB",
    "Number of files": 753693,
    "Number of files under 500 bytes": 129827,
    "Average file size in bytes": 4245,
}
```

## Dataset Creation

The dataset was created using Google Query for Github: 
https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code

The following steps were pursued for data
gathering:
1. Creation of a dataset and a table in Google Big Query Project.
2. Creation of a bucket in Google Cloud Storage.
3. Creation of a query in Google Big Query Project.
4. Running the query with the setting to output the results in the dataset and table
created at step one.
5. Exporting the resulting dataset into the bucket created in step 2. Export format of JSON with gzip compression.

The result of these steps leads to the following results:
* 2.7 TB Processed,
* number of extracted rows/Swift files was 464,215
* total logical bytes 1.46 GB.
* The result amounts to 7 json.gz files in a total of 700 MB

The SQL Query used is:
```sql
SELECT
  f.repo_name, f.path, c.copies, c.size, c.content, l.license
FROM
 (select f.*, row_number() over (partition by id order by path desc) as seqnum from `bigquery-public-data.github_repos.files` AS f) f
JOIN
  `bigquery-public-data.github_repos.contents` AS c
ON
  f.id = c.id AND seqnum=1
JOIN
  `bigquery-public-data.github_repos.licenses` AS l
ON
  f.repo_name = l.repo_name
WHERE
  NOT c.binary AND ((f.path LIKE '%.swift') AND (c.size BETWEEN 0 AND 1048575))
```

## Data Splits
The dataset only contains a train split.

Using the curated version of this dataset, a split was made into multiple repositories:
* Clean Version: https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean
* Clean Version Train: https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean-train
* Clean Version Valid: https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint-clean-valid

# Considerations for Using the Data

The dataset comprises source code from various repositories, potentially containing harmful or biased code, 
along with sensitive information such as passwords or usernames.

# Additional Information

## Dataset Curators

[mircea.dev@icloud.com](mircea.dev@icloud.com)

## Licensing Information

* The license of this open-source dataset is: other.
* The dataset is gathered from open-source repositories on [GitHub using BigQuery](https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code). 
* Find the license of each entry in the dataset in the corresponding license column.

## Citation Information

```json
@misc {mircea_vasiliniuc_2023,
	author       = { {Mircea Vasiliniuc} },
	title        = { iva-swift-codeint (Revision c09ebf8) },
	year         = 2023,
	url          = { https://huggingface.co/datasets/mvasiliniuc/iva-swift-codeint },
	doi          = { 10.57967/hf/0778 },
	publisher    = { Hugging Face }
}
```