
# Dataset Card for Shadertoys-fine

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Source Data](#source-data)
- [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** https://github.com/Vipitis/project (private placeholder)

### Dataset Summary

fine variant of the Shadertoys dataset (still WIP), where individual functions are avaialable as Datapoints.

### Supported Tasks and Leaderboards

 `language-modeling`: The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages.

### Languages

- English (names, comments)
- Shadercode **programming** language

## Dataset Structure

### Data Instances

A data point consists of the function string, it's name as well as a bit of metadata like the author and source URL. (in the future there might be a function string without comments)
```
{
  'name': '<type> <name>',
  'code': '<type> <name>(<inputs>) { <body> return <outputs>; }\n',
  'source': 'https://shadertoy.com/view/<shaderID>',
  'author': '<username>'
}
```

A data point in the `return_completion` subset for the return-completion task in [ShaderEval](https://huggingface.co/spaces/Vipitis/ShaderEval) includes just two features:
```
{
  'body': '<type> <name> <type> <name>(<inputs>) { <body> return',
  'return_statment': ' <outputs>: }\n',
}
```


### Data Fields

- 'name' funciton identifier composed of the type and the name of the function
- 'code' the raw code (including comments) of function.
- 'source' URL to the shader. It might be on a different renderpass
- 'author' username of the shader author

- 'body' the body of the function without the return statement (no comments)
- 'return_statment' the return statement of the function. everything infront of the semicolon is kept and white sapces are stripped in the custome Evaluator.

### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

These splits should be indexed the same across both subsets. So if you are fine-tuning on the `fine` subset you won't get exposed to the `return_completion` test split. However there are many duplicates among both subsets and splits.

## Dataset Creation

Data retrieved starting 2022-07-20

### Source Data

#### Initial Data Collection and Normalization

All data was collected via the [Shadertoy.com API](https://www.shadertoy.com/howto#q2) and then by looking for keywords and counting curly brackets to figure out what is part of a function and what isn't.

#### Who are the source language producers?

Shadertoy.com contributers which publish shaders as 'public+API'

## Licensing Information

The Default [licnese for each Shader](https://www.shadertoy.com/terms) is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached. The Dataset is currently not filtering for any licensis. 
task_categories:text-generation annotations_creators:no-annotation language_creators:machine-generated size_categories:100K<n<1M language:en language:code license:cc-by-nc-sa-3.0 code region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["machine-generated"], "language": ["en", "code"], "license": ["cc-by-nc-sa-3.0"], "multilinguality": [], "size_categories": ["100K<n<1M"], "source_datasets": [], "task_categories": ["text-generation"], "task_ids": [], "pretty_name": "Shadertoys-fine", "tags": ["code"], "dataset_info": [{"config_name": "default", "features": [{"name": "name", "dtype": "string"}, {"name": "code", "dtype": "string"}, {"name": "source", "dtype": "string"}, {"name": "author", "dtype": "string"}], "splits": [{"name": "train"}, {"name": "test"}], "download_size": 154529204, "dataset_size": 0}, {"config_name": "fine", "features": [{"name": "name", "dtype": "string"}, {"name": "code", "dtype": "string"}, {"name": "source", "dtype": "string"}, {"name": "author", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 119963236, "num_examples": 226910}, {"name": "test", "num_bytes": 20003783, "num_examples": 38356}], "download_size": 154529204, "dataset_size": 139967019}, {"config_name": "return_completion", "features": [{"name": "body", "dtype": "string"}, {"name": "return_statement", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 37597125, "num_examples": 84843}, {"name": "test", "num_bytes": 6360131, "num_examples": 14248}], "download_size": 154529204, "dataset_size": 43957256}]}
TAGS
#task_categories-text-generation #annotations_creators-no-annotation #language_creators-machine-generated #size_categories-100K<n<1M #language-English #language-code #license-cc-by-nc-sa-3.0 #code #region-us 


# Dataset Card for Shadertoys-fine

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information

## Dataset Description

- Repository: URL (private placeholder)

### Dataset Summary

fine variant of the Shadertoys dataset (still WIP), where individual functions are avaialable as Datapoints.

### Supported Tasks and Leaderboards

 'language-modeling': The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages.

### Languages

- English (names, comments)
- Shadercode programming language

## Dataset Structure

### Data Instances

A data point consists of the function string, it's name as well as a bit of metadata like the author and source URL. (in the future there might be a function string without comments)


A data point in the 'return_completion' subset for the return-completion task in ShaderEval includes just two features:



### Data Fields

- 'name' funciton identifier composed of the type and the name of the function
- 'code' the raw code (including comments) of function.
- 'source' URL to the shader. It might be on a different renderpass
- 'author' username of the shader author

- 'body' the body of the function without the return statement (no comments)
- 'return_statment' the return statement of the function. everything infront of the semicolon is kept and white sapces are stripped in the custome Evaluator.

### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

These splits should be indexed the same across both subsets. So if you are fine-tuning on the 'fine' subset you won't get exposed to the 'return_completion' test split. However there are many duplicates among both subsets and splits.

## Dataset Creation

Data retrieved starting 2022-07-20

### Source Data

#### Initial Data Collection and Normalization

All data was collected via the URL API and then by looking for keywords and counting curly brackets to figure out what is part of a function and what isn't.

#### Who are the source language producers?

URL contributers which publish shaders as 'public+API'

## Licensing Information

The Default licnese for each Shader is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached. The Dataset is currently not filtering for any licensis. 
# Dataset Card for Shadertoys-fine ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information ## Dataset Description

- Repository: URL (private placeholder) ### Dataset Summary

fine variant of the Shadertoys dataset (still WIP), where individual functions are avaialable as Datapoints. ### Supported Tasks and Leaderboards

 'language-modeling': The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages. ### Languages

- English (names, comments)
- Shadercode programming language ## Dataset Structure ### Data Instances

A data point consists of the function string, it's name as well as a bit of metadata like the author and source URL. (in the future there might be a function string without comments)


A data point in the 'return_completion' subset for the return-completion task in ShaderEval includes just two features: ### Data Fields

- 'name' funciton identifier composed of the type and the name of the function
- 'code' the raw code (including comments) of function.
- 'source' URL to the shader. It might be on a different renderpass
- 'author' username of the shader author

- 'body' the body of the function without the return statement (no comments)
- 'return_statment' the return statement of the function. everything infront of the semicolon is kept and white sapces are stripped in the custome Evaluator. ### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

These splits should be indexed the same across both subsets. So if you are fine-tuning on the 'fine' subset you won't get exposed to the 'return_completion' test split. However there are many duplicates among both subsets and splits. ## Dataset Creation

Data retrieved starting 2022-07-20 ### Source Data #### Initial Data Collection and Normalization

All data was collected via the URL API and then by looking for keywords and counting curly brackets to figure out what is part of a function and what isn't. #### Who are the source language producers?

URL contributers which publish shaders as 'public+API' ## Licensing Information

The Default licnese for each Shader is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached. The Dataset is currently not filtering for any licensis.
TAGS
#task_categories-text-generation #annotations_creators-no-annotation #language_creators-machine-generated #size_categories-100K<n<1M #language-English #language-code #license-cc-by-nc-sa-3.0 #code #region-us 
 # Dataset Card for Shadertoys-fine ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information ## Dataset Description

- Repository: URL (private placeholder) ### Dataset Summary

fine variant of the Shadertoys dataset (still WIP), where individual functions are avaialable as Datapoints. ### Supported Tasks and Leaderboards

 'language-modeling': The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages. ### Languages

- English (names, comments)
- Shadercode programming language ## Dataset Structure ### Data Instances

A data point consists of the function string, it's name as well as a bit of metadata like the author and source URL. (in the future there might be a function string without comments)


A data point in the 'return_completion' subset for the return-completion task in ShaderEval includes just two features: ### Data Fields

- 'name' funciton identifier composed of the type and the name of the function
- 'code' the raw code (including comments) of function.
- 'source' URL to the shader. It might be on a different renderpass
- 'author' username of the shader author

- 'body' the body of the function without the return statement (no comments)
- 'return_statment' the return statement of the function. everything infront of the semicolon is kept and white sapces are stripped in the custome Evaluator. ### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

These splits should be indexed the same across both subsets. So if you are fine-tuning on the 'fine' subset you won't get exposed to the 'return_completion' test split. However there are many duplicates among both subsets and splits. ## Dataset Creation

Data retrieved starting 2022-07-20 ### Source Data #### Initial Data Collection and Normalization

All data was collected via the URL API and then by looking for keywords and counting curly brackets to figure out what is part of a function and what isn't. #### Who are the source language producers?

URL contributers which publish shaders as 'public+API' ## Licensing Information

The Default licnese for each Shader is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached. The Dataset is currently not filtering for any licensis.