
# Dataset Card for Shadertoys

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Source Data](#source-data)
- [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** https://github.com/Vipitis/project (private placeholder)

### Dataset Summary

The Shadertoys dataset contains over 44k renderpasses collected from the Shadertoy.com API. Some shader programm contain multiple render passes.
To browse a subset of this dataset, look at the [ShaderEval](https://huggingface.co/spaces/Vipitis/ShaderCoder) space. A finer variant of this dataset is [Shadertoys-fine](https://huggingface.co/datasets/Vipitis/Shadertoys-fine).

### Supported Tasks and Leaderboards

 `text-generation` the dataset can be used to train generative language models, for code completion tasks.
 `ShaderEval` [task1](https://huggingface.co/spaces/Vipitis/ShaderEval) from ShaderEval uses a dataset derived from Shadertoys to test return completion of autoregressive language models.

### Languages

- English (title, description, tags, comments)
- Shadercode **programming** language, a subset of GLSL specifically for Shadertoy.com

## Dataset Structure

### Data Instances

A data point consists of the whole shadercode, some information from the API as well as additional metadata.
```
{
 'num_passes': 1,
 'has_inputs': False,
 'name': 'Image',
 'type': 'image',
 'code': '<full code>',
 'title': '<title of the shader>',
 'description': '<description of the shader>',
 'tags': ['tag1','tag2','tag3', ... ],
 'license': 'unknown',
 'author': '<username>',
 'source': 'https://shadertoy.com/view/<shaderID>'
}
```

### Data Fields
- 'num_passes' number of passes the parent shader program has
- 'has_inputs' if any inputs were used like textures, audio streams,
- 'name' Name of the renderpass, usually Image, Buffer A, Common, etc
- 'type' type of the renderpass; one of `{'buffer', 'common', 'cubemap', 'image', 'sound'}`
- 'code' the raw code (including comments) the whole renderpass.
- 'title' Name of the Shader
- 'description' description given for the Shader
- 'tags' List of tags assigned to the Shader (by it's creator); there are more than 10000 unique tags.
- 'license' currently in development
- 'author' username of the shader author
- 'source' URL to the shader. Not to the specific renderpass.

### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

## Dataset Creation

Data retrieved starting 2022-07-20

### Source Data

#### Initial Data Collection and Normalization

All data was collected via the [Shadertoy.com API](https://www.shadertoy.com/howto#q2) and then iterated over the items in 'renderpass' while adding some of the fields from 'info'.
The code to generate these datasets should be published on the GitHub repository in the near future. 

#### Who are the source language producers?

Shadertoy.com contributers which publish shaders as 'public+API'

## Licensing Information

The Default [license for each Shader](https://www.shadertoy.com/terms) is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached.
The Dataset is currently not filtering for any licenses but gives a license tag, if easily recognizeable by naive means.
Please check the first comment of each shader program yourself as to not violate any copyrights for downstream use. The main license requires share alike and by attribution.
Attribution of every data field can be found in the 'author' column, but might not include further attribution within the code itself or parents from forked shaders.
task_categories:text-generation task_categories:text-to-image annotations_creators:no-annotation language_creators:machine-generated size_categories:10K<n<100K language:en language:code license:cc-by-nc-sa-3.0 code region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["machine-generated"], "language": ["en", "code"], "license": ["cc-by-nc-sa-3.0"], "multilinguality": [], "size_categories": ["10K<n<100K"], "source_datasets": [], "task_categories": ["text-generation", "text-to-image"], "task_ids": [], "pretty_name": "Shadertoys", "tags": ["code"], "dataset_info": {"features": [{"name": "num_passes", "dtype": "int64"}, {"name": "has_inputs", "dtype": "bool"}, {"name": "name", "dtype": "string"}, {"name": "type", "dtype": "string"}, {"name": "code", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "description", "dtype": "string"}, {"name": "tags", "sequence": "string"}, {"name": "author", "dtype": "string"}, {"name": "license", "dtype": "string"}, {"name": "source", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 162960894, "num_examples": 37841}, {"name": "test", "num_bytes": 26450429, "num_examples": 6617}], "download_size": 86294414, "dataset_size": 189411323}}
TAGS
#task_categories-text-generation #task_categories-text-to-image #annotations_creators-no-annotation #language_creators-machine-generated #size_categories-10K<n<100K #language-English #language-code #license-cc-by-nc-sa-3.0 #code #region-us 


# Dataset Card for Shadertoys

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information

## Dataset Description

- Repository: URL (private placeholder)

### Dataset Summary

The Shadertoys dataset contains over 44k renderpasses collected from the URL API. Some shader programm contain multiple render passes.
To browse a subset of this dataset, look at the ShaderEval space. A finer variant of this dataset is Shadertoys-fine.

### Supported Tasks and Leaderboards

 'text-generation' the dataset can be used to train generative language models, for code completion tasks.
 'ShaderEval' task1 from ShaderEval uses a dataset derived from Shadertoys to test return completion of autoregressive language models.

### Languages

- English (title, description, tags, comments)
- Shadercode programming language, a subset of GLSL specifically for URL

## Dataset Structure

### Data Instances

A data point consists of the whole shadercode, some information from the API as well as additional metadata.


### Data Fields
- 'num_passes' number of passes the parent shader program has
- 'has_inputs' if any inputs were used like textures, audio streams,
- 'name' Name of the renderpass, usually Image, Buffer A, Common, etc
- 'type' type of the renderpass; one of '{'buffer', 'common', 'cubemap', 'image', 'sound'}'
- 'code' the raw code (including comments) the whole renderpass.
- 'title' Name of the Shader
- 'description' description given for the Shader
- 'tags' List of tags assigned to the Shader (by it's creator); there are more than 10000 unique tags.
- 'license' currently in development
- 'author' username of the shader author
- 'source' URL to the shader. Not to the specific renderpass.

### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%)

## Dataset Creation

Data retrieved starting 2022-07-20

### Source Data

#### Initial Data Collection and Normalization

All data was collected via the URL API and then iterated over the items in 'renderpass' while adding some of the fields from 'info'.
The code to generate these datasets should be published on the GitHub repository in the near future. 

#### Who are the source language producers?

URL contributers which publish shaders as 'public+API'

## Licensing Information

The Default license for each Shader is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached.
The Dataset is currently not filtering for any licenses but gives a license tag, if easily recognizeable by naive means.
Please check the first comment of each shader program yourself as to not violate any copyrights for downstream use. The main license requires share alike and by attribution.
Attribution of every data field can be found in the 'author' column, but might not include further attribution within the code itself or parents from forked shaders.
# Dataset Card for Shadertoys ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information ## Dataset Description

- Repository: URL (private placeholder) ### Dataset Summary

The Shadertoys dataset contains over 44k renderpasses collected from the URL API. Some shader programm contain multiple render passes.
To browse a subset of this dataset, look at the ShaderEval space. A finer variant of this dataset is Shadertoys-fine. ### Supported Tasks and Leaderboards

 'text-generation' the dataset can be used to train generative language models, for code completion tasks.
 'ShaderEval' task1 from ShaderEval uses a dataset derived from Shadertoys to test return completion of autoregressive language models. ### Languages

- English (title, description, tags, comments)
- Shadercode programming language, a subset of GLSL specifically for URL ## Dataset Structure ### Data Instances

A data point consists of the whole shadercode, some information from the API as well as additional metadata. ### Data Fields
- 'num_passes' number of passes the parent shader program has
- 'has_inputs' if any inputs were used like textures, audio streams,
- 'name' Name of the renderpass, usually Image, Buffer A, Common, etc
- 'type' type of the renderpass; one of '{'buffer', 'common', 'cubemap', 'image', 'sound'}'
- 'code' the raw code (including comments) the whole renderpass.
- 'title' Name of the Shader
- 'description' description given for the Shader
- 'tags' List of tags assigned to the Shader (by it's creator); there are more than 10000 unique tags.
- 'license' currently in development
- 'author' username of the shader author
- 'source' URL to the shader. Not to the specific renderpass. ### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%) ## Dataset Creation

Data retrieved starting 2022-07-20 ### Source Data #### Initial Data Collection and Normalization

All data was collected via the URL API and then iterated over the items in 'renderpass' while adding some of the fields from 'info'.
The code to generate these datasets should be published on the GitHub repository in the near future. #### Who are the source language producers?

URL contributers which publish shaders as 'public+API' ## Licensing Information

The Default license for each Shader is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached.
The Dataset is currently not filtering for any licenses but gives a license tag, if easily recognizeable by naive means.
Please check the first comment of each shader program yourself as to not violate any copyrights for downstream use. The main license requires share alike and by attribution.
Attribution of every data field can be found in the 'author' column, but might not include further attribution within the code itself or parents from forked shaders.
TAGS
#task_categories-text-generation #task_categories-text-to-image #annotations_creators-no-annotation #language_creators-machine-generated #size_categories-10K<n<100K #language-English #language-code #license-cc-by-nc-sa-3.0 #code #region-us 
 # Dataset Card for Shadertoys ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information ## Dataset Description

- Repository: URL (private placeholder) ### Dataset Summary

The Shadertoys dataset contains over 44k renderpasses collected from the URL API. Some shader programm contain multiple render passes.
To browse a subset of this dataset, look at the ShaderEval space. A finer variant of this dataset is Shadertoys-fine. ### Supported Tasks and Leaderboards

 'text-generation' the dataset can be used to train generative language models, for code completion tasks.
 'ShaderEval' task1 from ShaderEval uses a dataset derived from Shadertoys to test return completion of autoregressive language models. ### Languages

- English (title, description, tags, comments)
- Shadercode programming language, a subset of GLSL specifically for URL ## Dataset Structure ### Data Instances

A data point consists of the whole shadercode, some information from the API as well as additional metadata. ### Data Fields
- 'num_passes' number of passes the parent shader program has
- 'has_inputs' if any inputs were used like textures, audio streams,
- 'name' Name of the renderpass, usually Image, Buffer A, Common, etc
- 'type' type of the renderpass; one of '{'buffer', 'common', 'cubemap', 'image', 'sound'}'
- 'code' the raw code (including comments) the whole renderpass.
- 'title' Name of the Shader
- 'description' description given for the Shader
- 'tags' List of tags assigned to the Shader (by it's creator); there are more than 10000 unique tags.
- 'license' currently in development
- 'author' username of the shader author
- 'source' URL to the shader. Not to the specific renderpass. ### Data Splits

Currently available (shuffled):
  - train (85.0%)
  - test (15.0%) ## Dataset Creation

Data retrieved starting 2022-07-20 ### Source Data #### Initial Data Collection and Normalization

All data was collected via the URL API and then iterated over the items in 'renderpass' while adding some of the fields from 'info'.
The code to generate these datasets should be published on the GitHub repository in the near future. #### Who are the source language producers?

URL contributers which publish shaders as 'public+API' ## Licensing Information

The Default license for each Shader is CC BY-NC-SA 3.0. However, some Shaders might have a different license attached.
The Dataset is currently not filtering for any licenses but gives a license tag, if easily recognizeable by naive means.
Please check the first comment of each shader program yourself as to not violate any copyrights for downstream use. The main license requires share alike and by attribution.
Attribution of every data field can be found in the 'author' column, but might not include further attribution within the code itself or parents from forked shaders.
passage: TAGS
#task_categories-text-generation #task_categories-text-to-image #annotations_creators-no-annotation #language_creators-machine-generated #size_categories-10K<n<100K #language-English #language-code #license-cc-by-nc-sa-3.0 #code #region-us 
# Dataset Card for Shadertoys## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Source Data
- Licensing Information## Dataset Description

- Repository: URL (private placeholder)### Dataset Summary

The Shadertoys dataset contains over 44k renderpasses collected from the URL API. Some shader programm contain multiple render passes.
To browse a subset of this dataset, look at the ShaderEval space. A finer variant of this dataset is Shadertoys-fine.### Supported Tasks and Leaderboards

 'text-generation' the dataset can be used to train generative language models, for code completion tasks.
 'ShaderEval' task1 from ShaderEval uses a dataset derived from Shadertoys to test return completion of autoregressive language models.### Languages

- English (title, description, tags, comments)
- Shadercode programming language, a subset of GLSL specifically for URL## Dataset Structure### Data Instances

A data point consists of the whole shadercode, some information from the API as well as additional metadata.