# Dataset Card for "ts-eval"

This is a dataset of 744 high-quality TypeScript files, meant for evaluation of type prediction systems. The dataset is derived from TypeScript portion of [The Stack (dedup)](https://huggingface.co/datasets/bigcode/the-stack-dedup), version 1.1, and filtered.

## Filtering steps

We remove files that:

  1. Depend on external modules and do not type check.
  2. Have 0 type annotation locations.
  3. Have 50 or fewer lines of code.
  4. Have 0 functions.
  5. Average fewer than 5 lines of code per function.

Next, we compute a weighted quality score, based on the following factors:

  * function annotation density
  * variable annotation density
  * type definition density
  * trivial types density
  * predefined types density
  * lines of code per function
  * number of function usages

Files with a score 1 or more standard deviations below the mean are removed.

Next, we apply the training cutoff (December 31, 2021).

Finally, we remove type annotations. Some files become invalid during this process, so we exclude them from the dataset.

## Data fields

Most of the data fields come from [The Stack](https://huggingface.co/datasets/bigcode/the-stack/blob/main/README.md#data-fields).

We add the following fields:

  * `loc` (integer): lines of code, excluding comments and blanks
  * `functions` (integer): number of functions that have function bodies; includes function expressions, arrow functions, and methods
  * `function_signatures` (integer): number of function signatures, i.e., function declarations with no bodies
  * `function_parameters` (integer): number of function parameters
  * `variable_declarations` (integer): number of variable declarations
  * `property_declarations` (integer): number of property declarations, i.e., within classes and interfaces
  * `function_usages` (integer): number of functions (defined within the file) that are called
  * `trivial_types` (integer): number of trivial type annotations, e.g. `any`, `Function`
  * `predefined_types` (integer): number of predefined type annotations, e.g. `boolean`, `number`, `string`
  * `type_definitions` (integer): number of type definitions, e.g. classes, interfaces, type aliases
  * `dynamism_heuristic` (integer): a count of dynamic language features, e.g. `eval`, run-time type tests
  * `loc_per_function` (float): average lines of code per function
  * `estimated_tokens` (integer): estimated token count, using the SantaCoder tokenizer
  * `fun_ann_density` (float): a measure of function annotation density
  * `var_ann_density` (float): a measure of variable annotation density
  * `prop_ann_density` (float): a measure of property annotation density
  * `typedef_density` (float): a measure of type definition density
  * `dynamism_density` (float): a measure of dynamic feature usage density
  * `trivial_density` (float): a measure of trivial type annotation density
  * `predefined_density` (float): a measure of predefined type annotation density
  * `metric` (float): the quality score
  * `content_without_annotations` (string): content of the file, with type annotations removed

## Versions

The default version (`main`) is `v1.1`.

|Version|Description|
|-|-|
|`v1.1`       | Original version of the evaluation dataset, based on v1.1 of the Stack. Applies the training cutoff (December 31, 2021). |
|`v1.1full`   | Evaluation dataset, based on v1.1 of the Stack. Does not apply the training cutoff. |
|`v1.1subset` | A subset of v1.1 containing only 50 files. |

region:us
{"dataset_info": {"features": [{"name": "hexsha", "dtype": "string"}, {"name": "size", "dtype": "int64"}, {"name": "ext", "dtype": "string"}, {"name": "lang", "dtype": "string"}, {"name": "max_stars_repo_path", "dtype": "string"}, {"name": "max_stars_repo_name", "dtype": "string"}, {"name": "max_stars_repo_head_hexsha", "dtype": "string"}, {"name": "max_stars_repo_licenses", "sequence": "string"}, {"name": "max_stars_count", "dtype": "float64"}, {"name": "max_stars_repo_stars_event_min_datetime", "dtype": "string"}, {"name": "max_stars_repo_stars_event_max_datetime", "dtype": "string"}, {"name": "max_issues_repo_path", "dtype": "string"}, {"name": "max_issues_repo_name", "dtype": "string"}, {"name": "max_issues_repo_head_hexsha", "dtype": "string"}, {"name": "max_issues_repo_licenses", "sequence": "string"}, {"name": "max_issues_count", "dtype": "float64"}, {"name": "max_issues_repo_issues_event_min_datetime", "dtype": "string"}, {"name": "max_issues_repo_issues_event_max_datetime", "dtype": "string"}, {"name": "max_forks_repo_path", "dtype": "string"}, {"name": "max_forks_repo_name", "dtype": "string"}, {"name": "max_forks_repo_head_hexsha", "dtype": "string"}, {"name": "max_forks_repo_licenses", "sequence": "string"}, {"name": "max_forks_count", "dtype": "float64"}, {"name": "max_forks_repo_forks_event_min_datetime", "dtype": "string"}, {"name": "max_forks_repo_forks_event_max_datetime", "dtype": "string"}, {"name": "content", "dtype": "string"}, {"name": "avg_line_length", "dtype": "float64"}, {"name": "max_line_length", "dtype": "int64"}, {"name": "alphanum_fraction", "dtype": "float64"}, {"name": "loc", "dtype": "int64"}, {"name": "functions", "dtype": "int64"}, {"name": "function_signatures", "dtype": "int64"}, {"name": "function_parameters", "dtype": "int64"}, {"name": "variable_declarations", "dtype": "int64"}, {"name": "property_declarations", "dtype": "int64"}, {"name": "function_usages", "dtype": "int64"}, {"name": "trivial_types", "dtype": "int64"}, {"name": "predefined_types", "dtype": "int64"}, {"name": "type_definitions", "dtype": "int64"}, {"name": "dynamism_heuristic", "dtype": "int64"}, {"name": "loc_per_function", "dtype": "float64"}, {"name": "estimated_tokens", "dtype": "int64"}, {"name": "fun_ann_density", "dtype": "float64"}, {"name": "var_ann_density", "dtype": "float64"}, {"name": "prop_ann_density", "dtype": "float64"}, {"name": "typedef_density", "dtype": "float64"}, {"name": "dynamism_density", "dtype": "float64"}, {"name": "trivial_density", "dtype": "float64"}, {"name": "predefined_density", "dtype": "float64"}, {"name": "metric", "dtype": "float64"}, {"name": "content_without_annotations", "dtype": "string"}], "splits": [{"name": "test", "num_bytes": 5737779, "num_examples": 744}], "download_size": 2384891, "dataset_size": 5737779}, "extra_gated_prompt": "## Terms of Use for The Stack\nThe Stack dataset is a collection of source code in over 300 programming languages. We ask that you read and acknowledge the following points before using the dataset:\n1. The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant. We facilitate this by providing provenance information for each data point.\n2. The Stack is regularly updated to enact validated data removal requests. By clicking on \"Access repository\", you agree to update your own version of The Stack to the most recent usable version specified by the maintainers in [the following thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If you have questions about dataset versions and allowed uses, please also ask them in the dataset\u2019s [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new). We will also notify users via email when the latest usable version changes.\n3. To host, share, or otherwise provide access to The Stack dataset, you must include [these Terms of Use](https://huggingface.co/datasets/bigcode/the-stack#terms-of-use-for-the-stack) and require users to agree to it.\n\nBy clicking on \"Access repository\" below, you accept that your contact information (email address and username) can be shared with the dataset maintainers as well.", "extra_gated_fields": {"Email": "text", "I have read the License and agree with its terms": "checkbox"}}
TAGS
#region-us 

Dataset Card for "ts-eval"
==========================


This is a dataset of 744 high-quality TypeScript files, meant for evaluation of type prediction systems. The dataset is derived from TypeScript portion of The Stack (dedup), version 1.1, and filtered.


Filtering steps
---------------


We remove files that:


1. Depend on external modules and do not type check.
2. Have 0 type annotation locations.
3. Have 50 or fewer lines of code.
4. Have 0 functions.
5. Average fewer than 5 lines of code per function.


Next, we compute a weighted quality score, based on the following factors:


* function annotation density
* variable annotation density
* type definition density
* trivial types density
* predefined types density
* lines of code per function
* number of function usages


Files with a score 1 or more standard deviations below the mean are removed.


Next, we apply the training cutoff (December 31, 2021).


Finally, we remove type annotations. Some files become invalid during this process, so we exclude them from the dataset.


Data fields
-----------


Most of the data fields come from The Stack.


We add the following fields:


* 'loc' (integer): lines of code, excluding comments and blanks
* 'functions' (integer): number of functions that have function bodies; includes function expressions, arrow functions, and methods
* 'function\_signatures' (integer): number of function signatures, i.e., function declarations with no bodies
* 'function\_parameters' (integer): number of function parameters
* 'variable\_declarations' (integer): number of variable declarations
* 'property\_declarations' (integer): number of property declarations, i.e., within classes and interfaces
* 'function\_usages' (integer): number of functions (defined within the file) that are called
* 'trivial\_types' (integer): number of trivial type annotations, e.g. 'any', 'Function'
* 'predefined\_types' (integer): number of predefined type annotations, e.g. 'boolean', 'number', 'string'
* 'type\_definitions' (integer): number of type definitions, e.g. classes, interfaces, type aliases
* 'dynamism\_heuristic' (integer): a count of dynamic language features, e.g. 'eval', run-time type tests
* 'loc\_per\_function' (float): average lines of code per function
* 'estimated\_tokens' (integer): estimated token count, using the SantaCoder tokenizer
* 'fun\_ann\_density' (float): a measure of function annotation density
* 'var\_ann\_density' (float): a measure of variable annotation density
* 'prop\_ann\_density' (float): a measure of property annotation density
* 'typedef\_density' (float): a measure of type definition density
* 'dynamism\_density' (float): a measure of dynamic feature usage density
* 'trivial\_density' (float): a measure of trivial type annotation density
* 'predefined\_density' (float): a measure of predefined type annotation density
* 'metric' (float): the quality score
* 'content\_without\_annotations' (string): content of the file, with type annotations removed


Versions
--------


The default version ('main') is 'v1.1'.




TAGS
#region-us 

passage: TAGS
#region-us 
