# Dataset Card for "thevault-filtered"

Filtered version of [The Vault (function)](https://huggingface.co/datasets/Fsoft-AIC/the-vault-function). Restricted only to Python, then:
- Light AST filtering for self-contained functions
- Run through CodeBERT embeddings, clustered with k-means to 1024 clusters, and then the clusters were manually skimmed for seemingly uninformative functions.

The clusters excluded and their reasons are as follows:
```
  excluded = [
    4, # biochem stuff? DEcompiled code
    9, # Empty functions
    33, # Empty functions
    34, # UI stuff, just returns arguments
    37, # Empty functions
    40, # Empty functions
    42, # Empty functions
    44, # _namespace_SIO stuff
    55, # Trivial, e.g. add(a, b) = a + b
    66, # find_by class methods
    67, # Mostly methods, seems not very informative
    77, # openapi_types, returns a fixed dictionary
    78, # Minimal, method stuff
    83, # Locale configuration
    87, # Just returns argument
    101, # Incomplete
    102, # Class methods
    108, # openapi_types
    156, # Empty functions
    164, # Trivial, function aliases
    168, # Class methods
    172, # Empty functions
    173, # Class methods
    175, # Class methods
    181, # Empty functions
    182, # Fixed API stuff
    190, # Fixed specific stuff
    197, # from_dictionary class methods
    198, # Empty functions
    234, # Unimplemented
    246, # Fixed specific stuff
    277, # Empty functions
    280, # Empty functions
    282, # Empty functions
    287, # Trivial, e.g. helloWorld()
    299, # Mostly unfinished
    304, # Empty functions
    310, # Fixed API stuff
    313, # Just modifies globals
    320, # Empty functions
    329, # Takes a credentials object, and runs methods on it
    332, # MangoPi bot
    334, # Empty
    338, # namespace_SIO nonsense
    339, # fn(x) = x
    363, # Empty functions
    370, # Empty
    379, # Empty
    388, # Empty
    392, # Empty functions
    393, # Fixed lists
    409, # Fixed dictionaries
    416, # Aliases to print
    428, # Empty functions
    437, # Empty functions
    444, # Empty
    454, # Mostly just calls methods on arguments
    463, # Mostly just calls methods on arguments
    470, # Fixed dictionaries
    474, # Mostly fixed printing
    465, # OpenAPI fixed dictionaries
    476, # Empty
    477, # Fixed dictionaries
    491, # Trivial
    494, # Lots of fixed string stuff
    496, # Empty
    511, # Empty
    518, # OpenAPI
    521, # Fixed API stuff
    536, # Empty
    540, # Fixed API stuff
    553, # Empty
    555, # Empty
    564, # Empty
    566, # Empty
    568, # cls methods
    573, # Mostly fixed dict stuff
    574, # namespace_SO stuff, more biochem?
    582, # namespace_SO stuff, more biochem?
    602, # Fixed lists
    608, # Mostly cls methods
    617, # Mostly cls methods
    629, # cls methods, fixed lists
    641, # Fixed API stuff
    642, # Empty
    647, # Windows API stuff
    648, # jupyter stuff
    649, # mostly fixed dicts
    652, # Empty
    660, # Empty
    665, # cls methods
    666, # Empty
    672, # Empty
    680, # fixed dicts
    682, # Empty
    686, # Empty
    687, # Fixed lists elements_sequence
    692, # cls methods
    693, # ASCII art
    704, # Empty
    709, # mqtt send message
    712, # Empty
    715, # Fixed data recoding
    717, # Empty
    722, # cls methods
    725, # cls methods
    734, # cls methods
    737, # Empty
    741, # Trivial cls methods
    742, # Empty
    745, # Fixed strings
    752, # Empty
    758, # Mostly fixed printing
    768, # Empty
    783, # Empty
    784, # Mostly fixed dicts
    802, # Fixed printing
    806, # Empty
    821, # Empty
    824, # stuff like load_performance_win_x64_win_x64_vs2017_settings
    825, # Trivial
    835, # Empty
    851, # Empty
    862, # Empty
    876, # Trivial
    878, # Empty
    887, # Empty
    888, # Mostly fixed dicts
    890, # Mostly fixed dicts
    893, # Empty
    898, # cls methods
    899, # Fixed ['str'] stuff
    906, # Auto-generated or something
    912, # Empty
    924, # Empty
    933, # namespace_SO biochem stuff
    938, # Trivial
    959, # Mostly fixed printing
    963, # API-specific
    965, # cls methods
    967, # cls methods
    970, # Mostly fixed printing
    971, # cls methods
    972, # cls methods
    973, # Empty
    979, # cls methods
    982, # Empty
    983, # Empty
    989, # cls methods
    990, # API specific
    1007, # API specific
    1014, # Empty
]
```

MIT licensed, like the original dataset
task_categories:text-generation license:mit region:us
{"license": "mit", "task_categories": ["text-generation"], "dataset_info": {"features": [{"name": "hexsha", "dtype": "string"}, {"name": "repo", "dtype": "string"}, {"name": "path", "dtype": "string"}, {"name": "license", "sequence": "string"}, {"name": "language", "dtype": "string"}, {"name": "identifier", "dtype": "string"}, {"name": "return_type", "dtype": "string"}, {"name": "original_string", "dtype": "string"}, {"name": "original_docstring", "dtype": "string"}, {"name": "docstring", "dtype": "string"}, {"name": "docstring_tokens", "sequence": "string"}, {"name": "code", "dtype": "string"}, {"name": "code_tokens", "sequence": "string"}, {"name": "short_docstring", "dtype": "string"}, {"name": "short_docstring_tokens", "sequence": "string"}, {"name": "comment", "sequence": "string"}, {"name": "parameters", "list": [{"name": "param", "dtype": "string"}, {"name": "type", "dtype": "string"}]}, {"name": "docstring_params", "struct": [{"name": "returns", "list": [{"name": "docstring", "dtype": "string"}, {"name": "docstring_tokens", "sequence": "string"}, {"name": "type", "dtype": "string"}]}, {"name": "raises", "list": [{"name": "docstring", "dtype": "string"}, {"name": "docstring_tokens", "sequence": "string"}, {"name": "type", "dtype": "string"}]}, {"name": "params", "list": [{"name": "identifier", "dtype": "string"}, {"name": "type", "dtype": "string"}, {"name": "docstring", "dtype": "string"}, {"name": "docstring_tokens", "sequence": "string"}, {"name": "default", "dtype": "string"}, {"name": "is_optional", "dtype": "bool"}]}, {"name": "outlier_params", "list": [{"name": "identifier", "dtype": "string"}, {"name": "type", "dtype": "string"}, {"name": "docstring", "dtype": "string"}, {"name": "docstring_tokens", "sequence": "string"}, {"name": "default", "dtype": "string"}, {"name": "is_optional", "dtype": "bool"}]}, {"name": "others", "list": [{"name": "identifier", "dtype": "string"}, {"name": "docstring", "dtype": "string"}, {"name": "docstring_tokens", "sequence": "string"}]}]}, {"name": "code_with_imports", "dtype": "string"}, {"name": "idxs", "dtype": "int64"}, {"name": "cluster", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1555988881.6663418, "num_examples": 544627}], "download_size": 773215769, "dataset_size": 1555988881.6663418}}
TAGS
#task_categories-text-generation #license-mit #region-us 

# Dataset Card for "thevault-filtered"

Filtered version of The Vault (function). Restricted only to Python, then:
- Light AST filtering for self-contained functions
- Run through CodeBERT embeddings, clustered with k-means to 1024 clusters, and then the clusters were manually skimmed for seemingly uninformative functions.

The clusters excluded and their reasons are as follows:


MIT licensed, like the original dataset
# Dataset Card for "thevault-filtered"

Filtered version of The Vault (function). Restricted only to Python, then:
- Light AST filtering for self-contained functions
- Run through CodeBERT embeddings, clustered with k-means to 1024 clusters, and then the clusters were manually skimmed for seemingly uninformative functions.

The clusters excluded and their reasons are as follows:


MIT licensed, like the original dataset
TAGS
#task_categories-text-generation #license-mit #region-us 
 # Dataset Card for "thevault-filtered"

Filtered version of The Vault (function). Restricted only to Python, then:
- Light AST filtering for self-contained functions
- Run through CodeBERT embeddings, clustered with k-means to 1024 clusters, and then the clusters were manually skimmed for seemingly uninformative functions.

The clusters excluded and their reasons are as follows:


MIT licensed, like the original dataset
passage: TAGS
#task_categories-text-generation #license-mit #region-us 
# Dataset Card for "thevault-filtered"

Filtered version of The Vault (function). Restricted only to Python, then:
- Light AST filtering for self-contained functions
- Run through CodeBERT embeddings, clustered with k-means to 1024 clusters, and then the clusters were manually skimmed for seemingly uninformative functions.

The clusters excluded and their reasons are as follows:


MIT licensed, like the original dataset