
# Hugging Face Spaces Descriptions and Embeddings Dataset

I parsed all the available public 🤗 spaces as of May 22, 2023, generated concise descriptions of their functionality, and created embeddings for them.

The descriptions were generated using various LLMs from each space's app file (README.md -> app_file). The embeddings were created using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) SentenceTransformer model.

The dataset comprises approximately 30,000 spaces that meet specific criteria: having more than 40 lines of code and over 1000 characters in the app file.

The descriptions provide an overview of the spaces and their features.

## Dataset Details

- **Name**: HF Spaces Descriptions and Embeddings
- **Creator**: [anzorq](https://huggingface.co/anzorq)
- **License**: MIT

## Dataset Usage

You can use this dataset for various natural language processing (NLP) tasks such as semantic search, clustering, etc.

## Loading the Dataset
You can load the dataset using the datasets library:
```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("anzorq/hf-spaces-descriptions-embeddings")

# Access the different splits
train_split = dataset['train']
valid_split = dataset['valid']
test_split = dataset['test']
```

## Semantic Search Example
Performing a semantic search using the dataset's embeddings:
```python
import torch
from sentence_transformers import SentenceTransformer
from datasets import load_dataset
import numpy as np

# Load the dataset
dataset = load_dataset("anzorq/hf-spaces-descriptions-embeddings")

# Load the SentenceTransformer model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Example query
query = "Removing background from images"

# Encode the query
query_embedding = model.encode([query], convert_to_tensor=True)

# Get the space descriptions and embeddings
descriptions = dataset['train']['description']
embeddings = np.array(dataset['train']['embedding'])

# Calculate cosine similarity
cosine_scores = torch.nn.functional.cosine_similarity(query_embedding, torch.tensor(embeddings))

# Sort the results
top_k = torch.topk(cosine_scores, k=5)

# Print the top-k results
print("Query:", query)
for idx in top_k.indices[0]:
    print("Space ID:", dataset['train']['id'][idx])
    print("Description:", descriptions[idx])
    print("Score:", cosine_scores[idx].item())
```

## License

This dataset is distributed under the [MIT License](https://opensource.org/licenses/MIT).


license:mit region:us
{"license": "mit", "dataset_info": {"features": [{"name": "id", "dtype": "string"}, {"name": "description", "dtype": "string"}, {"name": "embedding", "sequence": "float64"}], "splits": [{"name": "train", "num_bytes": 94758018, "num_examples": 29718}], "download_size": 78891306, "dataset_size": 94758018}}
TAGS
#license-mit #region-us 


# Hugging Face Spaces Descriptions and Embeddings Dataset

I parsed all the available public  spaces as of May 22, 2023, generated concise descriptions of their functionality, and created embeddings for them.

The descriptions were generated using various LLMs from each space's app file (URL -> app_file). The embeddings were created using the all-MiniLM-L6-v2 SentenceTransformer model.

The dataset comprises approximately 30,000 spaces that meet specific criteria: having more than 40 lines of code and over 1000 characters in the app file.

The descriptions provide an overview of the spaces and their features.

## Dataset Details

- Name: HF Spaces Descriptions and Embeddings
- Creator: anzorq
- License: MIT

## Dataset Usage

You can use this dataset for various natural language processing (NLP) tasks such as semantic search, clustering, etc.

## Loading the Dataset
You can load the dataset using the datasets library:


## Semantic Search Example
Performing a semantic search using the dataset's embeddings:


## License

This dataset is distributed under the MIT License.


# Hugging Face Spaces Descriptions and Embeddings Dataset

I parsed all the available public  spaces as of May 22, 2023, generated concise descriptions of their functionality, and created embeddings for them.

The descriptions were generated using various LLMs from each space's app file (URL -> app_file). The embeddings were created using the all-MiniLM-L6-v2 SentenceTransformer model.

The dataset comprises approximately 30,000 spaces that meet specific criteria: having more than 40 lines of code and over 1000 characters in the app file.

The descriptions provide an overview of the spaces and their features. ## Dataset Details

- Name: HF Spaces Descriptions and Embeddings
- Creator: anzorq
- License: MIT ## Dataset Usage

You can use this dataset for various natural language processing (NLP) tasks such as semantic search, clustering, etc. ## Loading the Dataset
You can load the dataset using the datasets library: ## Semantic Search Example
Performing a semantic search using the dataset's embeddings: ## License

This dataset is distributed under the MIT License.
TAGS
#license-mit #region-us 
 # Hugging Face Spaces Descriptions and Embeddings Dataset

I parsed all the available public  spaces as of May 22, 2023, generated concise descriptions of their functionality, and created embeddings for them.

The descriptions were generated using various LLMs from each space's app file (URL -> app_file). The embeddings were created using the all-MiniLM-L6-v2 SentenceTransformer model.

The dataset comprises approximately 30,000 spaces that meet specific criteria: having more than 40 lines of code and over 1000 characters in the app file.

The descriptions provide an overview of the spaces and their features. ## Dataset Details

- Name: HF Spaces Descriptions and Embeddings
- Creator: anzorq
- License: MIT ## Dataset Usage

You can use this dataset for various natural language processing (NLP) tasks such as semantic search, clustering, etc. ## Loading the Dataset
You can load the dataset using the datasets library: ## Semantic Search Example
Performing a semantic search using the dataset's embeddings: ## License

This dataset is distributed under the MIT License.