Using Accelerated Inference API to produce sentense embeddings
Is it possible to useAccelerated Inference APIto produce sentense embeddings as describedhere?from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask



#Sentences we want sentence embeddings for
sentences = ['This framework generates embeddings for each input sentence',
             'Sentences are passed as a list of string.',
             'The quick brown fox jumps over the lazy dog.']

#Load AutoModel from huggingface model repository
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")
model = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")

#Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')

#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

#Perform pooling. In this case, mean pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
2021-05-18T13:08:39Z
hey@vitalii believe integration withsentence-transformersin the inference API is currently in progress, so maybe@osansevierocan share some details (or whether it’s currently possible)
2021-05-18T14:23:04Z
Hi@vitali!Your question comes in good time. You can already do this by callinghttps://api-inference.huggingface.co/pipeline/feature-extraction/MODEL_ID. This endpoint is inexperimentalstate at the moment, so things might not be stable.Note that, as of now, we’re working on deeply integratingsentence-transformerswith the Hub. This will be part of the v2 release of the library. Some details:Allow downloading sentence-transformer models from the Hub (PR, merged).Allow uploading sentence-transformer models from the Hub (PR).We expect to have more exciting results very soon
2021-05-18T15:08:45Z
Awesome! I will try to it.
2021-05-24T22:38:27Z
Can I infer from your answer that any pipeline can be inferred like this?That would be totally awesome.
2021-05-25T07:19:08Z
The pipeline is usually inferred from the tags in the model repo. Forcing the pipeline through the API has a risk of misusing a model for a different pipeline, but there are also certain models that support multiple pipelines, so you can use that as well.
2021-05-25T14:06:07Z
So If I add the tag “feature-extraction”  in the model repo then call to inference API will produce embeddings? what will happen if I add multiple tags, i.e “feature-extraction”, “fill-mask”, “zero-shot-classification”?
2021-05-27T09:57:21Z
Hi@vitali. We currently try to keep things simple: we usually have 1 task per model, and this holds for most models. But in case your model does support other tasks, you can use the API url as above.Right now there’s no way to validate that the model will work with the task, so that’s why this is not shared widely - it might lead to misusing and getting incorrect results.
2021-05-27T16:02:56Z
Thank you very much for your reply, I understand the issue: every transformer model can serve some tasks (make embeddings, MLM, zero-shot classification)  but no model can serve all tasks, thus comes a risk of misuse. Perhaps it would make sense to add “capabilities.json” to the model repo to define the list of supported tasks/pipelines based on the model architecture? I think this would clear some confusion amongst users, just a thought. Anyway, this way solves my need to make embeddings for downstream tasks perfectly, thank you very much again.
2021-05-28T15:01:08Z
For community reference, the issue of defining and using model pipelines is also discussed ongithub.
2021-12-07T10:01:23Z
Hey Omar, is this still an experimental API? I can’t seem to find any details about it. Would appreciate you pointing me to some resources if available as I am evaluating some APIs and would like to test yours. Thanks.
2021-12-08T10:32:13Z
Hey@abol3z.The API is not in experimental anymore, but we’re working in its documentation. You can usehttps://api-inference.huggingface.co/pipeline/feature-extraction/MODEL_IDto obtain the sentence embeddings.Let us know if you have any questions!
2022-04-23T15:03:35Z
Hello Omar,  I was trying to load embeddings via API as discussed in this thread, but I am struggling to find a model that actually supports this method.So for example, the modelhttps://api-inference.huggingface.co/models/sentence-transformers/all-MiniLM-L6-v2supports feature extraction, but the following URL is invalid:https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-MiniLM-L6-v2Can you help?Or maybe share a sample code snippet?Thanks.
2022-04-23T18:54:56Z
Hi there!Here is a working end-to-end example for the model you suggested.import requests

API_URL = " https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-MiniLM-L6-v2"
headers = {"Authorization": "Bearer TOKEN"}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()
	
output = query({
	"inputs": "I like you. I love you",
})
2022-09-22T09:53:22Z
Thanks sm! this helped a lot.
2023-04-12T18:10:22Z
Hello,I’ve used the feature extraction pipeline successfully with sentance-trasformer models.How do I use it with a model, which requiresmean_poolingto be applied to the result, such asE5?To clarify: get a single array of feature embeddings vs the current result that comes out - 3 arrays for the word “test”.Thank you!
2023-04-12T19:25:28Z
hi@mtomov, you might need acustom pipelineto process the result, here is a duplicated model implementing the mean_pooling on the requestradames/e5-large · Hugging Face, you can also tryimport requests

API_URL = "https://api-inference.huggingface.co/models/radames/e5-large"
headers = {"Authorization": "Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}


def query(payload):
    response = requests.post(API_URL, headers={}, json=payload)
    return response.json()


embeddings = query({
    "inputs": "query: how much protein should a female eat",
})
print(embeddings)