Sentence similarity models on Sagemaker
Hello,I am wondering if the “sentence-similarity” pipelines can be used on Sagemaker from Hub with the same ease as popular piplines like “question-answering”?I was trying to start an endpoint with sentence-similarity, but it gave me this error:2021-09-16 13:13:49,252 [INFO ] W-sentence-transformers__pa-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: "Unknown task sentence-similarity, available tasks are ['feature-extraction', 'text-classification', 'token-classification', 'question-answering', 'table-question-answering', 'fill-mask', 'summarization', 'translation', 'text2text-generation', 'text-generation', 'zero-shot-classification', 'conversational', 'image-classification', 'translation_XX_to_YY']" : 400This is how it was deployed inside Sagemaker Studio:from sagemaker.huggingface import HuggingFaceModel
import sagemaker 

role = sagemaker.get_execution_role()

# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'sentence-transformers/paraphrase-xlm-r-multilingual-v1',
    'HF_TASK':'sentence-similarity' # NLP task you want to use for predictions
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
)


# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.t2.medium"
)
2021-09-16T13:53:38Z
Hello@pavel-nesterov,Currently, the SageMaker Inference Toolkit only supports the NLP-based pipelines from thetransformerslibrary for zero-code deployments.But you can easily create your owninference.pywhich contains the code you would need for that.Here is Documentation on that:Deploy models to Amazon SageMakerHere the code for thesentence-simliaritypipeline:huggingface_hub/sentence_similarity.py at main · huggingface/huggingface_hub · GitHubThank you for the request I will add it as potential feature to the roadmap.
2021-09-16T14:26:15Z
Thank you, Philipp.I will try it as “next easiest step” in the search of the solution.The approach I am trying to explore now is “Are there any of the 186 Sentence Similarity models can be deployed  on sagemaker with no-code approach”. I checked all of them andI found some! (scerenshot is below)The only thing is that it doesn’t have “sentence similarity” pipeline, but I think I would try “token-classification” one. I am writing it because someone later may come to the same idea and I may save some time for that explorer.image1190×623 110 KB
2021-09-17T10:44:40Z
My 1st iterationfailed, but I hope it will help to improve the Model Hub. Therefore I am getting deeper into your suggestion, Philipp@philschmidI will post here how it goes (it may help someone like me, who tries to find answers in this forum).
2021-09-18T07:56:16Z
Hi Philipp!@philschmidI read the documentation twice and concluded, that I need to get deeper understanding of how the transformers work. The best way I found (as of know) - is to go through theHF Course.Even though I did many training jobs on Sagemaker with vanilla Pytorch, I feel like I need toget a deeper understanding of “pipelines”, “tokenisers” and “post-processing”.In my project I will also need to fine-tune a pre-trained model (because I am builing a product for not a widely-spoken language - slovenian).The link you shared leads to “Advanced features” of the HF-Sagemeker" documentation (read twice).Considering the three points above, I am timeboxing a week to go through the HF course to see if it will me to approach overriding default methods of HF inference toolkit.Not sure if there is a better way to approach the problem of making sentence-similarity endpoint on sagemaker. My candidate was to train bag-of-words + cosine similarity from scratch in Pytorch (and then deploy the endpoint), but I feel I will be able to benefit from learning the transformers’ basics.Thank you for helping me to find the direction for the solution!
2021-09-20T06:50:51Z
Sounds good!I found two solvenian model on the hub, which you can give a try to fine-tune and compare with yourbag-of-wordsmethod:EMBEDDIA/sloberta · Hugging FaceEMBEDDIA/crosloengual-bert · Hugging Face
2024-01-12T12:01:26Z
@philschmidI tried the custom method using inference.py, but I am faing the following error"You need to define one of the following ['audio-classification', 'automatic-speech-recognition', 'feature-extraction', 'text-classification', 'token-classification', 'question-answering', 'table-question-answering', 'fill-mask', 'summarization', 'translation', 'text2text-generation', 'text-generation', 'zero-shot-classification', 'conversational', 'image-classification', 'image-segmentation', 'object-detection'] as env 'HF_TASK’Will setting an environment variable solve this error?