Return all class labels from SageMaker invoke_endpoint
I’ve deployed the Hate-speech-CNERG/bert-base-uncased-hatexplain multi-class text classification model (along with others) to a SageMaker endpoint. Using the TextClassificationPipeline you’re able to pass in the return_all_scores=True parameter to see all class labels for the input.However when using the SageMakerRuntime.invoke_endpoint function, I can only get one class per input. Anyone have any thoughts on an equivalent to the return_all_scores parameter?
2022-01-14T06:18:24Z
Hi Will, when deploying a model to a Sagemaker endpoint you can provide a custom inference script in which you can control the behaviour of the model for inference requests. You can find the documentation on how to do this here:Deploy models to Amazon SageMakerIn your particular scenario I would think hat you can override thepredict_fnand/or theoutput_fnmethods to get to the desired result. You can find an example for aninference.pyfilehere(note that this particular file is for text summarization, so it won’t apply for your use case).To see what code you need to get all scores returned you can refer back to the code from the TextClassification Pipeline, that you have already mentioned:transformers/text_classification.py at master · huggingface/transformers · GitHub. If you implement this code into your inference script you should be all set.Hope that helps, let me know if any questions.CheersHeiko
2022-01-20T21:28:55Z
I would like to do the same too. Has anyone tried it yet?
2022-01-21T05:29:06Z
I just had a look into the code of the Huggingface Sagemaker Inference Toolkit and found an even easier way to accomplish this: You can passparametersin your inference request:sagemaker-huggingface-inference-toolkit/handler_service.py at main · aws/sagemaker-huggingface-inference-toolkit · GitHubThat’s awesome, because it means you don’t even have to provide aninference.pyscript!All you need to do is:predictor.predict({"inputs": "I love using the new Inference DLC.", "parameters": {"return_all_scores": True}})Capture840×101 4.16 KBHope that helps!
2022-01-21T07:34:27Z
We have also a documentation section available for how the API works:ReferenceThe Inference Toolkit accepts inputs in theinputskey, and supports additionalpipelinesparameters in theparameterskey. You can provide any of the supportedkwargsfrompipelinesasparameters.{
  "inputs": "This sound track was beautiful! It paints the senery in your mind so well I would recomend it
  even to people who hate vid. game music!", 
 "parameters": {
        "return_all_scores": True
  }
}
2022-01-21T11:37:00Z
Thanks both of you for your help! I will go ahead and try that
2022-01-22T01:19:33Z
@marshmellow77I tried it on Sagemaker studio and it still gives only one label. May I ask you to please look at the screenshot to see if anything is not right? Thank you2022-01-21_2016491920×893 246 KB
2022-01-22T04:46:56Z
All looking good except that you are using an old version of the transformers library. Trytransformers_version=“4.12.3”,pytorch_version=“1.9.1”, andpy_version=“py38”. That should workJust for reference, you can find the latest version of the DLCs here:ReferenceCheersHeiko
2022-01-22T05:01:30Z
It finally worked. Thanks Marshmellow!2022-01-22_0000351914×919 231 KB