Deploying a conversational pipeline on AWS
I am following the instructions for deploying a customMicrosoft/DialoGPT-mediumon AWS Sagemaker. As a first step, I am using the instructions included on the model hub, but they do not work and appear to be out of date.The code that follows leads to this error:ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "ConversationalPipeline expects a Conversation or list of Conversations as an input"
}Code from the documentation:from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'microsoft/DialoGPT-medium',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?",
	}
})I can make those lists intoConversationobjects, but even still I get an error. I am not even sure they will be json serializable anymore.The larger issue is that I want to use a finetunedMicrosoft/DialoGPT-mediumthat I trained on my local machine. I am following a Hugging Face youtube tutorial for that one, but once again the code presented in the video does not work out of the box.# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz",  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   transformers_version="4.6",                           # Transformers version used
   pytorch_version="1.7",                                # PyTorch version used
   py_version='py36',                                    # Python version used
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)This fails with the following error:ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "(\"You need to define one of the following [\u0027feature-extraction\u0027, \u0027text-classification\u0027, \u0027token-classification\u0027, \u0027question-answering\u0027, \u0027table-question-answering\u0027, \u0027fill-mask\u0027, \u0027summarization\u0027, \u0027translation\u0027, \u0027text2text-generation\u0027, \u0027text-generation\u0027, \u0027zero-shot-classification\u0027, \u0027conversational\u0027, \u0027image-classification\u0027] as env \u0027TASK\u0027.\", 403)"This is my fourth time deploying a Hugging Face model to AWS Sagemaker, and the process is so incredibly complex and non-intuitive. I feel like the entire process is held up by toothpicks. Thanks for any light you can shed.
2022-01-05T22:29:33Z
I think you forgot to add the env task when creating your HuggingFaceModel:huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz",  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   transformers_version="4.6",                           # Transformers version used
   pytorch_version="1.7",                                # PyTorch version used
   py_version='py36',                                    # Python version used
   env={ 'HF_TASK':'conversational' },
)
2022-01-07T08:06:27Z
Hey@JB2022,we added support forconversationalpipeline with a later release. Can you use instead oftransformers_version="4.6"=>4.12and forpytorch_version="1.7"=>1.9.You can find the whole list of available containers here:ReferenceThen your fist code snippet should work.from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'microsoft/DialoGPT-medium',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.12',
	pytorch_version='1.9',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?",
	}
})
2022-01-12T15:46:44Z
@JB2022I would LOVE to have a chat with you… you say you’ve done this four times?! Please tell me how?Any chance at all you could DM me on Twitter? My username / twitter page is on my Huggingface profile.
2023-04-13T03:36:28Z
I also faced the same issue today, and tried@CyranoBidea and decided to add the env variable.hub = {‘HF_TASK’:‘text-classification’}Then attach this value to the HuggingFaceModel class as belowhuggingface_model = HuggingFaceModel(model_data=“s3://xxxxxxxxxxxxxxx/model.tar.gz”,role=role,transformers_version=“4.6”,pytorch_version=“1.7”,py_version=‘py36’,env=hub,)This did not work… So I tried the next option. Where I decided to provide the model_id toohub = {‘HF_MODEL_ID’:‘distilbert-base-uncased-finetuned-sst-2-english’,‘HF_TASK’:‘text-classification’}Then deployed. After that the predict method worked.Ps:I have not reviewed the underlying code of predict method.
2023-04-13T07:53:05Z
Hello@Kamaljpcould you share how you created themodel.tar.gz? Also you normally don’t need to add theHF_TASKwhen deploying amodel.tar.gz.You should also update the versions you use to the latest ones, which are 4.26.0, 1.13.1, py39
2023-04-23T21:37:08Z
hi@philschmidIs there an easy way to add streaming to conversational endpoints using a bit of customizations? Streaming is quite handy as the models are now larger (Flan-t5 , etc), for this SSE probably is required. I saw this (GitHub - huggingface/text-generation-inference: Large Language Model Text Generation Inference) but it seems to be an over kill. Any advice is really appreciated.Cheers
2023-04-24T05:46:53Z
The SageMaker platform currently doesn’t support SSE (Server Sent Events), which is needed for streaming.
2023-07-08T17:41:36Z
hi@philschmid! I see you mentioned that SSE aren’t currently supported by SageMaker Endpoints, however, I was just wondering if this has changed now with the LLM inference container you released for aws (Introducing the Hugging Face LLM Inference Container for Amazon SageMaker).Currently looking into options for hosting LLMs with streaming enabled so any inputs are welcome.Cheers
2023-07-13T13:00:29Z
Inference endpoints is supporting streaming see:Deploy LLMs with Hugging Face Inference Endpoints