Deploying Huggingface Sagemaker Models with Elastic Inference
When I try to deploy a HuggingFace Sagemaker model with elastic inference (denoted by theaccelerator_typeparameter) I get an error.Deploy Snippet:predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.t2.medium",
    accelerator_type='ml.eia2.medium'
)Error Msg:~/miniconda3/envs/ner/lib/python3.8/site-packages/sagemaker/image_uris.py in _validate_arg(arg, available_options, arg_name)
    305     """Checks if the arg is in the available options, and raises a ``ValueError`` if not."""
    306     if arg not in available_options:
--> 307         raise ValueError(
    308             "Unsupported {arg_name}: {arg}. You may need to upgrade your SDK version "
    309             "(pip install -U sagemaker) for newer {arg_name}s. Supported {arg_name}(s): "

ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference.The model deploys successfully if I do not provide an accelerator (i.e., no Elastic Inference).Do the HuggingFace Sagemaker models support EI? If yes, how might I deploy the model successfully with EI? And if not, is EI support on the roadmap?Much thanks in advance!
2021-07-28T06:46:03Z
Hey@schopra,Sadly speaking we don’t haveEIDLCs yet. We are working on it and it is on the roadmap with one of the highest priorities.I would update this thread here when I got any news.
2021-09-03T16:35:06Z
Is there by any chance a list of supported instances at this time? Thanks!
2021-09-05T09:32:03Z
Hey@ujjirox,supported instances for what? Training or Inference or both? You can find an overview of supported instances type for sagemaker here:Amazon SageMaker Pricing – Amazon Web Services (AWS)
2021-09-05T16:14:39Z
Sorry! Should have been more clear. I meant for inference. I actually had tried running inference with ml.inf1.xlarge but it didn’t seem to work, hence the question.Thanks.
2021-09-06T06:19:01Z
Hey@ujjirox,Inferentia is also not yet supported, since we need to create a separate DLC for the Inferentia instances, but we are on it.Other than this every CPU / GPU machine should be supported.
2021-12-28T13:22:30Z
Hey, any news regarding the EI DLC/ INF DLC?
2021-12-29T10:21:21Z
Hey@asafab,Yes, i already opened a PR for INF DLCs:You can follow it here:[huggingface_pytorch][NEURON][build] Huggingface Neuron inference DLC by philschmid · Pull Request #1578 · aws/deep-learning-containers · GitHubWhen it is merged and available we will additionally share on social media + provide an example.
2021-12-29T10:35:44Z
very cool!thanks for the response, you have news about the EI DLC too?
2022-01-27T10:38:16Z
Hey there !!The PR regarding INF DLCs seems to have been merged, does it meanml.inf*instance family can now be used with HuggingFace models ?
2022-01-27T10:56:53Z
They are merged but yet not released. I hope they will be available in the next 2 weeks. We ll let you know on social media.
2022-03-18T15:50:42Z
Hello@philschmid,I read your articlehttps://www.philschmid.de/huggingface-bert-aws-inferentiaabout hugging face model deployment on inferentia instance (very good and clear btw).Can this method be used for all model types and all tasks ? I particularly think to Seq2Seq models (Bart, Pegasus, T5) for Summarization task
2022-03-21T12:53:17Z
Hello@YannAgora,Yes, it can also be used for T5 or pegasus. You can find more documentation here:Transformers MarianMT Tutorial — AWS Neuron documentation.You can use theNeuronGenerationcode inside theinference.pythen.
2022-05-18T11:42:52Z
Hi@philschmidAny updates on EI (Elastic Inference) DLCs for inference?Can we start using EI accelerators with HuggingFace models for inference?Thanks
2022-05-18T13:10:37Z
What is the error you are seeing when you are trying to deploy an EI backed endpoint?
2022-05-24T04:17:49Z
schopra:huggingface_model@philschmidAs requested, please find the error details below.ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference.Sagemaker SDK version = 2.87.0huggingface_model = HuggingFaceModel(model_data=model_data,  # path to your trained SageMaker modelrole=get_execution_role(),                            # IAM role with permissions to create an endpointentry_point=‘deploy_ei.py’,transformers_version=“4.12.3”,                        # Transformers version usedpytorch_version=“1.9.1”,                                # PyTorch version usedpy_version=‘py38’,                                    # Python version usedsagemaker_session=sagemaker_session)
2022-06-21T12:50:00Z
Hello@philschmidAny updates on this issue and EI (Elastic Inference) DLCs for inference? Thanks
2022-06-22T12:28:40Z
Let me reach out to the AWS team again. I ll report back here as soon as I hear something.Out of curiosity why are you interested in EIA and not using Inferentia?
2022-06-22T15:17:34Z
We are interested in cost effective solution and also interested in hosting multiple models in one container.But I think we can not host multiple models in one container behind one endpoint with both elastic inference and Inferentia but it’s possible with only cpu based instances. Thanks
2022-10-15T14:20:44Z
Hey@philschmidAny updates on this issue and EI (Elastic Inference) DLCs for inference?Thanks
No date available
No reply text available