Multi-Model Endpoint with Hugging Face
Hi Team,Good day!!I’m trying to deploy multiple BERT models in one container behind one endpoint using the boto3 API.If we unzip the model.tar.gz file then we have inference code, model artifacts in below structure.I observed that dependency libraries were not being installed when SageMaker started setting up the container in cloud watch logs.image1445×807 54.6 KBHence I’m getting the below error.ModuleNotFoundError: No module named ‘spacy’Note:I followed the above structure to deploy a single model endpoint and it works fine.Looks like there is a different structure we need to follow in case of multi-model endpoint deployment.Could you please help me with this issue?Thanks
2022-05-27T14:54:49Z
Can you please provide more information on how you create thezipfile and how you are deploying the endpoint? Also what is the content of yourrequirements.txt? it looks like you want to usespacyin the inference script but it is not installed
2022-05-27T15:05:44Z
As requested, please find the details below.Creating zip file: As shown in above image, “model” directory contains model artifacts and “code” directory contains inference code and requirements.txt file.Contents of requirements.txt filespacy==3.2.1spacy-alignments==0.8.4spacy-legacy==3.0.8spacy-loggers==1.0.1spacy-transformers==1.1.4tokenizers==0.10.3pytextspan==0.5.4validators==0.18.2Deployment:image1237×387 19.2 KB
2022-06-07T17:37:13Z
Could you try to create themodel.tar.gzfollowing the steps in this notebook?notebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHubthetar.add(model_path)is creating a wrong structure with a nestedmodel/directory but the artifacts need to be on top level.
2022-06-26T06:23:25Z
Hello@philschmidIssue was not with saving model artifacts on the top level.Solution: To install additional libraries on the container, libraries which are part of requirements.txt text file needs to be installed using pip in inference script. Within the archive, the HuggingFace container expects all inference code to be inside thecode/directory.I did not find any article on multi-model endpoint with hugging face and examples in “aws-samples” git repo hence I publish thisarticle on Medium.Hope that helps everyone. Thanks
2024-02-28T19:53:43Z
Vinayaks117 Medium article link:Multi-Model Endpoints with Hugging Face Transformers and Amazon SageMaker | by Vinayak Shanawad | Analytics Vidhya | Medium
2024-03-03T03:46:36Z
Medium – 31 Mar 23Multi-Model Endpoints with Hugging Face Transformers and Amazon SageMaker#HuggingFace #AWS #BERT #GPT-2 #SageMakerReading time: 6 min read