Save and deploy distilbert model in AWS SageMaker
I am trying to download the Hugging Face distilbert model, trying to save to S3. The model itself does not have a deploy method. So I am saving to S3, instantiating it and trying to deploy. May I know if this will work with Sagemaker. What I am doing wrong. Here are the steps:model_name = ‘distilbert-base-uncased-distilled-squad’ model = DistilBertForQuestionAnswering.from_pretrained(model_name) tokenizer = DistilBertTokenizerFast.from_pretrained(model_name)the below works and gives output context = “xxxx” question = “yyy?”nlp = pipeline(‘question-answering’, model=model, tokenizer=tokenizer)nlp({ ‘question’: ‘What organization is the IPCC a part of?’, ‘context’: context })save the model to local folder model.save_pretrained(’./scripts/mymodel’)zip the model file. with tarfile.open("./scripts/mymodel/model.tar.gz", “w:gz”) as tar: tar.add("./scripts/mymodel/pytorch_model.bin") tar.add("./scripts/mymodel/config.json")upload the zipped file to S3 sagemaker.Session().upload_data(bucket=sagemaker_session_bucket, path=’./scripts/mymodel/model.tar.gz’, key_prefix=‘model’)instantiating the saved modelbertmodel = PyTorchModel(entry_point=‘inference.py’, source_dir=‘scripts’, model_data=‘s3://’+sagemaker_session_bucket+’/model/model.tar.gz’, role=sagemaker.get_execution_role(), framework_version=‘1.5’, py_version=‘py3’)the below does not work nlp = pipeline(‘question-answering’, model=bertmodel, tokenizer=tokenizer)nlp({ ‘question’: ‘What organization is the IPCC a part of?’, ‘context’: context })error recd - AttributeError: ‘PyTorchModel’ object has no attribute ‘config’I am able to deploy the predictor predictor = bertmodel.deploy(initial_instance_count=1, instance_type=‘ml.m5.xlarge’)
2021-04-08T02:47:05Z
@gopalkr272I am afraid I don’t have a complete answer but review my support ticket here:https://github.com/huggingface/transformers/issues/11043I hope this will help you find an answer.
2021-04-09T08:06:17Z
There are couple public samples showing Hugging Face model deployment on SageMaker, for example those:amazon-sagemaker-deploy-nlp-huggingface/deploy_pretrained_BART_seq2seq_PyTorch.ipynb at main · aws-samples/amazon-sagemaker-deploy-nlp-huggingface · GitHubServing PyTorch models in production with the Amazon SageMaker native TorchServe integration | AWS Machine Learning BlogGitHub - aws-samples/amazon-sagemaker-bert-classify-pytorch: This sample show you how to train BERT on Amazon Sagemaker using Spot instances