Calling Sagemaker Endpoint for fine-tuned summarization model
Hi HuggingFace community,I’m attempting to deploy a fine-tuned T5 model for summarization using a SageMaker Endpoint. The endpoint is deployed successfully with the following code:from sagemaker.huggingface.model import HuggingFaceModel

huggingface_model = HuggingFaceModel(
   model_data="s3://my-s3-path/model.tar.gz",
   role=role,
   transformers_version="4.6",
   pytorch_version="1.7",
   py_version='py36',
)

predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge",
    endpoint_name="my-endpoint-name"
)I then try to call the endpoint:from sagemaker.huggingface.model import HuggingFacePredictor

predictor = HuggingFacePredictor(endpoint_name="my-endpoint-name", sagemaker_session=sess)

predictor.predict({'inputs': 'this is a string',
                  'parameters': {'max_length': 20,
                                'min_length': 1}
                  })And I get the following error:ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "(\"You need to define one of the following [\u0027feature-extraction\u0027, \u0027text-classification\u0027, \u0027token-classification\u0027, \u0027question-answering\u0027, \u0027table-question-answering\u0027, \u0027fill-mask\u0027, \u0027summarization\u0027, \u0027translation\u0027, \u0027text2text-generation\u0027, \u0027text-generation\u0027, \u0027zero-shot-classification\u0027, \u0027conversational\u0027, \u0027image-classification\u0027] as env \u0027TASK\u0027.\", 403)"
}Can anyone tell me where I should define that I want a summarization model? I can’t find anything in the docs to tell me this. Thanks!
2022-07-05T13:09:37Z
Hello@drcat101,It looks like themodel.tar.gzis malformed. How have you created it? You can check out the documentation:Deploy models to Amazon SageMaker
2022-07-05T13:50:04Z
Thanks@philschmidfor your comment, but that wasn’t the problem in this case, themodel.tar.gzwas fine.I just found the solution - I needed to add an extra parameter to the model like so:env = {'HF_TASK': 'summarization'}

huggingface_model = HuggingFaceModel(
   model_data="s3://my-s3-path/model.tar.gz",
   role=role,
    env=env,
   transformers_version="4.6",
   pytorch_version="1.7",
   py_version='py36',
)This is not in the documentation anywhere for fine-tuned models, but does appear in the tests for the inference package here:sagemaker-huggingface-inference-toolkit/test_models_from_hub.py at main · aws/sagemaker-huggingface-inference-toolkit · GitHub
2022-07-05T14:15:36Z
Interesting! Thanks for following up, normally if you train the model in sagemaker and then deploy the task should be detected automatically. can you share theconfig.jsoninside youmodel.tar.gz?
2022-07-05T14:50:12Z
Here’s theconfig.json. I don’t think I specified the task anywhere in the fine-tuning process, but it works…Screen Shot 2022-07-05 at 4.44.00 PM510×1256 102 KB
2022-07-05T15:08:17Z
This should normally detect thetext2text-generationpipeline. Can you please check that your model is loaded and not the default one from transformers? Meaning if you just provide ataskin the pipelinetransformerswill load a default model.Also, what is the structure of your archive/how have you created it?
2022-07-05T15:18:33Z
Yes, my model is loaded and not the default model, the archive is fine, and everything is now working great. I followed the steps in the O’Reilly book with the Pegasus example.
2023-02-22T08:26:30Z
I have a similar error: I am using a custom inference.py file with a custom model_fn.My model ismulti_label_classificationmodel and I have tried to pass the problem type as anENVvariable:env = {'HF_TASK': 'multi_label_classification'}but no luck.Any clue?
2023-02-22T08:43:45Z
Hello@Pidem,when you implement a custominference.pyyou don’t have to pass any ENV.
2023-02-22T13:29:01Z
I tried this as well but that didn’t work.  How can I make sure that the inference.py gets picked up ? It is in the code/ folder in my case.
2023-02-22T14:28:17Z
How have you created yourinference.py/ created themodel.tar.gz?Here is an example:Deploy FLAN-T5 XXL on Amazon SageMakermaybe that helps
2023-02-22T16:04:09Z
Yes, my inference.py file is in my model.tar.gz:from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

# Helper: Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


def model_fn(model_dir):
    # Load model from HuggingFace Hub
    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    model = AutoModel.from_pretrained(model_dir)
    return model, tokenizer

def predict_fn(data, model_and_tokenizer):
    # destruct model and tokenizer
    model, tokenizer = model_and_tokenizer
    
    # Tokenize sentences
    sentences = data.pop("inputs", data)
    encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

    # Compute token embeddings
    with torch.no_grad():
        model_output = model(**encoded_input)

    # Perform pooling
    sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

    # Normalize embeddings
    sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
    
    # return dictionary, which will be json serializable
    return [{"vectors": sentence_embeddings[i].tolist()} for i in range(len(sentence_embeddings))]and my model.tar.gz folder structure is:code/inference.py
pytorch_model.bin
tokenizer.jsonmy config.json file says:problem_type:"multi_label_classification"and here is the error message I am seeingimage1920×574 124 KB
2023-02-22T20:05:12Z
Found the error. In case someone runs into the same error in the future: I had renamed the model.tar.gz to model_inference.tar.gz and apparently that creates an error
2023-06-15T22:44:30Z
After breaking my head on this for one whole day. The issue is with how its suggested to generate the tar file.When I created the tar file using below, I got the error.!tar -zcvf model.tar.gzHowever, generating tar using below worked like a charmimport tarfile
import os

# helper to create the model.tar.gz
def compress(tar_dir=None,output_file="model.tar.gz"):
    parent_dir=os.getcwd()
    os.chdir(tar_dir)
    with tarfile.open(os.path.join(parent_dir, output_file), "w:gz") as tar:
        for item in os.listdir('.'):
          print(item)
          tar.add(item, arcname=item)
    os.chdir(parent_dir)

compress(str(model_id))
2024-02-01T06:53:32Z
When you created tarfile with “tar -zcvf model.tar.gz”, you just include the “model” directory in the target file. This would incur this error. You may try something like “tar -zcvf model.tar.gz -C model .”  to exclude the “model/” directory itself but only compress files inside it.
2024-03-22T13:20:42Z
Yeah you’re right. I’ve tested that. It works now. Thank you for sharing.