InternalServer Exception when deploying fine tuned model on Sagemaker
Hello everyone,I got a fine tuned german classification model which I want to deploy on Sagemaker. To test my model I want to predict a couple of texts in the json format. Here is my code:import csv
import json
import sagemaker
from sagemaker.s3 import S3Uploader,s3_path_join

# get the s3 bucket
sess = sagemaker.Session()
role = sagemaker.get_execution_role()
sagemaker_session_bucket = sess.default_bucket()


# uploads a given file to S3.
input_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/input")
output_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/output")
#s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)


# datset files
dataset_csv_file="test"
dataset_jsonl_file="test.jsonl"

with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+") as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        # remove @
        json.dump(row, outfile)
        outfile.write('\n')

s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)

#package my fine tuned model
!cd model_token && tar zcvf model.tar.gz * 
!mv model_token/model.tar.gz ./model.tar.gz

model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
model_uri = S3Uploader.upload('model.tar.gz',model_url)
print(f"Uploaded model to {model_uri}")

from sagemaker.huggingface.model import HuggingFaceModel

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',    
    split_type='Line')When I execute that batch tranform job I get the following error:“code”: 400,2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch_transform/input/test.jsonl:   “type”: “InternalServerException”,2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch_transform/input/test.jsonl:   “message”: “text input must of typestr(single example),List[str](batch or single pretokenized example) orList[List[str]](batch of pretokenized examples).”What am I missing?
2021-09-14T12:44:05Z
Hey@marlon89,could you share the cloudwatch logs? They should provide more information to why the batch transform job is failing.Additionally, can you share an example of yourtest.jsonl?
2021-09-14T13:11:21Z
Sure! I am super new to the whole AWS environment but I try…Here is a part of my test.json:{"input": "Machen Sie so weiter...."}
{"input": "."}
{"input": "Bleiben Sie so wie bisher!"}
{"input": "Weiter machen"}
{"input": "Passt alles. "}
{"input": "F\u00fcr alle Vergleiche, die bei mir anstehen, benutze ich Ihre Plattform schon seit Jahren. Ein Beweis f\u00fcr meine Zufriedenheit. "}
{"input": "alles bestens, vielen Dank!"}When I create a realtime endpoint with the predict function it works:predictor = huggingface_model.deploy(
   initial_instance_count=1, 
   instance_type="ml.m5.xlarge"
)

data = {
   "inputs": "Weiter machen"
}
predictor.predict(data)
2021-09-14T13:22:32Z
marlon89:{"input": "Machen Sie so weiter...."}inputneeds to beinputs. Then it should work.P.S. there might also be an additional issue with encoding, looking at"F\u00fcr alle Vergleiche,i guess your data is not properly encoded/decoded toutf-8
2021-09-14T13:43:20Z
It works! Thanks for your super fast help!!!