Missing schema for sagemaker deployment of zero-shot image classification models
Trying to get predictions on a deployed zero-shot image classification model and it appears as if the schema for the .predict() method is missing from the documentation. I have followed the schema example from the zero-shot text classification, but have changed the text input to an image and it doesn’t seem to be working. The model has been been deployed to an endpoint successfully, but fails when trying to produce an inference. I’m assuming its to do with the format of the inputs.I have also tried the following image representations and none have worked.Raw RGB data (Binary)Image itself - Object of type JpegImageFile is not JSON serializablebase64 encoded string is not a valid path 400 (invalid input parameters)bytes - Object of type bytes is not JSON serializablearray of bytes - Object of type bytes is not JSON serializableBytesIO object - Object of type bytes is not JSON serializableCode provided below:import torch, gcimport timefrom sagemaker.huggingface.model import HuggingFaceModelfrom sagemaker.serializers import DataSerializer, JSONSerializer, IdentitySerializerfrom sagemaker.deserializers import JSONDeserializerHub model configurationhttps://huggingface.co/modelshub = {‘HF_MODEL_ID’:‘openai/clip-vit-large-patch14’, # model_id fromModels - Hugging Face‘HF_TASK’:‘zero-shot-image-classification’     # Type of task your model is solving}create Hugging Face Model Classhuggingface_model = HuggingFaceModel(env=hub,                                                # configuration for loading model from Hubrole=role,                                              # IAM role with permissions to create an endpointtransformers_version=“4.17”,                             # Transformers version usedpytorch_version=“1.10”,                                  # PyTorch version usedpy_version=‘py38’,                                      # Python version used)#image_serializer= DataSerializer(content_type=‘image/x-image’) - didn’t workdeploy model to SageMaker Inferencepredictor1 = huggingface_model.deploy(initial_instance_count=1,instance_type=“ml.g4dn.4xlarge”,serializer=JSONSerializer(),deserializer=JSONDeserializer(),)image = Image.open(“000000039769.jpg”)byt = image.tobytes()mystr = base64.b64encode(byt).decode()_dict = {“bytes”: byt}newbytes = base64.b64decode(mystr)data = {“inputs”: mystr,“parameters”:{‘candidate_labels’:[“indoor”, “outdoor”]}}reponse = predictor1.predict(data)
2023-06-07T18:34:10Z
try{“input”: “http://someurl.com/someimage.jpg”,“parameters”: {“candidate_labels”: list_of_zero_shot_labels}}
2023-08-31T11:56:57Z
Hi, I tried supplying image as http url and it works. But if I want to host the model and send image byte string, how do I provide image? thanks