Inference Input for Vision Models
Hi,I set up a serverless inference endpoint for YOLOS but I’m failing a bit at using it for prediction.The example notebook uses a text model and that just passes a string, but for YOLOS (and other vision models) what do I put into the predictor?I tried a numpy array from an PIL Image but that gave me this Client Error:{
/home/johannes/Projects/Vision/object-detection/src/object-detection/sagemaker_serverless.ipynb Cell 12' in <cell line: 2>()
      1 pil_im = Image.open("20190601_14005820190911-1-u4n8ej.jpg")
----> 2 REDACTED_TOKEN.predict(np.asarray(pil_im))

File ~/.local/share/virtualenvs/object-detection-a0LJrJAA/lib/python3.9/site-packages/sagemaker/predictor.py:161, in Predictor.predict(self, data, initial_args, target_model, target_variant, inference_id)
    131 """Return the inference from the specified endpoint.
    132 
    133 Args:
   (...)
    155         as is.
    156 """
    158 request_args = self._create_request_args(
    159     data, initial_args, target_model, target_variant, inference_id
    160 )
--> 161 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
    162 return self._handle_response(response)

File ~/.local/share/virtualenvs/object-detection-a0LJrJAA/lib/python3.9/site-packages/botocore/client.py:508, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    504     raise TypeError(
    505         f"{py_operation_name}() only accepts keyword arguments."
    506     )
    507 # The "self" in this scope is referring to the BaseClient.
--> 508 return self._make_api_call(operation_name, kwargs)
...
--> 915     raise error_class(parsed_response, operation_name)
    916 else:
    917     return parsed_response

ClientError: An error occurred (413) when calling the InvokeEndpoint operation:
2022-07-23T12:16:42Z
Hello@johko,here is an example for image segmentation, which you can adopt for Yolos:notebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHub
2022-07-23T21:02:00Z
Thank you Philipp, I haven’t found that notebook before. I can run the example for the Segformer in there without a problem, also for serverless.But if I try to adapt this for YOLOS (serverless or not does not matter), I do get the above error.Here is the code I’m trying for the YOLOS serverless endpoint:hub = {
    'HF_MODEL_ID':'hustvl/yolos-tiny',
    'HF_TASK':'object-detection'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    sagemaker_session=sess,
    env=hub,                      # configuration for loading model from Hub
    role=role,                    # iam role with permissions to create an Endpoint
    transformers_version="4.17",  # transformers version used
    pytorch_version="1.10",        # pytorch version used
    py_version='py38',            # python version used
)

# Specify MemorySizeInMB and MaxConcurrency in the serverless config object
serverless_config = ServerlessInferenceConfig(
    memory_size_in_mb=4096, max_concurrency=10,
)

# deploy the endpoint
yolos_predictor = huggingface_model.deploy(
    endpoint_name="yolos-object-detection-serverless",
    serverless_inference_config=serverless_config,
    serializer=image_serializer
)

yolos_predictor.predict(data="example_resized.jpg")
2022-07-25T05:02:08Z
Ah my bad@johko! Yolos got added after4.17.0and we are still stuck on the release for the next DLC. What you could do is create arequriements.txtaddtransformers==4.20.1to it and deploy the model with a “custom” script.notebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHub
2022-07-25T08:03:34Z
I see, thank you@philschmid.I’ll try that and if it takes to much time I’ll try DETR, as I think that should already be included in transformers 4.17.0 and should be good enough for my needs
2022-07-25T10:46:37Z
Not sure if DETR will work since it requirestimmwhich is not installed by default. So there would be arequirements.txtneeded as well.
2022-07-26T10:43:13Z
I got YOLOS to work with the custom script (mainly following Niels from this notebook:Transformers-Tutorials/YOLOS_minimal_inference_example.ipynb at master · NielsRogge/Transformers-Tutorials · GitHub) and the newer transformer version.Thank you for the hints@philschmid