Image-To-Text task on Inference Endpoint
I would like to deploy an Image-To-Text model on an Inference Endpoint. For example, the following model.huggingface.coSalesforce/blip-image-captioning-base · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.When I try to deploy this model, the task is set tocustom, but it fails to start because there is no handler.py.The error log indicates that there is a task calledimage-to-text, but this one cannot be selected from the Inference Endpoint configuration screen.KeyError: “Unknown task custom, available tasks are [‘audio-classification’, ‘automatic-speech-recognition’, ‘conversational’, ‘depth- estimation’, ‘document-question-answering’, ‘feature-extraction’, ‘fill-mask’, ‘image-classification’, ‘image-segmentation’, ‘image-to- text’, ‘ner’, ‘object-detection’, ‘question-answering’, ‘sentiment-analysis’, ‘summarization’, ‘table-question-answering’, ‘text- classification’, ‘text-generation’, ‘text2text-generation’, ‘token-classification’, ‘translation’, ‘video-classification’, ‘visual- question-answering’, ‘vqa’, ‘zero-shot-audio-classification’, ‘zero-shot-classification’, ‘zero-shot-image-classification’, ‘zero-shot- object-detection’, ‘translation_XX_to_YY’]”Is it not possible to set image-to-text to task in my Inference Endpoint? And if the only way is to set the task to custom, is there a handler.py template for image-to-text somewhere?
2023-07-14T14:27:57Z
Hi, wondering if there was any update to this? Looking for the same problem
2023-07-19T07:09:51Z
Working on adding it as zero-code deployment. Will keep you posted.
2023-07-25T10:29:10Z
Hello,Is there any alternate solution to this?
2023-07-28T13:38:34Z
Any update?
2023-07-29T13:37:58Z
I have used this modelhttps://huggingface.co/prasanna2003/blip-image-captioning. You can use its own API url.
2023-10-04T14:16:56Z
I am trying to do the same. Is there any help here? If i put it to Image-Classification its asking for input_ids, which i am not able to provide neither ui nor through my request.
2023-10-10T00:10:31Z
Any updates on this?  I need to do the same thing and with no success.
2023-10-10T00:12:46Z
The problem when I try to generate an inference endpoint, I get this warning:Warning:deploying this model may fail because a“handler.py”file was not found in the repository. Try selecting a different model or creating acustom handler.
2023-10-10T00:13:27Z
Hi@philschmid,any updates on this?
2023-10-12T16:36:52Z
@pdichoneThis is how you do it  through custom endpoints:huggingface.cohandler.py · ckandemir/relyst at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.However, I think the optimum method now is do follow along what@philschmidhas shared today for  Sagemaker implementationphilschmid blog – 12 Oct 23Deploy Idefics 9B & 80B on Amazon SageMakerLearn how to deploy Hugging Face Idefics 9B & 80B to Amazon SageMaker and send requests with images and text to the model.
2023-10-13T07:05:06Z
@thorikawaalso you can use this one is for the blip model:huggingface.cohandler.py · ckandemir/blip-image-captioning-large-inference at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2023-10-14T16:23:04Z
@ckandemirYou are a legend!Thank you so much.My struggle was that I once I had cloned the repo (the blip model repo), I wasn’t able to generate the correct dependencies like the pythorch_bin and the weights needed for me to be able to then attach my custom handler.py and then create the inference endpoint.Do you mind sharing how you were able to do the whole process ( sorry, I am a newby when it comes to this stuff, if you can’t tell… lol).For the handler.py, I don’t want the payload to be a list of images, I just want to be able to attach 1 image to be processed…Anyway, you’ve been a lot of help already, I would totally understand if this is too much to ask
2023-10-17T00:22:30Z
@pdichonebasically when you are creating a custom endpoint handler your are reconfiguring the forward method to align with the generate function of the  model  you are using behind the endpoint, and thus tweaking the payload accordingly to the underlying mode’s accepted input type.@philschmidplease correct me if I am wrongI would recommend to follow  this documentationhereto build couple random endpoints to get a hold on it. Then you can easily spot whats going on  in the  logs when you deploy your endpoints.You can tweak this codehereto serve your use case