Image-to-image Stable Diffusion Inference Endpoint?
Hey there,I’m trying to setup an image-to-image, stable diffusion Inference Endpoint.I sawCompVis/stable-diffusion-v-1-4-originalwhich seems setup for the Inference Endpoint, but noticed this states it is only text-to-image, not image-to-image, so I assume this wouldn’t support generating images from an existing image.Then I looked at usinglambdalabs/sd-image-variations-diffusers, but upon getting ready to set this up, I noticed this message:Deploying this model with a custom task may fail as a handler.py file was not found in the repository. Refer to thedocumentationfor more details.Screenshot 2022-10-14 at 12.23.471910×2452 390 KBWhen searching for a stable-diffusion model,lambdalabs/sd-image-variations-diffuserswas the only result.Islambdalabs/sd-image-variations-diffuserssetup to work with the Inference Endpoint, or is there another model that is?Thanks folks!Fred.
2022-10-14T12:16:53Z
Hello Fred,Yes, lambdalabs/sd-image-variations-diffusers is supported by Inference Endpoints by using acustom handler.Hugging Face Inference Endpoints be default support all of theTransformers and Sentence-Transformers tasks. if you want to deploy a custom model or customize a task, e.g. for diffusion you can do this by creating aCreate custom Inference Handlerwith ahandler.py.The warning you saw indicates that the model doesn’t have a supported task and doesn’t include ahandler.py.We already have an example on how you would deploy stable-diffusion:philschmid/stable-diffusion-v1-4-endpoints · Hugging Face. With a custom handler you can customize the request payload to accept images and text, see thiszero-shot-image-classification example.
2022-10-14T13:09:57Z
Thanks so much for the swift reply here@philschmid.This all seems super powerful, and your comment makes sense. It seems like there’s two paths ahead:Uselambdalabs/sd-image-variations-diffusersand look to implement my own custom handler (handler.py)UseCompVis/stable-diffusion-v-1-4-originaland try to customise it to accept an image payloadI’ve used python alittlebefore, but I’m a front-end dev by trade, so a little out of my depth here, I’ll look into this and try and see how far I can get with it, I’ll probably prioritise trying#1.If you’ve any other tips or resources that are worth looking at, let me know!Cheers.
2022-10-21T13:06:44Z
Fred,Did you get this to work.  My install fails repeatedly with this in the tail of the  logs.tbghq 2022-10-21T12:58:51.065Z 2022-10-21 12:58:51,065 | INFO | Found custom pipeline at /repository/handler.pytbghq 2022-10-21T12:58:51.098Z File “/opt/conda/lib/python3.9/site-packages/starlette/routing.py”, line 566, inaentertbghq 2022-10-21T12:58:51.098Z 2022-10-21 12:58:51,098 | ERROR | Traceback (most recent call last):tbghq 2022-10-21T12:58:51.098Z File “”, line 850, in exec_moduletbghq 2022-10-21T12:58:51.098Z File “/app/./huggingface_inference_toolkit/handler.py”, line 44, in get_inference_handler_either_custom_or_default_handlertbghq 2022-10-21T12:58:51.098Z File “/opt/conda/lib/python3.9/site-packages/starlette/routing.py”, line 671, in lifespantbghq 2022-10-21T12:58:51.098Z 2022-10-21 12:58:51,098 | ERROR | Application startup failed. Exiting.tbghq 2022-10-21T12:58:51.098Z inference_handler = get_inference_handler_either_custom_or_default_handler(HF_MODEL_DIR, task=HF_TASK)tbghq 2022-10-21T12:58:51.098Z async with self.lifespan_context(app):tbghq 2022-10-21T12:58:51.098Z ValueError: need to run on GPUtbghq 2022-10-21T12:58:51.098Z File “/opt/conda/lib/python3.9/site-packages/starlette/routing.py”, line 648, in startuptbghq 2022-10-21T12:58:51.098Z File “/repository/handler.py”, line 13, intbghq 2022-10-21T12:58:51.098Z File “/app/./huggingface_inference_toolkit/utils.py”, line 202, in check_and_register_custom_pipeline_from_directorytbghq 2022-10-21T12:58:51.098Z spec.loader.exec_module(handler)tbghq 2022-10-21T12:58:51.098Z await self._router.startup()tbghq 2022-10-21T12:58:51.098Z await handler()tbghq 2022-10-21T12:58:51.098Z File “/app/./webservice_starlette.py”, line 56, in some_startup_tasktbghq 2022-10-21T12:58:51.098Z raise ValueError(“need to run on GPU”)tbghq 2022-10-21T12:58:51.098Ztbghq 2022-10-21T12:58:51.098Z File “”, line 228, in _call_with_frames_removedtbghq 2022-10-21T12:58:51.098Z custom_pipeline = check_and_register_custom_pipeline_from_directory(model_dir)
2022-10-21T13:30:15Z
Hello@dano1234,It seems that you are not using a GPU instance.raise ValueError(“need to run on GPU”)
2022-10-21T14:27:29Z
Phil,Thanks for quick reply.  I get that error before I have a chance to specify anything about a GPU.  I only filled in the “philschmid/stable-diffusion-v1-4-endpoints” AWS and region.  Then it goes to next screen and eventually fails before I can set anything else.  I am probably missing something.Dan
2022-10-21T14:41:44Z
@dano1234to select the instance type you need to click on “advanced configuration” and there you can select the instance type:image1201×1076 138 KBYou can learn more about in the documentation:Advanced Setup (Instance Types, Auto Scaling, Versioning)
2022-10-26T09:45:09Z
Thanks !  BTW when it says ~$450/month does it charge you that in a lump or is it pay as you go.  IE can I try this out without being on the hook for monthly charge?
2022-10-27T06:41:02Z
Hey@dano1234,At the end of each month the user or organization account will be charged for the compute resources used while Endpoints are up and running, that is, in the “running” state.So you will only pay the minutes of the uptime of you endpoint. You can learn more here:Hugging Face Inference Endpoint Pricing
2022-11-04T11:02:24Z
Thanks worked great.
2022-11-28T04:17:18Z
This is a total noob question, but I guess I have to ask it somewhereThe repophilschmid/stable-diffusion-v1-4-endpointsdoesn’t seem to show up as an available model in the Model Repository dropdown of theCreate a new Endpointpage.From this discussion, it looks like it used to show up since folks here seem to be using it as a starting point to set up a working stable diffusion based inference endpoint?
2022-11-29T00:34:09Z
Today thephilschmid/stable-diffusion-v1-4-endpointsrepo is showing up in the inference endpoint UI, along with a bunch of other diffusion models.  Maybe there was a bug that was fixed recently.  In any case, I’m unblocked now.
2022-11-29T04:34:05Z
Hey@dskill,We launched some new exciting features yesterday:Philipp Schmid on LinkedIn: Stable Diffusion with Hugging Face Inference EndpointsWhich added support for alltext-to-imagediffusion models based on StableDiffusionPipeline.
2022-11-29T06:19:24Z
Lucky timing for me!I did see your excellent blog post earlier today and was able to get an inference endpoint up and running with text-to-image.  It was extremely helpful.While I have you, one follow up question that’s been a bit confusing for me to untangle:  How similar is the inference API to the endpoint API?  Are they in fact identical?I also noticed that the inference API is turned off forhttps://huggingface.co/stabilityai/stable-diffusion-2.  Is there a way for me to develop using model with the inference API somehow, or is using endpoint API the only way for now?Again, many thanks.
2022-11-29T06:54:51Z
The Inference API is free and designed for testing and experimenting with models onhuggingface.co. It runs on shared infrastructure. This means that you are sharing the resources with other users, which could lead to high latency and low throughput there is no guarantee that the model has a GPU. The Inference API is not providing SLAs or other production-required features, like logs and monitoring.Inference Endpoints on the other side support all of this.
2022-11-29T07:09:44Z
OK got it - so in some cases the inference API may not be available, but that’s not something that I as an end user will have any control over.In general though, it sounds like the intended workflow is develop on the inference API where it’s available, and then graduate to endpoint API when you require the production solution.  Thanks again for the responses!