Is it possible to have streaming responses from inference endpoints?
I assume the answer is no. The inference endpoints seem like short lived connection endpoints only (similar to AWS lambdas).I understand it’s possible in huggingface spaces, for example via using server side events, but I would prefer to have the scaling capabilities of inference endpoints for my application.Is it possible?
2023-07-21T09:18:40Z
See for streaming responses:If you mean other protocols than HTTP then the answer is no.Deploy LLMs with Hugging Face Inference Endpoints
2023-07-21T16:51:56Z
Hi Phil. Thanks for the response!My use case is using a custom handler endpoint (handler.py), is there a way I can set that up to be able to work? I could setup, for example, a SSE streaming response within the handler.pyAny advice appreciated!Edit: For full context, my application is using llamaindex with a remote call to openai’s chatgpt-4. Chatgpt4 can give streaming output and I would like to forward the generator output using SSE to the client
2023-07-24T07:20:11Z
custom handler are only supporting traditional HTTP request ↔ response. To add this you would need to create a custom container yourself, that implements the feature.
2023-07-24T16:22:02Z
Is there a tutorial for this? I haven’t been able to get a custom container to work. It always ends up being stuck at initializing with the logs giving now insights. Does the model repository still need to have a handler.py file?
2023-07-24T16:49:08Z
huggingface.coUse a custom Container ImageWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2023-07-24T17:07:17Z
Hey Phil,It ended up being an issue on my end with setting the right AWS credentials.Vielen Dank für deine Hilfe.Volker