Streaming output text when deploying on Sagemaker
Hi,I’m working on fine-tuning bloom and deploying the model on Sagemaker and I wanted to know if it’s possible to stream the output generated text, with directly modifying the inference functions?I already tried with applying the TextIteratorStreamer within predict_fn function, but this doesn’t seem to be the solution.Any idea about this would be really appreciate
2023-05-12T13:58:44Z
SageMaker is currently not supporting streaming responses.
2023-05-12T14:39:22Z
Ok thank you@philschmid
2023-05-16T15:06:34Z
Hi@Ludivine, hi@philschmid,I’m also trying to stream in a GPT like style the output of my LLM. I’m trying to use https request and an iterator streamer. However, it would be great if it was a built-in possibility with huggingface/sagemaker.Thanks for your great job.
2023-06-11T09:55:22Z
Hey@RemiPthanks for your response. Can you pls elaborate how are you streaming outputs from the LLM deployed as HuggingFace inference endpoint? Appreciate you help:)
2023-10-06T09:48:17Z
Sagemaker real-time inference endpoints now supports streaming. Check thisblog