Streaming token output from models like T5
Hey, I’m looking to stream tokens out of text2textgeneration models (more specifically those from the T5 family). Similar to what you see in OpenAI playground when requesting generations.For text-generation models I’ve been able to emulate this (since we’re just predicting current token from the ones before it) by setting a max_time parameter and appending model generations to the prompt to continuously call until I’ve reached the number of tokens I wanted to generate. But I can’t do the same for text2text models.Any advice on how I could stream output from models like T5? Is it even possible with the architecture?
2023-02-03T22:46:56Z
Got a solution working, ingenerate()for the different types of sampling for examplegreedy_search()there is anext_tokenvariable you can incrementally get the subsequent tokens generated by the model as soon as they are done. You’ll have to decode it yourself and encode the special rules you’d get fromdecode()but it works well. Monkey patched it with a newgreedy_search()yieldingnext_tokenon each generation. Hope this helps for anyone looking to do the same.
2023-03-07T14:42:21Z
Inspired by the solution form@zhuda, I made a streaming generation service for Hugging Facetransformersthat is fully compatible with the OpenAI API:https://github.com/hyperonym/basaran
2023-03-14T08:40:02Z
Hey guys, why didn’t you register one of the callbacks in generate and choose instead to monkey patch?
2023-03-14T23:20:57Z
This is awesome, can’t wait take a proper look!
2023-03-27T07:11:41Z
Hi,We recently released a bunch of open-source tools to do this with any HuggingFace model.Check out our new library “text-generation-inference”:GitHub - huggingface/text-generation-inference: Large Language Model Text Generation Inference. It powers this Space:Chat Llm Streaming - a Hugging Face Space by olivierdehaene.cc@olivierdehaene
2023-06-05T11:10:05Z
Update:We now also support a newStreamerclass that works in tandem with thegeneratemethod.Here’s a great Twitter thread by@joaogantegoing over it:https://twitter.com/joao_gante/status/1643330507093196800
2023-06-07T22:27:13Z
This is perfect, will port over to this solution soon for OpenPlayground