

TITLE
Streamable way in model.generate

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### Feature request

As the llm becomes bigger and bigger, a quite slow speed in generate caused long time waiting for whole response.

would it be OK add a `model.stream_generate` or something like for users to use?

Actually some community guys implementated such a thing, but this could be a problem too: users don't which is reliable to use, it could be better for official introduce such as way.

I noticed there are some work form hf like text_generation, but what users might want is simple a tiny API along side model.geneate rather than a whole lib

### Motivation

For instance huge models inference.

### Your contribution

not yet