Multiple responses with async generate in TGI
I am using thegenerate()function fromAsyncClientfromText Generation Inferencet query a bunch of models. If I want to generate more than one candidate responses for a given prompt, how do I do that?I see anparameter (number of responses to generate) in thechat()function but none in thegenerate()function.The naive method I am working with right now is to just repeat the same prompt multiple times, but I am wondering if there is a better way?On a related note, what is the difference between using these two functions?
2024-04-23T20:12:49Z
Hey@shaily99,For thegeneratefunction, you can pass thebest_ofarg to the TGI client and that’ll make it return multiple candidates given your input prompt. For example:response = await client.generate(
    prompt,
    do_sample=True,
    best_of=4,
)

# this output is the highest probability response, which HF assumes is the "best"
best = output.generated_text

# you can get the other candidates like this
other_candidates = [seq.generated_text for seq in output.details.best_of_sequences]

# and just combine all of them
all_candidates = [best] + other_candidatesBy default, TGI only lets you generate up to 2 candidates. But when you’re starting up TGI, you can pass the--max-best-ofarg (reference in the docs) if you want more. For example:docker run \
    --rm \
    -it \
    --gpus '"device=0"' \
    -p $port \
    ghcr.io/huggingface/text-generation-inference:2.0.1 \
    --model-id $model_path \
    --sharded false \
    --dtype bfloat16 \
    --max-best-of 4 \ # <-- set this to whatever you wantOn a related note, what is the difference between using these two functions?I’ve never used thechatfunction before and didn’t even know it existed until now, but it looks to me like thechatfunction is intended specifically for chat-based models where you have dialog turns. And it lets you pass a list ofMessageobjects in and then the TGI code will handle formatting that into the prompt string given to the model.In contrast, thegeneratefunction expects the prompt to already be formatted correctly and ready to go.Sogenerateis more general purpose thanchat, but requires a little work on your end because you have to format the inputs correctly yourself