TextGeneration Inference Model
Hi,Is there a way to calculate time taken to first token generating using TGI Model? Does TGI gives this metric by default in the generated response output or some other way to capture this time?
2023-09-22T08:11:33Z
This is the prefill time that you can see either in the prom metrics ortext-generation-benchmark. Cheers.
2023-09-22T16:03:21Z
Narsil:text-generation-benchmarkWhat is text-generation-benchmark ? Is it separate repo?