Combining tokenizer.decode and model.generate scores for probability prediction
I want to use the input_id level scores provided bymodel.generate(return_scores=True)with thedecodefunction of theTokenizer.The logit scores are input_id level and the decoder decodes to token level.My desired output for a prompt of “hello, how are you” would be "doing (p=0.8) today (p=0.8 * 0.7) sir (p=0.8 * 0.7 * 0.7) for example.How can I achieve this?
2023-03-01T17:39:23Z
Hey@Shaike04Have a look at our documentationhere. I believe the first example does precisely what you want!
2023-03-01T19:54:30Z
Thank you so much, I am looking into it.