Why past_key_values is not in GreedySearchDecoderOnlyOutput?
I am using amodel = GPT2LMHeadModel()for generation. In my use case, I’ll need to callmodel.generate()for multiple times, and theinput_idshave a shared prefix.In my understanding, I could passpast_key_valuesas an argument inmodel.generate()so that it wouldn’t repeatedly compute the key, values of the shared prefix. However, how do I get thispast_key_values? Thegenerate()function returns aGreedySearchDecoderOnlyOutputobject (I set beam size = 1, no sampling), which does not containpast_key_values. It’s only in the return ofmodel.forward()?So I’m wondering what would be a typical example of usingpast_key_valuesin multiple calls ofmodel.generate(). Thanks!
2022-10-04T22:55:03Z
Never mind. There’s a PR for it.github.com/huggingface/transformers[generate] return past_key_valueshuggingface:main←patil-suraj:ret-cache-generateopened02:54PM - 06 Jun 22 UTCpatil-suraj+147-0# What does this PR do?

Allows returning `past_key_values` from `generate` when…`use_cache=True`.

Like other returned values, `past_key_values` are also returned as `Tuple`, one element per generated token.

Fixes #17016