What does the `use_cache` in `generate` actually do?
This is an offshoot fromWhy past_key_values is not in GreedySearchDecoderOnlyOutput? -Transformers - Hugging Face ForumsWhat does theuse_cacheingenerateactually do? Since there is no way to provide thepast_key_valuesto thegeneratefunction, does that mean that generate internally storespast_key_valuesbetween runs (say I rungeneratein a loop for a chatbot)?
2024-05-09T15:28:33Z
@nasheedYou can find the answerhereTL;DR: passreturn_dict_in_generate=Truetogenerate→generatereturnspast_key_values→ passpast_key_values=past_key_valuestogenerate