Incremental decoding with T5
Recently, we have seen evidence that in a variety of tasks, it may be helpful for a model to attend over intermediate computation steps when solving a task. An example isReAct: Synergizing Reasoning and Acting in Language Models – Google AI Blog (googleblog.com). The authors cite some work from the neural program synthesis community where this approach was found beneficial.Let’s assume we are processing conversations, where the context is progressively longer as the user and agent interact. Typically, we would re-encode the dialogue history and generate the answer from scratch for every interaction. Schematically, this could be represented as follows:step 1:[usr] sent_1→answer_1step 2:[usr] sent_1 [agent] sent_1 [usr] sent_2→answer_2…step k:[usr] sent_1 [agent] sent_1 [usr] sent_2 ... [agent] sent_k [user] sent_k→answer_kAbovesentis just an abbreviation for “sentence”. The LHS of “->” is the encoder input, the “RHS” is the decoder output. However, the answers are highly correlated, so arguably the model could predict more consistently if it was asked to show all the reasoning steps as the conversation progresses, instead of producing a single answer for the task. Schematically:step 1:[usr] sent_1→answer_1step 2:[usr] sent_1 [agent] sent_1 [usr] sent_2→answer_1<sep>answer_2…step k:[usr] sent_1 [agent] sent_1 [usr] sent_2 ... [agent] sent_k [user] sent_k→ answer_1<sep>answer_2<sep>…<sep>answer_kIn inference, this is problematic because concatenating the answers can lead to very long sequences if everything was generated from scratch. However, I was wondering if theuse_cachefeature together with thepast_key_valuecould be used to effectively implement a memory on the decoder side? In the above, after we decodeanswer_1we feed back the keys and values generated during decoding aspast_key_valuesto decodeanswer_2. Then we would feed back the outputs to generateanswer_3and so on.  So the model could attend over an updated conversational context and its past answers but would not “revise” all its previous answers.@patrickvonplaten, am I naive to think that the caching during inference could be implemented withhuggingfaceas is?