Why model.generate does encoding multiple times
Hi community, I have a question on the implementation ofmodel.generate. From my understanding, an encoder-decoder-based model first encodes the input context, and then autoregressively decodes the outputs, which means it should only encode once and decodes multiple times.This blog postalso supports my point of view. However, the detailed implementation of the high-level APImodel.generateis strange. Atthis line, theinput_idsis appended with new decoded output, which is then sent to the entire encoder-decoder model. Is it on purpose? Thanks!
2022-09-20T20:03:33Z
I think I got it. If it is an encoder-decoder based model, the algorithm would perform encoding first and save the result tomodel_kwargs, and at the meantimeinput_idswould always be the decoder’s output.