
Encoder Decoder Models
Overview
The [EncoderDecoderModel] can be used to initialize a sequence-to-sequence model with any
pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder.
The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks
was shown in Leveraging Pre-trained Checkpoints for Sequence Generation Tasks by
Sascha Rothe, Shashi Narayan, Aliaksei Severyn.
After such an [EncoderDecoderModel] has been trained/fine-tuned, it can be saved/loaded just like
any other models (see the examples for more information).
An application of this architecture could be to leverage two pretrained [BertModel] as the encoder
and decoder for a summarization model as was shown in: Text Summarization with Pretrained Encoders by Yang Liu and Mirella Lapata.
Randomly initializing EncoderDecoderModel from model configurations.
[EncoderDecoderModel] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [BertModel] configuration for the encoder and the default [BertForCausalLM] configuration for the decoder.
thon