
Speech Encoder Decoder Models
The [SpeechEncoderDecoderModel] can be used to initialize a speech-to-text model
with any pretrained speech autoencoding model as the encoder (e.g. Wav2Vec2, Hubert) and any pretrained autoregressive model as the decoder.
The effectiveness of initializing speech-sequence-to-text-sequence models with pretrained checkpoints for speech
recognition and speech translation has e.g. been shown in Large-Scale Self- and Semi-Supervised Learning for Speech
Translation by Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli,
Alexis Conneau.
An example of how to use a [SpeechEncoderDecoderModel] for inference can be seen in Speech2Text2.
Randomly initializing SpeechEncoderDecoderModel from model configurations.
[SpeechEncoderDecoderModel] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [Wav2Vec2Model] configuration for the encoder
and the default [BertForCausalLM] configuration for the decoder.
thon