I want to use EncoderDecoderModel.from_pretrained() where time-series-transformer is the encoder and gpt2 as decoder
I want to use EncoderDecoderModel.from_pretrained() where huggingface/time-series-transformer-tourism-monthly is the encoder and openai-community/gpt2 as decoder but as mentioned inLeveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models, TS transformer does not have a tokenizer and hence not sure what to use as encoder input ids
2024-02-19T15:55:41Z
Hi,The Time-Series Transformer itself already has an encoder-decoder architecture, and is not supported byEncoderDecoderModel. This class is only meant to stitch together text encoder-only models with text decoder-only models (and encoder-only models like BERT which have support to be used as decoder).
2024-02-19T16:40:32Z
@nielsrBut I saw a function called get_encoder() in the transformer encoder-decoder modelling files.
2024-02-20T13:22:48Z
Kalsel Pintar- To useEncoderDecoderModel.from_pretrained(), you can load the Transformer model for the encoder usingTimeSeriesTransformer.from_pretrained()and the GPT-2 model for the decoder usingGPT2LMHeadModel.from_pretrained(). Then, you can combine them usingEncoderDecoderModel.from_encoder_decoder_pretrained(). Here’s the code example:from transformers import TimeSeriesTransformer, GPT2LMHeadModel, EncoderDecoderModel

# Load pre-trained encoder (TimeSeriesTransformer)
encoder_model = TimeSeriesTransformer.from_pretrained("your_time_series_transformer_model_name")

# Load pre-trained decoder (GPT-2)
decoder_model = GPT2LMHeadModel.from_pretrained("gpt2")

# Combine encoder and decoder
model = EncoderDecoderModel.from_encoder_decoder_pretrained(encoder_model, decoder_model)Make sure to replace"your_time_series_transformer_model_name"with the name of the TimeSeriesTransformer model you want to use.