Fine Tuning GPT2 for machine translation

🤗Transformers
1

good evening everyone,

is it possible to fine-tune gpt2 for text translation?

if it is possible, how can I do it using my own data?

I want to translate from ASL to English, and the idea that came to me was to use gpt2 as the decoder (since it is trained in English) and use a BERT as an encoder (I would fine tune it and retrain with the ASL base)

Does anyone have a tutorial on how to do something like this?

2

hey @yansoares, you could try using the EncoderDecoderModel (docs) to create a BERT2GPT model. I’m not aware of any tutorials using this class, but the original paper (link) is quite well written and should give you a good idea on what needs to be done.