BERT2BERT Notebook for Models without GenerationMixin
Hello,I have just stambled upon the “Warm-starting BERT2BERT for CNN/Dailymail” Notebook@patrickvonplatenkindly shared with us:https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynbFrom my understanding, one can use this notebook for fine-tuning any modelForConditionalGeneration, such as T5, Pegasus, ProphetNet, etc. Can you please confirm this?Moreover, I am interested in the new Longformer and Reformer models, which you can feed much longer sequences. These two models do not have aForConditionalGenerationclass. However, I was wondering if they could be fine-tuned on summarization tasks using the same script, e.g. changingfrom transformers import EncoderDecoderModelbert2bert = EncoderDecoderModel.from_encoder_decoder_pretrained(“bert-base-uncased”, “bert-base-uncased”)Withfrom transformers import LongformerModel, ReformerModellongformer = LongformerModel.from_pretrained(“allenai/longformer-base-4096”)reformer = ReformerModel.from_pretrained(“google/reformer-enwik8”)Thank you for your help!