LongBlender embedding positions mismatch
I have successfully implemented Longformer’s self attention in BlenderBot Small. However, when I try to load the new model: BlenderbotSmallLongForConditionalGeneration .from_pretrained(‘facebook/blenderbot_small-4096’) , this error occurs:RuntimeError: Error(s) in loading state_dict for BlenderbotSmallLongForConditionalGeneration:size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([4096, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).The mismatch between the embedding positions is intentional. I have used this project-GitHub - patil-suraj/longbart: A long version of BART model based on Longformer modelas an example and apparently, during transformers-2.10.0 this embedding positions mismatch was not a problem, because the author of the paper successfully loaded the Long version of Bart(which is similar to Blender). I have tried loading BartForConditionalGeneration.from_pretrained as well, but the same problem occurred again.My question is: How should I load the new model?Thanks a lot