Use external embeddings
Hi,I’m trying to change GPT2 model in the following way:Instead of passing sentences through the tokenizer and tokenizer output to the model, I want to use my own defined external embeddings directly as input to the model, i.e. I want to get GPT2 12 pretrained layers as a model, and pass them as input to the embeddings vector. I saw that I can extract layers by:gpt.base_model.h, so I tried the following code (just for example):gpt = AutoModel.from_pretrained('gpt2')
model = nn.Sequential(*self.gpt.base_model.h)
model(torch.ones((1, 5, 768)))But I get an error:TypeError: layer_norm(): argument 'input' (position 1) must be Tensor, not tuple, has anyone encountered this scenario and know how to fix it?Thanks