New layer in bert embeddings
How to I add a new layer in bert embeddingThese are bert embeddings(embeddings): BertEmbeddings((word_embeddings): Embedding(30522, 768, padding_idx=0)(position_embeddings): Embedding(512, 768)(token_type_embeddings): Embedding(2, 768)(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)(dropout): Dropout(p=0.1, inplace=False))in which I would like to have(embeddings): BertEmbeddings((word_embeddings): Embedding(30522, 768, padding_idx=0)(position_embeddings): Embedding(512, 768)(token_type_embeddings): Embedding(2, 768)(custom_embeddings): Embedding(1,768)(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)(dropout): Dropout(p=0.1, inplace=False))My bert_embeddings= (word_embeddings)+(position_embeddings)+(custom_embeddings)
2022-04-01T23:22:51Z
Did you ever find an answer? I am hoping to do a similar thing