Weights in BERT model
I have a BERT modelmodel = BertForMaskedLM.from_pretrained('bert-base-uncased').I notice thatmodel.state_dict().keys()has 205 elements, while looking at the[i[0] for i in list(model.named_parameters())]only has 202 elements. The three missing elements are ‘bert.embeddings.position_ids’, ‘cls.predictions.decoder.weight’, and ‘cls.predictions.decoder.bias’. The first is inmodels.named_buffers(), but how about the other two? What are these doing?Another question: I noticed thatmodel.named_buffers()also containsbert.embeddings.token_type_ids, but this is not inmodel.state_dict(). Why isn’t this the case? Shouldn’t it be important to store this in the state as well?
2023-04-12T03:06:11Z
I have the same question.But I found thatcls.predictions.decoder.weightandbert.embeddings.word_embeddings.weighthave the same weights.