Missing keys in RobertaForMaskedLM state dict
Hello,I am training my model that is based on RobertaForMaskedLM and after saving it I get a pytorch_model.bin file from which I want to retrieve certain layer parameters.After I dostate_dict = torch.load(path_to_pytorch_model.bin, map_location=‘cpu’)and inspect the state_dict.keys() I observe that two parameters are missing: lm_head.decoder.weight and lm_head.decoder.biasFrom what I understand, lm_head.decoder.weight = roberta.embeddings.word_embeddings.weight which is due to the tied weights.What I have not figured out is where are stored the parameters of lm_head.decoder.bias. Are these unused parameters ?Thanks for your help !
2022-08-05T13:55:27Z
Those are not used since they are tied weight. You should usestrict=Falsewhen loading the model then retie the weights withroberta.tie_weights()(or jsut usefrom_pretrainedwhich will do all of that for you).
2022-08-05T14:20:18Z
Thank you for your reply. These two ways you describe indeed give me same resultscalling from_pretrainedor load_state_dict with strict=False followed by tie_weights()then the parameters for lm_head.decoder are the same both in terms of weight (same as that of the embeddings.word_embeddings) and bias.What still puzzle me is to what is tied the bias parameters (or where it is stored because it is not in the state_dict keys), this I could not get it from the source code.If for example I set to zeros the lm_head.decoder.bias then the model output is not the same anymore.And if I check for gradients during the backward pass, the param.grad from lm_head.decoder.bias are neither None or zero which would mean it is also trained.
2022-08-05T14:32:27Z
going through all parameters with torch.allclose against lm_head.decoder.weight and lm_head.decoder.bias seems to have given me the answertorch.allclose(roberta.embeddings.word_embeddings.weight, lm_head.decoder.weight) = Truetorch.allclose(lm_head.bias, lm_head.decoder.bias) = Trueso it seems that lm_head.decoder.bias and lm_head.bias are tied … is that right ?do you know why is this bias parameter replicated ?
2022-08-05T14:34:31Z
This is a very hack way of ensuringlm_head.biasgets resized when we add new tokens, since onlylm_head.decodergets passed to that function, coupled with some technical debt of having created that weight in the first place instead of just relying onlm_head.decoder. Not the nicest part of the library
2022-08-05T14:38:13Z
Ahah sorry for digging this out !But I needed to clarify that so thank you very much for helping.