DebertaForMaskedLM cannot load the parameters in the MLM head from microsoft/deberta-base
Hello, I’m trying to run this code:tokenizer = DebertaTokenizer.from_pretrained(‘microsoft/deberta-base’)model = AutoModelWithLMHead.from_pretrained(‘microsoft/deberta-base’)and get this warning:Some weights of the model checkpoint at microsoft/deberta-base were not used when initializing DebertaForMaskedLM: [‘lm_predictions.lm_head.dense.bias’, ‘lm_predictions.lm_head.LayerNorm.weight’, ‘lm_predictions.lm_head.dense.weight’, ‘lm_predictions.lm_head.bias’, ‘deberta.embeddings.position_embeddings.weight’, ‘lm_predictions.lm_head.LayerNorm.bias’]This IS expected if you are initializing DebertaForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).This IS NOT expected if you are initializing DebertaForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).Some weights of DebertaForMaskedLM were not initialized from the model checkpoint at microsoft/deberta-base and are newly initialized: [‘cls.predictions.transform.LayerNorm.bias’, ‘cls.predictions.transform.dense.weight’, ‘cls.predictions.transform.dense.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.bias’, ‘cls.predictions.decoder.weight’]You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.It seems that the checkpoints provided by microsoft/deberta-base doesn’t possess the weights needed for the MLM head, so that DebertaForMaskedLM cannot be directly used for masked token prediction.Is this a bug of the DebertaForMaskedLM class? Or the checkpoints provided by Microsoft? Or I do not understand something?Thank you!
2022-02-27T16:39:54Z
Same issue for me.help please?Thanks!
2022-04-29T11:40:00Z
I wouldn’t say it complete solution but it does initialize weights starting with “lm_predictions” checkout the codehereThanks to@nbroadfor his generous contribution.from new_deberta import NewDebertaForMaskedLMmodel = NewDebertaForMaskedLM.from_pretrained(model_name)after this.Some weights of the model checkpoint at microsoft/deberta-base were not used when initializing NewDebertaForMaskedLM: [‘deberta.embeddings.position_embeddings.weight’]This IS expected if you are initializing NewDebertaForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).This IS NOT expected if you are initializing NewDebertaForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).Some weights of NewDebertaForMaskedLM were not initialized from the model checkpoint at microsoft/deberta-base and are newly initialized: [‘lm_predictions.lm_head.decoder.weight’, ‘lm_predictions.lm_head.decoder.bias’]You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.I think we need some more hacks to initialize position_embedding weights tooHope this helps!
2022-04-29T12:46:38Z
There will be more changes coming soon to the main branch oftransformers. The new changes will allow for the MLM head to produce reasonable suggestions for [MASK] tokens. Thanks for your patience!