Clarify BERT model learnable parameters
Hello,I want to use HateBERT from the paper’srepository, which is a BERT model that its pre-training was extended in abusive language.In order to do that, I created a BERT model (bert-base-uncasedinPyTorch) and tried to load HateBERT’s weights withload_state_dict()(after having made minor changes in parameters names, to match BERT’s).load_state_dict()throws the error:RuntimeError: Error(s) in loading state_dict for BertModel:
	Missing key(s) in state_dict: "embeddings.position_ids".which means that the BERT model requiresembeddings.position_ids. I checked this tensor and it is not aPyTorchparameters tensor, just a tensor. From the error message, it is also evident that all other parameters match, otherwise their names would also be mentioned. Can someone explain?
2021-10-23T17:39:28Z
HateBERT is available on the hub.GroNLP/hateBERT · Hugging Facefrom transformers import AutoTokenizer, AutoModelForMaskedLM
  
tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")

model = AutoModelForMaskedLM.from_pretrained("GroNLP/hateBERT")
2021-11-04T10:11:37Z
Thank you@BramVanroy, and sorry for my late response.Is there a way to know that the Hugging Face library has that model, and do not make the OP in the first place? The question is to avoid future posts, and make a better search in advance.
2021-11-04T13:54:06Z
Sure, you can look for a model on the hub.huggingface.coModels - Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2021-11-04T14:29:46Z
Thank you@BramVanroy!