Questions on the `BertModelLMHeadModel`
Hello,I have a few questions about theBertModelLMHeadModel:IsBertModelLMHeadModelused to conduct the regular language modeling (next token prediction), as it is the case for theGPT2LMHeadModel?ForGPT2LMHeadModel, I can just specifylabels = input_idsfor convenience. I just specify thelabelsin this way for theBertModelLMHeadModelas well?Thanks,
2020-10-02T14:55:36Z
Hi@h56chodo you mean theBertLMHeadModel?If yes then, it’s intended to be used with theEncoderDecodermodel which allows you to use pre-trained encoder for as both encoder and decoder for seq2seq tasks.  It’s not intended for language modeling.While you can use that class as a standalone decoder by passingis_decoder=Truetoconfigit might not give you good results as it’s trained as an encoder.
2020-10-02T16:42:00Z
Hello,HuggingFace Transformer documentation seem to point out thatBertLMHeadModelcan be used for causal language modeling(https://huggingface.co/transformers/model_doc/bert.html#bertmodellmheadmodel). If you look at the returned values from this model, it includescausalLMoutput. doesn’t the term “causal language modeling” refer to regular language modeling, as in the case for GPT-2? I am not so interested in the accuracy of the results, my intention is to examine the distribution of the attention weights.Also, when providing “labels” for the causal language modeling with theBertLMHeadModel, can I just uselabels = input_idsas in the case for GPT-2, for convinence?Thank you,
2020-10-02T17:02:33Z
That’s what I said in the last comment,It can be used as a standalone decoder (standalone decoder = causal LM).Yes, you can passlabels = input_ids
2020-10-02T17:03:23Z
Thank you!
2020-10-02T20:27:00Z
Hello,Sorry I have some additional question. This question is about theBertForMaskedLMmodel.The documentation forBertForMaskedLMprovides the following example to illustrate the model’s usage:>>> from transformers import BertTokenizer, BertForMaskedLM
>>> import torch

>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = BertForMaskedLM.from_pretrained('bert-base-uncased', return_dict=True)
>>> input_ids = tokenizer("Hello, my dog is cute", return_tensors="pt")["input_ids"]

>>> outputs = model(input_ids, labels=input_ids)
>>> loss = outputs.loss
>>> prediction_logits = outputs.logitsIn the example above, I don’t see any[MASK]token in the input; can theBertForMaskedLMmodel really be used with an input string that does not include[MASK]token? If I provideBertForMaskedLMmodel an input string that does not include the[MASK]token, from which token will the output of the model be produced from? In this case, wouldBertForMaskedLMautomatically insert[MASK]token in the beginning of the input sequence?Thank you again,
2020-10-05T10:41:30Z
That’s probably a mistake,Thismighthelp.
2020-10-05T13:36:03Z
Yes it’s definitely a mistake. Will fix this morning.