Usind a fine-tuned sentence completion model in a Masked LM task
Is it possible to use a BERT model that has been fine-tuned already (e.g. SQUAD-tuned BERT) on a masked LM task? I suspect that the sentence-completion model that is added on top of BERT is fundamentally incompatible with a masked LM task, but I’d like to know for a fact.I’ve attempted to do this, using:from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("deepset/bert-base-cased-squad2")
model = AutoModelForMaskedLM.from_pretrained("deepset/bert-base-cased-squad2")but the results are very bad, so maybe I’m missing a step.Thanks!
2021-09-23T08:14:50Z
Hi,that’s not really possible, unless the model has a language modeling head that has been trained. If that’s not the case, it will load a randomly initialized language modeling head, which gives random predictions.The"deepset/bert-base-cased-squad2"checkpoint has a fine-tuned question-answering head, but not a trained language modeling head.
2021-09-23T21:57:42Z
Thank you!