How to correctly evaluate a Masked Language Model?
Hi All,my question is very simple. Starting from a pre-trained (Italian) model, I fine-tuned it on a specific domain of interest, say X, using masked language model (MLM) training. I then computed perplexity on a test text on domain X and checked that the final model performs better than the pre-trained one.Is it sufficient?People who trained this language modelUmbertousing MLM training, performed several tests on downstream tasks such as NER and POS.How did they do?My understanding is that starting from Umberto, they fine-tune it on NER and test it on WikiNER-ITA for instance. Is there a simple procedure to do it?Thanks!
2022-02-13T02:10:58Z
In RoBerta they use accuracy and f1 scores of the language model. O got this code that I think performs the accuracy:import sklearnfrom datasets import load_metricimport numpy as npmetric = load_metric(“accuracy”)def compute_metrics(eval_pred):logits, labels = eval_predpredictions = np.argmax(logits, axis=-1)indices = [[i for i, x in enumerate(labels[row]) if x != -100] for row in range(len(labels))]

labels = [labels[row][indices[row]] for row in range(len(labels))]
labels = [item for sublist in labels for item in sublist]

predictions = [predictions[row][indices[row]] for row in range(len(predictions))]
predictions = [item for sublist in predictions for item in sublist]

results = metric.compute(predictions=predictions, references=labels)
results["eval_accuracy"] = results["accuracy"]
print(results)
results.pop("accuracy")

return resultsThen create a trainer and use this as the compute_metric parameterfrom transformers import Trainertrainer = Trainer(model=model,…compute_metrics=compute_metrics,)Thenresults = trainer.evaluate()accuracy = results[‘eval_results’]
2023-07-03T16:55:25Z
Very much late to the party here, but I have been struggling to find perplexity with my BERT model. How exactly did you determine the perplexity of yours?
2023-08-11T14:05:08Z
@NHendrickson9616Perplexityis really only well-suited for causal/autoregressive language models, e.g., GPT. Since it only uses information from the tokens on the left-hand-side, it wouldn’t really provide valuable feedback for a model like BERT, which leverages bidirectional information.