Accuracy of MLM model
How to calculate the accuracy of the testing dataset when we build MLM model using scrach?
2021-07-07T20:16:21Z
have you got the answers?
2021-07-09T01:43:41Z
HiYes, I found a way but not sure whether it’s the best way.What I’m doing is for each sentence I’m masking a random word in the sentence and ask the model to predict it. If the actual word is in the predicted list I’m increasing the tp(positive) value or else the tn(negative) value. finally I’m calculating the accuracy (tp/tp+tn).and I’m calculating the perplexity value using the following codecolab.research.google.comGoogle ColaboratoryHope my answer is clear to you…
2021-07-09T09:25:57Z
Hey, thank you for your response,i am also calculating perplexity using the same code and i am using perplexity as a metric rather than accuracy.This is because i don’t thing accuracy would be a better choice(as a metric) for mlm because there can be may words which can be used for a given mask. (feel free the correct) , instead i am calculating perplexity in the compute metric function and printing it along with training_loss and validation loss along in the logs.Thanks again!!
2021-07-12T16:05:38Z
Also how much minimum data is required to fine tune roberta any idea?
2021-07-13T01:07:53Z
sorry. I don’t have an idea about it…do you know a way to build a n-gram model for word prediction and calculate the perplexity value of it?