Reproduce BERT and RoBERTa
Hi to all,I tried to use the runningrun mlm.pyto reproduce the result of the bert-base-uncased version. However, I found my reproduced results are always lower than the one reported in thiswebsiteprovided by the Huggingface team.TaskMetricReported by HuggingfaceOur reproduced resultCoLAMatthew’s corr56.5347.92SST-2Accuracy92.3287.56MRPCF1/Accuracy88.85/84.0782.03/80.97STS-BPerson/Spearman corr.88.64/88.4882.45/82.76QQPAccuracy/F190.71/87.4988.23/86.12MNLIMatched acc./Mismatched acc.83.91/84.1082.34/83.01QNLIAccuracy90.6685.45RTEAccuracy65.7056.95I think there must be some problems with my experiment. I ran my experiment by using:(1) I used the code in thisfilewithout any change.(2) I loaded the datasets of bookcorpus and wiki directly fromdatasetlibrary; the text is chunked into 512 tokens.(3) I set the batch size as 256 and ran 1M steps; and batch size as 8K and ran 50K steps. Both results are worse than the reported numbers.I really apprecitate if you could provide me a script that I can use to reproduce BERT or RoBERTa. Thank you very much!
2023-07-24T10:20:41Z
It’s very usual to have lower scores than the official team, probably due to randomness. But I’m still interested in why this is the case. Do you have any updates on this? Looking forward to it!@wyu1