Not all BLEU scores were created equal
While porting fairseq transformer and another model from allenai, I wasn’t getting the same BLEU scores as reported by the papers. At the end I learned that some of that difference was due to the fact that I was measuring the BLEU score in a different way from theirs. So when you see a BLEU number in a report, it could mean many different things. e.g. apparently you get a higher score if you measure tokenized outputs.Please see this paper for many more nuances:arXiv.orgA Call for Clarity in Reporting BLEU ScoresThe field of machine translation faces an under-recognized problem because of
inconsistency in the reporting of scores from its dominant metric. Although
people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose
values can vary...In your work and experiments, please, try to usesacrebleufor measuring as suggested in the paper. That’s what our seq2seqeval_run.pyuses.Thank you.