BLEU TER metric
49742509
A Structured Review of the Validity of BLEU
The BLEU metric has been widely used in NLP for over 15 years to evaluate NLP systems, especially in machine translation and natural language generation. I present a structured review of the evidence on whether BLEU is a valid evaluation technique, in other words whether BLEU scores correlate with real world utility and user satisfaction of NLP systems; this review covers 284 correlations reported in 34 papers. Overall, the evidence supports using BLEU for diagnostic evaluation of MT systems (which is what it was originally proposed for) but does not support using BLEU outwith MT, for evaluation of individual texts, or for scientific hypothesis testing.
Computational Linguistics
Ehud Reiter
203618521
Optimizing BLEU Scores for Improving Text Generation
In typical text generation settings, there exists a discrepancy between the training objective and evaluation criteria. Generally, researchers train text generation models by maximizing for loglikelihood on cross entropy while evaluating on a separate metric, such as BLEU score, that cannot be optimized by evaluating gradients. While some work has been done to use policy gradients to directly optimize for non differentiable rewards, these methods are difficult to train due to the amount of computation required. Instead, we focus on an alternative method of approximating for BLEU by evaluating Differentiable Expected BLEU (DEBLEU) We evaluate our method on machine translation, while comparing results to both cross entropy and policy gradient methods. Furthermore, we provide empirical results regarding the computational benefits of DEBLEU vs. policy gradients. Lastly, we focus on text style transfer and argue that DEBLEU loss is a more suitable objective than cross entropy for this task.
Terrance Liu
18035217
Some Improvements over the BLEU Metric for Measuring Translation Quality for Hindi
The BLEU translation quality evaluation metric is a modified n gram precision measure that uses a number of reference translations of a given candidate text in order to evaluate its quality of translation. In this paper, we propose some modifications to this metric so that it suits Indian languages; more specifically, Hindi. The problem of using BLEU in Hindi presents two difficulties: non availability of multiple references and prevalence of free word order in sentence construction. It is established that the validity of BLEU scores generally increases with the number of reference translations used. Further, Hindi being a free word order language, naive n gram matching as adopted by BLEU does not accurately predict the quality of a translated text. In our approach we have modified BLEU in order to take care of the above mentioned shortcomings. Our proposed metric has obtained a closer correlation with human judgment while using just one reference translation
2007 International Conference on Computing: Theory and Applications (ICCTA'07)
Niladri Chatterjee Anish Johnson Madhav Krishna
52009322
A Structured Review of the Validity of BLEU
The BLEU metric has been widely used in NLP for over 15 years to evaluate NLP systems, especially in machine translation and natural language generation. I present a structured review of the evidence on whether BLEU is a valid evaluation technique in other words, whether BLEU scores correlate with real world utility and user satisfaction of NLP systems; this review covers 284 correlations reported in 34 papers. Overall, the evidence supports using BLEU for diagnostic evaluation of MT systems (which is what it was originally proposed for) but does not support using BLEU outside of MT, for evaluation of individual texts, or for scientific hypothesis testing.
Computational Linguistics
Ehud Reiter
870211
Meteor, M BLEU and M TER: Evaluation Metrics for High Correlation with Human Rankings of Machine Translation Output
This paper describes our submissions to the machine translation evaluation shared task in ACL WMT 08. Our primary submission is the Meteor metric tuned for optimizing correlation with human rankings of translation hypotheses. We show significant improvement in correlation as compared to the earlier version of metric which was tuned to optimized correlation with traditional adequacy and fluency judgments. We also describe m bleu and m ter, enhanced versions of two other widely used metrics bleu and ter respectively, which extend the exact word matching used in these metrics with the flexible matching based on stemming and Wordnet in Meteor.
WMT@ACL
Abhaya Agarwal Alon Lavie
8617003
Improving AMBER, an MT Evaluation Metric
A recent paper described a new machine translation evaluation metric, AMBER. This paper describes two changes to AMBER. The first one is incorporation of a new ordering penalty; the second one is the use of the downhill simplex algorithm to tune the weights for the components of AMBER. We tested the impact of the two changes, using data from the WMT metrics task. Each of the changes by itself improved the performance of AMBER, and the two together yielded even greater improvement, which in some cases was more than additive. The new version of AMBER clearly outperforms BLEU in terms of correlation with human judgment.
WMT@NAACL HLT
Boxing Chen Roland Kuhn George F Foster
7120497
RED: A Reference Dependency Based MT Evaluation Metric
Most of the widely used automatic evaluation metrics consider only the local fragments of the references and translations, and they ignore the evaluation on the syntax level. Current syntaxbased evaluation metrics try to introduce syntax information but suffer from the poor parsing results of the noisy machine translations. To alleviate this problem, we propose a novel dependency based evaluation metric which only employs the dependency information of the references. We use two kinds of reference dependency structures: headword chain to capture the long distance dependency information, and fixed and floating structures to capture the local continuous ngram. Experiment results show that our metric achieves higher correlations with human judgments than BLEU, TER and HWCM on WMT 2012 and WMT 2013. By introducing extra linguistic resources and tuning parameters, the new metric gets the state of the art performance which is better than METEOR and SEMPOS on system level, and is comparable with METEOR on sentence level on WMT 2012 and WMT 2013.
COLING
Hui Yu Xiaofeng Wu Jun Xie Wenbin Jiang Qun Liu Shouxun Lin
5189165
LEPOR: A Robust Evaluation Metric for Machine Translation with Augmented Factors
In the conventional evaluation metrics of machine translation, considering less information about the translations usually makes the result not reasonable and low correlation with human judgments. On the other hand, using many external linguistic resources and tools (e.g. Part ofspeech tagging, morpheme, stemming, and synonyms) makes the metrics complicated, timeconsuming and not universal due to that different languages have the different linguistic features. This paper proposes a novel evaluation metric employing rich and augmented factors without relying on any additional resource or tool. Experiments show that this novel metric yields the state of the art correlation with human judgments compared with classic metrics BLEU, TER, Meteor 1.3 and two latest metrics (AMBER and MP4IBM1) which proves it a robust one by employing a feature rich and model independent approach.
COLING
Lifeng Han Derek F Wong Lidia S Chao
463326
LAYERED: Metric for Machine Translation Evaluation
This paper describes the LAYERED metric which is used for the shared WMT'14 metrics task. Various metrics exist for MT evaluation: BLEU (Papineni, 2002) METEOR (Alon Lavie, 2007) TER (Snover, 2006) etc. but are found inadequate in quite a few language settings like, for example, in case of free word order languages. In this paper, we propose an MT evaluation scheme that is based on the NLP layers: lexical, syntactic and semantic. We contend that higher layer metrics are after all needed. Results are presented on the corpora of ACL WMT, 2013 and 2014. We end with a metric which is composed of weighted metrics at individual layers, which correlates very well with human judgment.
WMT@ACL
Shubham Gautam Pushpak Bhattacharyya
12617973
Improving machine translation by training against an automatic semantic frame based evaluation metric
We present the first ever results showing that tuning a machine translation system against a semantic frame based objective function, MEANT, produces more robustly adequate translations than tuning against BLEU or TER as measured across commonly used metrics and human subjective evaluation. Moreover, for informal web forum data, human evaluators preferred MEANT tuned systems over BLEU or TER tuned systems by a significantly wider margin than that for formal newswire even though automatic semantic parsing might be expected to fare worse on informal language. We argue that by preserving the meaning of the translations as captured by semantic frames right in the training process, an MT system is constrained to make more accurate choices of both lexical and reordering rules. As a result, MT systems tuned against semantic frame based MT evaluation metrics produce output that is more adequate. Tuning a machine translation system against a semantic frame based objective function is independent of the translation model paradigm, so, any translation model can benefit from the semantic knowledge incorporated to improve translation adequacy through our approach.
ACL
Chi-kiu Lo Karteek Addanki Markus Saers Dekai Wu