What techniques have been developed to assess the robustness of automated evaluation techniques for neural text generation?
Analysis of NLG Metrics In comparison to the
vast literature on NLG metric development or
benchmarking (Mathur et al., 2020; Celikyilmaz
et al., 2020; Gehrmann et al., 2021; Kasai et al.,
2022b; Hämäläinen and Alnajjar, 2021), the robust-
ness analysis of PLM-based metrics is an under-
explored area, where exisiting work focused on
a relatively small subset of metrics or a limited
definition of robustness. For example, Vu et al.
(2022) explored BERTScore’s performance varia-
tion with changes in representation space and char-
acter perturbations. Kaster et al. (2021) propose a
regression-based global explainability technique to
disentangle metric scores along linguistic factors.
More related to our work, Hanna and Bo-
jar (2021) conducted a fine-grained analysis of
BERTScore on different error types. Caglayan et al.
(2020) discussed some curious phenomena for a
range of metrics. Chen et al. (2021) conducted diag-
nostic tests for factuality metrics with synthesized
errors. Sun et al. (2022) found that some metrics
are not robust to dialects. In comparison, this work
is more comprehensive in that the design of our
tests are inspired by a wider range of motivations,
e.g., the properties of the underlying PLMs.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Automatic evaluation of language generation systems is a well-studied problem in Natural Language Processing. While novel metrics are proposed every year, a few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation, despite their known limitations. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models by demonstrating important failure cases on multiple datasets, language pairs and tasks. Our experiments show that metrics (i) usually prefer system outputs to human-authored texts, (ii) can be insensitive to correct translations of rare words, (iii) can yield surprisingly high scores when given a single sentence as system output for the entire test set.
[{"first":"Ozan","last":"Caglayan","middle":[]},{"first":"Pranava","last":"Madhyastha","middle":[]},{"first":"Lucia","last":"Specia","middle":[]}]
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Evaluation metrics are a key ingredient for progress of text generation systems. In recent years, several BERT-based evaluation metrics have been proposed (including BERTScore, MoverScore, BLEURT, etc.) which correlate much better with human assessment of text generation quality than BLEU or ROUGE, invented two decades ago. However, little is known what these metrics, which are based on black-box language model representations, actually capture (it is typically assumed they model semantic similarity). In this work, we use a simple regression based global explainability technique to disentangle metric scores along linguistic factors, including semantics, syntax, morphology, and lexical overlap. We show that the different metrics capture all aspects to some degree, but that they are all substantially sensitive to lexical overlap, just like BLEU and ROUGE. This exposes limitations of these novelly proposed metrics, which we also highlight in an adversarial test scenario.
[{"first":"Marvin","last":"Kaster","middle":[]},{"first":"Wei","last":"Zhao","middle":[]},{"first":"Steffen","last":"Eger","middle":[]}]
A Fine-Grained Analysis of BERTScore
BERTScore, a recently proposed automatic metric for machine translation quality, uses BERT, a large pre-trained language model to evaluate candidate translations with respect to a gold translation. Taking advantage of BERT’s semantic and syntactic abilities, BERTScore seeks to avoid the flaws of earlier approaches like BLEU, instead scoring candidate translations based on their semantic similarity to the gold sentence. However, BERT is not infallible; while its performance on NLP tasks set a new state of the art in general, studies of specific syntactic and semantic phenomena have shown where BERT’s performance deviates from that of humans more generally. This naturally raises the questions we address in this paper: what are the strengths and weaknesses of BERTScore? Do they relate to known weaknesses on the part of BERT? We find that while BERTScore can detect when a candidate differs from a reference in important content words, it is less sensitive to smaller errors, especially if the candidate is lexically or stylistically similar to the reference.
[{"first":"Michael","last":"Hanna","middle":[]},{"first":"Ondřej","last":"Bojar","middle":[]}]
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
With the continuous upgrading of the summarization systems driven by deep neural networks, researchers have higher requirements on the quality of the generated summaries, which should be not only ﬂuent and infor-mative but also factually correct. As a result, the ﬁeld of factual evaluation has developed rapidly recently. Despite its initial progress in evaluating generated summaries, the meta-evaluation methodologies of factuality metrics are limited in their opacity , leading to the insufﬁcient understanding of factuality metrics’ relative advantages and their applicability. In this paper, we present an adversarial meta-evaluation methodology that allows us to (i) diagnose the ﬁne-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets, (ii) search for directions for further improvement by data augmentation. Our observations from this work motivate us to propose several calls for future research. We make all codes, diagnostic test datasets, trained factuality models available: https://github.com/zide05/ AdvFact .
[{"first":"Yiran","last":"Chen","middle":[]},{"first":"Pengfei","last":"Liu","middle":[]},{"first":"Xipeng","last":"Qiu","middle":[]}]
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
The evaluation of recent embedding-based evaluation metrics for text generation is primarily based on measuring their correlation with human evaluations on standard benchmarks. However, these benchmarks are mostly from similar domains to those used for pretraining word embeddings. This raises concerns about the (lack of) generalization of embedding-based metrics to new and noisy domains that contain a different vocabulary than the pretraining data. In this paper, we examine the robustness of BERTScore, one of the most popular embedding-based metrics for text generation. We show that (a) an embedding-based metric that has the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases, (b) taking embeddings from the first layer of pretrained models improves the robustness of all metrics, and (c) the highest robustness is achieved when using character-level embeddings, instead of token-based embeddings, from the first layer of the pretrained model.
[{"first":"Doan","last":"Vu","middle":["Nam","Long"]},{"first":"Nafise","last":"Moosavi","middle":["Sadat"]},{"first":"Steffen","last":"Eger","middle":[]}]
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Human assessment is the best practice at hand for evaluating language generation tasks such as machine translation (MT), dialogue systems, visual captioning and abstractive summarisation. In practice, however, we rely on automatic metrics which compare system outputs to human-authored references. Initially proposed for MT evaluation, metrics such as BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014) are increasingly used for other tasks, along with more task-oriented metrics such as ROUGE (Lin, 2004) for summarisation and CIDER  for visual captioning. Reiter and Belz (2009) remark that it is not sufficient to conclude on the usefulness of a natural language generation system's output by solely relying on metrics that quantify the similarity of the output to human-authored texts. Previous criticisms concerning automatic metrics corroborate this perspective to some degree. To cite a few, in the context of MT, Callison-Burch et al. (2006) state that human judgments may not correlate with BLEU and an increase in BLEU does not always indicate an improvement in quality. Further, Mathur et al. (2020) challenge the stability of the common practices measuring correlation between metrics and human judgments, the standard approach in the MT community, and show that they may be severely impacted by outlier systems and the sample size. In the context of image captioning, Wang and Chan (2019) claim that the consensus-based evaluation protocol of CIDER actually penalises output diversity. Similar problems have also been discussed in the area of automatic summarisation with respect to ROUGE (Schluter, 2017). Nevertheless, automatic metrics like these are a necessity and remain popular, especially given the increasing number of open evaluation challenges.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
In this paper, we further probe BLEU, METEOR, CIDER-D and ROUGE L metrics ( § 2) that are commonly used to quantify progress in language generation tasks. We also probe a recently proposed contextualised embeddings-based metric called BERTSCORE (Zhang et al., 2020). We first conduct leave-oneout average scoring with multiple references and show that, counter-intuitively, metrics tend to reward system outputs more than human-authored references ( § 3.1). A system output perturbation experiment further highlights how metrics penalise errors in extremely frequent n-grams ( § 3.2.1) while they are quite insensitive to errors in rare words ( § 3.2.2). The latter makes it hard to understand whether a model is better than another in its ability to handle rare linguistic phenomena correctly. Finally, we design ( § 3.3) an adversary that seeks to demonstrate metrics' preference for frequent n-grams by using a single training set sentence as system output for the entire test set. We observe strikingly high scores, such as the sentence "A man is playing" obtaining a BLEU score of 30.6, compared to 47.9 of a strong model, on a captioning task. We hope that our observations in this paper will lead the community towards formulation of better metrics and evaluation protocols in the future ( § 4).
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
In this section, we briefly describe the metrics we use in our experiments. To compute BLEU, ME-TEOR, CIDER and ROUGE-L, we provide pre-tokenised hypotheses and references to coco-caption utility 1 . For BERTSCORE, we use its official release 2 .
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
• Initially proposed for MT evaluation, BLEU is a prevalent metric based on n-gram matches between the candidate and reference sentences. The final score is the geometric mean of n-gram precisions with a brevity penalty to penalise outputs shorter than references.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
• In contrast, METEOR is a recall-based metric which uses explicit word alignments between candidate and references, allowing for exact matching, synonymy, and stemming. A fragmentation penalty rewards longer and fewer chunks of contiguously aligned tokens. The final score is computed between the best scoring reference and the candidate sentence.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
• Another recall-based metric is ROUGE-L which measures longest common sub-sequences between the candidate and references, i.e. a set of shared words with similar order even if not contiguous.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
• CIDER is a consensus-based captioning metric which uses term frequency inverse document frequency (TF-IDF) for n-gram weighting over all references. The final score is the average cosine similarity between the candidate sentence and references. We use the popular variant CIDER-D, which integrates a length-based Gaussian penalty and clipping.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
• Finally, BERTSCORE computes a token level similarity for each candidate token against each token in the reference sentence, using contextual BERT embeddings (Zhang et al., 2020). We report the Fscore, i.e. the harmonic mean of the precision and the recall of maximal per-token cosine similarities between the candidate and the reference(s).
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
This section describes our probing experiments and results 3 for different tasks and language pairs.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
We first explore the extent to which metrics reward human-authored texts. We hypothesise that metrics should prefer human-authored texts to machine-produced texts, as the former is considered to be the 'ground truth' reference. For a given multi-reference metric M, REF-VS-REF leave-one-out 4 average L M is computed across C human-authored reference corpora {R 1 , . . . , R C }. At each iteration i, the reference R i is taken as the candidate and is evaluated against the held-out C−1 references:
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Same principle holds when evaluating a trained system, except that SYS would now represent the system outputs. The metric computations inside the sum would still be using C−1 references at each iteration.  Tasks. We explore two visual captioning tasks, namely, COCO English image captioning (Chen et al., 2015) and the bilingual VATEX video captioning dataset . For the former, we use a state-of-the-art captioning model (Lu et al., 2018), and evaluate the 5k sentences in the test split (Karpathy and Fei-Fei, 2015). For English and Chinese VATEX, we train neural baselines and evaluate on the dev set which contains 3k videos. Finally, we also experiment with the Chinese→English NIST 2008 (MT08) test set which contains 1357 sentences, and translate it using Google Translate.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Results. Table 1 shows that human-authored references obtain relatively worse scores than systems in general. BLEU exhibits the biggest differences, with almost 11 points for COCO and 3.5 points for the NIST MT experiment. The results are consistent across different datasets and languages except for English VATEX and NIST-2008, which score slightly better than system outputs in METEOR and CIDER. Overall, the results suggest that the metrics do not necessarily reflect the quality of the generated captions or translations, as scores for human-authored texts are far from the upper bounds of the metrics (e.g. 100 for BLEU). Therefore, we suggest that this type of evaluation should not be used to draw conclusions on 'human parity', as previously done in some studies (Xu et al., 2015;Vinyals et al., 2017).
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
In this section, we propose two perturbation experiments to investigate the sensitivity of metrics to frequent and infrequent phenomena. In both experiments, the first reference corpus R 1 of COCO test set is considered to be the output of a hypothetical system from which perturbed versions are created. Each version is then evaluated against the four remaining references {R 2 , . . . , R 5 } of the COCO test set. Although we target unigrams for perturbation, they implicitly effect higher order n-grams as well.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
We create five independent variants of R 1 by substituting the most common words 'people', 'standing', 'sitting', 'man' and 'a' with the unknown token UNK. Table 2 shows that metrics react quite conservatively to this attack, until very high substitution percentages are reached. For example, substituting 'people' yields a drop of less than 0.4 points for BLEU, METEOR and ROUGE, a potentially uninteresting drop which can easily be overlooked during model development.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Although repeatedly missing indefinite articles 'a' is semantically much less critical than missing 'people', all metrics aggressively penalise the former, simply because the function word 'a' is extremely frequent: with 53.8% relative drop, BLEU is the most affected metric, whereas METEOR and BERTSCORE are more robust with relative drops of 17.9% and 12.9%, respectively. To find out more, we randomly substitute content words from R 1 (RANDOM in   .7) show substantial increases when compared to dropping 'a', supporting our semantic conjecture above. However, the decreases in METEOR (20.03→19.58) and BERTSCORE (0.480→0.413) highlight the sensitivity of these metrics to semantics rather than pure frequencies. The continuous and contextual nature of BERTSCORE seems to be an advantage here.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
We now approach to the problem from the other end to explore how sensitive metrics are to a set of rarely occurring words. We conjecture that this will provide insight into how much the metrics reward a hypothetical model that systematically translates a set of rare words correctly. Specifically, we create four variants of R 1 , with each one substituting a particular set of words by UNK, based on training set frequencies. 5 Table 3 shows that even the most aggressively short-listed hypothetical model (T = 30) obtains marginally worse scores than the full vocabulary model (T = 1). As a concrete example, the last row shows the non-substantial impact of systematically replacing each occurrence of 'woman' (0.54% of all unigrams) with 'man' (1.21% of all unigrams).
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Discussion. Although it is hard to interpret the magnitude of differences observed in both experiments, our empirical findings highlight interesting cases. Overall, the conclusions of our perturbation experiments are in line with the concurrent work of Mathur et al. (2020) which suggests that important conclusions, such as comparative judgments about systems, should not be drawn based only on small changes in automatic metrics.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Following the observations from the previous experiments, we search over the training set for a single representative sentence (SS) which maximises test set BLEU 6 when used as a system output for every test set instance. 7 We explore tasks and datasets which include the ones previously introduced ( § 3.1). For visual captioning, we add MSVD (Chen and Dolan, 2011), a widely known dataset of 1,970 videos, with up to 41 English captions per video. For image captioning, we use English Flickr30k (Young et al., 2014), and the STAIR (Yoshikawa et al., 2017) dataset which provides Japanese captions for COCO images. We also explore the multi-turn dialogue dataset DailyDialog (Li et al., 2017) which contains conversations that cover 10 different daily life topics, and its multi-reference test set (Gupta et al., 2019). Table 4 summarises the statistics about the datasets explored for this experiment. Table 5 draws a comparison between the scores obtained for the retrieved single sentences, baselines and state-of-the-art systems when available. We observe that: (i) MSVD exhibits the highest scores with 30.6 BLEU and 23.4 METEOR, the latter being very close to a strong captioning baseline (Venugopalan et al., 2015), (ii) the SS scores for the DAILYDIALOG are surprisingly on par with a recent baseline and a state-of-the-art system, and (iii) CIDER is more robust against the single sentence adversary as more references seem to affect its internal consensus-based scoring. Although BERTSCORE does not explicitly  (Lu et al., 2018) 75.2 34.4 26.5 1.06 55.5 0.628 FLICKR-EN "A man in a blue shirt standing in front of a building" 46.2 12.2 11.6 0.07 32.7 -Hard Attention (Xu et al., 2015) 66.9 19.9 18.5 ---Neural Baby Talk (Lu et al., 2018) 69.  Table 5: Comparison of single sentence scores to several baselines and state-of-the-art: "Single representative sentences are in italics." English translations are provided for non-English tasks.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
rely on n-gram statistics, it also exhibits surprisingly high scores for the SS setting. This contradicts the salient claims regarding the utility of the metric for model selection (Zhang et al., 2020).
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
In this work, we explore cases where commonly used language generation evaluation metrics exhibit counter-intuitive behaviour. Although the main goal in language generation tasks is to generate 'humanquality' texts, our analysis in § 3.1 shows that metrics have a preference towards machine-generated texts rather than human references. Our perturbation experiments in § 3.2.2 highlight potential insensitivity of metrics to lexical changes in infrequent n-grams. This is a major concern for tasks such as multimodal machine translation (Specia et al., 2016) or pronoun resolution in MT (Guillou et al., 2016), where the metrics are expected to capture lexical changes which are due to rarely occurring linguistic ambiguities. We believe that targeted probes (Isabelle et al., 2017) are much more reliable than sentence or corpus level metrics for such tasks. Finally, we reveal that metrics tend to produce unexpectedly high scores when each test set hypothesis is set to a particular training set sentence, which can be thought of as finding the sweet spot of the corpus ( § 3.3). Therefore, we note that a high correlation between metrics and human judgments is not sufficient to characterise the reliability of a metric. The latter probably requires a thorough exploration and mitigation of adversarial cases such as the proposed single sentence baseline.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Evaluation metrics are a key ingredient in assessing the quality of text generation systems, be it machine translation, summarization, or conversational AI models. Traditional evaluation metrics in machine translation and summarization, BLEU and ROUGE (Papineni et al., 2002a;Lin, 2004), have measured lexical n-gram overlap between system prediction and a human reference. While simple and easy to understand, early on, limitations of such lexical overlap metrics have been recognized (Callison-Burch et al., 2006), e.g., in that they can only measure surface level similarity, and they are especially inadequate when it comes to assessing current high-quality text generation systems (Rei et al., 2020;Mathur et al., 2020;Marie et al., 2021).
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Recently, a class of novel evaluation metrics based on BERT and its variants has been explored that correlates much better with human assessments of translation quality. For example, BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), BLEURT (Sellam et al., 2020), XMover-Score , and COMET (Rei et al., 2020) all use large-scale pretrained language models, but differ in whether they compare hypotheses to references, to source texts, to both, on the one hand, and whether they use human scores for supervision or not, on the other. Since these models all leverage large-scale language models which have pushed the state-of-the-art in many areas of NLP, their success comes with little surprise.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
To better understand these novel metrics based on black-box language representations is a prerequisite for identifying their limitations, e.g., to adversarial inputs. For example, if an evaluation metric is sensitive to lexical overlap, it can be fooled by using the same words but in different order.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
In this work, we fill the existing 'explainability gap' and introspect linguistic properties encoded in BERT-based evaluation metrics. Although there is already considerable work on introspecting and understanding BERT (see Rogers et al. (2020) for an overview), e.g., via probing (Tenney et al., 2019), analyzes by Hewitt and Liang (2019); ; Ravichander et al. (2021) indicate that probing results (based on supervision) are not always trustworthy. More importantly, the modern evaluation metrics sketched above rely on at least two factors: BERT (or its variants) and different aggregation schemes, such as Earth Mover Distance (Kusner et al., 2015;Zhao et al., 2019) or greedy alignment (Zhang et al., 2020), on top of BERT. Understanding BERT alone is thus not sufficient for explaining BERT-based evaluation metrics.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Here, we present a simple global explanation technique of BERT-based evaluation metrics which disentangles them on prominent linguistic factors, viz., syntax, semantics, morphology and lexical overlap. We find that all metrics capture these linguistic aspects to certain (but differing) degrees, and they are particularly sensitive to lexical overlap, which makes them prone to similar adversarial fooling (cf. Li et al., 2020;Keller et al., 2021) as BLEU-based lexical overlap metrics. Overall, our contributions are:
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• We disentangle a multitude of current BERTbased evaluation metrics on four linguistic factors using linear regression.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• We show that all metrics are sensitive to all factors and especially to lexical overlap, as confirmed both by the linear regression and an adversarial experiment.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• Based on the insight that different metrics capture different linguistic factors to varying degrees, we ensemble metrics and identify an average improvement of between 8 and 13% for the most heterogeneous metrics.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Our work concerns reference-based and referencefree metrics, on the one hand, and model introspection (or 'explainability'), on the other.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Evaluation metrics for Natural Language Generation In the last few years, several strong performing evaluation metrics have been proposed, the majority of which is based on BERT and similar high-quality text representations. They can be differentiated along two dimensions: (i) the input arguments they take, and (ii) whether they are supervised or unsupervised. Referencebased metrics compare human references to system predictions. Popular metrics are BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), and BLEURT (Sellam et al., 2020). In contrast, reference-free metrics directly compare source texts to system predictions, thus they are more resource-lean. Popular examples are XMover-Score , Yisi-2 (Lo, 2019), KoBE (Gekhman et al., 2020), and SentSim (Song et al., 2021). Rei et al. (2020) use all three information signals: source text, hypotheses and human references. There are also reference-free metrics outside the field of machine translation; for example, SUPERT  for summarization. Supervised metrics train on human sentence-level scores, e.g., Direct Assessment (DA) scores or postediting effort (HTER) for MT. These metrics include BLEURT and COMET (Rei et al., 2020). In MT, most metrics from the so-called 'Quality Estimation' (QE) tasks are also supervised reference-free metrics, e.g., TransQuest (Ranasinghe et al., 2020) and BERGAMOT-LATTE (Fomicheva et al., 2020b). Unsupervised metrics require no such supervisory signal (e.g., MoverScore, BERTScore, XMoverScore, SentSim).
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Model introspection There has been a recent surge in interest in explaining deep learning models. The techniques for explainability differ in whether they provide justification or information for model outputs on individual instances (local explainability) or focus on a model as a whole and disclose its internal structure (global explainability) (Danilevsky et al., 2020). Popular examples for local explainability are LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017) that find features from the input (such as particular words) relevant to model outputs.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Concerning (global) interpretability of text representations, previous works (Adi et al., 2017;Conneau et al., 2018) introspect the properties encoded in vector representations through probing classifiers-trained on external data to perform a certain linguistic task, such as inducing the dependency tree depth from a text representation (of a sentence). Tenney et al. (2019) extend this idea by inspecting BERT representations layer-by-layer, and find that BERT captures more semantic information in its higher layers and more syntactic and morphological information in its lower layers. However, probing results are not always trustworthy due to the sensitivity to probing design choices, e.g., data size and classifier choices , and data artefacts (Ravichander et al., 2021). More importantly, evaluation metrics use BERT differently: some are supervised and others are unsupervised, some fine-tune BERT on semantic similarity datasets, and they generally differ on how they aggregate and compare BERT representations. This means to understand these metrics it does not suffice to understand BERT alone.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
In our work, we disentangle BERT based evaluation metrics along linguistic factors as a form of global explainability of those metrics. This yields insights into which linguistic information signals specific metrics use, in general, and may expose their limitations.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
In our scenario, we consider different metrics m taking two arguments and assigning them a real-valued score m : (x, y) → s m ∈ R where x and y are source text and hypothesis text, respectively (for so-called reference-free metrics) or alternatively x and y are reference and hypothesis text, respectively (for so-called reference-based metrics). The scores s that metrics assign to x and y can be considered the similarity between x and y or adequacy of y given x. In our experiments in Section 4, we will focus on machine translation (MT) as use case; it is arguably the most popular and prominent text generation task. Thus, x is either a (sentence-level) source text in one language and y the corresponding MT output, or x is the human reference for the original source text.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
To better understand evaluation metrics, we decompose their scores s m along multiple linguistic factors. An example is outlined in Table 1.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We follow a long line of research in the applied sciences, and use a linear model to explain a target variable (in our case, the metric score), also called response variable, in terms of multiple regressors, also called explanatory variables. That is, we estimate the linear regression m(x, y) = α · sem(x, y) + β · syn(x, y)
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Here, sem(x, y), syn(x, y), morph(x, y) and lex(x, y) are scores which describe the semantic, syntactic, morphological, and lexical similarity between the two argument sentences. The real coefficients α, β, γ and δ are the regressors' weights, estimated from data. Finally, is an error term.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Linear regression assumes a linear relationship between the target variable and the regressors. It may fail when the relationship is non-linear, but its simple model structure provides interpretability: a larger positive coefficient means the respective regressor has higher positive impact on the target variable (fixing all other variables), a coefficient close to zero means no linear relationship, and a larger negative coefficient means an inverse linear relationship between regressor and target variable.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
The coefficient of determination R 2 describes how well the regression model reflects the data. It is defined as
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
where SSE denotes the sum of squared errors and SST denotes the sum of squared totals. They are defined as
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
respectively, whereŷ i is the prediction of the model, y i is the true score, and y i is the mean, y i = 1 N N i y i . R 2 is 1 for a perfect fit, 0 if it always predicts the mean and negative if the model is worse than this baseline.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
To ensure comparability of the different regressions, we normalize the scores of our regressors and the response variable with the z-normalization, i.e., subtracting the mean and dividing by the standard deviation, per variable. In the following, we define our regressors.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Semantic score (SEM) The semantic scores are provided by the datasets and were annotated by humans who rated e.g. the translation quality. See Section 4.2 for details.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Syntactic score (SYN) To measure the syntactic similarity of the argument sentences x and y, we compare their dependency trees. Both sentences are parsed by the Stanford dependency parser (Chen and Manning, 2014). Then, the tree edit distance (TED) (Bille, 2005) between the resulting trees is calculated. As an extension of string edit distance, TED measures how many operations are necessary to transform one tree into the other. Only the structures of the trees are considered in the calculation. We ignore the actual words.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We normalized the TED to ensure comparability between sentences with different lengths (Zhang and Shasha, 1989). The final score is calculated as
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
where l x and l y are the lengths of the sentences. Figure 1 shows an example of the TED calculation for sentences in the same language (monolingual) and Figure 2 shows a cross-lingual example.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Lexical overlap score (LEX) We measure the lexical overlap between x and y by the BLEU score (Papineni et al., 2002b): BLEU n calculates the precision based on how many n-grams of one sentence can be found in the other sentence. In the experiments below, we use BLEU 1 . Using unigrams assures that word order is ignored. The simple precision count is modified so that identical words are only counted once.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
It is a boy , likes to sport , but it cannot do it because of their very.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
He is a boy, he likes sports but he can't take part because of his knee.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
- He X X X X X X X X X X X X X X X unknowable very was he Figure 1: Monolingual tree edit distance example. The left-most tree is the first sentence of the sentence pair and the right-most tree is the second. To transform the left-most sentence into the right-most sentence, two leaves are removed. The unnormalized tree edit distance is thus 2. The normalized score is 1 − 2 6+4 = 0.8.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
For monolingual reference-based metrics, the BLEU score is calculated directly on the sentence pairs. To use BLEU for cross-lingual referencefree metrics, we choose to translate the non-English sentences into English via Google Translate, as it remains unclear how else to define lexical overlap between sentences from different languages. We compute BLEU scores on original English and translated sentence pairs.  Morphological score (MOR) We introduce a morphological score morph(x, y). To do so, we use static FastText word embeddings (Bojanowski et al., 2017) and increase morphological information in the original embeddings: (a) first, we produce two morphological lexicons based on words from WMT and STS, each containing word pairs with identical UD morphological tags (Nivre et al., 2020) (see Table 2). (b) Then, we finetune/retrofit the embeddings on the morphological lexicons using the method described in Faruqui et al. (2015), so that words with the same morphological tags have more similar representations. The final morphological score for a sentence pair is the cosine similarity between two averaged sentence embeddings over refined word vectors of each sentence. Note that, while we refer to these embeddings as morphological, they actually capture multiple linguistic factors and can only be considered more morphological than standard static vector spaces.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
If the overlap of morphological features between a language pair is very low, the morphological score will not be meaningful. We exclude the morphological score for such language pairs.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We analyze different evaluation metrics by calculating their score for sentence pairs. We use both reference-based metrics, which operate in a monolingual space, and reference-free metrics, which operate in a cross-lingual space.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Reference-based metrics We consider the following reference-based metrics.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• BERTScore (Zhang et al., 2020) aggregates and compares BERT embeddings by determining a greedy alignment between words in two sentences and summing up the cosine similarities of representations of aligned words.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• MoverScore (Zhao et al., 2019) computes an optimal alignment between words in the two sentences using word mover distance (Kusner et al., 2015) on top of BERT representations.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• Sentence BERT (SBERT) (Reimers and Gurevych, 2019) fine-tunes Siamese BERT networks on NLI data, and produces sentence embeddings by using pooling on top of BERT mother great a is it X X X X X X X X X X X X X X X ist Mutter großartige eine Sie Figure 2: Cross-lingual tree edit distance example. To transform the left-most into the right-most sentence, two leaves (shown in red and blue) are moved to other locations. This takes 2 operations. The normalized score is 1 − 2 5+5 = 0.8.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We compute the cosine similarity between SBERT representations.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• SBERT-WK (Wang and Kuo, 2020) is a variant of SBERT which weighs different layers of BERT.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• In contrast to the others, BLEURT (Sellam et al., 2020) is a supervised metric and fine-tunes BERT on the WMT datasets with available human assessment of translation quality.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Reference-free metrics We consider the following reference-free metrics:
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• Multilingual Sentence BERT (mSBERT) (Reimers and Gurevych, 2020 • LASER (Artetxe and Schwenk, 2019) is a BiL-STM encoder trained on parallel corpora. It produces language-agnostic representations. The encoder-decoder architecture is trained jointly on different languages.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• XMoverScore  extends Mover-Score to operate in the cross-lingual setup, and relies on re-aligned multilingual BERT representations. Note that we exclude a target-side lan-guage model integrated in XMoverScore to have a similar setup as for MoverScore.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Except for XMoverScore, all metrics are based on calculation of cosine similarity between the sourcelanguage and target-language sentence embeddings. Except for MUSE and LASER, all metrics are based on BERT representations. Note that some multilingual reference-free metrics can also be used in the monolingual reference-based case, especially those based on calculating cosine similarity on top of sentence embeddings, thus we will include them in both settings.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We use the datasets of the WMT shared task and the Semantic Text Similarity Benchmark (STSB) in our experiments. In the appendix, Table 8 shows the statistics in each dataset, and Table 9 shows examples for the sentences of the dataets.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
WMT The WMT datasets contain an input sentence in the source language, the hypothesis translation of an MT system and a human reference sentence in the target language. Humans have rated the similarity between human reference and MT hypothesis using so-called 'direct assessment' (DA) scores which are framed in terms of one sentence 'adequately expressing the meaning' of another. We use these ratings as semantic scores in our setup. For the reference-based case, we use the hypotheses and the references as sentence pairs. This data is collected over multiple language pairs which have English as target language (so both the human reference and the hypothesis are in English) of WMT15-WMT17. For the reference-free scenario, we pair the source texts with MT hypotheses and use the corresponding reference-to-hypothesis DA scores for similarity score. For German, we take the data from WMT15 (Bojar et al., 2015), WMT16 (Bojar et al., 2016) and WMT17 (Bojar et al., 2017). Chinese is only available in WMT17. (Cer et al., 2017) consists of English sentence pairs and a semantic similarity score for each pair. The scores were annotated by humans. The score is used as semantic score in the regression. In contrast to WMT, some sentence pairs in STSB are designed to have a different structure but a similar meaning.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
While we use the sentence pairs directly for the monolingual case, we translate the sentences in the cross-lingual case using Google Translate, following Chidambaram et al. (2018). Table 3 shows the results for the reference-based metrics. The R 2 values range from 0.43 to 0.76 on WMT, and from 0.58 to 0.91 on STS. This means we can reasonably well explain the metrics from our four explanatory variables and using a linear model. mSBERT can best be explained with an R 2 value of 0.91 on STS; however, since it has been trained on STS, this merely indicates overfitting. All metrics have positive coefficients for SEM, indicating that they all reflect semantic similarity and (semantic) 'adequacy' (as measured by DA), respectively: the weights range from 0.19 to 0.48 on WMT and from 0.12 to 0.76 on STS (ignoring mSBERT). The SYN coefficients are much lower and range from -0.05 to 0.16 for WMT and from -0.03 to 0.24 on STS. Mover-Score and BERTScore are most affected by syntactic similarity (0.11 to 0.24), while the sentence embedding based techniques have coefficients around zero. The coefficients for MOR are low on STS, except for SBERT-WK, and moderate for WMT.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
All metrics have comparatively large coefficients for lexical overlap, especially on WMT: the coefficient values range from 0.24 to 0.64 on WMT and from 0.14 to 0.67 on STS. Especially LEX dominates for MoverScore and BERTScore, indicating that these two metrics are most sensitive to lexical adversaries, potentially making them vulnerable to inputs such as 'man bites dog' vs. 'dog bites man'. Table 4 shows the results for ZH-EN in the reference-free setup (omitting the score for MOR as indicated earlier). Many SYN coefficients are now zero or negative, meaning that a larger syntactic difference between the input arguments leads to a higher metric score, indicating that metrics are sensitive to syntactic language differences. LEX is still significant in all cases. SEM has higher coefficient values than LEX in 6 out of 10 cases, and when it 'wins', it wins by a large margin. However, we note that the R 2 are low: they range from 0.3-0.39 on WMT and from 0.24-0.59 on STS. This means we can either not (well) explain the metrics given our current regressors or the relationship is not well explained by a linear model. The results for DE-EN are similar; we provide them in Table 10 (appendix).
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
To explore why R 2 scores are now lower, we note that reference-free metrics based on BERT might contain a form of cross-lingual bias (CLB) in that they do not properly score mutual translations, as Cao et al. (2020) and many others have shown that the multilingual subspaces induced by BERT are mis-aligned. We thus include a factor CLB as a regressor to measure how significant this bias is in different metrics. Note that the metrics use either different BERT variants or other representations such as LASER, which points to different sources of CLB. Therefore, we realize CLB differently across metrics. For each metric regression, we use the same metric but take a parallel sentence, i.e., source text and Google translation (as we assume that Google Translate has very high quality in general), as input arguments, and take the metric score as a proxy of the CLB factor. If a metric does not contain cross-lingual bias, it should assign almost full scores to parallel sentences; this constant would then be meaningless in the regression.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
In Table 5, we show that including the CLB factor in the regression improves the R 2 . We substantially improve the R 2 for XMoverScore (from 0.39 to 0.68), but observe little improvements for the remaining metrics (especially on STS). This is because XMoverScore is more problematic than the other metrics in terms of properly scoring mutual translations, given that the other metrics use BERT variants (or LASER) that have been finetuned (or trained) on parallel sentences. Apart from CLB, both SEM and LEX are the dominating factors in the regression. The DE-EN results are similar-see Table 11 (appendix).
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Limitations The R 2 scores for the WMT dataset are almost always lower than the corresponding STS scores. One may not forget that STS sentences are in a sense artificial sentences of the form 'a girl is playing a guitar' while WMT contains more realistic sentences as well as their (possibly faulty, non-grammatical) translations. The WMT datasets are furthermore inhomogeneous in that we used different years from 2015 to 2017, which has dif-    Table 5: Regression results for Chinese-English reference-free metrics. We add the CLB factor in the regression. ferent participating MT systems as well as slightly different task definitions, corresponding to an aggregation of different domains. The STS dataset is monolingual and was translated by Google Translate for the cross-lingual scenario. The latter may lower the quality of the data, but WMT data also contains translations.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
The WMT scores measure the similarity between the reference and the hypothesis but we compare the source with the hypothesis in the cross-lingual scenario, which reflects a mismatch. Fomicheva et al. (2020a) provide a dataset which gives human DA scores between source and hypothesis. We repeated the experiments with this dataset. The full results are shown in Table 12 in the appendix (omitting the CLB factor). The R 2 scores of 3 out of 5 metrics improve (slightly) compared to the WMT dataset for German-English but all R 2 scores are lower for Chinese-English. This means that mismatched DA scores are apparently not the main reason for our low regression fits. With the new DA scores, all coefficients for SEM are considerably lower. They are in the range of range 0.06 to 0.14 compared to the maximum of 0.47 for WMT. In contrast, all SYN (0-0.12) scores are higher especially vof German. All MOR (0.16-0.33) and most LEX scores are higher but they are still in a similar range as for the original DA scores.   In the following, we analyze two observations from our previous experiments in more depth: (i) the sensitivity of metrics to lexical overlap; (ii) the orthogonality of metrics in that they capture different linguistic signals. Kovacic did a quick give-and-go at midfield.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Kovacic managed a quick one-two in midfield.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Kovacic did a quick midfield at giveand-go.  
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
According to our results, all metrics rely on lexical overlap, which indicates that they may not be robust to adversarial examples. We check this by an additional experiment, for reference-based metrics, in which we query their pairwise preferences over three sentences: Sentence A is the anchor sentence; sentence B is a paraphrase of sentence A with little lexical overlap; sentence C is a non-paraphrase with high lexical overlap. A good metric m would have m(A, B) > m(A, C) but the high lexical overlap between A and C makes this task difficult.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Freitag et al. Sentences A are taken as source sentences from WMT19.  provided alternative references for WMT19; they instructed human professional translators to paraphrase the references as much as possible in terms of lexical choice and sentence structure but keep the same semantics. We take these as sentences B. We produce sentences C from sentences A: for each sentence A, we detect the nouns within the sentence using the NLTK POS tagger, and then we randomly permute them to produce a sentence C. Since  provided sentences in German, we translate all sentences into English using Google translate. We note that by inspection the translations are generally of high quality and satisfy our constraints of inducing paraphrases with low lexical overlap and non-paraphrases with high lexical overlap. Examples are shown in Table 7 and statistics in Table 6.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
PAWS We complement the analysis with the native English PAWS dataset (Zhang et al., 2019) which consists of paraphrase and non-paraphrase pairs that have high lexical overlap. For each sentence in the dataset, there are multiple paraphrases and non-paraphrases. For a given sentence A, we use the paraphrase with the smallest amount of lexical overlap as sentence B and sentence C is the non-paraphrase with the highest amount of lexical overlap with A. We note that PAWS is more problematic as even the paraphrases B with minimum amount of lexical overlap do have considerable lexical overlap. Therefore, we select the 100 sentence pairs with the smallest amount of lexical overlap between sentence A and B. Table 6 shows the lex-ical overlap and the size of the datasets. Indeed, for PAWS, sentences B have only a little less lexical overlap with A than sentences C, while the dataset of  has a much clearer separation between B and C in this respect. Figure 3 shows the distribution of m(A, B) and m(A, C) for selected metrics m. Overall, the adversarial results on translated and non-translated datasets point in a similar direction. We see that metrics clearly prefer the high lexical overlap sentences which are non-paraphrases (sentences C) in the translated dataset of . For non-translated PAWS, metrics are at least to some degree indifferent, but tend to prefer B on average, with MoverScore and BERTScore having higher preference for C than mSBERT and SBERT, which confirms our linear regression results.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Overall, these experiment show that metrics are indeed not robust to lexical adversarial examples.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Our experiments in Section 4 show that the different metrics use different information signals, even when they use the same underlying BERT representations. For example, BERTScore relies more on lexical overlap and mSBERT relies more on semantics; BERTScore and MoverScore both capture syntax, while the other metrics are less sensitive to it. This means that the metrics are to some degree orthogonal. Thus, we suspect that a combination of metrics yields a considerably better metric. We check this hypothesis through an extra experiment. We combine especially BERTScore and Mover-Score with SBERT and mSBERT based metrics.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We evaluate the metrics on segment-level. In segment-level evaluation, each sentence pair gets a score from m. The Pearson correlation coefficient is then calculated between these scores and the human judgement, disregarding the systems which generated the translations. To combine metrics, we simply average their scores. In the evaluation, we use the best performance of the single metrics as baseline and compare it to our combined metrics.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Table 13 (appendix) shows the improvements for different language pairs. In the reference-free case, the two best ensembles combine XMover-Score with mSBERT and LaBSE-the latter two rely less on lexical overlap than the first-leading to big improvements of 11-13% over the best individual metric. Combining metrics relying on similar factors shows less improvement, and often even leads to worse results. In the reference-based case, we combine BERTScore with mSBERT and observe an improvement of 8%, more than for any other combination we tested. These results show that combing metrics relying on different factors can largely improve their performance. 1
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
We disentangled BERT-based evaluation metrics along four linguistic factors: semantics, syntax, morphology, and lexical overlap. The results indicate that (i) the different metrics capture these different aspects to different degrees but (ii) they all rely on semantics and lexical overlap. The first observation indicates that combining metrics may be helpful, which we confirmed: simple parameterfree averaging of hetereogenous metric scores can improve correlations with humans by up to more than 13% in our experiments. The second observation shows that these metrics may be prone to adversarial fooling, just like BLEU and ROUGE, which we confirmed in an additional experiment in which we queried metric preferences over paraphrases with little lexical overlap and non-paraphrases with high lexical overlap. Future metrics should especially take this last aspect into account, and improve their robustness to adversarial conditions. There is much scope for future research, e.g., in developing better global explanations for referencefree metrics (as we cannot yet well explain these metrics), better linguistic factors (e.g., a clearer conceptualization of morphological similarity of two sentences) and in developing local explainability techniques for evaluation metrics (Fomicheva et al., 2021a,b). 2
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
The following tables contain remaining experimental results.       
A Fine-Grained Analysis of BERTScore
While manual, human evaluation of machine translation (MT) systems is still the gold standard, automatic evaluation metrics have long been used for their relative speed and inexpensiveness. Early automatic metrics were easy to implement and somewhat correlated with human judgements, but have clear limitations: BLEU (Papineni et al., 2002) relies on n-gram overlap, and is thus not robust to differing word order or choice. In contrast, ME-TEOR (Lavie and Agarwal, 2007) requires training, but depends on token alignment, which is also a fraught task.
A Fine-Grained Analysis of BERTScore
With the advent of deep learning, new automatic metrics have arisen, both in response to and making use of the technical advances brought by deep learning. In particular, metrics like COMET (Rei et al., 2020) and BERTScore use large pre-trained language models (LLMs) to generate scores for candidate sentences. The use of these LLMs allows for metrics that take advantage of the linguistic capabilities of these LLMs, and no longer rely solely on surface-level features such as n-grams.
A Fine-Grained Analysis of BERTScore
The expressiveness of these models is both a boon and a danger. While they can (and do, based on correlation with human judgments) generate more useful scores for translations, how they arrive at the score, and which types of sentences they will score accurately is not immediately obvious.
A Fine-Grained Analysis of BERTScore
Moreover, these LLMs are known to have flaws. BERT in particular has been shown to be, in certain scenarios, insensitive to negation (Ettinger, 2020) and word order (Pham et al., 2020). BERT also has inexact representations of numbers (Wallace et al., 2019) and fails to be robust to named entities (Balasubramanian et al., 2020). All of these phenomena could result in poor-quality scores from BERTScore. However, it is difficult to say for certain how these issues might manifest in BERTScore, as it employs BERT in an unsupervised scenario distinct from that of these analyses.
A Fine-Grained Analysis of BERTScore
Thus, in this paper, we analyze BERTScore. We first formally define desiderata for a MT metric. Then, we consider how BERTScore fulfills these requirements under conditions of interest. We find that BERTScore violates some of these requirements, specifically the requirement that incorrect translations be rated below correct ones; this occurs most often when the incorrect translation is lexically similar to the reference, or especially if the difference is only in function (not content) words.
A Fine-Grained Analysis of BERTScore
The most common method of measuring the quality of a MT metric is correlation with human judgments (Fomicheva and Specia, 2019); however, these correlations provide little information regard-ing when and why an MT metric differs from human judgment. In this paper, we consider three ways of examining MT metric quality, with the aim of determining the failure cases of MT metrics.
A Fine-Grained Analysis of BERTScore
In all of our experiments, we assume the following setup. We have a MT metric, which we take to be a function M that takes as input a reference and candidate translation, and outputs a score in [0, 1]. We also have a dataset D, consisting of triples (x, Y, B) where x is a source sentence, Y is a list of at least two reference translations, and B is a list of at least one "bad" translation, which contains errors.
A Fine-Grained Analysis of BERTScore
Then we state that a good MT metric 1 M fulfills, for any triple (x, Y, B) ∈ D, where Y = {y 1 , y 2 , . . . , y n }, and B = {b 1 , b 2 , . . . , b m }, the following conditions:
A Fine-Grained Analysis of BERTScore
(i) For any pair (y, y ) of reference translations from Y , M (y, y ) ≈ M (y, y ) ≈ 1.
A Fine-Grained Analysis of BERTScore
(ii) For any reference translation y ∈ Y and candidate translation b ∈ B, M (y, b) < 1 and M (b, y) < 1. It follows that given another reference translation y ∈ Y , we should have that M (y, b) < M (y, y ) and so on.
A Fine-Grained Analysis of BERTScore
(iii) If we know the relative quality of the bad translations in B, let B be a list sorted in decreasing order of translation quality, such that b 1 is better than b 2 , and so on. Then for any reference translation y, and bad transla-
A Fine-Grained Analysis of BERTScore
Put simply, (i) reference translations should be scored near 1 when compared to each other, (ii) bad candidate translations should be scored worse than reference translations, and (iii) the scoring of bad candidate translations should reflect their relative quality.
A Fine-Grained Analysis of BERTScore
To use this framework to investigate the failure points of MT metrics, we simply need a dataset that contains phenomena of interest; for example, we might be interested in knowing if a MT metric is able to distinguish between translations that do and do not correctly render negation. Then, we simply compute the quantities discussed in conditions (i) through (iii) for each example, and see which, if any, conditions are violated. If, for example, condition (i) is violated when two equivalent references employ different types of negation, this might imply that our metric is not robust to this sort of negation phenomenon.
A Fine-Grained Analysis of BERTScore
Note that these desiderata only concern the scores given to the reference and candidate sentences; the source-language sentence is ignored. This is because we define these desiderata keeping in mind that many MT metrics (including BERTScore) operate only in the target language 2 . We can thus avoid assuming the existence of a source sentence whatsoever, allowing the construction of datasets consisting only of reference and bad candidate translations that exhibit phenomena of interest. However, it may be desirable to use real translations, so that the dataset reflects the distribution of real-world translation errors.
A Fine-Grained Analysis of BERTScore
In this paper, our metric of interest is BERTScore (Zhang et al., 2020). To compute BERTScore, we first feed a reference and candidate translation for a given sentence into BERT, and retrieve their token level vector representations. Let z be the representations of the reference andẑ those of the candidate. Then we compute the precision and recall metrics for BERTScore by comparing each token representation z i of the reference translation to each token representationẑ j of the candidate translation as follows:
A Fine-Grained Analysis of BERTScore
The F 1 score can be defined as usual. As BERTScore can range from -1 to 1, but most often inhabits the upper end of that range, its creators suggest the use of baseline scaling, which generally leaves BERTScore in the range [0,1], as desired for use with our prior formalization. Baseline rescaling is performed for P BERT aŝ
A Fine-Grained Analysis of BERTScore
and likewise for R BERT ; a is an empirical lower bound on observed BERTScore.
A Fine-Grained Analysis of BERTScore
The form of BERTScore naturally leads to its interpretation as a similarity-based metric. It penalizes candidates containing words whose representations are not similar to any of the reference's words' representations (precision), and vice-versa (recall). As a result, the quality and characteristics of these representations, derived from BERT, will play a key role in the quality of BERTScore.
A Fine-Grained Analysis of BERTScore
What, then, is known about BERT, and its syntactic and semantic capabilities? Of the two, it is syntax that BERT is most widely claimed to capture within its internal representations: Hewitt and Manning (2019) use structural probing to find dependency trees in BERT's vector geometry, while Tenney et al. (2019) use probing to find part of speech tags and dependency arc labels, among other types of syntactic information. Analysis of BERT's attention has shown that certain heads attend to not only relevant linguistic units such as determiners of nouns and coreferent mentions (Clark et al., 2019), but also dependency relations (Htut et al., 2019).
A Fine-Grained Analysis of BERTScore
However, these analyses of internal representations and the information contained therein occasionally come at odds with targeted evaluations of BERT's syntactic abilities. Despite BERT's supposed knowledge of syntax, its predictions often remain the same, even when its inputs are shuffled (Pham et al., 2020). Moreover, BERT does not seem to understand negation (Ettinger, 2020); this may be due to BERT encoding syntactic information, but not necessarily using it in its predictions (Glavas and Vulić, 2021).
A Fine-Grained Analysis of BERTScore
For semantics, the situation is even more complicated. While BERT's performance on natural language understanding tasks set a new state of the art, more targeted tests of its semantic abilities have yielded less positive results. BERT has limited knowledge of lexical semantic relations such as hypernymy (Ravichander et al., 2020) and antonymy (Staliunaite and Iacobacci, 2020). Moreover, it has fragile representations of named entities (Balasubramanian et al., 2020), and imprecise representations of numbers (Wallace et al., 2019). These flaws comprise specific linguistic phenomena that BERTScore, due to its use of BERT, might be unable to handle, and thus merit investigation. 
A Fine-Grained Analysis of BERTScore
For our experiments, we utilize the framework described in Section 2 to investigate the questions about BERT described in Section 3. As the number of datasets that fit our framework is few, we limit ourselves to three primary experiments.
A Fine-Grained Analysis of BERTScore
First, we apply our framework to the TQ-AutoTest dataset (Macketanz et al., 2018). Originally used for targeted evaluation of MT systems, it includes German source sentences that each exhibit one of 14 different linguistic phenomena, such as ambiguity, composition, and subordination.
A Fine-Grained Analysis of BERTScore
Each example contains a source sentence, annotated with the broader and more specific phenomenon it exhibits, as well as 1-2 reference translations and 0-2 incorrect translations; see Figure 1 for a sample. As the dataset released is small, we filter out phenomena containing fewer than 5 examples.
A Fine-Grained Analysis of BERTScore
We can thus test condition (i) by computing the BERTScore between the two references, and verifying it is close to 1. We also test condition (ii) by comparing the BERTScore assigned (with respect to a given reference) to the other reference, and that which is assigned to a bad translation; the former should be greater than the latter. We cannot test condition (iii), as the dataset does not provide multiple bad translations sorted by quality.
A Fine-Grained Analysis of BERTScore
Second, we use the PE 2 rr dataset (Popović and Arčan, 2016), which is a manually annotated error analysis of MT output. Each example in the dataset consists of a source sentence, one MT output, and two correct translations, along with two error annotations. These annotations are word-level annotations into 8 broadly non-linguistic classes, such as  "addition", "lexical error", or "untranslated". See Figure 2 for an example.
A Fine-Grained Analysis of BERTScore
Despite the difference in annotation type, we apply our framework just as with the prior experiment. For consistency with the prior experiment, we use only the portion of the dataset where the target language is English (i.e., the German-English portion). We also filter out any examples in which the machine translation output is totally correct, as this would leave no bad examples with which to test conditions other than (i). This once more allows us to test conditions (i) and (ii).
A Fine-Grained Analysis of BERTScore
Finally, we use two non-MT datasets for grammatical error correction (GEC): the CoNLL 2014 shared task dataset (Ng et al., 2014), as well as additional annotations released by Bryant and Ng (2015). The former consists of non-native English speakers' essays on genetic testing, paired with annotated corrections from two annotators for each sentence. The latter adds additional annotators' corrections for each sentence, yielding 11 in total.
A Fine-Grained Analysis of BERTScore
Of special interest in this dataset is the presence of, for each example, an incorrect sentence and 11 sets of error-annotated corrections that can be applied to obtain correct sentences. Thus, for each original, ungrammatical sentence, we have 10 (often distinct) grammatical sentences, whose meaning should be roughly the same; these act as reference sentences. This allows us to test condi-
A Fine-Grained Analysis of BERTScore
We can also test condition (iii) by using the following moderate assumption: a sentence, originally grammatically incorrect, is more correct if more corrections have been applied to it. That is, applying one or more corrections (from the same annotator) brings an incorrect candidate sentence closer to the shared meaning of the reference sentences. This assumption can be false: sometimes a group of corrections, rather than one alone, is needed to increase the grammaticality of a sentence. But, this assumption allows us to apply an arbitrary number of corrections to an initially incorrect sentence, to generate intermediate incorrect sentences, with controlled, graded, levels of incorrectness.
A Fine-Grained Analysis of BERTScore
So, we generate two incorrect candidates by applying different numbers of edits from the same annotator to one original sentence. We then generate a reference sentence by applying all of another annotator's edits. Finally, we calculate the BERTScore of each candidate with respect to this reference; the candidate that received more edits should receive a higher BERTScore. Note that the reference must be generated by a different set of edits; comparing partially-corrected sentences to a correct sentence generated from same edits would make this a trivial string comparison problem. Figure 3 provides an example of this process.
A Fine-Grained Analysis of BERTScore
In the following section, we detail the experiments performed and their results. In all experiments, the original authors' implementation of BERTScore 3 is used, with default baseline rescaling.
A Fine-Grained Analysis of BERTScore
As discussed in Section 4.1, we test conditions (i) and (ii) with the TQ-Autotest dataset. First, we filter the dataset to include only those examples for which there are at least two good translations (y, y ). Then, to test condition (i), we compute the BERTScore (BERTScore(y, y )) assigned to the pair of good translations, then compute the mean score for each category.
A Fine-Grained Analysis of BERTScore
Note that one should not make absolute comparisons between the mean BERTScore assigned to each category and the desired value (1.0) of the mean score. Because BERTScore can easily be rescaled, it is more useful to verify that good translation pairs receive similar scores across categories;
A Fine-Grained Analysis of BERTScore
Original: As a result , if the situation keep go on in this unexpected trend , it will cause a bad effect on the young generation . Corrections (Annotator 2): (7, 8, 'keeps', subject-verb agreement) (8, 10, 'following', word choice) (10, 11, '', preposition) (17, 18, 'have', word choice) (23, 24, 'younger', word form) Original + 1 correction (b 1 ): As a result , if the situation keeps go on in this unexpected trend , it will cause a bad effect on the young generation . Original + 4 corrections (b 4 ): As a result , if the situation keeps following this unexpected trend , it will have a bad effect on the young generation . Alternate reference (Annotator 7) (y): As a result , if the situation keeps going on in this unexpected trend , it will have a bad effect on younger generations . Figure 3: Creation of graded ungrammatical sentences from GEC data. Each correction is a tuple of (start index, end index, new text, error made). We apply a correction via string replacement (i.e. in Python: original[start_index: end_index] = new_text).
A Fine-Grained Analysis of BERTScore
Having received more corrections, b 4 should be closer to y than b 1 is. Thus, we should have BERTScore(y, b 4 ) > BERTScore(y, b 1 ).  Table 2: TQ-AutoTest: Average Accuracy by broad linguistic phenomenon. Credit is given when BERTScore(y, y ) is greater than BERTScore(y, b), and BERTScore(y, b ), if a b is provided.
A Fine-Grained Analysis of BERTScore
these scores could then easily be rescaled to 1.0. Thus, in this section, we explore only which types of correct sentence pairs BERTScore is more or less able to assign a high score, compared to the average score given to correct sentence pairs. We see in table Table 1 that the average BERTScore for each category falls near the average. There are some notable exceptions: for example, the "function word" category, which falls well below the mean. This indicates that BERTScore gives different correct translations of the same sentence lower scores, when that sentence contains difficult function words. In contrast, the "coordination & ellipses" and "multi-word error" categories fall well above the mean, suggesting that alternate correct translations of sentences containing these phenomena are scored more highly.
A Fine-Grained Analysis of BERTScore
In the second experiment, we test condition (ii) via the following procedure. Once more, we filter the dataset, such that every example includes two good translations and one bad translation, (y, y , b), with potentially another bad translation b as well. Then, we test the accuracy of BERTScore on these examples. BERTScore is deemed to correctly answer an example if BERTScore(y, y ) > BERTScore(y, b). If b is present, as in 71% of examples, it is also necessary that BERTScore(y, y ) > BERTScore(y, b ). We then report the mean accuracy for each category of linguistic phenomenon.
A Fine-Grained Analysis of BERTScore
In this second experiment, see Table 2, the differences are more pronounced. In some categories, namely those such as "ambiguity", and "multi-  word error" which are likely to result in totally incorrect word choice (i.e. an obvious lexical difference), BERTScore has a high accuracy. In contrast, BERTScore struggles with difficult punctuation as well as composition errors and function words; notably, the last category also has the lowest score in Table 1. These errors are all somewhat subtle. Errors in function words are by definition not errors in more obvious content words. Similarly, in compositional phenomena like phrasal verbs, an error can appear simply as the omission or incorrect substitution of a mere preposition. In order to provide context for these results, we also run the second experiment (which corresponds to Table 2) using BLEU score 4 as our metric. As with BERTScore, we mark an example as correct when BLEU(y, y ) is greater than BLEU(y, b), and BLEU(y, b ), if a b is provided. We omit the category-level results for brevity, but find that the mean accuracy for BLEU is significantly lower, at 36.2%, compared to 75.5% in Table 2. This suggests that although BERTScore is flawed, it is at least more accurate than BLEU.
A Fine-Grained Analysis of BERTScore
Using the PE 2 rr dataset, we again test conditions (i) and (ii). We use an approach like that taken with the TQ-AutoTest dataset. First, we filter out examples in which the two references translations provided are identical; in this case, the BERTScore will be trivially 1. For all other examples (y, y ), we compute BERTScore(y, y ). Then, for each error category, we compute the mean BERTScore (F 1 ) among all examples that contain at least one 4 We calculate BLEU as implemented in the multi-bleu.perl script at https://github.com/ moses-smt/mosesdecoder/blob/master/ scripts/generic/multi-bleu.perl  Table 4: PE 2 rr: Average accuracy by error type, for both the easy and hard problem scenarios. Credit is given when BERTScore(y, y ) is greater than BERTScore(y, b). of that error. Results are reported in Table 3. We find that the average BERTScore assigned to pairs of correct translations in this dataset, 0.587, is much lower than in TQ-AutoTest, where the average BERTScore assigned to correct translations pairs was 0.815. Moreover, PE 2 rr examples with no errors in their machine translation have a higher BERTScore assigned to their correct translations.
A Fine-Grained Analysis of BERTScore
Testing condition (ii) with the PE 2 rr dataset is somewhat more challenging. First, we filter out any of the machine translations that have no errors (as we need a bad translation to test condition (ii)). Normally, the next step would be to compute and compare, using our two references translations y, y and one bad translation b, BERTScore(y, y ) and BERTScore(y, b).
A Fine-Grained Analysis of BERTScore
However, a difficulty arises: while each example has two correct translations, and one incorrect machine translation, the two correct translations are not generated in the same way. One is a post-edit of the machine translation, while the other is an original reference, generated from the German text without the machine translation.
A Fine-Grained Analysis of BERTScore
If we choose the original reference to be y, and the post-edit to be y , this is a fair comparison; however, if we choose the post-edit to be y, the task becomes much more challenging. This is because BERTScore(y, b) will be comparing a post-edited machine translation to the original machine translation, and there will naturally be a good deal of overlap, even though b contains errors. So, we report results in two cases in Table 4: the easy case, where y is the original reference, and the hard case, where y is the post-edit. This choice has a major effect on the ability of BERTScore to distinguish good translations from bad ones. When we choose the original reference as y, BERTScore has a high accuracy in all categories. In contrast, in the hard problem setting, BERTScore does poorly (at or below chance) in all categories except sentences with an untranslated word, which is likely easy for BERTScore to detect.
A Fine-Grained Analysis of BERTScore
We again compare BERTScore to BLEU to contextualize our results for condition (ii) (in Table 4). We find that once more BERTScore outperforms BLEU, which achieves accuracies below 50% in all categories in the easy setting, and below 13% in the hard setting. The disparity between the easy and hard setting performance reflects the fact that BLEU also struggles to penalize bad translations with high n-gram overlap, compared to those with less overlap.
A Fine-Grained Analysis of BERTScore
Finally, to provide an alternate explanation for the trend in BERTScore accuracies, we plot in Figure 4 the BERTScore accuracy for examples with n errors, in both the easy and hard scenarios. While in the easy scenario, BERTScore has an accuracy near 1 for n >= 5, in the hard scenario, the accuracy is lower, but increases with n. This suggests that it is easier for BERTScore to distinguish between translations that are good and those that are bad, but lexically similar to the postedited reference, when the latter contain more errors. Alternatively, we can view this as BERTScore being less sensitive to translation errors (except when they are numerous), and relatively more sensitive to the stylistic differences that exist between the post-edited and original reference sentences.
A Fine-Grained Analysis of BERTScore
For GEC, we test all three conditions. To test the first, we use the 11 annotators' corrections to create post-edited versions (y 1 , . . . , y 11 ) of the original sentences; these should all have the same mean-  Table 5: GEC: Average BERTScore (F 1 ) assigned when comparing a reference y to a) b n , a sentence with n errors in it, b) b 0 , the same sentence, but with all errors corrected, and c) y , an alternate reference sentence.
A Fine-Grained Analysis of BERTScore
ing. Then, we compute the mean BERTScore (F 1 ) for each pair of annotators' post edits; these are reported in Figure 5. The mean BERTScore falls in a very narrow range (0.84-0.88), except for Annotators 1 and 8, who provided similar annotations. We test conditions (ii) and (iii) jointly. To do so, we need two correct translations, created independently of one another. We also need two partially correct translations (of different levels of correctness) and one correct translation; these must be created from the same set of edits, independent of the first two. To create these, we first select three annotators, and assign them roles: reference annotator, alternate reference annotator, and "error" annotator. Next, for n = 1, . . . , 5, we filter out all examples where the error annotator made fewer than n error annotations. Then, for each example, the reference and alternate reference annotators' an-notations are used to create one reference sentence (y) and alternate reference sentence (y ). The error annotator's annotations are used to create both a third, fully corrected sentence (b 0 ), and a sentence with n errors in it (b n ); the errors that remain are chosen at random.
A Fine-Grained Analysis of BERTScore
Finally, we compute BERTScore(y, y ), BERTScore(y, b 0 ), and BERTScore(y, b n ); the mean values for each of these is reported in Table 5. Note that although b 0 and y have no errors, their mean BERTScore still changes with n because we filter out examples that have fewer than n errors according to the error annotator. Thus, the downward trend with growing n reflects the fact that, as the number of errors in a sentence increases, the annotators' corrections diverge.
A Fine-Grained Analysis of BERTScore
We can see that BERTScore does respect conditions (ii) and (iii) within the GEC data. The alternate reference and error-free sentence are always assigned a higher score than the sentence with errors (ii). Moreover, the sentences with more errors are assigned a lower score than those containing fewer errors (iii).
A Fine-Grained Analysis of BERTScore
Also of note is the absolute magnitude of the BERTScores assigned in this experiment, which is much higher than those in the prior PE 2 rr experiment. The mean BERTScore assigned to even examples containing 5 errors (0.812, Table 5), is much higher than the mean BERTScore assigned to PE 2 rr reference pairs (0.587, Table 3). As before, we suggest that this occurs due to sensitivity to style. Although b 5 contains errors, it is stylistically similar to the reference, as they are both edits of the same sentence; in contrast, the two references in the PE 2 rr dataset are very distinct from one another.
A Fine-Grained Analysis of BERTScore
Finally, we perform this same evaluation with BLEU, again omitting full results for brevity. We find that BLEU, regardless of n, gives the alternate reference y a higher score than the other sentences. However, the BLEU scores assigned to b n and b 0 were similar, and b 0 did not consistently receive higher scores. Thus, we conclude that BERTScore also performs better on this task.
A Fine-Grained Analysis of BERTScore
From our experiments on three datasets, we draw three main findings. First, we find that the performance of BERTScore with respect to conditions (i) and (ii) can vary based on linguistic phenomena. Second, while BERTScore is generally capable, it has difficulties on a challenge dataset that tasks it with penalizing lexically similar incorrect translations and preferring translations that are lexically different but more correct. Third, BERTScore does, in certain circumstances, respect our third condition-it ranks worse bad translations below better bad translations.
A Fine-Grained Analysis of BERTScore
With respect to the first finding, penalizing translations that incorrectly render function words seems to be the most difficult for BERTScore. In TQ-Autotest, this includes sentences with tag questions; in one example, the reference is "You're crazy, aren't you?", and secondary good translation is "You're crazy, right?", while incorrect sentences are "You're crazy, or?" and "You're crazy, are not you?". That is, the differences do not affect the main content of the sentence. In contrast, sentences with incorrectly resolved word ambiguity or larger, multi-word errors were easily penalized.
A Fine-Grained Analysis of BERTScore
The second finding also confirms that BERTScore can more easily detect bad translations when there is less lexical overlap. In the easy problem setting, where both the good and bad candidate translations were lexically distinct from the reference, BERTScore easily distinguished the good translation from the bad. But, in the hard problem setting, where the bad translation has high lexical overlap with and was stylistically similar to the reference, BERTScore struggled. Due to this style sensitivity, BERTScore may be better-suited to scoring candidates from widely-differing systems, as opposed to closely-related systems, or multiple candidates from one system. Our third finding, provides more positive results. In the GEC scenario, BERTScore was able to fulfill all three of the conditions we defined. It not only gave high scores to similar sentences and worse scores to sentences with errors; it also gave better scores to more grammatically correct sentences, even when they were not perfectly correct.
A Fine-Grained Analysis of BERTScore
Unfortunately, the GEC dataset does not necessarily reflect the kinds of errors that occur in machine translation. It focuses primarily on grammatical errors, and thus contains fewer semantic errors. Moreover, since the GEC annotators had only the incorrect text, and no source text to work with, their annotations are occasionally in disagreement, as they each independently inferred the intended meaning of the incorrect text.
A Fine-Grained Analysis of BERTScore
BERTScore is a new metric for MT evaluation that uses BERT, and is as a result difficult to interpret. We define desiderata for BERTScore and other such metrics' performance, and use targeted datasets to find when BERTScore fails. We find that BERTScore fails to assign low scores when a bad candidate sentence has high lexical overlap with the reference in terms of content words. Despite this, in less challenging scenarios, BERTscore does well, and is able to rank sentences in order of their quality. Moreover, BERTScore outperforms BLEU score across the datasets and conditions we tested. However, these experiments are limited in scope, due to limited available data with appropriate annotations. Development of datasets for MT metric evaluation with linguistic annotation would aid in further work on this topic.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
With the rapid development of neural networks in text summarization (Liu and Lapata, 2019;Zhong et al., 2019;Zhang et al., 2019;Lewis et al., 2019;Zhong et al., 2020;Liu and Liu, 2021), especially the use of contextualized pre-trained models (Devlin et al., 2019;Lewis et al., 2019), the state-of-the-art performance, measured by automated metrics such as ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2020) has been constantly updated. However, although these systems can generate informative, and fluent summaries, they suffer from the problem of making factual errors-generating incorrect facts that can not be supported by the source document (Cao et al., 2018a).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Among this background, a large body of recent works (Wang et al., 2020a;Kryscinski et al., 2020;Durmus et al., 2020;Cao et al., 2020) are trying to search for new automated metrics that can assess the factuality of generated summaries due to the fact that existing metrics (e.g., ROUGE) are not correlated well with factual consistency (Maynez et al., 2020;Goyal and Durrett, 2020).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Generally, the process of designing these evaluation metrics w.r.t factuality is commonly formulated into different forms of NLP tasks, ranging from text entailment (Falke et al., 2019;Kryscinski et al., 2020) at sentence level or more fine-grained level (Goyal and Durrett, 2020) to question answering (Wang et al., 2020a;Durmus et al., 2020). Improving the understanding of these factuality metrics with diverse paradigms is critical for further metric improvement. However, the evaluation methodologies of factuality metrics are limited in their opacity-they are opaque to their results, which are usually holistic scores (e.g., accuracy) and not interpretable. Specifically, different from traditional non-learnable metrics like ROUGE, whose scores are relatively straightforward to interpret, e.g., lower ROUGE-2 Recall implies fewer bigrams from reference summaries are covered by generated summaries, there are diverse factors that could lead to lower score of factuality metrics (e.g., entity replacement, number inference). However, most of existing meta-evaluation strategies fail to tell (i) which types of factual errors the metric evaluated at hand are better at identifying, (ii) on which categories the error recognition ability of factuality metrics can not be well generalized. As a result, (1) the relative advantages between a better-and worse-performing systems w.r.t factuality are unclear.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(2) the lack of understanding of factuality metrics' applicability reduces their reliability, and users may take the risk of over-estimating their generalization ability so as to apply them to inappropriate evaluation samples. (3) it's unclear how to improve the metric further.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Thus instead of further pursuing a new method, we take a step back to understand the shortcomings of existing metrics. We present an adversarial meta-evaluation framework which can perform fine-grained evaluation of factuality metrics. Methodologically, we (i) first conduct error analysis of existing state-of-the-art factuality metrics, (ii) define effective adversarial transformations based on the results of error analysis. We (iii) construct diagnostic examples by applying adversarial transformations to test datasets with different distributions and then diagnose existing top-scoring factuality metrics. (iv) We finally show that, the technique of data augmentation, driven by adversarial transformations, can increase the diversity of training samples, making factuality metrics more robust and reliable.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Our contributions can be summarized as follows: (1) We figure out several representative errors made by the existing top-performing factuality metrics ( §4.2), inspiring the direction for further improvement.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(2) We propose effective adversarial transformations that can either be applied to test set for model diagnosis ( §5) or applied to training set for data augmentation ( §6.2), by which we further improve the performance of current checkers. (3) We propose a fine-grained meta-evaluation methodology for factuality metrics and re-evaluate existing top-performing metrics to assess their relative strengths and weaknesses. (4) We call for a more fine-grained and interpretable meta-evaluation of factuality metrics for future research. As a first step, we released our constructed diagnostic test sets with various characteristics, as well as augmented training data and more robust factuality metrics.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Although researchers have slightly different definitions of factuality (Maynez et al., 2020;Kryscinski et al., 2020). In this paper, we consider factuality as how well generated summaries are supported by source documents without using any external knowledge. A factual error happens when generated summaries contain salient facts (Kryscinski et al., 2020) that can not be inferred from source documents. The summary sentences that need to be verified are also called claims below to keep consistent with the field of fact verification (Zhou et al., 2019;Schuster et al., 2019;Liu et al., 2020).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Type Train data 
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
There are two major task formulations of factuality metrics: natural language inference (NLI) and question answering (QA). Model types and training data are summarized in Tab. 1.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
NLI-based metrics consider factual consistency as a natural language inference problem, the core idea of which is to infer if facts from generated summaries can be entailed by its source documents.  (Williams et al., 2018). The neutral class samples are deleted in the dataset for fair comparison following Goyal and Durrett (2020).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
The basic idea behind QA-based metrics is whether similar answers can be replied when we ask the same question to a generated summary S and its source document D ( 
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
To get a holistic overview of factuality metrics performances, we collect four different human judgment datasets that can be used to meta-evaluate the correctness of factuality metrics.   (1) FACTCC has achieved the best performance in most of the test sets except in FaithFact. The reason for this is that the claims in this set are highly paraphrased (its novelty is 99.2% in Tab. 2) thus will mislead FACTCC which is trained on less abstractive claims (CNNDM-G as shown in Tab. 1).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(2) FEQA underperforms FACTCC most of the time.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(3) With the same pre-trained model (ELECTRA), DAE outperforms MNLIELECTRA in FaccTe and QagsC. However, DAE with dependency information doesn't show constant superiority over MNLI-ELECTRA in all evaluation sets.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Setup and Error Typology To get a more finegrained understanding of factuality checkers and define the upper bound of the difficulty for the task, we choose FACTCC as the representative factuality checker (for its superior performance as described in §4.1) and perform error analysis on it. We examine 140 samples 4 that the checker fails to predict correctly in FaccTe and QagsC, and divide the reasons into diverse categories. Examples are pre-• R1: VANs replacement: the checker is hard to detect Verb, Adjective and Noun replacements (e.g., antonym, synonym) thus producing the wrong prediction. Here noun represents noun or noun phrase excluding entity. • R2: Numerical inference the checker obtains worse performance when verifying samples that require numerical inference (e.g., date). Similar results are also observed in (Zhao et al., 2020b). • R3: Entity coreference: a slight change of person name or replacing the pronoun with its reference name will mislead factuality checker which suggests the lack of entity coreference resolution ability. • R4: Missing details: when the claim lacks some detailed information (e.g., location), the checker tends to predict it as inconsistent though it is not. While this is frequently occurring in the scenario of summarization when the summarizer only extracts the most important information. • R5: Paraphrase The more complex paraphrase patterns (e.g., complex reorder, passive-active transformation, sentence fusion and so on) other than simple token replacement or omission that cause the model to make wrong predictions. • R6: Background knowledge The checker is fragile when extra knowledge is required. • R7: Truncate The checker truncates long documents and will ignore the information of evidence sentences in later part of documents, therefore making wrong judgment. • R8: Wrong label Incorrect annotated label. • R9: Others Other reasons.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
As presented in Tab. 3, VANs replacement and Missing details account for a large proportion in all error reasons. It is because verb, adjective and noun (besides entity) replacement and detail omission are not included in the training data for FACTCC. Moreover, misclassifications that caused by paraphrase are account for 11.8%, which lies in the lack of paraphrase for training data of FACTCC as the only paraphrase pattern is introduced by backtranslation (Edunov et al., 2018). While entity and number swap are included in negative sample construction in (Kryscinski et al., 2020), FACTCC still makes wrong prediction facing samples requiring entity coreference resolution and numerical inference.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Source document Claim Ratio R1: VANs replacement (inco → co)
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
...Japanese court issued a landmark injunction halting plans to restart two nuclear reactors in a western prefecture... japanese court orders to restart two nuclear reactors in a western prefecture.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R2: Numerical inference (co → inco)
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
...On October 31, 2014, the Italian government announced the end of "Mare Nostrum" ... the italian government announced the end of "mare nostrum" in 2014.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R3: Entity coreference (co → inco)
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
...Ahmed Farouq didn't have the prestige.....Before that, Farouq was the deputy emir of al Qaeda....
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
ahmed farouq was the deputy emir of al qaeda in the indian subcontinent.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R4: Missing details (co → inco)
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
...Phil Rudd, the drummer for legendary hard rock band AC/DC, has pleaded guilty to charges of... rudd has pleaded guilty to threatening to kill and possession of drugs in a court.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R5: Paraphrase (inco → co)
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
...A police motorcycle stopped the rest of the pack, before organisers of the 151-mile race slowed the leaders to allow the pack to catch up...
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Leaders of the tour de france were stopped by police as they crossed a railway line to avoid a train.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R6: Background knowledge (co → inco)
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Scientists from harvard medical school have discovered a way of turning stem cells into killing machines ...
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Scientists in the us have developed a stem cell therapy for brain tumours. These days we are increasingly using outdoor space for the occasional barbecue or to relax in a hot tub rather than for tending flowers.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
these days we are increasingly using outdoor space for tending flowers.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
12.4% Table 3: Error reasons with their corresponding examples and the ratio of them. The bold span is corresponding to the error reason. co → inco represents the gold label is factually correct while checker misclassifies it as factually incorrect (inco → co means the opposite).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
[>512] means there are more than 512 subwords before this position.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
It is not realistic to produce large scale human annotated test sets with multiple error reasons observed above. As a consequence, former work (Hidey et al., 2020) and (Naik et al., 2018) construct diagnostic test sets automatically. In this section, we first introduce automatic rule-based transformation methods based on error analysis ( §5.1). Then we construct 24 diagnostic test sets based on three types of baseline test sets.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
We introduce four types of automatic transformation methods corresponding to the R1-4 error reasons in error analysis ( §4.2). Paraphrasing (R5) is not included here for it is hard to produce simply with rule, thus we introduce it in another way-using gold references as claims in §5.2. The rest four error reasons are either too hard for models (R6, R9) or correspond to systematic error (R7) or lie in annotation error (R8), and also will not be included here. The adversarial transformation examples are shown in Tab. 4.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
We first use Stanza (Qi et al., 2020) to do Part-of-Speech tagging and then use WordNet wrapped in NLTK (Bird et al., 2009) package to find antonyms for verb and adjective. Negative samples are produced by replacing the original word with its antonyms. The reason we do not include synonyms replacement is that simply replacing word with its synonyms can introduce factual error and cause the gold label ambiguous.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R2: Numerical Editing FACTCC exhibits worse performance when it needs numerical reasoning to derive the result as §4.2 shows, which motivates us to design a numerical adversarial transformation. Specifically: (1) to produce negative samples, we replace numerical entity 5 with a randomly chosen entity of the same type in source document and guarantee the transformed claim differs from the origin. On the other hand, we also add preposition (e.g.,"after") before date and timing type entities while adding "more than" and "less than" before other types of numerical entities;
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(2) For positive samples, we change the number or date 6 and add "before", "after", "more than" and "less than" properly (e.g., "in 2019" to "two years before 2021"). We include more complex negative and positive transformations for numerical inference compared with Kryscinski et al. (2020).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R3: Entity Replacement At the phase of error analysis, we discover FACTCC fails to understand the equivalence between named entities referring to R3: EntRep pos actor isaiah washington → isaiah tweeted : ' okay , watching the #walterscott video was horrible , but i think the brave person who captured the murder is a hero and a godsend #truthdom . ' neg actor isaiah washington → michelle williams tweeted : ' okay , watching the #walterscott video was horrible , but i think the brave person who captured the murder is a hero and a godsend #truthdom . '
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
R4: SynPrun prepo. the queen and the duke of edinburgh appeared in good spirits as they arrived to a red carpet at the event .
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
clause the mystery hero who raced to the edge of a cliff and pulled a driver from his precariously-balanced car has been identified as a 29-year -old man who fled the scene to go to work . R4: Syntactic Pruning Syntactic pruning is used to produce positive examples with detail omitted. Despite using dependency parsing, we choose constituency parsing to disentangle the summary sentence for it is more suitable to capture clauses and phrases. To produce positive examples, clauses with label "S" and "SBAR" and prepositional phrases with label "PP" are deleted based on the assumption that the lack of sub-clause will not affect the factual consistency.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
We construct 24 diagnostic datasets 7 based on three types of base test sets as follows: Besides only using sentences in source document (DocAsClaim) as input to transformation as previous work (Kryscinski et al., 2020) does, we propose to use another two base test sets: gold sum- 7 We have released the datasets on our Github repository. And the detailed information of it is included in the appendix. mary (RefAsClaim) and generated summary (FaccTe, QagsC,RankTe and FaithFact) to serve as input to the adversarial transformation. Reasons are: (i) the diagnostic set constructed based on reference summaries corresponds to the error reason R5 in §4.2, which is a more challenging test set for factuality checkers due to its more complex paraphrase patterns. (ii) the distribution of generated summaries will be more closed to summaries verified by factuality checkers in real scenarios (e.g., generated summaries from BART). Finally we obtain 6 base test sets and 24 diagnostic test sets (4 adversarial transformations on every base test set).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
In order to explore the reliability of the automatically generated diagnostic test sets, we conduct human examination on whether the generated claim is grammatically correct and maintains correct label. This is carried out on 50 randomly chosen samples for each type of adversarial transformation. Results 8 show that all the diagnostic sets are grammatically correct (ratio around 85%) and possess correct factuality labels (ratio higher than 90%).  shows (nearly all entry values of AntoSub columns are negative). However, FEQA and FACTCC obtain obvious performance improvement in the AntoSub diagnostic set of RefAsClaim. It is because claims in RefAsClaim original set are highly paraphrased which will mislead the checkers to produce negative labels and cause lower accuracy. While Antonym Substitution introduces factual inconsistent samples, thus instead, model performance improves. Models transferred from MNLI and DAE are more robust to samples with highly paraphrased claims.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Numerical Editing Nearly all factuality checkers get worse performance with NumEdit transformation (almost all results of NumEdit columns are negative in Tab. 5). Even FACTCC is not the exception though it may possess numerical inference ability to some extend. It emphasizes the importance to improve numerical inference ability for factuality checkers. However, FEQA and FACTCC get better performances when tested in NumEdit diagnostic set of RefAsClaim because the numerical editing transformation introduces more negative samples (reason is similar as described above).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Entity Replacement Similar to numerical Editing, the entity replacement transformation also tends to mislead six factuality checkers as nearly all values of EntRep columns in Tab. 5 are negative. Although FACTCC is trained with data that also includes entity replacement transformation, it still obtains worse performance in EntRep diagnostic test sets of QagsC and RankTe. This implies the incompleteness of entity replacement in (Kryscinski et al., 2020). It shows the same pattern as Anto-Sub when models are tested in EntRep diagnostic sets of RefAsClaim and the reason is similar as described above.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Syntactic Pruning The diagnostic test sets of SynPrun can lead to more performance drop when the base test sets are RefAsClaim and FaccTe because the last columns of these subtables get more negative values. Transformation of this type will be more confusing when the claims are highly paraphrased.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
As observed in Tab. 5, models transferred from MNLI dataset and DAE are more robust when syntactic pruning are introduced, while FACTCC and FEQA are constantly misled by SynPrun diagnostic test sets. This can be attributed to the lack of highly paraphrased claims in FACTCC training set. DAE tends to extract dependency triples of summary and make prediction based on them, thus is more robust when evaluated in SynPrun diagnostic sets. As for models transferred from MNLI, it may because the training set of MNLI already possesses pattern of detail omission and the trained models have the capability to recognize it.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Takeaways (1) Most factuality checkers obtain poor performance in AntoSub and NumEdit diagnostic sets, which suggests that current factuality metrics are not faithful when dealing with antonym substitution and numerical editing samples. (2)  FACTCC can handle entity replacement diagnostic sets to some extent, but can not maintain the performance constantly over all EntRep sets. (3) MNLIBERT, MNLIROBERTA, MNLIELECTRA and DAE are more reliable to deal with highly paraphrased claims and are more robust to syntactic pruning transformation.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Besides utilizing adversarial transformation to construct test sets, it can also be used to create more training data, i.e., data augmentation, to improve the model performance. Here we choose FACTCC to conduct adversarial training 9 due to the excellent performance of FACTCC in § 4.1.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
As the original training data of FACTCC has more than 100 million samples, we first subsample 50 million data to train FACTCC sub . Moreover, we add 34,912 adversarial training data to the subsampled set and train another checker called FACTCC adv sub . Also, we investigate whether introducing references as claims to the training set will enhance model performance. We include references as claims and make negative transformations in (Kryscinski et al., 2020) 
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
In this paper, we present an adversarial metaevaluation methodology driven by our fine-grained analysis, which not only allows us to re-evaluate existing top-performing factuality metrics, diagnosing their limitations, but also instructs us to further improve current metrics by data augmentation. Based on what we have explored and observed in this work, we suggest following potentially promising future directions:
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(1) Knowledge-guided factuality metric: One error reason in §4.2 is the lacking of extra knowledge reference ability for factuality metrics. It would be promising to explore the effectiveness of external knowledge like knowledge base (Bordes et al., 2013), citation graph (Lo et al., 2020) (for scientific summarization).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(2) Long document Modeling: Lengths of most of summarization documents are over 512, which brings great challenge for pretrain based factuality metrics (R7 in §4.2). Various methodologies (e.g., first retrieval then verification (Zhou et al., 2019)) should be put forwards to deal with the problem.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(3) Fine-grained meta-evaluation and more diverse human judgments: To reliably evaluate factuality metrics, human judgments over diverse distribution are needed. Moreover, fine-grained metaevaluation for metrics is beneficial to further identify their drawbacks and suggest future directions. NLI transferred models We train three NLI transferred models (MNLIBERT, MNLIROBERTA and MNLIELECTRA) on MNLI dataset (Williams et al., 2018) and the samples with neutral label are deleted for fair comparison. Every model is trained on 4 TITAN Xp for 15 epochs. We choose the AdamW as optimizer and set the learning rate to 2e-5. The training batch size for each gpu is 8. The code and the trained checkpoints can be found in our github https://github.com/ zide05/AdvFact. FEQA The trained FEQA in (Durmus et al., 2020) are used in this paper and the checkpoints and codes can be found in https://github.com/ esdurmus/feqa.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Detailed information for baseline and diagnostic datasets. We introduce the basic information for the baseline datasets in Tab. 7. The more detailed statistics for baseline and diagnostic datasets are displayed in Tab. 11. Detailed holistic meta-evaluation Following conclusions can be drawn from the holistic metaevaluation results in Fig. 2: (1) FACTCC has achieved the best performance in most of the test sets except in FaithFact. The reason for this is that the claims in this set are highly paraphrased (novelty of it is 99.2% in Tab. 7) thus will mislead FACTCC which is trained on less abstractive claims (CNNDM-G as shown in Tab. 10).
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(2) FEQA underperforms FACTCC most of the time. In FaithFact, however, FEQA gets higher accuracy. Because the claims in FaithFact are highly paraphrased, thus FEQA tends to label samples as factually inconsistent. On the other hand, the negative samples account for 92% in FaithFact. Thus the tendency of producing negative labels helps to improve the accuracy of FEQA.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
(3) With the same pre-trained model ELECTRA, DAE outperforms MNLIELECTRA in FaccTe and QagsC. However DAE with dependency information doesn't show constant superiority over NLI based model MNLIELECTRA in all evaluation sets. It shows especially worse performance in FaithFact. Opposite to FEQA, DAE averages the factuality scores of all dependency arc triples as the claim-level factuality score, which is biased towards the label of factually correct. Therefore it will obtain lower accuracy in the test set with more negative samples.    Table 9: Quality examination of four diagnostic evaluation sets. "CoLabel" and "CoGrammar" represent the correctness rate of automatically generated labels and grammar.  10: The model types and training data of factuality metrics. NLI-A and NLI-S represent NLI-based metrics defining facts as dependency arcs and span respectively. PARANMT-G and CNNDM-G mean the automatically generated training data from PARANMT (Wieting and Gimpel, 2018) and CNN/DailyMail (Nallapati et al., 2016) 
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Evaluating the quality of generated outputs by Natural Language Generation (NLG) models is a challenging and open problem. Human judgments can directly assess the quality of generated texts (Popović, 2020;Escribe, 2019). However, human evaluation, either with experts or crowdsourcing, is expensive and time-consuming. Therefore, automatic evaluation metrics, which are fast and cheap, are commonly used alternatives for the rapid development of text generation systems (van der Lee et al., 2019). Traditional metrics such as BLEU (Papineni et al., 2002), METEOR (Banerjee and 1 The code of our experiments is available at https://github.com/long21wt/ robust-bert-based-metrics Lavie, 2005), and ROUGE (Lin, 2004) measure n-gram overlap between generated and reference texts. While these metrics are easy to use, they cannot correctly assess generated texts that contain novel words or a rephrasing of the reference text.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Recent metrics like BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), COMET (Rei et al., 2020), BARTScore (Yuan et al., 2021), and BLEURT (Sellam et al., 2020) adapt pretrained contextualized word embeddings to tackle this issue. These novel metrics have shown higher correlations with human judgments on various tasks and datasets (Ma et al., 2019;Mathur et al., 2020). However, the correlations are measured on standard benchmarks containing text domains similar to those used for pretraining the embeddings themselves. As a result, it is unclear how reliable these metrics are on domains and datasets containing words outside the vocabulary of the pretraining data.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
The goal of this paper is to investigate the robustness of embedding-based evaluation metrics on new and noisy domains that contain a higher ratio of unknown tokens compared to standard text domains. 2 We examine the robustness of BERTScore, one of the most popular recent metrics for text generation. 3 In order to perform a systematic evaluation on the robustness of BERTScore with regard to the ratio of unknown tokens, we use characterbased adversarial attacks (Eger and Benz, 2020) that introduce a controlled ratio of new unknown tokens to the input texts. Our contributions are:
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
• We investigate whether the use of characterbased embeddings instead of token-based embeddings improves the robustness of embedding- 2 We connect to recent research that investigates the behavior of metrics in adversarial situations (Sai et al., 2021;Kaster et al., 2021;Leiter et al., 2022;Zeidler et al., 2022). 3 E.g., as of September 2022, BERTScore is cited ∼1200 times while it is ∼200 and 400 for MoverScore and BLEURT, respectively. based generation metrics. Our results show that the evaluations based on character-level embeddings are more robust.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
• We examine the impact of the hidden layer used for computing the embeddings in BERTScore. We show that the choice of hidden layer affects the robustness of the evaluation metric.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
• We show that by using character-level embeddings from the first layer, we achieve the highest robustness, i.e., similar correlation with human evaluations for different ratios of unknown tokens.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
BERTScore (Zhang et al., 2020) computes the pairwise cosine similarity between the reference and hypothesis using contextual embeddings. It forward-passes sentences through a pretrained model, i.e., BERT (Devlin et al., 2019), and extracts the embedding information from a specific hidden layer. To select the best hidden layer, BERTScore uses average Pearson correlation with human scores on WMT16 (Bojar et al., 2016) over five language pairs. For instance, the best layer is the ninth layer for BERT base−uncased .
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
BERTScore with character-level embeddings.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Existing embedding-based metrics, including BERTScore, use token-based embeddings that are taken from pretrained models like BERT (Devlin et al., 2019). In this paper, we investigate the impact of using character-level embeddings instead of token-level embeddings in BERTScore (Zhang et al., 2020). We use ByT5 (Xue et al., 2021), which encodes the input at the byte level. It tokenizes a word into a set of single characters or converts it directly to UTF-8 characters before forwarding the input sequence into the model. Xue et al. (2021) show that ByT5 is more robust to noise compared to word-level embeddings. For computing BERTScore using character-level embeddings, we use ByT5 instead of BERT in BERTScore computations. We adapt three variants of ByT5 (small, base, large) in BERTScore.  guage pairs. Each language pair has 2800 sentences, each corresponding to one reference, plus the systems' output sentences. Totally, the human evaluation in WMT19 has 281k segment sample scores for each of the output translation in to-English language pairs. Table 2 shows the language pairs considered, as well as the number of segments per language pair.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Evaluation on different ratios of unknown tokens. To evaluate the robustness of evaluation metrics on new domains, we use character-level attacks to introduce a controlled ratio of unknown tokens in the corresponding reference texts of the evaluation sets. 4 We examine five different attacks from Eger and Benz (2020): (a) intruders: inserting a character-e.g., '.', '/', ':'-in between characters of a word, (b) disemvoweling: removing vowels-e.g., 'a', 'e', 'i'-from the word, (c) keyboard typos: randomly replacing letters of a word with characters that are nearby the original characters on an English keyboard, (d) phonetic: changing a word's spelling in such a way that its pronunciation stays the same, and (e) visual: replacing characters with a symbol that is its visually nearest neighbor ( (Eger and Benz, 2020;Keller et al., 2021) at perturbation level p = 0.3, i.e., the probability that each letter in a sentence is attacked is 0.3. the ratio of tokens that are affected by the adversarial attack by the perturbation level (p), e.g., p = 0 denotes no attack and p = 0.3 indicates that each letter in the sentence is attacked by the probability of 0.3. Table 3 shows an example of each of these attacks at p = 0.3. Figure 1 shows the average number of unknown tokens, as determined based on BERT's tokenizer, per segment across seven to-English language pairs given different attacks and perturbation levels. We count a token as an unknown token if (1) BERT represents it as [UNK], or (2) BERT splits it into subwords, e.g., 'pre-trained' to 'pre','##train','##ed'. 5 As we see from the figure, the number of unknown tokens increases as we apply these character-level attacks with higher perturbation levels. In our experiments in Section 4, we report the results using visual attacks. The results using other attacks are also reported in Appendix B, and they follow the same patterns as those using the visual attack.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Evaluation on low-resource language pairs. Apart from the experiments on WMT19, we also perform the evaluations on the (Xhosa, Zulu) and 5 Please refer to the detailed algorithm in Appendix A.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
No. unknown tokens bn-hi (Bengali-Hindi) 19.235 hi-bn (Hindi-Bengali) 23.478 xh-zu (Xhosa-Zulu)
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
28.930 zu-xh (Zulu-Xhosa) 28.743 Table 4: The number of average unknown tokens per segment for each language pair in our low-resource datasets.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
(Bengali, Hindi) language pairs from WMT21 (Freitag et al., 2021). BERTScore uses multilingual BERT for evaluating non-English languages. Multilingual models contain a higher ratio of unknown tokens for low-resource languages, and therefore, evaluating the correlation of embedding-based metrics with human judgments on low-resource languages is also an indicator of their robustness. Table 4 shows the number of unknowns tokens per segment to multilingual BERT in four different lowresources language pairs in WMT21 dataset. We refer to the number of segments of low-resources dataset in Table 7 in Appendix C. Table 5 shows the results of BERTScore using different embeddings on WMT19's to-English language pairs (using p = 0). Figure 2 shows the average correlation score over all seven to-English language pairs given different perturbation level from p = 0 to p = 0.3 using the visual attack. We observe that computing BERTScore using the ByT5-small models results in a slightly lower average correlation with human scores over the seven to-English pairs at p = 0 compared to BERTScore using BERT and larger ByT5 models.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
However, the average correlation using ByT5small remains around the same value given different ratio of unknown tokens, indicating higher robustness of the metrics using ByT5-small. On the other hand, while using BERT-large embeddings results in the highest average correlation with human scores in Table 5, its correlation drops considerably in the presence of more unknown tokens in Figure 2.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
For Hindi-Bengali and Zulu-Xhosa, we compare the results against using the BERT-basemultilingual model in Table 6. We observe that the BERTScore metric that uses ByT5-small achieves  Table 5: Segment-level Kendall correlation results for to-English language pairs in WMT19 without any attack, i.e. p = 0. The correlation of BERTScore with human are reported using different embeddings including bert-baseuncased, bert-large-uncased, ByT5-small, ByT5-base, and ByT5-large.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
BERTScore-bert-base-uncased BERTScore-bert-large-uncased BERTScore-byt5-small BERTScore-byt5-base BERTScore-byt5-large  higher correlations with humans throughout. Given that low resources languages contain more outof-vocabulary words for pretrained models, this observation confirms our previous results using character-level attacks on the WTM19 dataset.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Our results in Section 4.1 show the robustness of BERTScore when using the ByT5-small model for computing the embeddings. However, as Table 1 shows, the selected hidden layer for getting embeddings varies when using different pretrained models. For instance, when using ByT5-small embeddings, the model uses the embeddings of the first layer while it uses the embeddings of the 30th layer for ByT5-large. Zhang et al. (2020) show that BERTScore correlation scores with humans drop as they select the last few layers of BERT for getting the embeddings. Therefore, the robust-ness of examined metrics may also depend on their corresponding selected layers for computing embeddings.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
In this section, we evaluate the impact of the selected hidden layer on the robustness of the metric. We evaluate three settings where we use: (a) the embeddings of the first layer for all models, (b) the embeddings of the best layer for each model (cf .  Table 3), and (c) the mean of aggregated embeddings over all layers. We perform the robustness evaluations using the visual attack at p = 0.3. Figure 3 shows the average results of this experiment 6 . We make the following observations. First, using the embeddings of the first layer closes the gap between the correlations of different variations of the ByT5 model, i.e., small, base, and large, in the presence of more unknown tokens, i.e., p = 0.3.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Second, using the embeddings of the first layer improves the robustness of BERTScore using BERT embeddings, i.e., improving the correlation from 0.033 to 0.174 for BERT-base given p = 0.3. However, the correlation of the resulting BERTScore is still considerably lower than using ByT5 embeddings at the presence of more unknown tokens. This indicates that both the choices of the hidden layer as well as the pretrained model play an important role in the robustness of the resulting embedding-based metric. A reason why the first layer may be more effective in our setup is that, in the presence of input noise or unknown tokens, embeddings of higher layers may become less and less meaningful, as the noise may propagate and accumulate along layers. We provide an example from the similarity matrix of the resulting embeddings for different layers in Figure 5 in the Appendix E.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Overall, our results indicate that optimizing the layer on a standard data set such as WMT16 may be suboptimal in terms of the generalization of the resulting metrics. Concerning efficiency of the resulting metrics (a core aspect of modern NLP (Moosavi et al., 2020)), BERT-base has 110 million parameters, while ByT5-small has 300 million parameters. With the default BERTScore setting, passing the input through 9 layers results in a longer inference time. However, using the embeddings of the first layer results in a very fast inference for both models.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Embedding-based evaluation metrics will be used across different tasks and datasets that may contain data from very different domains. However, such metrics are only evaluated on standard datasets that contain similar domains as those used for pretraining embeddings. As a result, it is not clear how reliable the results of such evaluation metrics will be on new domains. In this work, we investigate the robustness of embedding-based metrics in the presence of different ratios of unknown tokens. We show that (a) the results of the BERTScore using BERT-based embeddings is not robust, and its correlation with human evaluations drops significantly as the ratio of unknown tokens increases, and (b) using character-level embeddings from the first layer of ByT5 significantly improves the robustness of BERTScore and results in reliable results given different ratios of unknown tokens. We encourage the community to use this setting for their embeddingbased evaluations, especially when applying the metrics to less standard domains.
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
In future work, we aim to address other aspects of robustness of evaluation metrics beyond an increased amount of unknown tokens as a result of spelling variation, such as how metrics cope with varying factuality (Chen and Eger, 2022) or with fluency and grammatical acceptability issues (Rony et al., 2022). We also plan to investigate the impact of pixel-based representations (Rust et al., 2022) (which are even more lower-level) for enhancing the robustness of evaluation metrics.  contains "##".
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
The results of other attacks are illustrated in Figure 4. Table 7 shows the number of provided human annotations in FLORES. 
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
In Figure 5, we show four different settings and their cosine similarity matrix computed by BERTScore using bert-base-uncased. In both normal reference with 1st or 9th setups, matched tokens get higher similarity score. 9th layer setting gathers information for relevant tokens, which results in higher similarity score across the matrix. As in the case with attacked reference, 1st layer setting penalizes the unmatched tokens and the magnitude of matched tokens are as high as using normal reference with 1st layer setup. However, by using 9th layer for attacked reference, we can observe the hue color of matched tokens with low score. Thus, we conclude the accumulated noise to higher layer cause the problem with effectiveness in our previous setup with WMT19 dataset. 
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
BERTScore-bert-base-uncased BERTScore-bert-large-uncased BERTScore-byt5-small BERTScore-byt5-base BERTScore-byt5-large   Table 9: Segment-level correlation metric results Kendall for seven to-non-English language pairs in WMT19 with respect to fist layer, default layer and mean of aggregated embeddings setting under visual attack at 0.3 perturbation level.