What kind of behavior tests have been created to evaluate NLP models on languages other than English?
The CheckList framework (Ribeiro et al., 2020) enables the generation of behavioral tests for NLP models but its templates are English-centric. English Checklist tests have been extended to other languages via translation (Ruder et al., 2021; K et al., 2022). Such approaches, however, struggle\ with comprehensively covering linguistic features specific to a language and are not able to easily represent morphological variation. Relatedly, Jiang et al. (2020) create templates that integrate morphology for simple knowledge retrieval queries while Kassner et al. (2021) automatically translate knowledge retrieval queries into other languages. Compared to their approach, our framework allows for integrating morphology into a broader range of tests and is more scalable and flexible.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT’s performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.
[{"first":"Nora","last":"Kassner","middle":[]},{"first":"Philipp","last":"Dufter","middle":[]},{"first":"Hinrich","last":"Schütze","middle":[]}]
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
A challenge towards developing NLP systems for the world’s languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically-aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models’ behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.
[{"first":"Ester","last":"Hlavnova","middle":[]},{"first":"Sebastian","last":"Ruder","middle":[]}]
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as"Punta Cana is located in _."However, while knowledge is both written and queried in many languages, studies on LMs' factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for 23 typologically diverse languages. To properly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algorithms to generate multi-token predictions. Extensive experimental results provide insights about how well (or poorly) current state-of-the-art LMs perform at this task in languages with more or fewer available resources. We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages. Benchmark data and code have been released at https://x-factr.github.io.
[{"first":"Zhengbao","last":"Jiang","middle":[]},{"first":"Antonios","last":"Anastasopoulos","middle":[]},{"first":"Jun","last":"Araki","middle":[]},{"first":"Haibo","last":"Ding","middle":[]},{"first":"Graham","last":"Neubig","middle":[]}]
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite and fine-grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models.
[{"first":"Sebastian","last":"Ruder","middle":[]},{"first":"Noah","last":"Constant","middle":[]},{"first":"Jan","last":"Botha","middle":[]},{"first":"Aditya","last":"Siddhant","middle":[]},{"first":"Orhan","last":"Firat","middle":[]},{"first":"Jinlan","last":"Fu","middle":[]},{"first":"Pengfei","last":"Liu","middle":[]},{"first":"Junjie","last":"Hu","middle":[]},{"first":"Dan","last":"Garrette","middle":[]},{"first":"Graham","last":"Neubig","middle":[]},{"first":"Melvin","last":"Johnson","middle":[]}]
Multilingual CheckList: Generation and Evaluation
Multilingual evaluation benchmarks usually contain limited high-resource languages and do not test models for specific linguistic capabilities. CheckList is a template-based evaluation approach that tests models for specific capabilities. The CheckList template creation process requires native speakers, posing a challenge in scaling to hundreds of languages. In this work, we explore multiple approaches to generate Multilingual CheckLists. We device an algorithm - Template Extraction Algorithm (TEA) for automatically extracting target language CheckList templates from machine translated instances of a source language templates. We compare the TEA CheckLists with CheckLists created with different levels of human intervention. We further introduce metrics along the dimensions of cost, diversity, utility, and correctness to compare the CheckLists. We thoroughly analyze different approaches to creating CheckLists in Hindi. Furthermore, we experiment with 9 more different languages. We find that TEA followed by human verification is ideal for scaling Checklist-based evaluation to multiple languages while TEA gives a good estimates of model performance.
[{"first":"K","last":"Karthikeyan","middle":[]},{"first":"Shaily","last":"Bhatt","middle":[]},{"first":"Pankaj","last":"Singh","middle":[]},{"first":"Somak","last":"Aditya","middle":[]},{"first":"Sandipan","last":"Dandapat","middle":[]},{"first":"Sunayana","last":"Sitaram","middle":[]},{"first":"Monojit","last":"Choudhury","middle":[]}]
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Pretrained language models (LMs) (Peters et al., 2018;Howard and Ruder, 2018;Devlin et al., 2019) can be finetuned to a variety of natural language processing (NLP) tasks and generally yield high performance. Increasingly, these models and their generative variants are used to solve tasks by simple text generation, without any finetuning (Brown et al., 2020). This motivated research on how much knowledge is contained in LMs: Petroni et al. (2019) used models pretrained with masked language to answer fill-in-the-blank templates such as "Paris is the capital of [MASK]." * Equal contribution -random order.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Two most frequent predictions en X was created in MASK.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
[Japan (170) Table 1: Language bias when querying (TyQ) mBERT. Top: For an Italian cloze question, Italy is favored as country of origin. Bottom: There is no overlap between the top-ranked predictions, demonstrating the influence of language -even though the facts are the same: the same set of triples is evaluated across languages. Table 3 shows that pooling predictions across languages addresses bias and improves performance. WW = "Wirtschaftswissenschaftler". This research so far has been exclusively on English. In this paper, we focus on using multilingual pretrained LMs as knowledge bases. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT's performance as knowledge base language-independent or does it vary from language to language? To answer these questions, we translate English datasets and analyze mBERT for 53 languages. (iii) A multilingual model is trained on more text, e.g., BERT's training data contains the English Wikipedia, but mBERT is trained on 104 Wikipedias. Can mBERT leverage this fact? Indeed, we show that pooling across languages helps performance.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
In summary our contributions are: i) We automatically create a multilingual version of TREx and GoogleRE covering 53 languages. ii) We use an alternative to fill-in-the-blank querying -ranking entities of the type required by the template (e.g., cities) -and show that it is a better tool to investigate knowledge captured by pretrained LMs. iii) We show that mBERT answers queries across languages with varying performance: it works reasonably for 21 and worse for 32 languages. iv) We give evidence that the query language affects results: a query formulated in Italian is more likely to produce Italian entities (see Table 1). v) Pooling predictions across languages improves performance by large margins and even outperforms monolingual English BERT. Code and data are available online (https://github.com/ norakassner/mlama).
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We follow the LAMA setup introduced by Petroni et al. (2019). More specifically, we use data from TREx (Elsahar et al., 2018) and GoogleRE. 1 Both consist of triples of the form (object, relation, subject). The underlying idea of LAMA is to query knowledge from pretrained LMs using templates without any finetuning: the triple (Paris, capital-of, France) is queried with the template "Paris is the capital of [MASK]." In LAMA, TREx has 34,039 triples across 41 relations, GoogleRE 5528 triples and 3 relations. Templates for each relation have been manually created by Petroni et al. (2019). We call all triples from TREx and GoogleRE together LAMA.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
LAMA has been found to contain many "easyto-guess" triples; e.g., it is easy to guess that a person with an Italian sounding name is born in Italy. LAMA-UHN is a subset of triples that are hard to guess introduced by Poerner et al. (2020).
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We translate both entities and templates. We use Google Translate to translate templates in the form "[X] is the capital of [Y]". After translation, all templates were checked for validity (i.e., whether they contain "[X]", "[Y]" exactly once) and corrected if necessary. In addition, German, Hindi and Japanese templates were checked by native speakers to assess translation quality (see Table 2). To translate the entity names, we used Wikidata and Google knowledge graphs.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
mBERT covers 104 languages. Google Translate covers 77 of these. Wikidata and Google Knowledge Graph do not provide entity translations for all languages and not all entities are contained in the knowledge graphs. For English we can find a total of 37,498 triples which we use from now on. On average, 34% of triples could be translated (macro average over languages). We only consider languages with a coverage above 20%, resulting in the final number of languages we include in our study: 53. The macro average of translated triples in these 53 languages is 43%. Figure 1 gives statistics. We call the translated dataset mLAMA.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We work with mBERT (Devlin et al., 2019), a model pretrained on the 104 largest Wikipedias. We denote mBERT queried in language x as mBERT [x]. As comparison we use the English BERT-Base model and refer to it as BERT. In initial experiments with XLM-R (Conneau et al., 2020) we observed worse performance, similar to Jiang et al. (2020a). Thus, for simplicity we only report results on mBERT.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Petroni et al. (2019) use templates like "Paris is the capital of [MASK]" and give arg max w∈V p(w|t) as answer where V is the vocabulary of the LM and p(w|t) is the (log-)probability that word w gets predicted in the template t. Thus the object of a triple must be contained in the vocabulary of the language model. This has two drawbacks: it reduces the number of triples that can be considered drastically and hinders performance comparisons across LMs with different vocabularies. We refer to this procedure as UnTyQ.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We propose to use typed querying, TyQ: for each relation a candidate set C is created and the prediction becomes arg max c∈C p(c|t). For templates like "[X] was born in [MASK]", we know which entity type to expect, in this case cities. We observed that (English-only) BERT-base predicts city names for MASK whereas mBERT predicts years for the same template. TyQ prevents this.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We choose as C the set of objects across all triples for a single relation. The candidate set could also be obtained from an entity typing system (e.g., (Yaghoobzadeh and Schütze, 2016)), but this is beyond the scope of this paper. Variants of TyQ have been used before (Xiong et al., 2020).
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Assuming that objects are in the vocabulary (Petroni et al., 2019) is a restrictive assumption, even more in the multilingual case as e.g., "Hamburg" is in the mBERT vocabulary, but French "Hambourg" is tokenized to ["Ham", "##bourg"]. We consider multitoken objects by including multiple [MASK] tokens in the templates. For both TyQ and UnTyQ we compute the score that a multitoken object is predicted by taking the average of the log probabilities for its individual tokens.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Given a template t (e.g., "[X] was born in [Y].") let t 1 be the template with one mask token, (i.e., "[X] was born in [MASK].") and t k be the template with k mask tokens (i.e., "[X] was born in [MASK] [MASK] . . . [MASK]."). We denote the log probability that the token w ∈ V is predicted at ith mask token as p(m i = w|t k ), where V is the vocabulary of the LM. To compute p(e|t) for an entity e that is tokenized into l tokens 1 , 2 , . . . , l we simply average the log probabilities across tokens:
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
If k is the maximum number of tokens of any entity e ∈ C gets split into, we consider all templates t 1 , . . . , t k , with C being the candidate set. The prediction is then the word with the highest average log probability across all templates t 1 , . . . , t k . Note that for UnTyQ the space of possible predictions is V × V × · · · × V whereas for TyQ it is the candidate set C.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We compute precision at one for each relation, i.e., 1/|T | t∈T 1{t object = t object } where T is the set of all triples andt object is the object predicted by TyQ or UnTyQ. Note that T is different for each language. Our final measure (p1) is then the precision at one averaged over relations (i.e., macro average). Results for multiple languages are the macro average p1 across languages. 
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We first investigate TyQ and UnTyQ and find that TyQ is better suited for investigating knowledge in LMs. After exploring the translation quality, we use TyQ on mLAMA and observe rather stable performance for 21 and poor performance for 32 languages. When investigating the languages more closely, we find that prediction results highly depend on the language. Finally, we validate our initial hypothesis that mBERT can leverage its multilinguality by pooling predictions: pooling indeed performs better. Figure 2 shows the distribution of p1 scores for single and multitoken objects. As expected, TyQ works better, both for single and multitoken objects. With UnTyQ, performance not only depends on the model's knowledge, but on at least three extraneous factors: (i) Does the model understand the type constraints of the template (e.g., in "X is the capital of Y", Y must be a country)? (ii) How "fluent" a substitution is an object under linguistic constraints (e.g., morphology) that can be viewed as orthogonal to knowledge? Many English templates cannot be translated into a single template in many languages, e.g., "in X" (with X a country) has different translations in French: "à Chypre", "au Mexique", "en Inde". But the LAMA setup requires a single template. By enforcing the type, we reduce the number of errors that are due to surface fluency. (iii) The inadequacy of the original LAMA setup for multitoken answers. Figure 2 (right) shows that the original UnTyQ struggles with multitokens (mean p1 .03 vs. .17 for TyQ).
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Overall, TyQ allows us to focus the evaluation on the core question: what knowledge is contained in LMs? From now on, we report numbers in the TyQ setting.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Manual template tuning or automatic template  mining (Jiang et al., 2020b) has been investigated in the literature to approach the typing problem. We had native speakers check templates for German, Hindi and Japanese, correct mistakes in the automatic translation and paraphrase the template to obtain predictions with the correct type. Table 2 shows that corrections do not yield strong improvements. We conclude that template modifications are not an effective solution for the typing problem.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Contemporaneous work by Jiang et al. (2020a) provides manual translations of LAMA templates for 23 languages respecting grammatical gender and inflection constraints. We evaluate our machine translated templates by comparing performance on a common subset of 14 languages using TyQ querying on the TREx subset. Surprisingly, we find a performance difference of 1 percentage points (0.23 vs. 0.24, p1 averaged over languages) in favor of the machine translated templates. This indicates that the machine translated templates in combination with TyQ exhibit comparable performance but come with the benefit of larger language coverage (53 vs. 23 languages).
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
In mLAMA, not all triples are available in all languages. Thus absolute numbers are not comparable across languages and we adopt a relative performance comparison: we report p1 of a modellanguage combination divided by p1 of mBERT's performance in English (mBERT[en]) on the exact same set of triples and call this rel-p1. A rel-p1 score of 0.5 for mBERT [fi] means that p1 of mBERT on Finnish is half of mBERT[en]'s performance on the same triples. rel-p1 of English BERT is usually greater than 1 as monolingual BERT tends to outperform mBERT [en]. Figure 3 shows that mBERT performs reasonably well for 21 languages, but for 32 languages  rel-p1 is less than 0.6 (i.e., their p1 is 60% of English's p1). We conclude that mBERT does not exhibit a stable performance across languages. The variable performance (from 20% to almost 100% rel-p1) indicates that mBERT has no common representation for, say, "Paris" across languages, i.e., mBERT representations are language-dependent.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
If mBERT captured knowledge independent of language, we should get similar answers across languages for the same relation. However, Table 1 shows that mBERT exhibits language-specific biases; e.g., when queried in Italian, it tends to predict Italy as the country of origin. This effect occurs for several relations: Table 4 in the supplementary presents data for ten relations and four languages.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We investigate pooling of predictions across languages by picking the object predicted by the majority of languages. Table 3 shows that pooled mBERT outperforms mBERT[en] by 6 percentage points on LAMA, presumably in part because the language-specific bias is eliminated. mBERT[pooled] even outperforms BERT by 3 percentage points on LAMA-UHN. This indicates that mBERT can leverage the fact that it is trained on 104 Wikipedias vs. just one and even outperforms the much stronger model BERT.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We presented mLAMA, a dataset to investigate knowledge in language models (LMs) in a multilingual setting covering 53 languages. While our results suggest that correct entities can be retrieved for many languages, there is a clear performance gap between English and, e.g., Japanese and Thai. This suggests that mBERT is not storing entity knowledge in a language-independent way. Experiments investigating language bias confirm this finding. We hope that this paper and the dataset we publish will stimulate research on investigating knowledge in LMs multilingually rather than just in English. Table 4 shows the language bias for 10 relations. For each relation we aggregated the predictions across all triples and show the most common two predicted entities together with its count (in brackets). The querying language clearly affects results. The effect is drastic for relations that ask for a country (e.g., P495 or P1001). P39 yields very different results without exhibiting a clear pattern. Other relations such as P463 or P178 are rather stable. Table 4 and Table 5 show randomly sampled entries from the data.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
We investigate whether performance across languages is correlated with the amount of pretraining data for each language. To this end we investigate the number of articles per language as of January 2021 2 and p1 for TyQ in Figure 6. We do not have access to the original pretraining data of mBERT. Thus, the number of articles we consider in the analysis might be different to the actual data used to train mBERT.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
2 https://meta.wikimedia.org/wiki/List_ of_Wikipedias Figure 4: Three randomly sampled data entries from mLAMA per language. Due to the automatic generation of the dataset not all of them are fully correct. Table 4: Most frequent object predictions (TyQ) in different languages. Some relations exhibit language specific biases. WW = "Wirtschaftswissenschaftler".  
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
In natural language processing (NLP), there is a need to build systems that serve more of the world's approximately 6,900 languages. As one measure of linguistic diversity, the World Atlas of Language Structures (WALS;Haspelmath et al., 2005) records 192 linguistic features along which languages differ. These range from the order of subject, object, and verb (Dryer, 2013) to the number of basic color categories (Kay and Maffi, 2013). Languages present in existing NLP datasets mostly lie in low-density regions of the space of possible typological features (Ponti et al., 2021). In other words, many linguistic features that are common across the world's languages are not observed in languages that are the focus of NLP research. 2 It is thus important to investigate to which linguistic features models can generalize and where they face challenges. However, existing datasets 1 We make all code publicly available at https://github. com/google-research/multi-morph-checklist.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
2 For instance, while tone is present in around 80% of African languages (Adebara and Abdul-Mageed, 2022), few Indo-European languages can be considered tonal. do not allow for a fine-grained cross-lingual evaluation and mainly permit comparisons on a language level (Hu et al., 2020). Prior studies focused on syntax and grammar through the lens of acceptability judgements (Ravfogel et al., 2018;Ahmad et al., 2019;Mueller et al., 2020;Papadimitriou et al., 2022). While these enable the evaluation of what a model deems 'natural' in a given language, it is often unclear how such biases relate to real-world applications of NLP technology.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We propose Multilingual Morphological Checklist (M2C) to enable the investigation of a broader set of cross-lingual differences in practical scenarios. Specifically, we create a morphologicallyaware behavioral testing framework (Ribeiro et al., 2020) that allows for the specification of tests in a diverse set of languages. Using this framework, we design tests that probe model's behavior in light of specific capabilities and typological features in 12 typologically diverse languages. We focus on a question answering setting as it represents one of the most general and widely useful NLP applications (McCann et al., 2018) and enables zero-shot evaluation of models. We create tests that cover a diverse set of reasoning capabilities involving general linguistic features that are expressed differently across languages-negation, numerals, spatial and temporal expressions, and comparatives-as well as features unique to certain languages such as time in Swahili, measure words in Chinese, compounding possessives in Finnish, and motion verbs in Russian. We evaluate state-of-the-art language models on the generated tests in zero-shot and oneshot settings. Our findings shed light on generalization failures to specific typological features. For instance, all models struggle with time expressions in Swahili and measure words in Chinese. We show the workflow of using M2C, from template creation to model evaluation, in Figure 2.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Our contributions are: (1) We create a new morphologically-aware multilingual behavioral testing framework. (2) We highlight linguistic features that are challenging in different languages.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
(3) We design tests that probe model capabilities in light of practically relevant typological differences. (4) We evaluate state-of-the-art language models on the generated tests. (5) We shed light on the challenges posed by typological differences in multilingual scenarios.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
CheckList (Ribeiro et al., 2020) relies on templates to generate a large amount of samples in order to evaluate models' behavior regarding different tasks and capabilities in a controlled manner. A template consists of a string with placeholders such as {first_name} delimited by curly brackets, e.g., "{first_name} is {adj}". The user provides a set of values for each placeholder, for instance, {first_name} = {Michael, John, ... } and {adj} = {busy, friendly, ... }, which are used to populate the templates with their Cartesian product. The generated samples can then be applied to systematically test a model's performance in a specific setting.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Multilingual tests CheckList has been designed for English and provides mainly English-specific functionality. For example, it matches indefinite articles with nouns based on their starting letter, i.e., the placeholder {a:job} generates "a lawyer" and "an engineer". As a consequence, CheckList is not capable of effectively generating tests in languages with richer morphology, which require maintaining agreement between multiple parts of the templatea feature that is beyond the scope of CheckList.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
While multilingual tests can be generated by translating English tests (Ruder et al., 2021;K et al., 2022), optionally including template extraction and human verification, such generated templates struggle with handling rich morphology. In addition, in order to systematically probe linguistic features specific to a language, it is crucial to be able to efficiently generate in-language tests from scratch.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We propose the M2C (Multilingual Morphological Checklist) framework in order to enable the generation of tests in a broad set of languages, including languages with rich morphology. A user provides a template as a string, a list of values for each placeholder, and an optional configuration dictionary in case of duplicate placeholders. The placeholder values can either be passed without inflections (for example, names in English) as a list of strings, or as a list of dictionaries with their corresponding inflected values. Each key of the dictionary is a feature combination (e.g., MASC.PL) and the value is the corresponding string (e.g. "apples"). As such, each entity can have multiple inflections, for instance, in English "apple" and "apples". We show the general M2C workflow in Figure 2.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Morphological categories Our library follows the UniMorph Schema representation (Sylak-Glassman, 2016), which decomposes morphology into 23 dimensions and over 212 features. For example, Gender is one dimension, which contains features such as Feminine (FEM), Masculine (MASC), and Neuter (NEUT).
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
The ability to indicate these dimensions using a clear codification allows us to describe both the value attributes given to placeholders and their dependence on one another. As an example, in order to differentiate between "Juliette est grande" and "Julien est grand" in French, it is necessary to ensure gender agreement between noun and adjective by including the Gender attribute in the template. To cover such functionality, we introduce a syntax describing the morphological dependence between placeholders: {X.<Y.D>} signifies that X should have the same feature for dimension D as Y. In the above example, this is realized by "{first_name} est {adj.<first_name.GENDER>}".
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Language-specific dimensions While initially relying on the UniMorph schema, we found cases where the existing dimensions are not sufficient to describe morphology of placeholders within the templates, which is especially necessary for dealing with exceptions. For instance, the trifold article distinction in Italian masculine gender-il treno, l'hotel, lo studente-depends on whether the noun starts with a consonant, vowel or h, or a specific consonant combination 3 respectively. In order to lexically encode such exceptions, we provide the ability to add dimensions, in this case STARTSWITH, which includes features VOW, CONS, and CONS2. While the goal of M2C is not to be exhaustive, it should enable encoding a sufficient number of dimensions to allow the user to write templates for diverse use cases. 4
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Advanced templating system To cover the variety of morphological phenomena, we designed a templating system with a rich syntax. When describing dependence rules, features can be added sequentially and are commutative, e.g., <first_name.GENDER.NUMBER> is equivalent to <first_name.NUMBER.GENDER> where NUMBER = {singular, plural}. Often, only two or three output values are necessary, which directly depend on a placeholder's feature. We allow a simple expression to be passed directly in the template to make this rule explicit: {val_1:placeholder.feature_1 | ... | val_n:placeholder.feature_n}, e.g., {is:first_name.SG|are:first_name.PL}, which produces "is" for a singular {first_name} and "are" for a plural one. Finally, we allow multiple placeholders with the same type, e.g., {first_name1} and {first_name2}, to be populated by values of a common type, i.e., first_name. In the case of multiple placeholders, we can provide a configuration for each placeholder type that specifies boolean repetition and order fields to, for instance, avoid having examples like "John and John" (repetition) or "John and Mary" and "Mary and John" (order).
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Manual enumeration of features and their corresponding values is a barrier to scaling. To circumvent this, we integrate UnimorphInflect (Anastasopoulos and Neubig, 2019), which uses mod-els trained on Unimorph data using the Unimorph Schema to generate inflections in 55 languages. As Unimorph models are imperfect-test accuracies range from 90%+ in many languages to 23% in Arabic-we envision a workflow where inflections are generated at scale using UnimorphInflect and then manually inspected by annotators for correctness. We expect the increase in productivity, and thus reduction in cost, to be significant by leveraging semi-automated as opposed to manual generation for languages with good performance. 5
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Answer validation Most prior benchmarks for behavioral testing of language models have focused on classification tasks (Talmor et al., 2020;Ribeiro et al., 2020). As M2C aims to support the evaluation of generative models using arbitrary templates, we implement functionality to match a range of outputs for each template, based on morphology, string matching and regex. 6
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Summary Overall, the M2C framework enables the systematic and controlled generation of highquality tests at scale in a broad set of languages. As such, it occupies a middle ground between libraries such as SimpleNLG (Gatt and Reiter, 2009) that generate high-quality data but require encoding each language-specific rule, and template expansion via generative language models (Honovich et al., 2022), which are highly scalable but less reliable and underperform on languages with limited data (Hu et al., 2020). M2C enables modular design by allowing the addition of user-specified dimensions and features for specific templates and languages without requiring to encode all possible rules of a language. Furthermore, an advanced templating syntax and the semi-automatic generation of inflections may improve user productivity.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Languages We generate tests targeting capabilities and typological features in 12 typologically diverse languages: English (EN), Spanish (ES), Italian (IT), French (FR), German (DE), Swedish (SV), Finnish (FI), Slovak (SK), Russian (RU), Swahili (SW), Mandarin Chinese (ZH), and Arabic (AR).
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Recent models have excelled at a wide range of tasks in English requiring a diverse set of reasoning  and understanding capabilities (Wang et al., 2019;Hendrycks et al., 2021). As most languages are morphologically richer than English, they encode the linguistic features representing such capabilities in more complex ways. The features we investigate are relevant in a variety of real-world applications including sentiment analysis (Wiegand et al., 2010) We investigate capabilities and linguistic features present in all our investigated languages as well as linguistic features unique to certain languages. For each feature, we highlight differences in its cross-lingual instantiation and challenges for natural language understanding and generation. We create templates using the M2C framework to test a model's understanding of each capability and feature. We show a subset in Table 1.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Negation In Indo-European languages, negation is often expressed via a separate particle such as not (English), inte (Swedish), etc. In contrast, in Swahili, for instance, negation morphemes are fused with the verb root and thus harder to identify. For other negation terms such as kein (German) models need to produce the correct agreement when generating text. In addition to gender and number agreement with the subject, Ara-bic negation takes up to five forms in singular, three forms in dual, and five forms in plural, e.g., (SG.MASC) and (SG.FEM).
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Numerals Models must be able to recognize and reason with numbers in their spelled-out and numerical forms across different writing and numeral systems, e.g., seventeen (English) and 17 (Western Arabic numerals) and and (Eastern Arabic numerals). For generation in Russian and Slovak, models must inflect the noun depending on the quantity of the object. Slovak, for instance, has separate inflections for quantities of one, two/three-/four, and five and more, which also vary based on the object's animacy.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Spatial expressions In Russian, prepositions are associated with different cases, for example the instrumental case for за (behind) and the prepositional case for on. Such case agreement needs to be taken into account when generating text in Russian. Finnish, in addition to prepositions, follows a system of postpositions, which relate the location of one thing to another and require objects to be inflected in either partitive or genitive case.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Temporal expressions Some languages with rich morphology such as Finnish and Swahili encode temporal expressions in less complex ways than their inflection-sparser counterparts. In Swahili, verbal structure follows a simple compounding schema of subject marker + tense marker Prompt: Svara på frågan.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Kontext: Pennan är under stolen och telefonen är på fönstret. Fråga: Var är telefonen? Svar: På fönstret Kontext: Boken är under soffan och pennan är på hyllan. Fråga: Var är pennan? Svar: Table 2: Zero-shot and few-shot prompt example in Swedish spatial template. The zero-shot prompt only includes the information in bold while the one-shot prompt also includes the additional exemplar. + verb, e.g. a-na-soma (he reads) or u-ta-soma (you will read).
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Comparatives Commonly, comparatives are expressed by a suffix or using a quantifier, e.g., more/less. Spanish and French follow the latter approach by placing más/menos and plus/moins before the adjective with only a few standard exceptions. On the other hand, in Finnish, for example, the formation of comparatives follows a complex system of rules for compounding that includes categories depending on the endings of adjectives and a suffix mpi.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Time in Swahili In many languages, the day is divided into two periods: a.m. and p.m., with the daily cycle starting at midnight (0:00) and running through noon (12:00). In Swahili, time is based on sunset and sunrise, defined to be 6 pm and 6 am respectively in standard time. For example, 11.30 am in standard time is 5.30 in the morning in Swahili time. Understanding different time systems is key not only for in-language reasoning but also for cross-lingual applications.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Possessives in Finnish Compounding in Finnish along with its system of 15 cases is one of the most challenging aspects of the language. One relevant feature are the possessive suffixes, which attach to the stem of nouns, e.g., koulu (school) becomes kouluni (my school) and koulumme (our school). Possession is expressed via a suffix -lla, which compounds with other suffixes, e.g., siskollani (my sister has), which must be correctly inflected by models in order to achieve the intended meaning.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Another language specific-feature are measure words in Mandarin Chinese, which include over 150 cases and are used for different types of objects depending on their characteristics, e.g., "本" for books, "双" for pairs, or "辆" for vehicles.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
In most Slavic languages, motion verbs are a challenging concept as they behave differently than other verb categories. While most verbs have two forms (imperfective and perfective), motion verbs have three forms: one perfective form and two imperfective forms. Of the imperfective forms, the definite form indicates unidirectional or current one-time motion while the indefinite form represents multi-directional or habitual motion.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Experimental setting We evaluate models on the generated tests in a question answering setting as can be seen in Figure 2. Each test consists of a context, a question, and an answer that needs to be predicted by the model. For each template, we generate 2,000 test examples on which the model is evaluated. A model's performance on a template is its accuracy of predicting a valid answer for a test averaged across all tests of the template.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We evaluate models in both zero-shot and oneshot settings for each capability and language. In the one-shot setting, a test randomly generated using the same template is used as the exemplar. This simplifies the task in two ways: i) it provides the model with a clear format for generating the answer and may enable the model to infer the answer's relationship to the rest of the template. While we conduct one-shot experiments to show the impact of additional instructions, zero-shot evaluation is the only setting that fully tests the model's understanding and generative capabilities independent of confounders such as the exemplar choice (Zhao et al., 2021), in line with prior work on behavioral testing (Ribeiro et al., 2020;Efrat et al., 2022). We provide an example of both settings in Table 2.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Models We evaluate five state-of-the-art pretrained language models of different sizes: an LMadapted version (Vu et al., 2022)    been trained on large amounts of web text but have not been otherwise fine-tuned for instructionfollowing or few-shot learning.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Generation Predictions are generated using greedy decoding with a temperature of 0 and a maximum of 20 decoding steps.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We show the average results across tests covering language-agnostic features across languages and models in Table 3. We present the detailed results across test types for mT5-XXL and PaLM 2 in Table 4 and for PaLM-S, PaLM-M, and PaLM-L in Appendix A. We show results on language-specific features for all models in Table 5.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
M2C tests are challenging, particularly for smaller models and for certain languages.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
mT5-XXL and PaLM-S achieve comparatively poor performance on average across languages. While performance is highest for English, across the other languages both models only pass at most 50% of tests-and less than a third for Slovak (SK), Swahili (SW), and Arabic (AR) for PaLM-S. These results highlight that the tests generated with M2C are challenging for the majority of state-of-the-art models and demonstrate that a clear gap between performance on English and performance in other languages remains for most models.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Competence with language-agnostic features emerges at scale. We observe a 20 point improvement in average performance from PaLM-S to PaLM-M to PaLM-L, highlighting that model robustness to linguistic features improves with scale. The strongest model, PaLM 2, reaches almost perfect performance on English and on the Indo-European languages. Compared to PaLM-L, PaLM 2 achieves the largest improvements on Slovak, Russian, Swahili, and Arabic. On Finnish, Slovak, Chinese, and Swahili average performance of PaLm 2 is still below 90%, however, indicating that there is headroom left in terms of competence with regard to language-agnostic features for even the strongest current models.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Language-agnostic features The most challenging test types for mT5-XXL and PaLM 2 in Table  4 are numerals and comparatives. mT5 performs poorly on addition and only slightly better on subtraction while PaLM 2 achieves around 90% performance on most languages. On comparatives, both models have more difficulty in the conditional case. While PaLM 2 passes negation tests with almost perfect accuracy across different languages, mT5 displays reduced performance, particularly when the question is negated and for non-Indo-European languages. This highlights that robust reasoning with negation only emerges at scale. On spatial and temporal tests, mT5 achieves reasonable performance in most languages, while PaLM 2 achieves perfect performance in most cases and only underperforms in Swahili.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Language-specific features We show the results on the language-specific feature tests in Table 5. All models have acquired a reasonable ability to distinguish between different forms of motion verbs in Russian. Small and medium-sized models generally fail to reason with compounding possessives in Finnish and time expressions in Swahili while all models are unable to perfectly employ the correct measure words in Chinese, despite it being a high-resource language. Similarly, even PaLM 2 is unable to correctly reason with time expressions in Swahili. We show examples of errors in model predictions for each test type together with English glosses in Appendix B.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
The generated tests focus on evaluating a model's understanding capabilities with regard to specific capabilities and linguistic features. As the linguistic features are often expressed via morphology, we additionally calculate the fraction of errors due to morphology in the models' output for the tests with morphological variation in the answer. This enables us to assess a model's ability to generate morphologically correct forms. For instance, in   Slovak, a model must generate the correct accents and suffixes, e.g., it is an error if the model predicts the Trináste (13th) instead of Trinást' (13). We automatically identify and manually curate these errors for PaLM-L and report the proportion of morphology-related errors for a subset of tests and languages in Table 6. We show examples of errors in model predictions that are due to morphology in Appendix C. For certain tests with morphological variation in the answer, a non-negligible fraction of errors are due to producing incorrect morphological forms. For negation in Slovak, around half of PaLM-L's errors are due to morphology such as an incorrect use of diacritics or suffixes, highlighting a weakness of subword-based models. For numerical reasoning, models frequently produce incorrectly inflected numerals. Similarly, models generate outputs with an incorrect case or number for tests related to spatial and temporal expressions and comparatives.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We show one-shot results for all models in Appendix D. The one-shot setting generally improves results as it allows the model to infer the format of the answer and potentially its relationship to the rest of the template. Improvements are larger for smaller models, which benefit more from information about the template. Nevertheless, even in this setting models are unable to achieve perfect accuracy across all languages. Reasoning with numerals and comparatives are still challenging for most models while improvements on numerals are also relatively smaller than on other test types. Models struggle particularly in Swahili across different test types. Overall, these results demonstrate that even in one-shot settings, large language models are not able to systematically generalize to certain typological features in multilingual settings.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
In this paper, we have introduced M2C, a multilingual morphological framework for targeted behavioral evaluation of language-specific capabilities. As world languages present different challenges, M2C aims to provide flexibility in defining a suitable templating system with its individual dimensions and features. We have conducted experiments on state-of-the-art large language models, highlighted typological features that models struggle with, and quantified errors occurring due to morphology. We hope M2C inspires further research focused on tackling typological and morphological challenges with large language models.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Accessibility Our new behavioral testing framework enables the generation of tests that incorporate morphology, which makes the systematic and fine-grained evaluation of NLP models more accessible across a diverse set of languages. For many such languages, it was previously not feasible to gain a fine-grained understanding of a model's capabilities.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Risks Risks are limited and mainly relate to obtaining a biased view of a capability due to the use of limited templates.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Limitations The creation of templates still requires native speaker expertise and an understanding of a language's grammar. Morphological inflection models are imperfect so morphological forms may need to be enumerated to ensure highquality tests. We leave model-in-the-loop template creation and improving morphological inflection models for future work. While we design representative templates with thousands of permutations for each capability, a larger set of templates and arguments may be necessary to ensure a comprehensive coverage. 
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We show zero-shot results for PaLM-S, PaLM-M, and PaLM-L across different tests and languages in Table 7.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We show examples of errors on language-specific feature tests with PaLM-L together with English glosses in Table 8.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We show example errors in predictions of PaLM-L that are due to morphology in Table 9.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We show one-shot results for all models in Table  10. We show summary statistics of the average relative change in performance of the one-shot setting compared to the zero-shot setting for each language and model in      In the Broader Impact Statement.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
A2. Did you discuss any potential risks of your work?
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
In the Broader Impact Statement.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
A3. Do the abstract and introduction summarize the paper's main claims?
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Section 1.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
A4. Have you used AI writing assistants when working on this paper?
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Left blank.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
B Did you use or create scientific artifacts?
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Section 4. A new behavioral testing library.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Section 4.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
The library will be released under a Creative Commons license.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
We describe how the library should be used in Section 4 and Figure 1.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No created data contains personally identifying information.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Sections 4 and 5.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in Section 6.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Section 6.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 6.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Language models (LMs; (Church, 1988;Kneser and Ney, 1995;Bengio et al., 2003)) learn to model the probability distribution of text, and in doing so capture information about various aspects of the syntax or semantics of the language at hand. Recent works have presented intriguing results demonstrating that modern large-scale LMs also capture a significant amount of factual knowledge (Petroni et al., 2019;Jiang et al., 2020;Poerner et al., 2019). This knowledge is generally probed by having the LM fill in the blanks of cloze-style prompts such as * : Work done at Carnegie Mellon University. The first two authors contributed equally. en fr nl ru es jp vi zh hu ko tr he Figure 1: X-FACTR contains 23 languages, for which the data availability varies dramatically. Prompts get instantiated to produce grammatical sentences with different numbers of mask tokens and are used to obtain predictions for [Y]. In this Spanish example, the verb gerund "fundar" to found is rendered as "fundada" to agree in gender and number with the subject "Bloomberg L.P.". The final prediction is in bold.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
"Obama is a _ by profession.", where these prompts are invariably written in English. However, it goes without saying that there are many languages of the world other than English, and it is quite conceivable that (1) users may want to query this factual knowledge in other languages, and (2) some facts will be written in non-English languages and thus multilingually trained LMs (hereinafter, M-LMs) may be more equipped to recall these facts in the languages of the original data. In this paper, we study the intersection of multilinguality and the factual knowledge included in LMs.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We create a new multilingual benchmark for probing factual knowledge in LMs -the Crosslingual FACTual Retrieval benchmark (X-FACTR). X-FACTR shares a similar formulation as the LAMA benchmark of Petroni et al. (2019), which assesses whether LMs have memorized a fact (i.e., a subject-relation-object triple) by having LMs predict the blank (i.e. object) in a cloze-style prompt for each relation after filling in the subject. We manually create such prompts for 23 languages spanning different language families and different levels of data availability ( § 3.1). Because many languages that we handle are morphologically rich, we design a morphology-sensitive annotation schema (see example in Fig. 1) that can properly instantiate prompts using entity metadata (e.g. gender) and a morphological inflection model ( § 3.3).
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
In addition, while previous works (Petroni et al., 2019;Jiang et al., 2020;Poerner et al., 2019) have limited examination to single-token entities (e.g. "France"), we expand our setting to include multi-token entities (e.g. "United States"), which comprise more than 75% of facts included in our underlying database (Wikidata; § 3.2). We propose several decoding algorithms for prediction of these multi-token entities using masked LMs ( § 4). We discuss the related work in depth in § 7.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We perform experiments on X-FACTR ( § 5), comparing and contrasting across languages and LMs to answer the following research questions:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
(1) How and why does performance vary across different languages and models? (2) Can multilingual pre-training increase the amount of factual knowledge in LMs over monolingual pre-training? (3) How much does knowledge captured in different languages overlap? We find that the factual knowledge retrieval of M-LMs in high-resource languages is easier than in low-resource languages, but the overall performance is relatively low, indicating that this is a challenging task. We analyze the types of failure cases, shedding light on future directions to improve factual knowledge in M-LMs. In addition, multilingual pre-training does not necessarily lead to a higher recall of facts compared to language-specific monolingual pre-training. The knowledge memorized by M-LMs in fact is largely distinct across languages, with almost 50% of facts being recalled in only one language. Inspired by the above observations, we propose a code-switching-based objective function to improve the ability of M-LMs to access knowledge using queries from a variety of languages. We replace entities in a sentence from the original language with counterparts in another lan-guage, and further fine-tune the LM on these codeswitched data ( § 6). We perform experiments on three languages (French, Russian, and Greek, codeswitched with English). Results demonstrate that this code-switching-based learning can successfully improve the knowledge retrieval ability with low-resource language prompts.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
In this paper we follow the protocol of Petroni et al. (2019)'s English-language LAMA benchmark, which targets factual knowledge expressed in the form of subject-relation-object triples from Wikidata 1 curated in the T-REx dataset (ElSahar et al., 2018). The cloze-style prompts used therein are manually created and consist of a sequence of tokens, where [X] and [Y] are placeholders for subjects and objects (e.g. "[X] is a [Y] by profession."). To assess the existence of a certain fact, [X] is replaced with the actual subject (e.g. "Obama is a mask by profession.") and the model predicts the object in the blankŷ i = argmax y i p(y i |s i:i ), where s i:i is the sentence with the i-th token masked out. Finally, the predicted fact is compared to the ground truth. In the next section, we extend this setting to more languages and predict multiple tokens instead of a single one.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Retrieval Benchmark
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
In sampling the languages to create our multilingual benchmark, we attempted to create a subset as diverse as possible with regards to data availability, typology, and script -within the constraints of requiring inclusion in Wikidata and standard pretrained M-LMs. To this end, we created prompts in 23 languages: English, French, Dutch, Spanish, Russian, Japanese, Chinese, Hungarian, Hebrew, Turkish, Korean, Vietnamese, Greek, Cebuano, Marathi, Bengali, Waray, Tagalog, Swahili, Punjabi, Malagasy, Yoruba, and Ilokano. Our subset includes languages from 11 families (the Indo-European ones include members of the Germanic, Romance, Greek, Slavic, and Indic genera), using 10 different scripts. Our languages display high variance with respect to Wikipedia presence, a proxy for overall data availability, ranging from very large to very small (see Fig. 1 13.0 9.4 8.2 7.9 7.3 7.1 6.8 5.5 4.9 4.6 4.1 #single-token 18.9 13.9 12.8 13.5 3.4 1.3 0.2 6.2 1.1 2.5 2.0 3.9 0.7 0.1 3.3 0.2 3.0 3.2 2.8 0.1 1.7 0.9 2.1 #multi-token 26.8 26.4 25.5 23.6 22.9 23.8 22.9 14.2 16.0 13.6 14.1 9.7 12.3 9.3 4.9 7.7 4.4 3.9 4.0 5.4 3.2 3.7 2.0 Table 1: X-FACTR benchmark statistics (in thousands). More details in the Appendix (Tab. 5 and Fig. 6).
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
While Petroni et al. (2019) and follow-up works focus on entities that can be represented by a single token, since many popular entities consist of multiple tokens (e.g. "United States"), we argue that it is crucial to include multi-token entities in the benchmark to make the evaluation unbiased. Similar to Petroni et al. (2019), we use the T-REx dataset to collect facts for our benchmark. Since T-REx aligns facts from Wikidata with sentences in abstract sections from DBpedia, we can estimate the commonality of each fact based on its frequency of being grounded to a sentence in these abstracts.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
For each of the 46 relations in T-REx, we sample 1000 subject-object pairs with probability proportional to their frequency. Frequency-proportional sampling makes the distribution of the facts in our benchmark close to real usage and covers facts of different popularity. To keep the benchmark unbiased, we did not constrain the facts with any language-related criteria (e.g., require the entities to have translations in all languages we considered). As a result, some entities (either subjects or objects) might not have translations in all languages. The number of facts in different languages in our multilingual multi-token X-FACTR benchmark is shown in Tab. 1. Because many modern pre-trained M-LMs almost invariably use some variety of subword tokenization, the number of tokens an entity contains will depend on the tokenization method used in the LM. We report the statistics based on the WordPiece tokenization used in multilingual BERT (Devlin et al., 2019). The tokenization scheme statistics for the other M-LMs are similar.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Some languages we include in the benchmark require additional handling of the prompts to account for their grammar or morphology. For example, (some) named entities inflect for case in languages like Greek, Russian, Hebrew, or Marathi. In some languages syntactic subjects and objects need to be in particular cases. Similarly, languages often require that the verb or other parts of the sentence agree with the subject or the object on some morphological features like person, gender, or number.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Our prompts provide the necessary information in order to generate grammatical sentences, given the gender and number of the entities. For example, the Russian prompt for " [X] was born in [Y]" is:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
The prompt denotes that the subject ([X]) needs to be in the nominative (Nom) case and the object ([Y]) needs to be inflected in the essive case (Ess). The prompt also accounts for the variation of the gender of [X] providing options (separated by |) for the subject being masculine, feminine, or neuter (MASC, FEM, NEUT respectively). Everything within square brackets gets concretely instantiated given the subject and object. Grammatical gender is assigned through a combination of Wikidata information and languagespecific heuristics, constructed based on feedback from native speakers of each language. When the entity corresponds to a person, we retrieve their "sex_or_gender" properties from Wikidata. In addition, for languages like Greek or French, the gender of an entity can be inferred with fairly high certainty given the form of the word (e.g. looking at the ending). Last, some categories of entities (such as cities, countries, organizations, etc, which can be obtained using the "instance_of" Wikidata property) often get assigned a general grammatical case based on the category.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Once all the morphological features have been specified as detailed above, we use the unimorph_inflect package (Anastasopoulos and Neubig, 2019) to generate the appropriately inflected surface form of the bracketed words. 3 We note that the target entity ([Y]) might also need to be inflected, as in the above Russian example, in which case we require the model's predictions to match the inflected target forms.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
To verify the quality of the prompts we performed user studies with native speakers, finding that 88% on average were judged as natural and grammatically correct. Details are shown in Appendix B, but it is worth noting that the majority of errors are due to prompts being awkward or incorrect for some senses captured by the relation, and not due to our gender heuristics or automatic inflection. This issue is also present in the LAMA English prompts (Jiang et al., 2020).
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
As noted in Petroni et al. (2019), because some subject-relation pairs might have multiple correct objects (e.g., America maintains diplomatic relations with multiple countries), we collect all valid objects and judge a prediction as correct if it can match any object (e.g., both France and Canada are correct). Since an entity might have multiple aliases (e.g., "America" and "the US"), we collect all aliases for each entity from Wikidata, and the prediction is marked as correct if it can match any one of them after lowercasing.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
As Tab. 1 shows, many facts involve multi-token entities and thus a LM would need to predict these entities in multiple steps. Generating multiple predictions is straightforward for traditional left-toright LMs (Sundermeyer et al., 2015;Radford et al., 2019), where we can autoregressively decode the next token conditioned on previous tokens. However, many pre-trained LMs such as BERT (Devlin et al., 2019) are masked LMs that predict individual words given left and right contexts, and decoding from such masked LMs remains an open problem (Lawrence et al., 2019;Salazar et al., 2020;Ghazvininejad et al., 2019;Wang and Cho, 2019;Cho, 2019). We systematically examined different multi-token decoding algorithms from three orthogonal perspectives: (1) how the initial predictions are produced, (2) how to refine the predictions, and (3) other commonly used components in neural text generation systems. We assume that the following conditional probability distribution is defined by the masked LM for a sentence with n tokens:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
where the subscript of mask indicates its position, and the surrounding token x · can either be an actual word x · or mask . We aim to handle sentences containing multiple mask tokens conditioning on the surrounding actual words:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Barack Obama is a by profession United 1 of 1 president 1 (a) Independent:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Barack Obama is a by profession United 1 State 2 President 3 Barack Obama is a by profession minister 2 of 3 cabinet 1 Figure 2: Illustration of three initial prediction and refinement methods. Green boxes are mask tokens to be filled, and subscripts indicate the prediction order.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Given a sentence with multiple mask tokens, e.g., Eq. 2, we can either generate outputs in parallel independently or one at a time conditioned on the previously generated tokens. These methods are similar to the prediction problems that BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019b) perform in their pre-training stages respectively. We define c ∈ R n as the probability of each prediction, with details varying by prediction methods.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
After all mask tokens are replaced with the initial predictions, i.e.,ŝ i:j = x 1 , ...,ŷ i , ...,ŷ j , ..., x n , we can further refine the predictions by iteratively modifying one token at a time until convergence or until the maximum number of iterations is reached. Here we outline the algorithms with high-level descriptions, and provide concrete details in Appendix C. Independent. For independent initial prediction ( Fig. 2a), the mask tokens are all predicted in parallel (at once). We also consider two autoregressive methods for initial prediction or refinement. Order-based. Mask tokens are predicted from left to right, in each step conditioning also on the previously generated tokens (Fig. 2b). In the refinement stage, we modify predictions also from left to right, and convergence is reached when there are no changes in a left-to-right scan. Confidence-based. In each step, we choose the prediction with the highest probability, so the order of predictions can be arbitrary (Fig. 2c). In the refinement stage, we choose from all predicted tokens the one with the lowest confidence (i.e., the lowest probability) and re-predict it similarly to Ghazvininejad et al. (2019). Convergence is reached when the re-predicted token is the same as the original token.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Because we do not know the number of tokens of the ground truth in advance, we enumerate from 1 to M mask tokens and choose the final prediction based on the confidence. Given the prompt in Eq. 2, the simplest way to compute the confidence is pseudo log likelihood, which is the sum be easily adapted to non-consecutive cases. of log probabilities of each predicted token conditioned on the other tokens (Salazar et al., 2020): v(j − i + 1) = j k=i log c k , where c k is the confidence (probability) of the k-th predicted token, and v(m) is the overall prediction confidence with m initial mask tokens. Among M predictions, we choose the one with the highest confidence.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We also investigate additional components commonly used in neural generation systems. Specifically, we consider length normalization in computing the final confidence (i.e., divide v(m) by the number of mask tokens m) because a simple sum might favor short predictions. In addition, the confidence value c in previous methods contains probabilities when the predictions are first generated, which will become stale once the surrounding tokens change (Ghazvininejad et al., 2019). We consider re-computing confidence c whenever a change happens. Last, we attempted beam search to keep track of the most plausible B predictions at each step. Details of these components can be found in Appendix C, along with a general schema of the overall decoding algorithm in Alg. 1.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Implementation Details. We use the implementations of different multilingual/monolingual pretrained LMs in the Transformers library (Wolf et al., 2019). We examine 3 multilingual pre-trained LMs, M-BERT, XLM, XLM-R (Devlin et al., 2019;Conneau and Lample, 2019;, 5 and 8 monolingual pre-trained LMs, BERT (en), CamemBERT (fr), BERTje (nl), BETO (es), Ru-BERT (ru), Chinese BERT (zh), BERTurk (tr), and GreekBERT (el) (Martin et al., 2020;de Vries et al., 2019;Cañete et al., 2020;Kuratov and Arkhipov, 2019;Schweter, 2020). Details of these models can be found in Appendix D.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We set the maximal number of mask tokens to M = 5 for English, French, Dutch, and Spanish. In these languages more than 90% of the entities are split into ≤5 tokens. For all other languages we use M = 10. This is expected because the vocabulary of M-LMs based on WordPiece tokenization is dominated by frequent words and low-resourcelanguage words tend to split into more pieces (Ács, 2019). We set the maximal number of iterations to T = 2M , so that we can approximately refine all the predicted tokens once for a sentence with 5 Yoruba is not in the training data of XLM and XLM-R.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
M mask tokens (the initial prediction takes exactly M iterations). In our main results, we report results with two decoding algorithms: the simplest independent generation method and the confidencebased method for both initial and refinement predictions. The latter performs better than order-based methods, as we will show in Tab. 3. To save computation time, we only use confidence re-computation for M = 5. We discuss computation complexity in Appendix C.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Evaluation Metrics. We follow Petroni et al. (2019), computing the accuracy of predicted objects for each relation and macro-average them as final scores. For fine-grained analysis of different decoding methods, pre-trained LMs, and languages, we report results on all facts as well as on subsets consisting only of single-token objects (single) and multi-token objects (denoted as multi).
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We run both the independent and confidence-based decoding methods with 3 M-LMs, and when available 8 monolingual LMs, across 23 languages, 6 with results shown in Fig. 3. Overall, even in the most favorable settings, the performance of state-of-that-art M-LMs at retrieving factual knowledge in the X-FACTR benchmark is relatively low, achieving less than 15% on high-resource languages (e.g., English and Spanish) and less than 5% for some low-resource languages (e.g., Marathi and Yoruba). This may initially come as a surprise, given the favorable performance reported in previous papers (Petroni et al., 2019;Jiang et al., 2020), which achieved accuracies over 30% on English. We justify this discrepancy in our following analysis. We note that, although we provide baseline results in almost all languages, we perform our extensive analysis on a representative subset, consisting of 13 languages.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Performance on Different Languages. Performance on high-resource languages is usually better than performance on middle-or low-resource languages regardless of the (M-)LMs. This is probably due to high-resource languages having more data in the pre-training stage. It is also possible that even if the fact of low-resource languages is written in the available data for these languages, it is not appropriately memorized due to lack of model capacity or forgetting (Kirkpatrick et al., 2017 Figure 3: Accuracy on different languages using different LMs (%). Independent prediction (solid bars) outperforms confidence-based prediction (no-fill bars) on high-resource languages but not on low-resource languages. Different models are color-coded, with missing/unsupported models marked with ×. Languages are ranked by the total number of facts in our benchmark. Details in Appendix Tab. 10.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
is worth noting that the best results are in Indo-European languages which not only have the most data, but also share the same (Latin) script which could further facilitate cross-lingual learning.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Performance of Different LMs. Comparing the performance of different M-LMs, we found that M-BERT outperforms XLM and XLM-R on highresource languages, while on low-resource languages performance is similar. This is contradictory to the conclusion on other cross-lingual tasks, such as natural language inference and syntactic prediction, as reported in Hu et al. (2020). Our conjecture is that because factual knowledge probing requires retrieving the identity and relations of individual entities, it is more fine-grained than more coarse-grained understanding of syntactic and semantic classes that are required to solve other tasks. We posit that pre-training methods that show superior performance on inference and syntactic prediction tasks (i.e., XLM-R) might achieve good syntactic/semantic abstraction at the cost of making less concrete lexical distinctions.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Comparing M-BERT with language-specific LMs, we find M-BERT outperforms the monolingual BERT on Dutch, Spanish, and Greek, while underperforming on English, Russian, Chinese, and Turkish. Since most of the LMs follow the architecture and pre-training settings of BERT (Devlin et al., 2019) or RoBERTa , we hypothesize that training corpus is the major contributor to the final performance, and summarize those corpora in Tab. 8 in the Appendix. Another potential explanation is that model capacity limita-en fr nl es ru zh he tr ko vi el mr yo Single-token vs Multi-token. Since we choose among M candidate predictions with different numbers of mask tokens based on confidence, it is possible that the prediction with the correct number of mask tokens has lower confidence than the other predictions. To investigate the errors introduced by this step, we conduct an ablation experiment that assumes we know the ground-truth number of mask tokens. As shown in Fig. 4, performance improves significantly by 75% on average across all languages using the oracle mask number, indicating that pre-trained LMs have difficulties in choosing the correct number of mask tokens. The performance on single-token facts (i.e., the setting of previous works that only predicts a single token) is even higher, demonstrating the difficulty of multi-token prediction. 7  The most prominent error type, about one-fourth of mistakes for all LMs, was repeating subjects, whereby the prediction repeats either the full or partial subject. Predicting the wrong entities is also fairly common, especially in Spanish (29%). Interestingly, we find that wrong predictions are often a language-specific "common" entity such as 'Αθήνα' (Athens, the capital of Greece) in Greek location prompts, while the Spanish model insisted most musicians play 'flauta' (flute). Another error type, particularly common in Greek (27%), is producing non-informative output, where the predictions are function words that could never be an entity. Type errors when the semantic type of the prediction is different than expected (e.g. predicting dates instead of locations) are fairly common (English: 8%, Spanish 6%), as are related concepts predictions (English: 7%), where the model predicts relevant, possibly factually correct entities (e.g. predicting a country or a state instead of a city). Worryingly, in a fair amount of cases (English: 5%, Spanish: 8%, Greek: 11%) the models output non-existent words (unk). Errors of the last 4 types could potentially be avoided by limiting the allowed outputs of the model to specific entity classes; we leave this for future work. Last, we identified around 3% of false negatives, where the prediction is actually correct but is not part of our aliases list and less than 1% of inflection errors where the prediction is the correct entity but improperly inflected.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Overall, the confidence-based decoding method improves the accuracy in middle-and low-resource languages, while it hurts the performance on high-  resource languages. To better understand the effect of different components on the final performance, we conduct a comprehensive comparison on English and Chinese. We compare the three initial prediction methods and the three refinement options (including not performing refinement), for a total of nine decoding methods ( § 4.1). We further apply additional improvements ( § 4.3) on the confidence-based decoding method.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
By comparing the performance in Tab. 3, we first see advanced decoding methods improve performance on multi-token objects, but hurt performance on single-token ones. The best-performing decoding method on English improves the multitoken accuracy from 5.57% to 11.06%, indicating that advanced decoding methods have a better chance to elicit multi-token facts from M-BERT. Some examples are shown in Tab. 7 in the Appendix. The lower performance on single-token objects is probably caused by the fact that advanced decoding methods discover multi-token predictions that have higher confidence than single-token ones ( § 4.2). For example, the single-token prediction for "Enrique Iglesias used to communicate in _." is "Spanish", while the best decoding method outputs "his own words" with higher confidence. Second, initial prediction methods have a greater effect on the final performance than refinement methods. We  hypothesize that this is because the greedy decoding process heavily depends on previous predictions, and refinement cannot recover from unsatisfactory initial predictions. Third, length normalization was not found useful in either case. There are also observations not consistent across the two languages. First, since Chinese has a larger portion of multi-token objects than English (as shown in Tab. 1), the overall performance on Chinese increases while it decreases on English, which is consistent with the observation in Fig. 3. Second, confidence re-computation and beam search are not as effective on Chinese, which we conjecture is because that the distribution over English sentences exhibits more multimodality than the distribution over Chinese sentences due to more training data.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
As the performance of M-LMs is relatively low, especially on low-resource languages, an obvious endeavor is to refine the model to improve fact retrieval performance in various languages. We analyze how similarly M-BERT performs on queries in different languages. We collect correctly predicted facts across all languages, and count in how many languages each fact was retrieved correctly. As shown in the bottom-left histogram of Fig. 5, half of the correctly predicted facts were correct in a single language, indicating little overlap across languages (Lin et al., 2018). Only 3% of facts were correct in more than 5 languages, and objects in those facts are usually sub-strings of subjects, making them easy to retrieve regardless of the language. This observation is also confirmed by the overlap between pairs of languages in the top-right chart of Fig. 5; even the most similar languages (i.e., English and Dutch) only have 34% of correct predictions in common.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We find that facts retrievable only in a single language tend to be knowledge that is mainly mentioned in a certain language. For example, M-BERT mistakenly predicts "QQ" in the English sentence "Tencent QQ is developed by _.", while the prediction "腾讯" (Tencent) in the corresponding Chinese sentence "腾讯QQ是由_开发的。" is correct. This is probably because Tencent, a Chinese company, is more frequently mentioned in the Chinese training corpus.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Inspired by these observations, we propose to use code-switching to create data to fine-tune pretrained LMs, replacing entity mentions in one language (e.g., English/Greek) with their counterparts in another language (e.g., Greek/English). Through this bi-directional code-switching, entity mentions serve as pivots, enabling knowledge that was originally learned in one language to be shared with others. Given a pair of languages, we first identify Wikipedia sentences that mention entities from our benchmark using SLING (Ringgaard et al., 2017). The M-LM is then finetuned on these sentences. Following Wu et al. (2020), with 30% of probability we switch all the entity mentions (can be one or multiple) from the original language to their counterparts in the other language, ending up with sentences like "Οµπάµα later reflected on his years ...", where we substituted "Obama" with a Greek mention of the entity, and vice-versa for Greek-to-English. 70% of the sentences remain the same. If there are multiple mention texts for an entity, we sample proportionally to their frequencies, which we found in our preliminary experiments performed better than using a fixed translation. We fine-tune M-BERT using the masked LM objective on this data, with 15% of non-mention words and 50% of mention words masked out. 8
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We choose three languages with different data availability, namely French, Russian, and Greek, and pair them with English, producing 560k, 396k, and 129k code-switched sentences respectively. We compare M-BERT after code-switched fine-tuning (denoted as cs) with both the original M-BERT and with fine-tuning only on raw text (raw). We vary the evaluation settings to illustrate the effect of code-switching: on top of matching predictions to  ground truth aliases in the prompt language (singleeval), we evaluate with targets in both languages (double-eval; English and prompt). As shown in Tab. 4, continued fine-tuning on raw text outperforms the original M-BERT, likely due to our fine-tuning on a subset of sentences with mentions of entities from our benchmark. Results on code-switched text are slightly worse when only matching entities in the original target language, but significantly better if we allow matching in both the original language and English. This indicates that code-switched fine-tuning allows M-BERT to retrieve facts, albeit in English rather than in the prompt language. Encouragingly, the increase is larger for low-resource (Greek) and typologically distant-to-English (Russian) languages. For example, the prediction for the Greek prompt "η Θεωρία κατηγοριών είναι µέρος των ." ("Category theory is part of _.") is "mathematics" (in English!), while the prediction without code-switching is the non-informative "οποίων" ("which"). Considering that we have more raw than code-switched sentences in the dataset, this seems to indicate that English entities are easier to predict than their promptlanguage counterparts, which might be because facts expressed in English are better learned in the pre-trained model due to training data abundance.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We examine the intersection of multilinguality and the factual knowledge included in LMs by creating a multilingual and multi-token benchmark X-FACTR, and performing experiments comparing and contrasting across languages and LMs. The results demonstrate the difficulty of this task, and that knowledge contained in LMs varies across languages. Future directions include other pre-training or fine-tuning methods to improve retrieval performance and methods that encourage the LM to predict entities of the right types.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Tab. 5 shows the detailed number of facts in each language in our X-FACTR benchmark. Fig. 6 demonstrates the ratio of facts with respect to the number of tokens of the object in different languages, where high-resource languages (e.g., English, French, Dutch, and Spanish) have more portion of single-token facts than low-resource languages.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
The prompts generated in different languages may not be perfectly natural. This could be due to awkwardness of attempting to express relational phrases that were originally devised for English in languages where the semantic distinctions of the underlying words may differ, or due to our errors in our automated approach to grammatical attribute inference and subsequent inflection. To this end, we evaluated our prompts on a sample of languages, providing native speakers with 10 sentences per prompt with the missing slots filled by our inflection models. Our approach produces sentences that are annotated as correct 97.9% of the cases in Spanish, 90.5% in Yoruba, 86.7% in Greek, 82.3% in Marathi, and 81.9% in Russian.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We present an analysis of the annotations on the erroneous prompts in Table 6. The error types differ drastically across languages. Russian and Marathi have comparatively large percentages of inflection-related errors, but for different reasons: the prediction of non-human entity grammatical gender in Russian is difficult and this results in mistakes in the inflection. In Marathi, this issue is also exacerbated by the inflection model, which is of slightly lower quality due to the scarcity of training data availability.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Despite these two outliers, we consider the rest of our prompts to be of high quality. Even if small inflection or grammatical gender assignment mistakes occur (e.g. in Greek) this should not render the prompt unintelligible to native speakers -the burden is on the model to be robust to such slight variations, just as humans are. We point out that the prompts can be awkward or incorrect for some senses captured by the relation, an issue unrelated to our gender heuristics or automatic inflection. This issue, though, is also present in the LAMA English prompts (Petroni et al., 2019;Jiang et al., 2020) and is the result of the original Wikidata annotation.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We outline here the exact concrete formulation of our multi-token decoding algorithms. Given a sentence with multiple mask tokens, e.g., Eq. 2, we can either generate outputs in parallel independently or one at a time conditioned on the previously generated tokens. These methods are similar to the prediction problems that BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019b) perform in their pre-training stages respectively. We define c ∈ R n as the confidence of each prediction, with details varying by prediction method.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Independent For independent initial prediction, the mask tokens are all predicted in parallel:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
∀k ∈ {i, ..., j}.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We also consider two autoregressive methods for initial prediction or refinement.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Order-based Mask tokens are predicted from left to right, conditioned on previously generated tokens in each step:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
In the refinement stage, we modify the predicted tokens from left to right by replacing the token with a mask and re-predicting it:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
where s\i means that the i-th token in s is replaced with mask . Convergence is reached when there are no changes in a left-to-right scan.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Confidence-based Among all the predictions for masked positions, we choose the one with the highest confidence (i.e., the highest probability), so the actual order of predictions can be arbitrary, as shown in Fig. 2:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
In the refinement stage, we choose from all predicted tokens the one with the lowest confidence (i.e., the lowest probability) and re-predict it (Ghazvininejad et al., 2019):
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Convergence is reached when the re-predicted token is the same as the original token.   
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Length Normalization Since the sum used in § 4.2 might favor short predictions, we consider normalizing it by the number of the mask tokens:
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Confidence Re-computation Note that the confidence of each predicted token c in previous equations is the probability when the token is predicted. However, the probability will become stale once the surrounding tokens change because of the bidirectional conditional distributions, and this is also noted in (Ghazvininejad et al., 2019). To make the confidence up-to-date, given the prompt in Eq. 2, when a new token is predicted (in the initial stage) or a token is modified (in the refinement stage), we re-compute c i to c j . This makes the time complexity quadratic to the number of mask tokens, because every time we make a modification, we have to recompute the confidence values of all predictions. As a result, the final confidence becomes: c k = p(ŷ k |ŝ i:j \ k),
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
whereŝ i:j = x 1 , ...,ŷ i , ...,ŷ j , ..., x n contains the final predictions.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Beam Search All of the previous methods use the most plausible prediction at each masked position. We also consider performing beam search that keeps track of the most plausible B predictions. Our beam search algorithm is very similar to the case of conventional left-to-right decoding, except that the decoding order might be arbitrary if we use confidence-based initial or refinement prediction methods. As a result, extending different samples in the beam might lead to the same results so we need an additional deduplication step. The time complexity with all the above components is O(M 2 BT ), where M is the maximal number of mask tokens, and T is the maximal number of iteration. Alg. 1 outlines the overall multi-token decoding algorithm. The confidence-based decoding method takes 20 minutes to 2 hours on a Nvidia Geforce RTX 2080 Ti GPU depending on the number of facts of each language.  
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
LMs examined in this paper share similar architecture and pre-training setting as BERT (Devlin et al., 2019) or RoBERTa , but are trained on different corpora. We provide the shortcut name of each LM in the Hugging-Face's Transformer library (https://huggingface. co/transformers/pretrained_models.html) and their training corpora in Tab. 8, from which you can find more information.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Detailed performance across LMs and languages and error cases in Spanish and Greek are shown in Tab. 10 and Tab. 9 respectively.    Table 10: Accuracy on different languages using different LMs (%). We use M = 5 mask tokens for en, fr, nl es, vi (on the left) and M = 10 mask tokens for the other languages on the right. Best results for each language-part combination are in bold. "-" denotes missing/unsupported models.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Most research in natural language processing (NLP) to date has focused on developing methods that work well for English and a small set of other high-resource languages (Joshi et al., 2020). In contrast, methods for other languages can be vastly more beneficial as they enable access to language technology for more than three billion speakers of low-resource languages and prevent the NLP community from overfitting to English. Motivated by these benefits, the area of multilingual NLP has attracted increasing interest recently.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
However, evaluating multilingual models is challenging as it requires assessing performance on a wide range of typologically distinct languages in the face of limited heterogeneous data sources. Recently large-scale benchmarks such as XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020) XTREME XTREME-R # of languages 40 50 # of tasks 9 − 2 + 3 = 10 Task categories Classification, structured prediction, QA, retrieval +language-agnostic retrieval Analysis tools -MULTICHECKLIST, Explainaboard Leaderboard Static Interactive, +metadata Table 1: Overview of XTREME and XTREME-R.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
have been introduced, consolidating existing multilingual tasks and covering tens of languages. When XTREME was released, the gap between the best-performing baseline, XLM-R Large (Conneau et al., 2020), and human-level performance was roughly 25. This has since shrunk to less than 12 points, a much smaller but still substantial gap compared to the difference from human-level performance observed in English transfer learning (Wang et al., 2019a), which has recently been closed entirely on some evaluation suites (He et al., 2021). In order to examine the nature of this progress, we first perform an analysis of state-of-the-art multilingual models on XTREME. We observe that progress has not been uniform, but concentrated on cross-lingual retrieval tasks where fine-tuning on other tasks and pre-training with parallel data lead to large gains. On other task categories improvements are more modest. Models still generally perform poorly on languages with limited data and non-Latin scripts. Fine-tuning on additional translated data generally leads to the best performance.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Based on this analysis, we propose XTREME-R (XTREME Revisited), a new benchmark with the dual purpose of ensuring that research in multilingual NLP focuses on the most challenging problems and equipping researchers with a broader set of tools to better understand their models (see Table 1 for a brief overview). XTREME-R follows in its predecessor's footsteps by being massively multilingual, diverse, and accessible. It expands on XTREME by covering 50 typologically diverse languages and 10 challenging, diverse tasks. To make retrieval more difficult, we introduce two new tasks that focus on "language-agnostic" retrieval (Roy et al., 2020), where targets must be retrieved from a large multilingual candidate pool. We additionally establish new state-of-the-art mT5 (Xue et al., 2021) and translate-train baselines for our tasks. XTREME-R aims to move away from a single aggregate metric summarizing a model's performance and towards a more nuanced evaluation and comparison of multilingual models (Ethayarajh and Jurafsky, 2020;Linzen, 2020). To this end, we introduce an extensible multilingual diagnostic and evaluation suite that consists of two main components: a) MULTICHECKLIST, a test suite (Ribeiro et al., 2020) for probing question answering capabilities in 50 languages. This test suite is the first of its kind and enables direct evaluation of finegrained capabilities in a massively multilingual setting. b) We extend the multi-dataset evaluation framework EXPLAINABOARD (Fu et al., 2020;Liu et al., 2021) to additional tasks and the multilingual setting. This framework allows us to break down performance based on language and task-specific attributes, which enables a more nuanced diagnosis of a model's behaviour.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We also make several logistic improvements to improve XTREME-R's utility as a leaderboard. To make it easier to choose the best model for a use case, each submission is required to provide metadata such as the number of parameters and amount of pre-training data, which we make available via an interactive leaderboard. We also introduce task and language-specific sub-leaderboards to invite submissions of dedicated models.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
In sum, we make the following contributions: a) an analysis of progress in cross-lingual modeling; b) an improved benchmark covering 50 languages, including a newly created retrieval task (Mewsli-X); c) a massively multilingual diagnostic suite; d) fine-grained evaluation capabilities; e) experiments and analyses of state-of-the-art models; and f) an interactive metadata-rich leaderboard.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
2 Examining the State of Multilingual Benchmarking
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Benchmarking is critical to evaluate generalpurpose language understanding technologies. To this end, benchmarks like GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) provide a way to assess the transfer learning capabilities of various models. However, these benchmarks focus only on English. On the other hand, crosslingual approaches have been evaluated on a wide range of disparate tasks (Hu et al., 2020). XTREME was proposed as a platform to unify this fragmented evaluation landscape and to catalyze advances in cross-lingual learning by including a diverse set of tasks and languages. It consists of 9 tasks covering 40 diverse languages, which can be grouped into 4 broad task types (see §3.1 for details): classification (XNLI, PAWS-X), structured prediction (UD-POS, WikiANN-NER), question answering (XQuAD, MLQA, TyDiQA-GoldP), and retrieval (Tatoeba, BUCC). XTREME focuses on zero-shot cross-lingual transfer, i.e. models can be pretrained on any multilingual data and are fine-tuned only in English. Similarly, XGLUE (Liang et al., 2020), another cross-lingual benchmark focuses on a smaller number of less typologically diverse languages, and includes generation tasks. Other non-English benchmarks focus on specific linguistic phenomena, e.g. code-switching (Khanuja et al., 2020); languages, e.g. Indonesian (Willie et al., 2020) and Persian (Khashabi et al., 2020); and language families, e.g. Indian languages (Kakwani et al., 2020).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
As of April 15, 2021, all submissions to the XTREME leaderboard are large-scale Transformers (Vaswani et al., 2017) trained with masked language modeling (MLM; see Appendix A for further details). We analyze the performance of these models on the XTREME leaderboard in Figure 1. 1 Overall, multilingual models have improved the average performance on XTREME from 55.8 to 81.4. Much of this improvement is concentrated in the retrieval-based tasks where performance increased from 47.7 (mBERT) to 92.7 (VECO). In contrast, performance on question answering and structured prediction tasks has improved only slightly. Breaking down performance by language family, on Tatoeba (Figure 1c) recent models still struggle with a few low-resource languages. Models perform well for most other languages and their scores are concentrated in a relatively small range. On MLQA (Figure 1b), scores have increased slightly (a) Performance on XTREME (b) Performance on MLQA (c) Performance on Tatoeba (d) Performance on UD-POS Figure 1: Performance of models (a) on the XTREME leaderboard across all nine XTREME tasks, (b) on the MLQA question answering dataset, (c) on the Tatoeba retrieval task, and (d) on Universal Dependencies POS tagging across language families. Models are ordered based on their XTREME score (a). Results of models that do not evaluate on a task category are omitted, i.e. RemBERT for retrieval and mT5 for retrieval and tagging.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
but remain well below performance on English. On POS tagging (Figure 1d), scores remain largely the same; performance is lower for some languages with non-Latin scripts and low-resource languages. We show the scores for the remaining tasks in Appendix B. The remaining gap to English performance on these tasks is partially an artefact of the evaluation setup: zero-shot cross-lingual transfer from English favors English representations whereas models fine-tuned on in-language monolingual data perform more similarly across languages (Clark et al., 2020;Hu et al., 2020).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Overall, representations from token-level MLM pre-training are of limited use for cross-lingual sentence retrieval, as evidenced by the comparatively poor performance of the mBERT and XLM-R models. Fine-tuning on sentence-level tasks (Phang et al., 2020;Fang et al., 2021) can mitigate this. The strong performance of recent models such as VECO and ERNIE-M on the retrieval tasks can be attributed to a combination of parallel data and new pre-training objectives that make use of it. Pretraining on parallel data improves performance on retrieval by making the pre-training task more sim-ilar to the downstream setting but does not significantly improve performance on other tasks. Finetuning on automatically translated task-specific data yields strong gains and is used by most recent models to achieve the best performance (Hu et al., 2020;Ouyang et al., 2020;Luo et al., 2020). Nevertheless, key challenges such as how to learn robust cross-lingual syntactic and semantic processing capabilities during pre-training remain.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
In order to encourage the NLP community to tackle challenging research directions in pursuit of better cross-lingual model generalization, we propose XTREME-R (XTREME Revisited). XTREME-R shares its predecessor's core design principles for creating an accessible benchmark to evaluate crosslingual transfer but makes some key changes.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
First, XTREME-R focuses on the tasks that have proven to be hardest for current multilingual models. To this end, it drops XTREME's PAWS-X and BUCC tasks since recent advances have left less room for further improvement, and they cover only a small number of less diverse languages. They  are replaced instead by three new, more challenging tasks: one focusing on causal commonsense reasoning ( §3.2.1) and two focusing on harder retrieval scenarios ( §3.2.2), as this has been the task category where gains have been easiest to achieve. We retain XTREME's seven other tasks as each still presents substantial challenges for state-of-the-art cross-lingual models ( §3.1). Overall, XTREME-R includes 10 diverse tasks, summarized in Table 2. We also make changes to the structured prediction tasks, NER and POS. Instead of only providing examples as lists of tokens, XTREME-R always provides the full text of an input sentence, thus ensuring that the entire benchmark now supports research on models that operate directly from the raw input string (Clark et al., 2021). Furthermore, XTREME-R adopts a more realistic version of the NER task in which no gold tokenization is provided at all, meaning that systems will either have to use model-predicted tokens or embrace tokenizationfree approaches. Finally, XTREME-R provides a multilingual diagnostic and evaluation suite ( §3.4).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We retain the XNLI (Conneau et al., 2018), UD-POS (Nivre et al., 2018), WikiANN-NER (Pan et al., 2017), XQuAD (Artetxe et al., 2020a), MLQA (Lewis et al., 2020), TyDiQA-GoldP (Clark et al., 2020), and Tatoeba (Artetxe and Schwenk, 2019) tasks from XTREME (see Appendix C).  (Raffel et al., 2020), the state-of-the-art on XCOPA is only around 70%.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Many previous retrieval benchmarks assume that the entire candidate pool is in a single language. For instance, a French query will be used to search over only English candidates. However, practical settings often violate this assumption, e.g. the answer to a question may be available in any number of languages, possibly different from the query language. Models that cannot compare the appropriateness of retrieval results across languages are thus ineffective in such real-world scenarios. XTREME-R includes two new related crosslingual retrieval tasks. The first seeks to measure the extent to which cross-lingual representations are "strongly aligned" (Roy et al., 2020), i.e. they place the semantically most related text pairs (e.g. a question and its answer) closest together in representation space, regardless of their language identities. The second analogously frames entity linking as retrieving from a multilingual pool of entity descriptions, given an entity mention in context (Botha et al., 2020). For both, we report performance as mean average precision at 20 (mAP@20). LAReQA Language Agnostic Retrieval Question Answering (Roy et al., 2020) is a sentence retrieval task. Each query has target answers in multiple languages, and models are expected to rank all correct answers above all incorrect answers, regardless of language. We use the LAReQA XQuAD-R dataset which contains 13,090 questions each of which has 11 target answers (in 11 distinct languages) within the pool of 13,014 candidate answer sentences. Following Roy et al. (2020), we fine-tune models on the SQuAD v1.1 train set. The fine-tuned model is used to rank the 13K candidates for each question.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Mewsli-X Mewsli (Multilingual Entities in News, linked) is an automatically extracted dataset that requires linking a contextual entity mention to its entry in a language-agnostic knowledge base by retrieving the entity's description from a multilingual candidate pool (Botha et al., 2020). For XTREME-R, we derive Mewsli-X as a new variant of Mewsli-9, still linking against WikiData (Vrandečić and Krötzsch, 2014). Mewsli-X features 15K mentions in 11 languages: given a mention in context, the task is to retrieve the single correct target entity description from a candidate pool ranging over 1M candidates across all 50 languages of XTREME-R. Fine-tuning is done on a predefined set of English-only mention-entity pairs randomly sampled from English Wikipedia hyperlinks (see Appendix E for further details).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
For our baseline systems on both tasks, we follow previous work (Roy et al., 2020;Botha et al., 2020) and train a dual encoder initialized from the pre-trained model weights, optimizing for an inbatch sampled softmax loss (Gillick et al., 2018).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
XTREME-R adds the following ten languages to XTREME: Haitian Creole, Cusco Quechuan, Wolof, Lithuanian, Punjabi, Gujarati, Polish, Ukrainian, Azerbaijani, and Romanian. In total, XTREME-R covers the following 50 languages (shown using their ISO 639-1 codes for brevity; new languages are bolded) belonging to 14 language families and two isolates: af, ar, az, bg, bn, de, el, en, es, et, eu, fa, fi, fr, gu, he, hi, ht, hu, id, it, ja, jv, ka, kk, ko, lt, ml, mr, ms, my, nl, pa, pl, pt, qu, ro, ru, sw, ta, te, th, tl, tr, uk, ur, vi, wo, yo, zh. 2 XTREME-R is similarly typologically and genealogically diverse as XTREME while covering a larger number of languages (see Appendix D).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
To increase the language coverage of low-resource languages in XTREME-R and to enable us to systematically evaluate a model's cross-lingual generalization ability, we augment XTREME-R with a massively multilingual diagnostic and evaluation suite. Challenge sets and diagnostic suites in NLP (Wang et al., 2019a,b;Belinkov and Glass, 2019) are mostly limited to English, with a few exceptions (Gulordava et al., 2018). As challenge sets are generally created with a human in the loop, the main challenge for creating a large multilingual diagnostic suite is to scale the annotation or translation effort to many languages and to deal with each language's idiosyncrasies. MULTICHECKLIST To address this, we build on the CHECKLIST (Ribeiro et al., 2020) framework, which facilitates creating parameterized tests for models. CHECKLIST enables the creation of test cases using templates, which test for specific behavioral capabilities of a model with regard to a downstream task. Importantly, by relying on template-based tests, we can efficiently generate a large number of diverse multilingual test cases by creating a relatively small number of templates in 50 languages. 3 We focus on translating English tests, which consist of templates and their fill-in values. 4 To study the feasibility of creating multilingual test cases at scale, we translate the minimum functionality tests (MFT) of CHECKLIST, which probe for general vocabulary and taxonomic knowledge in question answering. We instruct translators to create separate variants of a template to disambiguate linguistic phenomena, such as gender of fillin values, question type, semantics of properties, etc. We automatically fill names in each language based on data from Wikidata and programmatically consolidate different templates in each language.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We show examples of templates and the tests that they generate in different languages in Table 3.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We highlight statistics of the dataset and translation process, instructions to translators, and general challenges of template translation in Appendix F. We believe that parameterized tests are a powerful tool to obtain diverse diagnostics data for otherwise resource-starved languages. We view participatory research (∀ et al., 2020) with native speakers to create template-based test cases testing for languagespecific behaviour as particularly promising. Multilingual EXPLAINABOARD The standard practice in leaderboards is to average performance across different settings (Wang et al., 2019b,a). While this provides discriminative power, it has limited utility for examining the relative advantages of systems, the characteristics of different datasets and languages, and how these factors relate to each other. To provide more granular evaluation capabilities, we extend Fu et al. (2020); Liu et al.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
(2021)'s EXPLAINABOARD to the task categories and languages in XTREME-R. EXPLAINABOARD provides a more nuanced impression of a model's performance on a task by defining task-specific attributes (e.g. entity length for NER). The test set is partitioned into different buckets based on the defined attributes and performance is broken down over different attribute values. We define new taskspecific attributes for the four task types as well as task-independent attributes (see Appendix K). Metadata We additionally would like to enable practitioners to rank submissions based on other information. To this end, we ask each submission to XTREME-R for relevant metadata such as the number of parameters, the amount of pre-training data, etc. We will show this information in an interactive leaderboard (see Appendix H for the metadata of current XTREME submissions).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Training and evaluation setup XTREME-R focuses on zero-shot cross-lingual transfer from En-glish. While recent work (Hu et al., 2020;Lauscher et al., 2020;Hedderich et al., 2020) demonstrates the benefits of fine-tuning on in-language data, we believe the zero-shot scenario remains the most effective way to evaluate the amount of a priori multilingual knowledge a pre-trained model captures. Due to variation in cross-lingual evaluation (Keung et al., 2020), we recommend researchers to use the validation set of a single target language for development (Artetxe et al., 2020b).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We employ established pre-trained multilingual and models using translations as baselines. mBERT Multilingual BERT (Devlin et al., 2019) has been pretrained on the Wikipedias of 104 languages using MLM. XLM-R XLM-R Large (Conneau et al., 2020) uses the same MLM objective with a larger model, and was trained on a magnitude more web data from 100 languages. mT5 Multilingual T5 (Xue et al., 2021) is an encoder-decoder transformer that frames NLP tasks in a "text-to-text" format. It was pre-trained with MLM on a large multilingual web corpus covering 101 languages. We employ the largest mT5-XXL variant with 13B parameters. Translate-train To evaluate the impact of MT, we fine-tune mBERT on translations of English training data from Hu et al. (2020). We create new translations for the XCOPA and SIQa data using an in-house MT system. 5 Translate-train multilingual In addition, we finetune both mBERT and mT5 on the combined translated training data of all languages (including the original English data) jointly. Human performance We use the human performance estimates from XTREME for the retained tasks. For XCOPA we average the proportion of annotated labels disagreeing with the majority label across all languages (Ponti et al., 2020). We are not able to obtain human performance estimates for the new retrieval tasks as identifying a translation among a large number of candidates is too time-consuming for a human to perform.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We show the main results in Table 4. As in prior work, XLM-R Large generally outperforms mBERT. Fine-tuning helps significantly on Tatoeba  Table 4: Overall results of baselines across all XTREME-R tasks. *: Due to compute limitations, mT5-XXL language-agnostic retrieval results are obtained with a frozen rather than a fine-tuned model.  compared to the zero-shot setting (Hu et al., 2020). The new tasks are challenging for current models, which show relatively lower performance compared to other tasks. XCOPA presents a challenging classification task that requires cross-lingual common sense reasoning while the languageagnostic nature of Mewsli-X and LAReQA puts the cross-lingual alignment of multilingual representations to the test. Analysis of the language-agnostic retrieval results show a large gap remains between cross-lingual and same-language test cases. XLM-R Large improves significantly over mBERT on the cross-lingual case in exchange for a slight drop for the same-language case. This points to XLM-R Large inducing more "strongly-aligned" representations (see Appendix I for details). The stateof-the-art mT5 improves performance on classification and QA tasks but performs less well on structured prediction and retrieval, highlighting settings where advances beyond scale are needed. 6 Training on task-specific translations is beneficial in all cases and generally performs best, although improvements on QA tasks are diminishing. To obtain a more fine-grained understanding of the performance of current models, we conduct several analyses using our multilingual diagnostic suite.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We show the results of XLM-R fine-tuned on English SQuAD v1.1 on the 6 tests of MULTICHECK-LIST in Table 5 (see Appendix F for the full results, example failure cases, and mBERT results). While mBERT's average error rate is greater than 85% on 4/6 test categories, XLM-R demonstrates a substantially more robust cross-lingual understanding ability. XLM-R performs worst on tests in lowresource languages with limited or no pre-training data such as gu, ha, ht, qu, sw, wo, and yo and in languages with non-Latin scripts such as he, ja, 6 Due to compute limitations, we extract mT5 embeddings by averaging the encoder outputs of a frozen mT5 model fine-tuned on SQuAD v1.1, as opposed to fine-tuning a dual encoder. For this reason, the mT5 language-agnostic retrieval scores are not directly comparable to those of mBERT and XLM-R.   "what" (A), "how" (B), "who" (C), "when" (D) and "which" (E). In the single system diagnosis histogram, blue (red) x ticklabels represent the bucket category (e.g., XS) of a specific attribute on which a system achieved the best (worst) performance. In the pairwise system diagnosis histogram, blue (red) x ticklabels represent the bucket value of a specific attribute where system M1 surpasses (under-performs) M2 by the largest margin illustrated by a blue (red) bin. Blue-only x ticklabels (e.g., -D) indicate that M1 outperforms M2 in all categories of an attribute.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
th, and zh. In addition, XLM-R displays interesting variation across languages, for instance failing in modeling comparisons in some languages, like Basque (eu), where it otherwise succeeds. We release the tests and test outputs to encourage deeper analysis and extension to other tasks and languages.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We showcase how nuanced multilingual evaluation enables us to perform single and pairwise system diagnosis on XQuAD in Table 6 (see Appendix K for analyses of the other tasks). We choose two systems: ERNIE-M, one of the top systems on XTREME, and XLM-R in eight languages: English, Chinese, Hindi, Greek, Russian, Turkish, Arabic, and Vietnamese (en, zh, hi, el, ru, tr, ar, vi).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Attributes We denote (X c , X q , X a ) as a tuple of a context, question and answer, and refer to cLen, qLen, aLen as their lengths (i.e., the number of tokens). We use BLEU (Papineni et al., 2002) to measure lexical overlap between (X a , X q ) and (X q , X c ) as BLEU-AQ and BLEU-QC. We report the top 5 most frequent question types (qType), which cover 85% of questions in the training set.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Single System Analysis For almost all languages, ERNIE-M achieves the highest performance on shorter answers (XS), but the worst performance on longer answers (XL). Especially in el, the performance difference between long and short an-swers is more than 40 absolute points. The influence of question and context length is languagedependent. For example, in zh the system favors long questions and contexts while in hi, it is the opposite. If the answer is lexically similar to the question (larger BLEU-AQ), the system tends to make more mistakes in all eight languages. However, a higher lexical overlap between questions and contexts (BLEU-QC) is helpful for some languages: el, ru, ar. Surprisingly, ERNIE-M struggles to answer relatively frequent question types (i.e., what, and how), while it performs better on less frequent questions, indicating that although questions about person, place and choice are less frequent, they are easier than abstract questions. Pairwise System Analysis Although ERNIE-M outperforms XLM-R by a large margin, it is surpassed by XLM-R on a few buckets. In en, XLM-R is better at dealing with longer answers and questions. In tr, XLM-R surpasses ERNIE-M on samples with shorter answers and contexts. In zh, XLM-R performs better when dealing with questions that are lexically similar to the answers.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Our analyses and experiments have shed light on important directions where scale alone is not sufficient such as "strong" alignment, syntactic transfer, fine-grained natural language understanding, and answering of abstract questions. We encourage the development of better inductive biases, pre-training objectives, and evaluation resources. We make our data, translations, evaluation resources, and interactive leaderboard supporting detailed comparative analyses available to help the community gain a better understanding of multilingual models.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
7 Ethical Considerations 7.1 Language representation XTREME-R seeks to improve language representation and language diversity in NLP research, which has been identified as a large challenge (Joshi et al., 2020). We tried to cover a set of languages that is as diverse as possible, while still providing access to evaluation data in multiple tasks for each language. Despite this, XTREME-R has little representation of languages of the Americas and Africa due to a lack of labeled datasets for these languages. In addition, some languages included in XTREME-R with few data available online are only covered in a small number of datasets (see Table  7). To ameliorate this, we release training data of tasks translated into other languages, as well as the new MULTICHECKLIST. We reiterate the ongoing need for creating labeled datasets for diverse tasks in under-represented languages, to facilitate the development and evaluation of NLP models for such languages. We emphasize the importance of participatory research (∀ et al., 2020) as a modus operandi for such work in order to involve marginalized communities in the research process.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
New benchmarks incentivize researchers to hillclimb on aggregate metrics (Ethayarajh and Jurafsky, 2020). In addition, new benchmarks create new opportunities for models to reach "superhuman" performance, which may lead people outside the field to erroneously conclude that some model has "solved language". We hope that our inclusion of EXPLAINABOARD and MULTICHECKLIST help to prevent such a fallacy, by enabling more fine-grained evaluation that goes beyond a single aggregate metric.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Multilingual models have been shown to reflect biases similar to their monolingual counterparts (Zhao et al., 2020). In addition, multilingual models are biased towards languages with more pre-training data (Hu et al., 2020). Zero-shot crosslingual transfer additionally introduces a bias towards the source language (Søgaard et al., 2018;Anastasopoulos and Neubig, 2020). Due to the paucity of training data in other languages, we nevertheless focus on English-centric transfer and encourage future dataset creation efforts to include training data in multiple languages.  
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We show the performance of the models on the XTREME leaderboard broken down by language family on the remaining XTREME tasks in Figure  2.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
C XTREME tasks retained in XTREME-R is another cross-lingual question answering task. The evaluation data in seven languages was automatically mined from Wikipedia, annotations were crowd-sourced, and answer spans aligned. For both XQuAD and MLQA, we use their respective data for evaluation and train on SQuAD v1.1. TyDiQA-GoldP We use the gold passage (GoldP) version of TyDiQA (Clark et al., 2020), a benchmark for information-seeking question answering, which covers nine typologically diverse languages. The GoldP version is a simplification of the primary task, using only the gold passage as context and excluding unanswerable questions. We use the English training data for training and evaluate on the test sets of the target languages. Tatoeba We evaluate on the Tatoeba dataset (Artetxe and Schwenk, 2019), which consists of up to 1,000 English-aligned sentence pairs covering 122 languages. We find the nearest neighbor using cosine similarity. To make the setting more realistic, we move away from zero-shot retrieval and fine-tune models on SQuAD v1.1.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Language characteristics We show a detailed overview of languages in XTREME-R including interesting typological differences in Table 7. Wikipedia information is taken from Wikipedia 8 and linguistic information from WALS Online 9 . XTREME-R includes members of the Afro-Asiatic, Austro-Asiatic, Austronesian, Dravidian, Indo-European, Japonic, Kartvelian, Kra-Dai, Niger-Congo, Sino-Tibetan, Turkic, Uralic, Creole 10 , and Quechuan language families as well as of two isolates, Basque and Korean. Language diversity indices We measure the language diversity of XTREME-R according to the typology and language family indices of Ponti et al.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
(2020), which we show in Table 8 for XTREME-R, XTREME ( 
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Mewsli-X is constructed specifically for XTREME-R and is a more carefully sampled variant of the Mewsli-9 dataset (Botha et al., 2020), derived from WikiNews in the same way. Compared to Mewsli-9, Serbian is dropped and Polish, Romanian and Ukrainian are added to obtain 11 languages, while the entity descriptions   Table 8: Diversity indices with regard to typology and language family of XTREME-R, XTREME, and XGLUE. to be retrieved range over all 50 languages in XTREME-R (Table 9). To broaden accessibility, the mention queries, candidates and Wikipedia-based training instances are all downsampled compared to the previous work. Mention Extraction Viable mentions were taken as hyperlinks in the WikiNews articles pointing to Wikipedia entity pages (in any language) that could be mapped successfully to items in our base WikiData entity vocabulary. V base is defined as the set of entities that have a Wikipedia page in any of the 50 languages in XTREME-R (|V base | ≈ 12M). The latter condition ensures that entity descriptions are available. We also extended the WikiData filter used by Botha et al. (2020) to additionally exclude entities that are instances of Wikipedia List Pages (Q13406463) or Wikimedia (Human) Disambiguation Pages (Q22808320, Q4167410), which are not of large interest for entity linking. The resolved, viable mentions were filtered to drop duplicate surface mentions of an entity in the same article, and mentions of years (e.g. 2014), which are commonly linked in WikiNews but not of great interest. We then performed stratified sampling by both mention language and entity frequency bins, seeking uniform sizes across strata. Entity frequency is estimated as the number of times an entity is referenced on pages in the 50-language Wikipedia collection, and then binned into five intervals: The resulting set of 15,000 test mentions covers 9,647 distinct gold entities (V gold ). Candidate Set Doing vector encoding and nearest neighbor search over a full knowledge base of millions of entities is relatively time-consuming, but searching only among V gold is also unrealistic. We thus strike a balance by defining a candidate set V cand ⊃ V gold , by sampling additional items from V base \ V gold , this time only stratified by entity frequency, to obtain |V cand | = 1, 000, 000.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Each entity e ∈ V cand is assigned a single description: we randomly sample a language from among the L e ≤ 50 Wikipedia pages corresponding to e, and take the page's first sentence as entity description.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Fine-tuning Data The fine-tuning data constitutes 115K English-only (mention, entity)-pairs, which were sampled from English Wikipedia hyperlinks that map to V base \ V cand . Sampling is according to the natural distribution. The Mewsli-X evaluation setting in XTREME-R is thus doubly zero-shot: no candidate or test entities are observed during finetuning, nor is non-English text. 3. If there is not a literal translation or the same translation has already been used for another word, feel free to use a translation that is similar in meaning.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
4. If the translation of the template differs based on the substituted words, please create multiple translations for the template and indicate which substituted words correspond to it. For instance, if one translation of the template assumes that some substituted words have male gender but others have female gender, create a separate translation of the template that conforms to the female gender.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
5. In one template, {p.p1} and {p.p2} refer to a property (either "shape", "color", "size", "age", or "material"). {p.v1} and {p.v2} refer to an attribute of each property such as "old", "new", "red", "blue", etc.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Challenges of template translation We highlight some of the challenges and linguistic phenomena we encountered during the process of creating MULTICHECKLIST in the following. Unless specified otherwise, we create separate templates to disambiguate each phenomenon.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Gender agreement: Adjectives and nationalities need to be declined to match the gender of their referring expression. To keep the translation effort manageable and avoid creating separate templates for each gender, we control for gender and restrict fill-in values to male names for the affected tests (3/6). We sample genders equally for the other tests. We welcome dedicated test suites analyzing multilingual gender bias as future extensions.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Declination: In Russian, animal and vehicle names require Accusative and Nominative in different cases.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Normalization: For appropriate substitution, fill-in values often need to include articles. We normalize answers and predictions by removing language-specific articles in order to ensure a consistent comparison.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Names: Our use of names based on data in Wikidata leads to certain biases. Names that are more common in Wikidata are more likely to be chosen. In some cases, names in Wikidata are not written in the native script. Japanese names from Wikidata are often written in hiragana or katakana rather than kanji. Our choice of using the first name is also not applicable to all languages. In Japanese, people are usually not referred to by their first name, e.g. Masanori Suzuki would be called Suzuki-san instead of Masanori.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Declension of names: In some languages, a suffix is appended to a name depending on its spelling. For instance, in Turkish the suffix changes based on the vowel of the last syllable, e.g. Ahmet'in, Ali'nin, Umut'un, Seyma'nın, Özge'nin, etc., and Ahmet'ten, Ali'den, Umut'tan,Şeyma'dan, Özge'den, etc. In Finnish, names are appended with a variation of "lla", e.g. Peterillä, Lisalla, Mattilla, etc.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Professions: Words for certain professions are gendered (e.g. waiter/waitress), so they only occur with male or female names.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Question syntax: In some languages, the syntax of the question changes depending on the property or adjective one asks about.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Syntax of adjectives: In some languages, the syntax changes depending on what adjective is used. In German, the translations of "happy", "excited", and "passionate" require different prepositions.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• Stilted language: Some text appears stilted when values are filled into the translated templates. For instance, the question "どちらの 方が冷静でないですか。" is an unusual way to do negation in Japanese; if directly translated to English, it would mean "Who is more not calm?".
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We tried to address most of these challenges by instructing translators to create additional templates to disambiguate linguistic phenomena and by consolidating different templates programmatically. However, as this process was relatively laborintensive, we recommend the usage of morphologically aware templates similar to Jiang et al. (2020) for future work. Note that morphologically aware templates may not be able to resolve some of the finer linguistic differences. For this reason, we also advise working closely with native speakers to design tests that reflect natural language as closely as possible. Full results We show the full results of XLM-R and mBERT on the MULTICHECKLIST tests in Tables 10 and 11 respectively. mBERT only shows limited amounts of cross-lingual taxonomic knowledge. While it is able to distinguish between job and nationality and animals and vehicles in some languages, it fails to do this consistently across all languages. In addition, it completely fails to distinguish between different properties and intensifiers and is not able to perform comparisons. In contrast, while XLM-R struggles with intensifiers, it demonstrates the other capabilities much more consistently across languages.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Example failure cases We provide example failure cases of XLM-R on a subset of languages in Table 12. We will publicly release a comprehensive list of failure cases for XLM-R and mBERT, the complete tests and model outputs for further analysis.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
mBERT We use the cased version, which covers 104 languages, has 12 layers, 768 hidden units per layer, 12 attention heads, a 110k shared Word-Piece vocabulary, and 110M parameters. 12 The model was trained using Wikipedia data in all 104 languages, oversampling low-resource languages with an exponential smoothing factor of 0.7. We generally fine-tune mBERT for two epochs, with a training batch size of 32 and a learning rate of 2e-5. We build on the Transformers library (Wolf et al., 2019) for training on each task. XLM-R We use the XLM-R Large version that covers 100 languages, uses a 200k shared BPE vocabulary, and has been trained with masked language modelling. 13 We fine-tune XLM-R generally for two epochs with a learning rate of 3e-5 and an effective batch size of 16. We use the Transformers library for training XLM-R on all tasks. mT5 We use the publicly released mT5-XXL version that has nearly 13 billion parameters with a vocabulary size 250k (Xue et al., 2021). It has been trained on multilingual C4 (mC4) corpus which has 6.3 trillion tokens spanning 101 languages 14 . For all downstream tasks, we fine-tune mT5-XXL for 10k steps with a constant learning rate of 0.001, dropout rate of 0.1 and a batch size of 2 17 tokens. For early stopping, we save checkpoints every 200 steps and choose the checkpoint with the highest performance on the validation set.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We intend to ask each submission to XTREME-R for relevant metadata. Such metadata includes the number of parameters, amount of pre-training data, amount of fine-tuning data, etc. We are doing this to enhance transparency and to increase utility of our benchmark for practitioners with varying needs. As a first step in this direction, we provide information about the number of parameters and the amount of monolingual and parallel pre-training data used by all submissions to XTREME in Table 13. Note that the different systems report their training data in different ways (e.g. number of tokens, number of examples, size of the data). We plan to standardize this by asking submissions to 12 https://github.com/google-research/ bert/blob/master/multilingual.md 13 https://github.com/facebookresearch/ XLM 14 https://www.tensorflow.org/datasets/ catalog/c4#c4multilingual XTREME-R to report training data in terms of number of tokens seen.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
The multiway cross-language nature of Mewsli-X and LAReQA enables closer analysis of model performance by input and target language pairs. Mewsli-X can directly be split by language pair as it has a single correct target per input mention. For LAReQA, we follow the "Limit to One Target" strategy of Roy et al. (2020): instead of asking the model to retrieve all correct answers in one pass, we evaluate on each target separately, with all the other correct answers removed from the candidate pool, allowing us to report splits by language pair. Table 14 summarizes these pairwise mAP@20 scores (here, micro-averaged), showing that XLM-R Large improves substantially over mBERT on the cross-lingual case (+38% on Mewsli-X and +137% for LAReQA) in exchange for a slight drop for the same-language case. Even so, performance on the cross-lingual case is still low at 29-36 mAP@20, and remains a challenging area for future work. Figures 3 and 4 show the detailed breakdowns.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We show the detailed results for each task and language in Tables 15 (XNLI), 
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We perform nuanced multilingual evaluations by categorizing testing examples into different attribute buckets and measuring the system performance on each attribute bucket. In the following, we describe the available attributes for tasks in XTREME-R and provide additional analysis on different attributes.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
QA We denote (X c , X q , X a ) as a tuple of the corresponding context, question and answer, and refer to cLen, qLen, aLen as their lengths (i.e., the number of tokens). We use BLEU (Papineni et al., 2002) to measure the lexical overlap between (X a , X q ) and (X q , X c ) as BLEU-AQ and BLEU-QC. We classify questions based on their first tokens and report the top 5 most frequent question types as qType (i.e., what, how, when, where, ar de en es fa ja pl ro ta tr uk ar  de  en  es  fa  ja  pl  ro  ta  tr  uk  af  az  bg  bn  el  et  eu  fi  fr  gu  he  hi  ht  hu  id  it  jv  ka  kk  ko  lt  ml  mr  ms  my  nl  pa  pt  qu  ru  sw  te  th  tl  ur  vi  wo    which), which cover 85% of questions in the training set. We list the six attributes as follows.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ aLen (X a ) = len(X a ),
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ qLen (X q ) = len(X q ),
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ cLen (X c ) = len(X c ),
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ BLEUaq (X a , X q ) = BLEU(X a , X q ),
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ BLEUqc (X q , X c ) = BLEU(X q , X c ),
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ type (X q ) = type(X q ), question type.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Given a sentence X, we define the i-th word token as x i and a span of words in the range of [i, j) as X i:j in the sentence. We then define five attributes including the label of a span (tag), the token length of a sentence (sLen), the token length of an entity span (eLen), the character length of an entity span (tLen) and the relative token position of an entity (rPos) in the sentence as follows.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ tag (X i:j ) = label(X i:j )
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ sLen (X) = len(X)
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ eLen (X i:j ) = len(X i:j )
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• φ tLen (X i:j ) = |X i:j | • φ rPos (X i:j ) = i/φ sLen(X) , relative position
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
where |x| represents the number of characters.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
We   Table 27 and 28 illustrate the single system diagnosis of ERNIE-M and XLM-R respectively on the WikiANN-NER task in three languages (i.e., en, es, fr). We make the following observations. ERNIE-M In Table 27, first, we observe that the effects of some attributes for ERNIE-M are language-independent. For example, based on the attribute rPos, the system is good at predicting entities located within the first 1/3 part of the English sentences, while it is relatively bad at predicting entities within the first 1/3 part of the sentences for other languages. Second, the system favors long sentences based on the attribute sLen. We even observe that performance increases as the sentence length increases on es and fr. Third, across all languages, the system performs relatively bad at predicting long entities (eLen) and entities belonging to the organization class (tag). Finally, the system is good at predicting sentences with fewer entities based on the attribute for entity density (eDen). XLM-R In Table 28, we observe that the influence of some attributes such as sLen,eLen, eDen with respect to the system performance are similar between ERNIE-M and XLM-R, although ERNIE-M performs significantly better than XLM-R at generalizing its predictions on es, fr. Table 29 shows the pairwise system analysis of ERNIE-M and T-URLv2 for the XQuAD task. We find that although the overall performance of T-URLv2 outperforms ERNIE-M, it is surpassed by ERNIE-M on a few buckets. For example, in zh, ERNIE-M is better at dealing with samples that have long answers, long questions, and a high lexical overlap between questions and answers. In ru, ERNIE-M is better at dealing with samples with long answers, long questions, and lower lexical overlap between questions and answers, questions and contexts. Figure 5 shows the interface of EXPLAINABOARD containing possible selection options to observe the fine-grained analysis for submitted systems on XTREME. We also demonstrate how to perform Single System and Pair Systems analysis on Figure 6 and 7 respectively. Specifically, to generate a fine-grained overview, we first select models in the table, then click one of the three Analysis Buttons, which generates a fine-grained analysis such as in Figure 6 (single system analysis) and Figure 7 (pair-wise system analysis).  the table represents the performance of a system on a specific dataset and a specific language. Relevant pieces of information such as the paper title also are provided. Figure 6: Single system analysis. Each histogram represents the fine-grained evaluation results of a given system, which are broken down based on a pre-defined attribute (e.g., answer length ).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Figure 7: Pairwise system analysis. Each histogram illustrates the fine-grained performance gap between system 1 and system 2, which has been broken down by different pre-defined attributes (e.g., answer length).       Table 13: Metadata for the current submissions to XTREME. Note that monolingual pre-training data is reported in either number of tokens or size of the data (in GB/TB). The amount of parallel data is reported in either number of pairs or size of the data (in GB/TB).       , who (C), when (D) and which (E), ranked by their frequencies in training set. In the pairwise system diagnosis histogram, blue (red) x ticklabels represents the bucket value of a specific attribute on which system M1 surpasses (under-performs) M2 by the largest margin that is illustrated by a blue (red) bin. The blueonly x ticklabels (e.g., -D) indicate that M1 outperforms M2 in all categories of an attribute.
Multilingual CheckList: Generation and Evaluation
Multilingual transformer based models (Devlin et al., 2019;Conneau et al., 2020;Xue et al., 2021) have demonstrated commendable zero & few-shot capabilities. Their performance is typically evaluated on benchmarks like XNLI (Conneau et al., 2018), XGLUE (Liang et al., 2020), XTREME (Hu et al., 2020b) & XTREME-R (Ruder et al., 2021). However, this evaluation paradigm has a number of limitations including: First, most of these datasets are limited to a few high resource languages (Hu et al., 2020a;Wang et al., 2020;Vulić et al., 2020), except for a few tasks (e.g., NER, POS (Ahuja et al., 2022;Bhatt et al., 2021a)). Second, creating high quality test sets of substantial size for many tasks and languages is prohibitively expensive. Third, state-of-art models are known to learn spurious patterns to achieve high accuracies, saturating performance on these test-benches, yet performing poorly on often much simpler real world cases (Goyal et al., 2017;Gururangan et al., 2018;Glockner et al., 2018;Tsuchiya, 2018;Geva et al., 2019). Fourth, these benchmarks do not evaluate models for language specific nuances (Ribeiro et al., 2020). Lastly, this evaluation approach does not provide any insights into where the model is failing (Wu et al., 2019). These limitations lead to the need of interactive, challenging, and much larger testing datasets (like (Srivastava et al., 2022;Kiela et al., 2021)) and more holistic approaches to evaluation (like Ribeiro et al. (2020)).
Multilingual CheckList: Generation and Evaluation
CheckList (Ribeiro et al., 2020) is an evaluation paradigm that systematically tests the various (linguistic) capabilities required to solve a task. It allows creation of large and targeted test sets easily using various abstractions. Specifically, users can generate templates, essentially sentences with slots that can be filled in with a dictionary of lexicons to generate test instances. CheckList templates are created by native speakers. Ruder et al. (2021) introduce Multilingual Checklists created by human translation from English CheckList for 50 languages for a subset of tests on Question Answering. However, since CheckLists are task & language specific, human creation or translation of CheckLists remains extremely resource-intensive.
Multilingual CheckList: Generation and Evaluation
In this paper, we introduce an automatic approach to creating Multilingual CheckLists. We devise the Template Extraction Algorithm (TEA) for extracting templates in a target language from the translated instances of a source language CheckList (here English) automatically ( §2). We also experiment with semi-automatic and manual approaches for Multilingual CheckList creation ( §3). In the semi-automatic approach (TEA-ver), we ask human annotators to verify and correct the templates created by TEA. In the manual approach, we ask annotators to create CheckLists in two ways: first, by translation of English CheckList to the target language (t9n) (same as Ruder et al. (2021)); Second, by giving a description of the task and capabilities to create CheckLists from scratch (SCR) (same as original English CheckLists creation (Ribeiro et al., 2020)).Using these four approaches, we create CheckLists for Sentiment Analysis (SA) and Natural Language Inference (NLI) in Hindi ( §5). We demonstrate broad applicability of TEA by generating CheckLists in additional 9 typologically diverse languages (Gujarati, French, Swahili, Arabic, German, Spanish, Russian, Vietnamese, Japanese) and TEA-ver CheckLists in 3 of them ( §6).
Multilingual CheckList: Generation and Evaluation
Evaluation of CheckLists is non-trivial. For thorough comparisons, we propose evaluation metrics along four axes: utility, diversity, cost & correctness ( §4). Our evaluation indicates that CheckLists created using TEA are not only cost-effective but also useful and diverse, with comparable quality to the manually and semi-automatically created CheckLists. Experiments on typologically diverse languages show that TEA CheckLists provide a good estimate of the failures of the model, and thus can be used even in the absence of resources to verify them or create human-annotated gold test-sets.
Multilingual CheckList: Generation and Evaluation
To summarize, our contributions are: a) We propose TEA (Template Extraction Algorithm) to extract templates in a target language using translated instances of a source CheckList. b) We experiment with varying degrees of human intervention, comparing semi-automatic & manual approaches of Multilingual CheckList creation with TEA, to understand the best utilization of the human effort. c) We introduce evaluation metrics along the axes of utility, diversity, cost, and correctness for indepth comparison of the the CheckLists. d) We will release all the 4 CheckLists in Hindi for SA and NLI, TEA CheckLists in 9 languages for SA and TEA-ver CheckLists in 3 languages for SA.
Multilingual CheckList: Generation and Evaluation
We release the code of TEA and the CheckLists created at aka.ms/multilingualchecklist 2 TEA: Template Extraction Algorithm Terminology (consistent with Ribeiro et al. (2020)): Linguistic capabilities: These are capabilities tested for a particular task. For e.g, negation.
Multilingual CheckList: Generation and Evaluation
Templates: These are sentences with slots. For e.g, '{CITY} is beautiful'. Here, '{CITY}' is a slot. Templates can have any number of slots. Lexicon keys and values: This a dictionary of values. In the above example, 'CITY' is the key. Values are the words that would be filled in the slots (replacing the keys) like 'New Delhi', 'New York', 'London', etc. We use the notation 'CITY = ['New Delhi', 'London', 'New York'] ' for lexicons. Instances: These are test sentences created by inserting lexicon values in templates . In the above example, the instances formed are: 'New York is beautiful', 'London is beautiful', etc.
Multilingual CheckList: Generation and Evaluation
The CheckList paradigm allows creation of large number of test instances. For multilingual evaluation, these can then be translated to the target languages using Machine Translation. However, there are limitations to this approach. Firstly, a large machine translated test set is difficult to be verified by humans, as one would have to go through every example. Second, it defeats the purpose of abstraction that CheckLists facilitates. And third, the quality of this test set will be directly impacted by the quality of the MT system. This results in the need to generate templates in the target language so that these can be utilized and verified in the same fashion as the template sets in the source language.
Multilingual CheckList: Generation and Evaluation
Our early experiments suggested that due to word order and syntactic differences between languages, both: 1) a word-to-word or heuristic translation of the template and 2) extraction of template from a single source instance (such as by simply replacing one word with other in a single translated instance) do not work well for template translation. This necessities a non-trivial algorithm that can extract templates given a set of instances.
Multilingual CheckList: Generation and Evaluation
We propose the Template Extraction Algorithm or TEA, to automatically extract template sets given an input a set of instances. In this paper, these input instances for TEA are obtained by machine translating instances created from the source CheckList template sets. We use machine translation to reduce cost and human effort, but the algorithm can be used with any input set of instances, i.e it would work with human-translated instances.
Multilingual CheckList: Generation and Evaluation
Briefly, TEA is a recursive approach to extract templates from input instances by treating every input instance as directed acyclic graph of the words. TEA combines the instances with similar structure into a single template by recursively merging instances and replacing terminals (or lexicon values) with non-terminals (or lexicon keys). Figure 1 shows how TEA creates lexicon keys by combining instances from the translated instances. We assume English (EN) to be the source language and Hindi (HI) to be the target language. The pipeline starts with an EN template, the instances are created by replacing the lexicon values in the templates, that are then machine translated to get HI instances. These instances function as input to TEA which then recursively groups instances using non-terminals to form templates. The entire process of template extraction is repeated for every EN template, resulting in the HI template set. The TEA has 3 steps which we describe as follows (pseudocode and details are in appendix A):
Multilingual CheckList: Generation and Evaluation
Step 1: Grouping Terminals into Non-Terminals: First, we convert the Hi instances into a directed acyclic graph whose nodes are unique words (or tokens). There is an edge from node A to B if word B follows word A in at least one of the input instances. In this directed graph (see Fig. 1), between any two nodes, if there are multiple paths of length less than equal to k + 1 (we set k to 2), we concatenate the intermediate words in the path (with space in between them) and treat them as terminals. This set of terminals, between the two nodes, are grouped together represented by a nonterminal symbol (for example Key_1 and Key_2 in Fig. 1). This step corresponds to lexicon formation; the non-terminal extracted here are essentially keys of the lexicon & the terminals constituting them are the lexicon values for the slots of a template.
Multilingual CheckList: Generation and Evaluation
Step 2: Template Extraction Using a set of Hi instances, S = {s 1 , s 2 , . . . s N }, and all non-
Multilingual CheckList: Generation and Evaluation
where w ij are terminals (obtained step 1), TEA outputs a set of tem-platesT = {t 1 , t 2 , ...} such thatT can generate all the examples in S using the given non-terminal and their corresponding terminals. For each sentence s i , we generate a set of candidate templates, T i = {t i1 , t i2 , ...}, such that s i belongs to the set of examples generated by each t ij . To find the minimal template set, i.eT that covers all examples is treated as a set cover problem and we use a greedy approximation to find this set.
Multilingual CheckList: Generation and Evaluation
Step 3: Combine Steps 1 and 2 The above template extraction process, while resulting in correct outputs, may be computationally expensive due to translation noise 1 and its time-complexity which is exponential on the number of non-terminals. To mitigate this, we follow an iterative approach where instead of using all the extracted non-terminals (along with their terminals), we initialize the set of non-terminals with an empty set and iteratively add the most useful non-terminals (with their cor-responding terminals) to this set. Note that, TEA can generate multiple templates for the set of instances (all of which might be generated from a single source template). This design is intentional and desirable as due to morphosyntactic complexities (e.g, grammatical gender), it is likely that all instances in a target language will not fit into a single template.
Multilingual CheckList: Generation and Evaluation
We now describe the various ways in which multilingual checklists can be generated, ranging from fully automatic to fully manual approaches.
Multilingual CheckList: Generation and Evaluation
Using TEA We start with a source language (En) CheckList template and generate instances by replacing lexicon values in templates. These instances are translated using an MT system. The translated instances now serve as the input to the TEA and target language (HI) CheckList template is extracted. The process ( Fig. 1) is repeated for all En templates to form the complete HI CheckList.
Multilingual CheckList: Generation and Evaluation
TEA with Verification (TEA-ver) This is a semi-automatic approach, where we ask a humanverifier to verify and correct the CheckLists generated using TEA. The verifiers (or annotators) are provided with a set of templates and lexicons generated using the TEA pipeline, along with the original source langauge CheckList and description of the capabilities. The annotators are instructed to verify the target language templates for (grammatical) correctness. They can delete or edit the incorrect templates. They can also add any missing templates that they think are significantly important (cover too many missed instances).
Multilingual CheckList: Generation and Evaluation
Translating source CheckList (t9n) This is a completely manual approach, but relies on a source language (here, En) CheckList. The annotators are provided with the En templates, lexicons and the descriptions of capabilities. They are tasked to translate the templates and lexicons into the target language. If a source template cannot be translated to a single target template (such as due to divergent grammatical agreement patterns), annotators are instructed to include as many variants as necessary. This approach is same as that used by Ruder et al. (2021) to create multilingual CheckList.
Multilingual CheckList: Generation and Evaluation
Generating CheckList from scratch (SCR) This is a completely manual approach of creating CheckLists from scratch, not relying on any source CheckList. Here, the CheckList templates are generated in the same manner as generated in by humans in Ribeiro et al. (2020). That is, human annotators are provided with a description of the task and capabilities and are instructed to develop the templates and lexicon, directly in the target language. In our pilot we found that users were better able to understand the capabilities with some examples as opposed to only from the description, so we also provided them with a couple of examples, in English, for each capability.
Multilingual CheckList: Generation and Evaluation
Comparison of CheckLists is non-trivial. Firstly, CheckLists cannot be evaluated using absolute metrics, comparisons can only be relative (Bhatt et al., 2021b). Further, the question of what constitutes a better CheckList can be answered in multiple ways. For example, if a CheckList A can help discover (and/or fix) more bugs than CheckList B, CheckList A could be more useful. On the other hand, variability of instances may be desirable. If CheckList B generates more diverse instances as compared CheckList A, even though it discovers less bugs, B could be considered better as it allows testing of the system on a broader variety of instances. Finally, in practical scenarios, cost and correctness are both important factors for generating the CheckList.
Multilingual CheckList: Generation and Evaluation
We thus propose evaluation metrics along 4 dimensions: a) utility for discovery and fixing bugs; b) diversity in the generated instances; c) cost of generating templates. d) correctness of templates.
Multilingual CheckList: Generation and Evaluation
Failure Rate (FR) Here, we measure the percentage of instances generated by the CheckList that the model failed on averaged over all the capabilities. 2 The numbers are reported for XLM-R fine-tuned with English task data from standard datasets (SST-2 for SA and mNLI for NLI). Effectively, we measure the FR on zero-shot transfer from English to the target language. For FR, the higher the value the better the CheckList.
Multilingual CheckList: Generation and Evaluation
Augmentation Utility (Aug) These metrics aims to test the utility of CheckList in fixing failures using data augmentation following Bhatt et al. (2021b). This is done in two ways:
Multilingual CheckList: Generation and Evaluation
(a) From Scratch (Aug-0): Here, we fine-tune XLM-R directly using CheckList instances.
Multilingual CheckList: Generation and Evaluation
(b) On Fine-tuned model (Aug-CFT): Here, XLM-R is first fine-tuned with English task data (SST-2 for SA and mNLI for NLI) and then further continually fine-tuned using CheckList instances.
Multilingual CheckList: Generation and Evaluation
In both cases, we first generate all instances using the CheckLists being compared. We retain a maximum of 10k instances per capability for each CheckList. The instances are then randomly split into train and test sets in 70:30 ratio. The training data (of the corresponding CheckList) is used for the augmentation as described above. The test sets, generated from all the CheckLists being compared are combined together to form a common test set and accuracy on this set is reported. Intuitively, this aims to determine the utility of the CheckList's instances for fixing failures using augmentation. For both the Augmentation metrics, higher is better.
Multilingual CheckList: Generation and Evaluation
Number of templates (#temp) and lexicon values (#lexv) The simplest way to measure the diversity is the number of distinct templates and lexicon values (or terminals). Higher number of templates and lexicon values means more diversity.
Multilingual CheckList: Generation and Evaluation
To measure the diversity between the templates, we measure the BLEU score (or similarity) for every instance generated by a template with the the instances generated by all other templates in the CheckList . Since this score is sensitive to the number of templates in the Checklist, we normalize the score by the number of templates in the set. Lower CT-BLEU is indicative of better CheckList as it indicates more diverse instances from templates.
Multilingual CheckList: Generation and Evaluation
Time per template (TpT) We define the cost of creation of these Checklists simply as the human time required. Since different methods or users can create substantially different number of templates per capability, we measure the mean time taken (TpT) for creation from scratch (SCR), translation (t9n) and verification (TEA-ver) of a template as the measure of the cost. A better CheckList for practical purposes would have lower TpT.
Multilingual CheckList: Generation and Evaluation
Here, we assume that templates generated with any amount of human intervention (manual or semiautomatic) would always be correct. As a result, we calculate correctness only for TEA templates.
Multilingual CheckList: Generation and Evaluation
We define the correctness of TEA templates with respect to TEA-ver templates. This is because during creation of templates by the TEA-ver process annotators correct or remove templates. Thus, only correct TEA template are left unedited. Therefore, in order to estimate the correctness of the TEA templates, we compute the following two metrics.
Multilingual CheckList: Generation and Evaluation
Failure Rate Difference (FR-diff) It is possible that the model fails in some cases if the input instance is not well-formed. As a result, the difference between the failure rates induced by TEA-ver templates (which always lead to well-formed instances) and that of TEA templates (which could lead to some ungrammatical instances) will give an estimate to the correctness of TEA templates. As a result, we define this metric as simply the difference between the FR of TEA and TEA-ver.
Multilingual CheckList: Generation and Evaluation
Precision and Recall (P/R) Since during the TEA-ver process, annotators edit or remove incorrect templates, only the correct templates that were generated by TEA are left as is. Therefore, in order to estimate the correctness of the TEA, we compute the precision and recall of the TEA template set, with respect to TEA-ver template set. We define match when the templates are same and the lexicon values of either one is a subset of the other, implying they will generate similar set of examples.
Multilingual CheckList: Generation and Evaluation
We start with Hindi (Hi) as the target language, create CheckLists using all 4 methods from §3 and evaluate them using the metrics from §4. Hi has significant syntactic divergence from the source language (here English (En)) and uses a different script. Hi is a mid-resource language with reasonably good publicly available En-Hi MT systems. We argue that if TEA works well in the En-Hi pipeline, it would also work for most other high to mid resource languages with reasonable MT systems and similar or less syntactic divergence from En, which we also substantiate by performing additional multilingual experiments in §6.
Multilingual CheckList: Generation and Evaluation
We create and evaluate Hi CheckLists for 2 tasks, Sentiment Analysis (SA) and Natural Language Inference (NLI). For SA, we choose 5 capabilities namely Vocabulary, Negation, Temporal, Semantic Role Labeling and Relational, and their associated Minimum Functionality Test (MFT) templates from Ribeiro et al. (2020) as our source Check-List. For NLI, we choose co-reference resolution, spatial, conditional, comparative and causal reasoning as capabilities and their associated templates from Tarunesh et al. (2021). We refer readers to Appendix B for details about these capabilities.
Multilingual CheckList: Generation and Evaluation
Following Ribeiro et al. (2020), we chose 6 software developers as our annotators, who are knowledgeable in NLP. All users are native speakers of Hi and have near-native En fluency. 3 We expect developers to be the actual users of the approach, as it is usually a developer's job to find and fix bugs. The annotators were given a detailed description of expectations along with examples (both in En and Hi). Furthermore, during our pilot study, we found some of the common errors users make, and to mitigate those we provided a list of common errors illustrated with simple examples.
Multilingual CheckList: Generation and Evaluation
Each of the 6 annotators was randomly assigned a CheckList creation approach that requires human intervention. Thus, we had 2 annotators each for the SCR, t9n and TEA-Ver setups. They carry out the process independently for both SA and NLI. The same description of capabilities and examples are used for all the experimental setups. Similarly, the same source templates and lexicons are used for t9n, TEA-ver and TEA. For the TEA pipeline, we used Bing Translator API for translating En instances to Hi. While reporting the results, we report the average metrics of both annotators. Table 1 reports the metrics ( §4) for the 4 methods.
Multilingual CheckList: Generation and Evaluation
The trends for cost or TpT are consistent with expectations. Creating CheckLists from Scratch (SCR) takes the most time, as the user has to think and create the templates. t9n requires manual translation and is quicker than SCR but slower than TEA-ver, which just requires verification and correction on templates generated by TEA. We do not factor in the time required to create the source En Checklist, because 1) It is common to all of these 4 approaches and sourced from existing literature; and 2) it is a one-time effort which can be reused for generation of CheckLists in many target languages, leading to a very low amortized cost.
Multilingual CheckList: Generation and Evaluation
In diversity metrics TEA generates the most diverse templates, closely followed by TEA-Ver. t9n is much less diverse, and SCR has the least diversity. We found that, the users created very few 3 Educated for 15+ years in English templates for SCR, perhaps because it is difficult to decide what would be a good number of templates. We also observe that TEA generates a largest number of templates. The source checklists had 32 (74) and 18 (76) templates (lexicon values) for SA and NLI, respectively. Thus on average, a source template generates around 3 target templates, which is primarily due to syntactic divergence between the En and Hi. These numbers are reduced in TEA-ver, most likely because not all of the TEA templates are perfect and human annotators merge or delete some of them during the verification.
Multilingual CheckList: Generation and Evaluation
The trends in utility metrics are varied. In SA, TEA-ver templates induce highest FR and TEA is a close second. However, for NLI, SCR Check-List induces the highest failure, followed by t9n. This might be due to the task complexity. We leave further exploration on the co-relation of task complexity and efficacy of TEA to future work. TEA has the highest Aug-0 and Aug-CFT values except one case where it is a close second, indicating that the instances generated by TEA CheckLists are effective in fixing failure by augmentation. TEA-ver has values that close to TEA for these metrics 4 .
Multilingual CheckList: Generation and Evaluation
In terms of correctness, based on P/R of TEA with respect to TEA-ver, we find that that around a third of the TEA templates had to be significantly edited or removed. Despite this, from FR-diff, we see that the FR generated by TEA is fairly close to the FR generated by TEA-ver. Additionally, even the numbers of other utility metrics are also comparable. This indicates that even the unverified templates (from TEA) which may generate some ungrammatical instances, can give very close estimates of the failure rates and augmentation accuracy to human-verified template sets. This is a positive finding, because while TEA-ver is more reliable, but when resources to get TEA templates verified are not available, despite imperfections, TEA CheckLists can be used for evaluation.
Multilingual CheckList: Generation and Evaluation
Finally, we would like to point out some of the qualitative differences that we saw in the Check-Lists created by these different methods which are hard to articulate through metrics. In particular, we saw that CheckLists created from scratch tend to capture cultural context better. For example, annotators use Indian names in the lexicon values as opposed to western names that get generated due to translations in all other 3 approaches. However, while difficult for TEA, this entity recontextualization is fairly easy for the other two approaches where humans are involved. We also find that the template sets of TEA-ver and t9n are overlapping. This is because of the setup, where t9n is directly translated at template level and TEA-ver is obtained after correcting the templates obtained from translated instances. The major difference occurs in the amount of time taken as correcting templates is faster than translating them. Thus overall, we conclude that TEA followed by human verification, or TEA-ver would be an ideal approach for scaling CheckList evaluation to multiple languages. That said, the fully automatic TEA approach is even more cost-effective and almost equally reliable to the TEA-ver approach, making it suitable for large-scale multilingual CheckList generation with extremely limited resources.
Multilingual CheckList: Generation and Evaluation
So far, we see that TEA is cost-efficient in producing effective Hi CheckLists. We now experiment with 9 more typologically diverse languages -Arabic, French, German, Gujarati, Japanese, Russian, Spanish, Swahili and Vietnamese to evaluate the efficacy of scaling TEA to may languages. We use TEA to automatically generate CheckLists across these languages from the same set of source templates in English for SA across 6 capabilities: Vocabulary, Temporal, Fairness, Negation, Semantic role labeling (SRL) and Robustness. We use the same source En CheckList from the Ribeiro et al. (2020) and use Bing Translator in the TEA pipeline to translate En instances to the target language.
Multilingual CheckList: Generation and Evaluation
In Table 2 we report the FR on XLM-R model fine-tuned with SST-2 data; thus, except for En, all other values are for zero-shot transfer to the respective language. The average FR for AMCG is highest for Swahili (59%), Vietnamese and Gujarati (around 52%), and lowest for French (43%), Spanish and German (around 45%). For English, average FR is 41%. These trends are consistent with expectation of performance as English, French, and other European languages are high-resourced while Swahili and Vietnamese are very low-resourced.
Multilingual CheckList: Generation and Evaluation
For 3 of the target languages, namely French, Gujarati and Swahili, native speakers verified the generated templates and thus, we also report the FR for TEA-ver. 5 We observe that the Pearson (Spearman) correlation between TEA and TEA-ver FR values for French, Gujarati and Swahili are 0.99 (1.0), 0.98 (0.89) and 0.97 (0.94) respectively. Furthermore, the difference between FR (FR-Diff) is also low. This implies, similar to our observations from section 5, that one can obtain an extremely accurate assessment of the capabilities of multilingual models just from TEA CheckLists even for low resource languages like Swahili. This re-affirms that despite noise, TEA is able to generate CheckLists that are useful without any human supervision.
Multilingual CheckList: Generation and Evaluation
In this paper, we introduced the TEA to generate target language CheckList (templates + lexicon) from the translated instances of source language CheckList. We show that with drastically reduced human effort required for creating CheckList in a new language, the TEA CheckLists provide an accurate estimate of the models' capabilities. However, some of the generated templates/lexicons are noisy and were removed or edited by humans through the TEA-ver process. In this section, we summarize the limitations, common error patterns  and suggest some possible ways to resolve them. Agnostic to Semantics TEA is agnostic of the semantics of the lexicon keys. So, when faced with a set of sentences: Las Vegas is good., New York is good., New Delhi is good. and Las Palmas is good., it is unclear whether it should design 1 template CITY is good. with lexicon CITY={Las Vegas, New York, New Delhi, Las Pal-mas} or 2 templates: Las CITY1 is good., CITY1={Vegas, Palmas} and New CITY2 is good., CITY2={York, Delhi}. This problem is hard to solve without heuristics. One possibility is to use the translation alignment information however, such alignments are often imperfect even for high-resource languages. We leave improvements to TEA for handling this to future work.
Multilingual CheckList: Generation and Evaluation
Handling Morphology Creating good templates for morphologically rich languages (Sinha et al., 2005;Dorr, 1994) is more challenging due to inflections. For e.g, in Hindi a verb may take different form for different tenses and gender. While TEA can handle such cases by creating multiple templates, but with still a third of Hi templates needed correcting. We leave morphologically informed CheckList creation to future work.
Multilingual CheckList: Generation and Evaluation
Translation Errors Translation errors are a frequent pattern, affecting the input target language instances. In some cases, due to the statistical nature of TEA, we are able to naturally filter out such erroneous templates. For e.g, for an En template 'I used to think this {air_noun} was {neg_adj}, {change} now I think it is {pos_adj}', translated Hi templates 'Mujhe lagta hai ki us {udaan} {ghatia} tha, ab mujhe lagta hai ki yeh asadharan hai' (correct) matches 187 translations, and 'Mujhe lagta hai ki us {udaan} {ghatia} tha karte the, ab mujhe lagta hai ki yeh bohut achha hai' (noisy) matches only 35. While TEA can remove some noisy patterns, errors due to misunderstood context are much harder to fix. For e.g 'the service is poor' translated as 'vah seva garib hai' but 'garib' in Hindi means "lacking sufficient money" and not "lower or insufficient standards". We leave comparisons of TEA for human v.s machine translated input instances and methods to measure and reduce the effect of translation errors on TEA to future work.
Multilingual CheckList: Generation and Evaluation
Metric Limitations Quantifying the quality of generated template and verification of the relevance of templates with respect to provided description is non-trivial . While we suggest a set of metrics quantifying utility, diversity and cost, these should be extended and further studied for efficacy across tasks and languages. Lastly, soundness and completeness of a template sets (or a test-suite in general) is another unexplored aspect in our current work and an important future direction of research. Furthermore, we acknowledge the limitation of Failure Rate as a metric in the sense that the model could also fail if an example is ungrammatical. In other words, FR is conditional to correctness of the CheckList. However, in our experimentation in both Hindi and other languages, we have found that the the difference between the FR of human verified TEA-ver and TEA is typically small (with a few exceptions) across languages. This means that high FR being caused due to ungrammatical instances here is unlikely. Thus, as stated before, the closeness of the FRs of TEA and TEA-ver points to the reliability of the TEA algorithm.
Multilingual CheckList: Generation and Evaluation
In this paper we proposed TEA (Template Extraction Algorithm) to automatically generate multilingual CheckLists in a target language without any human supervision ( §2). This algorithm recursively extracts templates and lexicon from an input set of instances by treating sentences as a directed acyclic graph of words and combining them.
Multilingual CheckList: Generation and Evaluation
We additionally experimented with 3 other approaches with varying degrees of human intervention, 2 manual and 1 semi-automatic for CheckList generation ( §3). For comparing these CheckLists, we introduced metrics along the dimensions of utility, diversity, cost and correctness ( §4).
Multilingual CheckList: Generation and Evaluation
We performed in-depth analysis of all the 4 methods, with varying degree of human interventions, to create CheckLists for Sentiment Analysis and NLI in Hindi ( §5). In addition to Hindi, we experimented with 9 more typologically diverse languages to demonstrate the efficacy of TEA along with comparison with human-verified CheckLists in 3 of them ( §3). We found that TEA is costeffective, useful, and diverse in the CheckLists that it generates. While around one-third of the TEA templates required correction by humans, making the semi-automatic approach more reliable, we find that the model performance estimates provided by unverified CheckLists are very close to that of the human-verified (or semi-automatically created) CheckLists and are also significantly correlated to it. We also substantiated the finding of TEA being effective as well as reliable in the other languages.
Multilingual CheckList: Generation and Evaluation
Our overall recommendation is that TEA followed by human verification is the most reliable and cost-effective way to scale CheckList evalu-ation to multiple languages. But in case of very limited resources, TEA is still good enough to test system performance. We end with a discussion on the limitations of this work and propose directions that will, hopefully, inspire research in scaling and improving multilingual evaluation using Check-Lists. Finally, we note that TEA is general purpose algorithm of template extraction that can be used for other template-based evaluations such as bias evaluation (Webster et al., 2020;Bhatt et al., 2022)  Here, the keywords (CITY-0,CITY-1) are the non-terminals and their corresponding lexicons are the terminal symbols. Also, CITY-1 should be different than CITY-0; and hence the non-terminal symbols cannot be replaced independently of each other, establishing the context-sensitive nature of templates. This is a why we need to look beyond probabilistic context free grammar induction to learn the templates.
Multilingual CheckList: Generation and Evaluation
We use terminal and non-terminal to denote lexicons and keywords respectively. In a template, if the non-terminals are appended with cardinals from 0 to k, then they can not be replaced with same terminal while generating sentences. Also, if a template contains an instance of a non-terminal with cardinal k, (k > 0) then at least one instance of the same non-terminal with cardinal k − 1 should have occurred before its occurrence in the template.
Multilingual CheckList: Generation and Evaluation
We first briefly recap the pipeline of TEA for ease of exposition. First we discuss our approach to extract potential set of terminal words, i.e., we group a set of words (terminals) and give them a symbol/name (non-terminal). Then we extract the templates using the terminals and non-terminals that are extracted in previous step. Towards the end of this section, we briefly discuss the scalability issues and the approximations that we used to make it more scalable.
Multilingual CheckList: Generation and Evaluation
First, we convert the given Hi examples into a directed graph whose nodes are unique words (or tokens, if we use a different tokenizer) from the examples and there is an edge from word A to word B if word B follows word A in at least one of the examples. In this directed graph (as shown in Fig. 1), between any two nodes, if there are multiple paths of length less than equal to k+1, we group all those paths and give the group a name or a non-terminal symbol (for example Key_1 and Key_2 in Fig. 1). 6 By grouping the paths, we meant to concatenate the intermediate words in the path (with space in between them) and then to group the concatenated RENAME-NONTERMINAL-CARDINALS This function renames the cardinals to make sure that an instance of a non-terminal with cardinal k − 1 occurs before the instance of that non-terminal with cardinal k, (k > 0). For example, after re-naming the cardinals, the above three templates become the following three, respectively. (1) "#CITY-0 is beautiful. CITY-1 is cold. Paris is bigger." (2) "#Paris is beautiful. CITY-0 is cold. CITY-1 is bigger." (3) "#CITY-0 is beautiful. CITY-1 is cold. CITY-0 is bigger."
Multilingual CheckList: Generation and Evaluation
First, we find all the potential terminals and nonterminals (using § A.2.1) for all Hi examples, and then use them to extract template following the algorithm outlined in § A.2.2. While this simple procedure is possible, it is often computationally expensive; one of the reasons is that due to noise (many of the translated sentences may not fit into a template), the algorithm to extract terminals and non-terminals ( § A.2.1) often gives a lot of different non-terminals that share many common terminals. For example, we may get two non-terminals with their corresponding terminals such as "{Paris, New York, Delhi}" and "{London, New York, Delhi}". Moreover, the complexity of the algorithm in § 1 to extract templates can be increased exponentially with the number of non-terminals. To mitigate this problem, we follow an iterative approach where instead of using all the extracted non-terminals (along with their terminals), we initialize the set of non-terminals with an empty set and iteratively add the most useful non-terminals (with their corresponding terminals) to the existing set of nonterminals. Vocabulary This capability tests whether the model can appropriately handle the impact of words with different parts of speech on the task. In particular, sentences with neutral adjectives are expected to have a neutral prediction and sentences sentiment-laden (positive or negative) adjectives are expected to have the corresponding label. For example, "This is a private (NEUTRAL_ADJ) aircraft" should be labelled neutral; and "This is a great (POSITIVE_ADJ) aircraft" "This is a bad (NEGATIVE_ADJ) aircraft" should be labelled positive and negative respectively.
Multilingual CheckList: Generation and Evaluation
Negation This capability tests that the negation of a positive adjective in the sentence should be labelled as positive or neutral, for example: "This is not a great (POSITIVE_ADJ) aircraft" should be labelled negative or neutral. Similarly, sentence with negation of negative adjective should be positive our neutral and those with negation of neutral adjectives should remain neutral.
Multilingual CheckList: Generation and Evaluation
Semantic Role Labeling (SRL) SRL aims to test that the model understands the agent, object etc in an instance. That is sentiment of the correct role in the instance is parsed. Here, there are two distinct capabilities MFTs. The first one is to test that the sentiment author sentiment is given more importance than of sentiment of others. For example, "Some people think this aircraft is bad, but I thought it was great (POSITIVE_ADJ)" should be labelled as Positive. The second test is related to parsing yes/no questions with the correct sentiment. For example, "Do I think this aircraft is great? Yes" should be labelled as positive, whereas if the answer was No, it should be negative.
Multilingual CheckList: Generation and Evaluation
Temporal This capability is used to test whether the model understands the sequence of events correctly. In other words that the most recent sentiment is correctly parsed in labelling. For example, "I used to hate this aircraft, but now I love it" should be labelled positive.
Multilingual CheckList: Generation and Evaluation
Robustness There are two tests for robustness: First changing of values within semantically equivalent classes should not change the prediction. For example, "I flew in from Delhi" and "I flew in from New York" should have the same label as the change here is within the semantically equivalent class of 'CITY'. Secondly, typos (or random character exchange) should not flip labels. For example, "This is a graet aircraft" should still remain positive.
Multilingual CheckList: Generation and Evaluation
Fairness Fairness is used to test that prediction should be the same for various adjectives within a protected class. For example, "Mary is a black (RACE) woman" and "Mary is a white (RACE) woman" should have same sentiment prediction.
Multilingual CheckList: Generation and Evaluation
We use the template sets from Tarunesh et al. (2021) which in turn rely on the taxonomy of capabilities from (Joshi et al., 2020) for their selection of capabilities. In examples that follow, P stands for Premise and H for hypothesis.
Multilingual CheckList: Generation and Evaluation
Co-reference resolution Test the model for resolving pronouns between the premise and hypothesis correctly. For example, P: Angelique and Ricardo are colleagues. He is a minister and she is a model. H: Angelique is a model. Here H should 'entail' P.
Multilingual CheckList: Generation and Evaluation
Spatial reasoning Tests the model for reasoning using spatial properties. For example, P: Manchester is 67 miles from Pittsburg and 27 miles from Kansas. H: Manchester is nearer to Kansas than Pittsburg. Here H should 'entail' P.
Multilingual CheckList: Generation and Evaluation
Causal reasoning Tests the model for using causation in the premise to infer the hypothesis. For example, P: Katherine taught science to Nancy. H: Nancy learnt science from Katherine. Here H should 'entail' P.
Multilingual CheckList: Generation and Evaluation
Conditional reasoning Tests the model for logically inferring the hypothesis given conditional premise. For example, P: If the baby is fed on time, he does not get cranky. H: The baby gets crancky when he is hungry. Here H should 'entail' P.
Multilingual CheckList: Generation and Evaluation
Comparative reasoning Tests models for reasoning involving comparisons of objects. For example, P: The earth is larger than the moon but smaller than sun. H: The moon is smaller than sun.
Multilingual CheckList: Generation and Evaluation
Here H should 'entail' P.