What are the different approaches for extending existing LLMs to languages with limited resources?
Recent research has attempted to extend existing LLMs to languages with limited resources. Wang et al. (2019) propose vocabulary extension; Ebrahimi and Kann (2021) investigate adaptation methods, including MLM and Translation Language Model (TLM) objectives and adapters; Alabi et al. (2022) adapt XLM-R to 17 African languages; Wang et al. (2022) expand language models to low-resource languages using bilingual lexicons.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Pretrained multilingual models (PMMs) enable zero-shot learning via cross-lingual transfer, performing best for languages seen during pretraining. While methods exist to improve performance for unseen languages, they have almost exclusively been evaluated using amounts of raw text only available for a small fraction of the world’s languages. In this paper, we evaluate the performance of existing methods to adapt PMMs to new languages using a resource available for close to 1600 languages: the New Testament. This is challenging for two reasons: (1) the small corpus size, and (2) the narrow domain. While performance drops for all approaches, we surprisingly still see gains of up to 17.69% accuracy for part-of-speech tagging and 6.29 F1 for NER on average over all languages as compared to XLM-R. Another unexpected finding is that continued pretraining, the simplest approach, performs best. Finally, we perform a case study to disentangle the effects of domain and size and to shed light on the influence of the finetuning source language.
[{"first":"Abteen","last":"Ebrahimi","middle":[]},{"first":"Katharina","last":"Kann","middle":[]}]
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Multilingual pre-trained language models (PLMs) have demonstrated impressive performance on several downstream tasks for both high-resourced and low-resourced languages. However, there is still a large performance drop for languages unseen during pre-training, especially African languages. One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) — fine-tuning a multilingual PLM on monolingual texts of a language using the pre-training objective. However, adapting to target language individually takes large disk space and limits the cross-lingual transfer abilities of the resulting models because they have been specialized for a single language. In this paper, we perform multilingual adaptive fine-tuning on 17 most-resourced African languages and three other high-resource languages widely spoken on the African continent to encourage cross-lingual transfer learning. To further specialize the multilingual PLM, we removed vocabulary tokens from the embedding layer that corresponds to non-African writing scripts before MAFT, thus reducing the model size by around 50%. Our evaluation on two multilingual PLMs (AfriBERTa and XLM-R) and three NLP tasks (NER, news topic classification, and sentiment classification) shows that our approach is competitive to applying LAFT on individual languages while requiring significantly less disk space. Additionally, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods.
[{"first":"Jesujoba O.","last":"Alabi","middle":[]},{"first":"David Ifeoluwa","last":"Adelani","middle":[]},{"first":"Marius","last":"Mosbach","middle":[]},{"first":"Dietrich","last":"Klakow","middle":[]}]
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Recently, pre-trained language models have achieved remarkable success in a broad range of natural language processing tasks. However, in multilingual setting, it is extremely resource-consuming to pre-train a deep language model over large-scale corpora for each language. Instead of exhaustively pre-training monolingual language models independently, an alternative solution is to pre-train a powerful multilingual deep language model over large-scale corpora in hundreds of languages. However, the vocabulary size for each language in such a model is relatively small, especially for low-resource languages. This limitation inevitably hinders the performance of these multilingual models on tasks such as sequence labeling, wherein in-depth token-level or sentence-level understanding is essential. In this paper, inspired by previous methods designed for monolingual settings, we investigate two approaches (i.e., joint mapping and mixture mapping) based on a pre-trained multilingual model BERT for addressing the out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech tagging, named entity recognition, machine translation quality estimation, and machine reading comprehension. Experimental results show that using mixture mapping is more promising. To the best of our knowledge, this is the first work that attempts to address and discuss the OOV issue in multilingual settings.
[{"first":"Hai","last":"Wang","middle":[]},{"first":"Dian","last":"Yu","middle":[]},{"first":"Kai","last":"Sun","middle":[]},{"first":"Janshu","last":"Chen","middle":[]},{"first":"Dong","last":"Yu","middle":[]}]
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a target language. Thus, the majority of the world’s languages cannot benefit from recent progress in NLP as they have no or limited textual data. To expand possibilities of using NLP technology in these under-represented languages, we systematically study strategies that relax the reliance on conventional language resources through the use of bilingual lexicons, an alternative resource with much better language coverage. We analyze different strategies to synthesize textual or labeled data using lexicons, and how this data can be combined with monolingual or parallel text when available. For 19 under-represented languages across 3 tasks, our methods lead to consistent improvements of up to 5 and 15 points with and without extra monolingual text respectively. Overall, our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology.
[{"first":"Xinyi","last":"Wang","middle":[]},{"first":"Sebastian","last":"Ruder","middle":[]},{"first":"Graham","last":"Neubig","middle":[]}]
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Pretrained multilingual models (PMMs) are a straightforward way to enable zero-shot learning via cross-lingual transfer, thus eliminating the need for labeled data for the target task and language. However, downstream performance is highest for languages that are well represented in the pretraining data or linguistically similar to a well represented language. Performance degrades as representation decreases, with languages not seen during pretraining generally having the worst performance.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
In the most extreme case, when a language's script is completely unknown to the model, zero-shot performance is effectively random. While multiple methods have been shown to improve the performance of transfer to underrep- resented languages (cf. Section 2.3), previous work has evaluated them using unlabeled data from sources available for a relatively small number of languages, such as Wikipedia or Common Crawl, which cover 316 1 and 160 2 languages, respectively. Due to this low coverage, the languages that would most benefit from these methods are precisely those which do not have the necessary amounts of monolingual data to implement them as-is. To enable the use of PMMs for truly low-resource languages, where they can, e.g., assist language documentation or revitalization, it is important to understand how state-of-the-art adaptation methods act in a setting more broadly applicable to many languages.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
In this paper, we ask the following question: Can we use the Bible -a resource available for roughly 1600 languages -to improve a PMM's zero-shot performance on an unseen target language? And, if so, what adaptation method works best? We investigate the performance of XLM-R (Conneau et al., 2020) when combined with continued pretraining (Chau et al., 2020), vocabulary extension, (Wang et al., 2020), and adapters (Pfeiffer et al., 2020b) making the following assumptions: (1) the only text available in a target language is the New Testament, and (2) no annotated training data exists in the target language.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
We present results on 2 downstream tasks -partof-speech (POS) tagging and named entity recognition (NER) -on a typologically diverse set of 30 languages, all of which are unseen during the pretraining of XLM-R. We find that, surprisingly, even though we use a small corpus from a narrow domain, most adaptation approaches improve over XLM-R's base performance, showing that the Bible is a valuable source of data for our purposes. We further observe that in our setting the simplest adaptation method, continued pretraining, performs best for both tasks, achieving gains of up to 17.69% accuracy for POS tagging, and 6.29 F1 for NER on average across languages.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Additionally, we seek to disentangle the effects of two aspects of our experiments on downstream performance: the selection of the source language, and the restricted domain of the New Testament. Towards this, we conduct a case study focusing on three languages with Cyrillic script: Bashkir, Chechen, and Chuvash. In order to understand the effect of the choice of source language, we use a more similar language, Russian, as our source of labeled data. To explore the effect of the New Testament's domain, we conduct our pretraining experiments with an equivalent amount of data sampled from the Wikipedia in each language. We find that changing the source language to Russian increases average baseline performance by 18.96 F1, and we achieve the highest results across all settings when using both Wikipedia and Russian data.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Prior to the introduction of PMMs, cross-lingual transfer was often based on word embeddings (Mikolov et al., 2013).  present monolingual embeddings for 294 languages using Wikipedia, succeeded by  who present embeddings for 157 languages trained on additional data from Common Crawl. For crosslingual transfer, monolingual embeddings can then be aligned using existing parallel resources, or in a completely unsupervised way (Bojanowski et al.,   2017; Artetxe et al., 2017;Artetxe et al., 2016). Although they use transformer based models, Artetxe et al. (2020) also transfer in a monolingual setting. Another method for cross-lingual transfer involves multilingual embeddings, where languages are jointly learned as opposed to being aligned (Ammar et al., 2016;Artetxe and Schwenk, 2019). For a more in-depth look at cross-lingual word embeddings, we refer the reader to Ruder et al. (2019). While the above works deal with generally improving cross-lingual representations, task-specific cross-lingual systems often show strong performance in a zero-shot setting. For POS tagging, in a similar setting to our work, Eskander et al. (2020) achieve strong zero-shot results by using unsupervised projection (Yarowsky et al., 2001) with aligned Bibles. Recent work for cross-lingual NER includes Mayhew et al. (2017) who use dictionary translations to create target-language training data, as well as Xie et al. (2018) who use a bilingual dictionary in addition to self-attention. Bharadwaj et al. (2016) use phoneme conversion to aid cross-lingual NER in a zero-shot setting. More recently, Bari et al. (2020) propose a model only using monolingual data for each language, and Qi et al. (2020) propose a language-agnostic toolkit supporting NER for 66 languages. In contrast to these works, we focus on the improvements offered by adaptation methods for pretrained models for general tasks.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
PMMs can be seen as the natural extension of multilingual embeddings to pretrained transformerbased models. mBERT was the first PMM, covering the 104 languages with the largest Wikipedias. It uses a 110k byte-pair encoding (BPE) vocabulary (Sennrich et al., 2016) and is pretrained on both a next sentence prediction and a masked language modeling (MLM) objective. Languages with smaller Wikipedias are upsampled and highly represented languages are downsampled. XLM is a PMM trained on 15 languages. XLM similarly trains on Wikipedia data, using a BPE vocabulary with 95k subwords and up-and downsamples languages similarly to mBERT. XLM also introduces translation language modeling (TLM), a supervised pretraining objective, where tokens are masked as for MLM, but parallel sentences are concatenated such that the model can rely on subwords in both languages for prediction. Finally, XLM-R is an improved version of XLM. Notable differences include the larger vocabulary of 250k subwords created using SentencePiece tokenization (Kudo and Richardson, 2018) and the training data, which is taken from CommonCrawl and is considerably more than for mBERT and XLM. XLM-R relies solely on MLM for pretraining and achieves stateof-the-art results on multiple benchmarks (Conneau et al., 2020). We therefore focus solely on XLM-R in our experiments.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Downstream Performance of PMMs While Pires et al. (2019) and Wu and Dredze (2019) show the strong zero-shot performance of mBERT, Wu and Dredze (2020) shine light on the difference in performance between well and poorly represented languages after finetuning on target-task data. Muller et al. (2020) observe varying zero-shot performance of mBERT on different languages not present in its pretraining data. They group them into 'easy' languages, on which mBERT performs well without any modification, 'medium' languages, on which mBERT performs well after additional pretraining on monolingual data, and 'hard' languages, on which mBERT's performs poorly even after modification. They additionally note the importance of script, finding that transliterating into Latin offers improvements for some languages. As transliteration involves language specific tools, we consider it out of scope for this work, and leave further investigation in how to best utilize transliteration for future work. Lauscher et al. (2020) focus on PMM finetuning, and find that for unseen languages, gathering labeled data for few-shot learning may be more effective than gathering large amounts of unlabeled data.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Additionally, Chau et al. (2020), Wang et al. (2020), and Pfeiffer et al. (2020b) present the adaptation methods whose performance we investigate here in a setting where only the Bible is available. We give a general overview of these methods in the remainder of this section, before describing their application in our experiments in Section 3.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Continued Pretraining In a monolingual setting, continued pretraining of a language representation model on an MLM objective has shown to help downstream performance on tasks involving text from a domain distant from the pretraining corpora (Gururangan et al., 2020). In a multilingual setting, it has been found that, given a target language, continued pretraining on monolingual data from that language can lead to improvements on downstream tasks (Chau et al., 2020;Muller et al., 2020).
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Vocabulary Extension Many pretrained models make use of a subword vocabulary, which strongly reduces the issue of out-of-vocabulary tokens. However, when the pretraining and targettask domains differ, important domain-specific words may be over-fragmented, which reduces performance. In the monolingual setting,  show that extending the vocabulary with indomain tokens yields performance gains. A similar result to that of continued pretraining holds in the multilingual setting: downstream performance of an underrepresented language benefits from additional tokens in the vocabulary, allowing for better representation of that language. Wang et al. (2020) find that extending the vocabulary of mBERT with new tokens and training on a monolingual corpus yields improvements for a target language, regardless of whether the language was seen or unseen. Chau et al. (2020) have similar results, and introduce tiered vocabulary augmentation, where new embeddings are learned with a higher learning rate. While both approaches start with a random initialization, they differ in the amount of new tokens added: Wang et al. (2020) limit new subwords to 30,000, while Chau et al. (2020) set a limit of 99, selecting the subwords which reduce the number of unknown tokens while keeping the subword-totoken ratio similar to the original vocabulary.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Adapters Adapters are layers with a small number of parameters, injected into models to help transfer learning (Rebuffi et al., 2017). Houlsby et al. (2019) demonstrate the effectiveness of taskspecific adapters in comparison to standard finetuning. Pfeiffer et al. (2020b) present invertible adapters and MAD-X, a framework utilizing them along with language and task adapters for crosslingual transfer. After freezing model weights, invertible and language adapters for each language, including English, are trained together using MLM. The English-specific adapters are then used along with a task adapter to learn from labeled English data. For zero-shot transfer, the invertible and language adapters are replaced with those trained on the target language, and the model is subsequently evaluated.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Unlabeled Data We use the Johns Hopkins University Bible Corpus (JHUBC) from McCarthy et al. (2020), which contains 1611 languages, providing verse-aligned translations of both the Old and New Testament. However, the New Testament is much more widely translated: 86% of translations do not include the Old Testament. We therefore limit our experiments to the New Testament, which accounts to about 8000 verses in total, although specific languages may not have translations of all verses. For the 30 languages we consider, this averages to around 402k subword tokens per language. The specific versions of the Bible we use are listed in Table 5.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Labeled Data For NER, we use the splits of Rahimi et al. (2019), which are created from the WikiAnn dataset (Pan et al., 2017). For POS tagging, we use data taken from the Universal Dependencies Project (Nivre et al., 2020). As XLM-R utilizes a subword vocabulary, we perform sequence labeling by assigning labels to the last subword token of each word. For all target languages, we only finetune on labeled data in English.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Language Selection To select the languages for our experiments, we first compile lists of all lan-guages for which a test set exists for either downstream task and we have a Bible for. We then filter these languages by removing those present in the pretraining data of XLM-R. See Table 1 for a summary of languages, their attributes, and the downstream task we use them for.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Our goal is to analyze state-of-the-art PMM adaption approaches in a true low-resource setting where the only raw text data available comes from the New Testament and no labeled data exists at all. We now describe our implementation of these methods. We focus on the Base version of XLM-R (Conneau et al., 2020) as our baseline PMM.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
We consider three models based on continued pretraining. In the simplest case, +MLM, we continue training XLM-R with an MLM objective on the available verses of the New Testament. Additionally, as Bible translations are a parallel corpus, we also consider a model, +TLM, trained using translation language modeling. Finally, following the findings of Lample and Conneau (2019), we also consider a model using both TLM and MLM, +{M|T}LM. For this model, we alternate between batches consisting solely of verses from the target Bible and batches consisting of aligned verses of the target-language and source-language Bible. For NER, we pretrain +MLM and +TLM models for 40 epochs, and pretrain +{M|T}LM models for 20 epochs. For POS tagging, we follow a simlar pattern, training +MLM and +TLM for 80 epochs, and +{M|T}LM for 40 epochs.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Vocabulary Extension To extend the vocabulary of XLM-R, we implement the process of Wang et al. (2020). We denote this as +Extend. For each target language, we train a new SentencePiece (Kudo and Richardson, 2018) tokenizer on the Bible of that language with a maximum vocabulary size of 30,000. 3 To prevent adding duplicates, we filter out any subword already present in the vocabulary of XLM-R. We then add additional pieces representing these new subwords into the tokenizer of XLM-R, and increase XLM-R's embedding matrix accordingly using a random initialization. Finally, we train the embeddings using MLM on the Bible. For NER, we train +Extend models for 40 epochs, and for POS tagging, we train for 80 epochs.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Adapters For adapters, we largely follow the full MAD-X framework (Pfeiffer et al., 2020b), using language, invertible, and task adapters. This is denoted as +Adapters. To train task adapters, we download language and invertible adapters for the source language from AdapterHub (Pfeiffer et al., 2020a). We train a single task adapter for each task, and use it across all languages. We train language and invertible adapters for each target language by training on the target Bible with an MLM objective. As before, for NER we train for 40 epochs, and for POS we train for 80 epochs.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
For finetuning, we train using 1 Nvidia V100 32GB GPU, and use an additional GPU for adaptation methods. Experiments for NER and POS take around 1 and 2 hours respectively, totalling to 165 total training hours, and 21.38 kgCO 2 eq emitted (Lacoste et al., 2019). All experiments are run using the Huggingface Transformers library (Wolf et al., 2020). We limit sequence lengths to 256 tokens.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
We select initial hyperparameters for finetuning by using the English POS development set. We then fix all hyperparameters other than the number of epochs, which we tune using the 3 languages which have development sets, Ancient Greek, Maltese, and Wolof. We do not use early stopping. For our final results, we finetune for 5 epochs with a batch size of 32, and a learning rate of 2e-5. We use the same hyperparameters for both tasks.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
For each task and adaptation approach, we search over {10, 20, 40, 80} epochs, and select the epoch which gives the highest average performance across the development languages. We use the same languages as above for POS. For NER we use 4 languages with varying baseline performances: Bashkir, Kinyarwanda, Maltese, and Scots. We pretrain with a learning rate of 2e-5 and a batch size of 32, except for +Adapters, for which we use a learning rate of 1e-4 (Pfeiffer et al., 2020b).
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
We present results for NER and POS tagging in Tables 2 and 3, respectively. We additionally provide plots of the methods' performances as compared to the XLM-R baseline in Figures 2 and 3  (+MLM, +TLM, +{M|T}LM), perform best, with 3.93 to 6.29 F1 improvement over XLM-R. Both +Extend and +Adapters obtain a lower average F1 than the XLM-R baseline, which shows that they are not a good choice in our setting: either the size or the domain of the Bible causes them to perform poorly. Focusing on the script of the target language (cf. Table 1), the average performance gain across all models is higher for Cyrillic languages than for Latin languages. Therefore, in relation to the source language script, performance gain is higher for target languages with a more distant script from the source. When considering approaches which introduce new parameters, +Extend and +Adapters, performance only increases for Cyrillic languages and decreases for all others. However, when considering continued pretraining approaches, we find a performance increase for all scripts. Looking at Figure 2, we see that the lower the baseline F1, the larger the improvement of the adaption methods on downstream performance, with all methods increasing performance for the language for which the baseline is weakest. As baseline performance increases, the benefit provided by these methods diminishes, and all methods underperform the baseline for Scots, the language with the highest baseline performance. We hypothesize that at POS Tagging Our POS tagging results largely follow the same trend as those for NER, with continued pretraining methods achieving the highest increase in performance: between 15.81 and 17.61 points. Also following NER and as shown in Figure 3, the largest performance gain can be seen for languages with a low baseline performance, and, as the latter increases, the benefits obtained from adaptation become smaller. However, unlike for NER, all methods show a net increase in performance, with +Adapters, the lowest performing adaptation model, achieving a gain of 9.01 points. We hypothesize that a likely reason for this is the domain and style of the Bible. While it may be too restrictive to significantly boost downstream NER performance, it is still a linguistically rich resource for POS tagging, a task that is less sensitive to domain in general.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Additionally, there is a notable outlier language, Coptic, on which no model performs better than random choice (which corresponds to 6% accuracy). This is because the script of this language is almost completely unseen to XLM-R, and practically all non-whitespace subwords map to the unknown token: of the 50% of non-whitespace tokens, 95% are unknown. While +Extend solves this issue, we believe that for a language with a completely unseen script the Bible is not enough to learn representations which can be used in a zero-shot setting.  
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
As previously stated, using the Bible as the corpus for adaptation is limiting in two ways: the extremely restricted domain as well as the small size.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
To separate the effects of these two aspects, we repeat our experiments with a different set of data. We sample sentences from the Wikipedia of each target language to simulate a corpus of similar size to the Bible which is not restricted to the Bible's domain or content. To further minimize the effect of domain, we focus solely on NER, such that the domain of the data is precisely that of the target task. Additionally, we seek to further investigate the effect on the downstream performance gains of these adaptation methods when the source language is more similar to the target language. To this end, we focus our case study on three languages written in Cyrillic: Bashkir, Chechen, and Chuvash. We break up the case study into 3 settings, depending on the data used. In the first setting, we change the language of our labeled training data from English to Russian. While Russian is not necessarily similar to the target languages or mutually intelligible, we consider it to be more similar than English; Russian is written in the same script as the target languages, and there is a greater likelihood for lexical overlap and the existence of loanwords. In the second setting, we pretrain using Wikipedia and in the third setting we use both Wikipedia data as well as labeled Russian data. To create our Wikipedia training data, we extract sentences with WikiExtractor (Attardi, 2015) and split them with Moses SentenceSplitter (Koehn et al., 2007). To create a comparable training set for each language, we first calculate the total number of subword tokens found in the New Testament, and sample sentences from Wikipedia until we have an equivalent amount. In the setting where we use data from the New Testament and labeled Russian data, for +TLM and +{M|T}LM we additionally substitute the English Bible with the Russian Bible. When using Wikipedia, we omit results for +TLM and +{M|T}LM, as they rely on a parallel corpus.  Table 4: Case study: Cyrillic NER (F1). Setting describes the source of data for adaptation, either the (B)ible or (W)ikipedia, as well as the language of the finetuning data, (E)nglish or (R)ussian.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
We present the results of our case study in Table  4. In the sections below, we refer to case study settings as they are described in the table caption.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Effects of the Finetuning Language We find that using Russian as the source language (the "Russian baseline"; B-R w/ XLM-R) increases performance over the English baseline (B-E w/ XLM-R) by 18.96 F1. Interestingly, all of the adaptation methods utilizing the Bible do poorly in this set-ting (B-R), with +MLM only improving over the Russian baseline by 1.09 F1, and all other methods decreasing performance. We hypothesize that when adaptation data is limited in domain, as the source language approaches the target language in similarity, the language adaptation is mainly done in the finetuning step, and any performance gain from the unlabeled data is minimized. This is supported by the previous NER results, where we find that, when using English as the source language, the adaptation methods lead to higher average performance gain over the baseline for Cyrillic languages, i.e., the more distant languages, as opposed to Latin languages. The adaptation methods show a larger improvement when switching to Wikipedia data (W-R), with +MLM improving performance by 11.85 F1 over the Russian baseline. Finally, the performance of +Extend when using Russian labeled data is similar on average regardless of the adaptation data (B-R, W-R), but noticeably improves over the setting which uses Wikipedia and English labeled data.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Fixing English as the source language and changing the pretraining domain from the Bible to Wikipedia (W-E) yields strong improvements, with +Adapters improving over the English baseline by 20.9 F1 and +MLM improving by 14.68 F1. However, we note that, while the average of +Adapters is higher than that of +MLM, this is due to higher performance on only a single language. When compared to the best performing pretraining methods that use the Bible (B-E), these methods improve by 11.29 F1 and 5.30 F1 respectively. When using both Wikipedia and Russian data, we see the highest overall performance, and +MLM increases over the English baseline by 30.81 F1 and the Russian baseline by 11.85 F1.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
One limitation of this work -and other works which involve a high number of languages -is task selection. While part-of-speech tagging and named entity recognition 4 are important, they are both low-level tasks largely based on sentence structure, with no requirement for higher levels of reasoning, unlike tasks such as question answering or natural language inference. While XTREME (Hu et al., 2020) is a great, diverse benchmark covering these higher level tasks, the number of languages is still limited to only 40 languages, all of which have Wikipedia data available. Extending these benchmarks to truly low resource languages by introducing datasets for these tasks will further motivate research on these languages, and provide a more comprehensive evaluation for their progress. Additionally, while the Bible is currently available in some form for 1611 languages, the available text for certain languages may be different in terms of quantity and quality from the Bible text we use in our experiments. Therefore, although we make no language-specific assumptions, our findings may not fully generalize to all 1611 languages due to these factors. Furthermore, this work focuses on analyzing the effects of adaptation methods for only a single multilingual transformer model. Although we make no model-specific assumptions in our methods, the set of unseen languages differs from model to model. Moreover, although we show improvements for the two tasks, we do not claim to have state-of-the-art results. In a low-resource setting, the best performance is often achieved through task-specific models. Similarly, translation-based approaches, as well as few-shot learning may offer additional benefits over a zero-shot setting. We also do not perform an extensive analysis of the target languages, or an analysis of the selected source language for finetuning. A better linguistic understanding of the languages in question would allow for a better selection of source language, as well as the ability to leverage linguistic features potentially leading to better results.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Finally, by using a PMM, we inherit all of that model's biases. The biases captured by word embeddings are well known, and recent work has shown that contextual models are not free of biases either (Caliskan et al., 2017;Kurita et al., 2019). The use of the Bible, and religious texts in general, may further introduce additional biases. Last, we acknowledge the environmental impact from the training of models on the scale of XLM-R (Strubell et al., 2019).
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
In this work, we evaluate the performance of continued pretraining, vocabulary extension, and adapters for unseen languages of XLM-R in a realistic lowresource setting. Using only the New Testament, we show that continued pretraining is the best per-forming adaptation approach, leading to gains of 6.29 F1 on NER and 17.69% accuracy on POS tagging. We therefore conclude that the Bible can be a valuable resource for adapting PMMs to unseen languages, especially when no other data exists. Furthermore, we conduct a case study on three languages written in Cyrillic script. Changing the source language to one more similar to the target language reduces the effect of adaptation, but the performance of the adaptation methods relative to each other is preserved. Changing the domain of the adaptation data to one more similar to the target task while keeping its size constant improves performance.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
In Table 5, we provide the number of subwords created by the XLM-R tokenizer from the New Testament of each target language, in addition to the specific version of the Bible we use, as found in the JHU Bible Corpus. In Table 7 and 6 we provide the relative performance of all adaptation methods as compared to baseline performance.
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Bible Size (thousands) ace ace-x-bible-ace-v1 536k arz
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
arz-x-bible-arz-v1 828k bak bak-BAKIBT 453k bam
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
bam-x-bible-bam-v1 429k ceb
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
ceb-x-bible-bugna2009-v1 384k che
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
che-x-bible-che-v1 523k chv chv-CHVIBT 519k cop
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
cop-x-bible-bohairic-v1 259k crh crh-CRHIBT 347k eng eng-x-bible-kingjames-v1 461k glv
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
glv-x-bible-glv-v1 196k grc
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
grc-x-bible-textusreceptusVAR1-v1 322k gsw
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
gsw-x-bible-alemannisch-v1 351k hak
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
hak-x-bible-hak-v1 598k ibo
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
ibo-x-bible-ibo-v1 458k ilo
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
ilo-x-bible-ilo-v1 378k kin
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
kin-x-bible-bird-youversion-v1 344k mag mag-MAGSSI 388k mhr mhr-x-bible-mhr-v1 398k min min-x-bible-min-v1 505k mlt mlt-x-bible-mlt-v1 389k mri mri-x-bible-mri-v1 411k myv myv-x-bible-myv-v1 463k nds
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
nds-x-bible-nds-v1 333k ory ory-x-bible-ory-v1 386k rus
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
rus-x-bible-kulakov-v1 283k sco
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
sco- x-bible-sco-v1  30k  tat  tat-TTRIBT  438k  tgk  tgk-TGKIBT  233k  war war-x-bible-war-v1 401k wol
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
wol-x-bible-wol-v1 383k yor
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
yor-x-bible-yor-v1 450k
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Avg. 402k   
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Recent advances in the development of multilingual pre-trained language models (PLMs) like mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), and RemBERT (Chung et al., 2021) have led to significant performance gains on a wide range of cross-lingual * * Equal contribution. transfer tasks. Due to the curse of multilinguality (Conneau et al., 2020) -a trade-off between language coverage and model capacity -and non-availability of pre-training corpora for many low-resource languages, multilingual PLMs are often trained on about 100 languages. Despite the limitations of language coverage, multilingual PLMs have been shown to transfer to several low-resource languages unseen during pretraining. Although, there is still a large performance gap compared to languages seen during pre-training.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) -fine-tuning a multilingual PLM on monolingual texts in the target language using the same pre-training objective. This has been shown to lead to big gains on many cross-lingual transfer tasks (Pfeiffer et al., 2020), and low-resource languages (Muller et al., 2021;Chau & Smith, 2021), including African languages Adelani et al., 2021). Nevertheless, adapting a model to each target language individually takes large disk space, and limits the cross-lingual transfer abilities of the resulting models because they have been specialized to individual languages (Beukman, 2021).
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
An orthogonal approach to improve the coverage of low-resource languages is to include them in the pretraining data. An example for this approach is AfriB-ERTa (Ogueji et al., 2021), which was trained from scratch on 11 African languages. A downside of this approach is that it is resource intensive in terms of data and compute.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Another alternative approach is parameter efficient fine-tuning like Adapters (Pfeiffer et al., 2020) and sparse fine-tuning (Ansell et al., 2021), where the model is adapted to new languages by using a sparse network trained on a small monolingual corpus. Similar to LAFT, it requires adaptation for every new target language. Although it takes little disk space, all target language-specific parameters need to be stored.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
In this paper, we propose multilingual adaptive finetuning (MAFT), a language adaptation to multiple languages at once. We perform language adaptation on the 17 most-resourced African languages (Afrikaans, Amharic, Hausa, Igbo, Malagasy, Chichewa, Oromo, Naija, Kinyarwanda, Kirundi, Shona, Somali, Sesotho, Swahili, isiXhosa, Yorùbá, isiZulu) and three other high-resource language widely spoken on the continent (English, French, and Arabic) simultaneously to provide a single model for cross-lingual transfer learning for African languages. To further specialize the multilingual PLM, we follow the approach of Abdaoui et al. (2020) and remove vocabulary tokens from the embedding layer that correspond to non-Latin and non-Ge'ez (used by Amharic) scripts before MAFT, thus effectively reducing the model size by 50%.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Our evaluation on two multilingual PLMs (AfriB-ERTa and XLM-R) and three NLP tasks (NER, news topic classification and sentiment classification) shows that our approach is competitive to performing LAFT on the individual languages, with the benefit of having a single model instead of a separate model for each of the target languages. Also, we show that our adapted PLM improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods like Adapters (Pfeiffer et al., 2020) and sparse finetuning (Ansell et al., 2021).
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
As an additional contribution, and in order to cover more diverse African languages in our evaluation, we create a new evaluation corpus, ANTC -African News Topic Classification -for Lingala, Somali, Naija, Malagasy, and isiZulu from pre-defined news categories of VOA, BBC, Global Voices, and Isolezwe newspapers. To further the research on NLP for African languages, we make our code and data publicly available. 1 Additionally, our models are available via HuggingFace. 2
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We perform MAFT on 17 African languages Afrikaans, Amharic, Hausa, Igbo, Malagasy, Chichewa, Oromo, Naija, Kinyarwanda, Kirundi, Shona, Somali, Sesotho, Swahili, isiXhosa, Yorùbá, isiZulu) covering the major African language families and 3 high resource languages (Arabic, French, and English) widely spoken in Africa. We selected the African languages based on the availability of a (relatively) large amount of monolingual texts. We obtain the monolingual texts from three major sources: the mT5 pre-training corpus which is based on Common Crawl Corpus 4 (Xue et al., 2021), the British Broadcasting Corporation (BBC) News, Voice of America News 5 (Palen-Michel et al., 2022), and some other news websites based in Africa. Table 9 in the Appendix provides a summary of the monolingual data, including their sizes and sources. We pre-processed the data by removing lines that consist of numbers or punctuation only, and lines with less than six tokens.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We run our experiments on two sentence level classification tasks: news topic classification and sentiment classification, and one token level classification task: NER. We evaluate our models on English as well as diverse African languages with different linguistic characteristics.  (Tjong Kim Sang & De Meulder, 2003) also containing texts from the news domain. For isiXhosa, we use the data from Eiselen (2016). Lastly, to evaluate on Arabic we make use of the AN-ERCorp dataset (Benajiba et al., 2007;Obeid et al., 2020).
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We use existing news topic datasets for Amharic (Azime & Mohammed, 2021), English -AG News corpus - (Zhang et al., 2015), Kinyarwanda -KINNEWS -(Niyongabo et al., 2020), Kiswahili -new classification dataset- , and both Yorùbá and Hausa (Hedderich et al., 2020). For dataset without a development set, we randomly sample 5% of their training instances and use them as a development set.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We use the NaijaSenti multilingual Twitter sentiment analysis corpus (Muhammad et al., 2022). This is a large code-mixed and monolingual sentiment analysis dataset, manually annotated for 4 Nigerian languages: Hausa, Igbo, Yorùbá and Pidgin. Additionally, we evaluate on the Amharic, and English Twitter sentiment datasets by Yimam et al.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
(2020) and Rosenthal et al. (2017), respectively. For all datasets above, we only make use of tweets with positive, negative and neutral sentiments.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We created a novel dataset, ANTC -African News Topic Classification for five African languages. We obtained data from three different news sources: VOA, BBC 6 , Global Voices 7 , and isolezwe 8 . From the VOA data we created datasets for Lingala and Somali. We obtained the topics from data released by Palen-Michel et al. (2022) and used the provided URLs to get the news category from the websites. For Naija, Malagasy and isiZulu, we scrapped news topic from the respective news website (BBC Pidgin, Global Voices, and isolezwe respectively) directly base on their category. We noticed that some news topics are not mutually exclusive to their categories, therefore, we filtered such topics with multiple labels. Also, we ensured that each category has at least 200 samples. The categories include but are not limited to: Africa, Entertainment, Health, and Politics. The pre-processed datasets were divided into training, development, and test sets using stratified sampling with a ratio of 70:10:20. Table 1 provides details about the dataset size and news topic information.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
For our experiments, we make use of different multilingual PLMs that have been trained using a masked language model objective on large collections of monolingual texts from several languages. Table 2 shows the number of parameters as well as the African languages covered by each of the models we consider.  Hyper-parameters for baseline models We finetune the baseline models for NER, news topic classification and sentiment classification for 50, 25, and 20 epochs respectively. We use a learning rate of 5e-5 for all the task, except for sentiment classification where we use 2e-5 for XLM-R-base and XLM-R-large. The maximum sequence length is 164 for NER, 500 for news topic classification, and 128 for sentiment classification. The adapted models also make use of similar hyper-parameters.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We introduce MAFT as an approach to adapt a multilingual PLM to a new set of languages. Adapting PLMs has been shown to be effective when adapting to a new domain (Gururangan et al., 2020) or language (Pfeiffer et al., 2020; Muller et al., 2021;Adelani et al., 2021). While previous work on multilingual adaptation has mostly focused on autoregressive sequence-to-sequence models such as mBART (Tang et al., 2020), in this work, we adapt non-autoregressive masked PLMs on monolingual corpora covering 20 languages. Crucially, during adaptation we use the same objective that was also used during pre-training. The models resulting from MAFT can then be finetuned on supervised NLP downstream tasks. We name the model resulting after applying MAFT to XLM-R-base and XLM-R-miniLM as AfroXLMR-base and AfroXLMR-mini, respectively. For adaptation, we train  Hyper-parameters for MAFT The PLMs were trained for 3 epochs with a learning rate of 5e-5 using huggingface transformers (Wolf et al., 2020). We use of a batch size of 32 for AfriBERTa and a batch size 10 for the other PLMs.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Multilingual PLMs come with various parameter sizes, the larger ones having more than hundred million parameters, which makes fine-tuning and deploying such models a challenge due to resource constraints. One of the major factors that contributes to the parameter size of these models is the embedding matrix whose size is a function of the vocabulary size of the model. While a large vocabulary size is essential for a multilingual PLM trained on hundreds of languages, some of the tokens in the vocabulary can be removed when they are irrelevant to the domain or language considered in the downstream task, thus reducing the vocabulary size of the model. Inspired by Abdaoui et al. (2020), we experiment with reducing the vocabulary size of the XLM-R-base model before adapting via MAFT. There are two possible vocabulary reductions in our setting:
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
(1) removal of tokens before MAFT or (2) removal of tokens after MAFT. From our preliminary experiments, we find approach (1) to work better. We call the resulting model, AfroXLMR-small.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
To remove non-African vocabulary sub-tokens from the pretrained XLM-base model, we concatenated the monolingual texts from 19 out of the 20 African languages together. Then, we apply sentencepiece to the Amharic monolingual texts, and concatenated texts separately using the original XLM-R-base tokenizer. The frequency of all the sub-tokens in the two separate monolingual corpora is computed, and we select the top-k most frequent tokens from the separate corpora. We used this separate sampling to ensure that a considerable number of Amharic sub-tokens are captured in the new vocabulary, we justify the choice of this approach in Section 5.3. We assume that the top-k most frequent tokens should be representative of the vocabulary of the whole 20 languages. We chose k = 52.000 from the Amharic sub-tokens which covers 99.8% of the Amharic monolingual texts, and k = 60.000 which covers 99.6% of the other 19 languages, and merged them. In addition, we include the top 1000 tokens from the original XLM-R-base tokenizer in the new vocabulary to include frequent tokens that were not present in the new top-k tokens. 9 We note that our assumption above may not hold in the case of some very distant and low-resourced languages as well as when there are domain differences between the corpora used during adaptation and fine-tuning. We leave the investigation of alternative approaches for vocabulary compression for future work. Tables 3, 4, and 5), we directly fine-tune on each of the downstream tasks in the target language: NER, news topic classification and sentiment analysis.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Performance on languages seen during pre-training For NER and sentiment analysis we find XLM-R-large to give the best overall performance. We attribute this to the fact that it has a larger model capacity compared to the other PLMs. Similarly, we find AfriBERTa and XLM-R-base to give better results on languages they have been pre-trained on (see Table 2), and in most cases AfriBERTa tends to perform better than XLM-R-base on languages they are both pre-trained on, for example amh, hau, and swa. However, when the languages are unseen by AfriBERTa (e.g. ara, eng, wol, lin, lug, luo, xho, zul), it performs much worse than XLM-R-base and in some cases even worse than the XLM-R-miniLM. This shows that it may be better   to adapt to a new African language from a PLM that has seen numerous languages than one trained on a subset of African languages from scratch.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
LAFT is a strong baseline The results of applying LAFT to the XLM-R-base model are shown in the last row of Tables 3, 4, and 5. We find that applying LAFT on each language individually provides a significant improvement in performance across all languages and tasks we evaluated on. Sometimes, the improvement is very large, for example, +7.4 F1 on Amharic NER and +9.5 F1 for Zulu news-topic classification. The only exception is for English since XLM-R has already seen large amounts of English text during pre-training. Additionally, LAFT models tend to give slightly worse result when adaptation is performed on a smaller corpus. 10
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
While LAFT provides an upper bound on downstream performance for most languages, our new approach is often competitive to LAFT. On average, the difference on NER, news topic and sentiment classification is −0.5, −0.2, and −0.3 F1, respectively. Crucially, 10 We performed LAFT on eng using VOA news corpus with about 906.6MB, much smaller than the CC-100 eng corpus (300GB) compared to LAFT, MAFT results in a single adapted model which can be applied to many languages while LAFT results in a new model for each language. Below, we discuss our results in more detail.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
PLMs pre-trained on many languages benefit the most from MAFT We found all the PLMs to improve after we applied MAFT. The improvement is the largest for the XLM-R-miniLM, where the performance improved by +4.2 F1 for NER, and +5.6 F1 for news topic classification. Although, the improvement was lower for sentiment classification (+0.8). Applying MAFT on XLM-R-base gave the overall best result. On average, there is an improvement of +2.6, +3.6, and +1.5 F1 on NER, news topic and sentiment classification, respectively. The main advantage of MAFT is that it allows us to use the same model for many African languagesinstead of many models specialized to individual languages. This significantly reduces the required disk space to store the models, without sacrificing performance. Interestingly, there is no strong benefit of applying MAFT to AfriBERTa. In most cases the improvement is < 0.6 F1. We speculate that this is probably due to AfriBERTa's tokenizer having a limited coverage. We leave a more detailed investigation of this for future work.  More efficient models using vocabulary reduction Applying vocabulary reduction helps to reduce the model size by more than 50% before applying MAFT. We find a slight reduction in performance as we remove more vocabulary tokens. Average performance of XLM-R-base-v70k reduces by −1.6, −1.5 and −0.6 F1 for NER, news topic, and sentiment classification compared to the XLM-R-base+LAFT baseline. Despite the reduction in performance compared to XLM-R-base+LAFT, they are still better than XLM-R-miniLM, which has a similar model size, with or without MAFT. We also find that their performance is better than that of the PLMs that have not undergone any adaptation. We find the largest reduction in performance on languages that make use of non-Latin scripts i.e. amh and ara -they make use of the Ge'ez script and Arabic script respectively. We attribute this to the vocabulary reduction impacting the number of amh and ara subwords covered by our tokenizer. In summary, we recommend XLM-R-base+MAFT (i.e. AfroXLMR-base) for all languages on which we evaluated, including high-resource languages like English, French and Arabic. If there are GPU resource constraints, we recommend using XLM-R-base-v70k+MAFT (i.e. AfroXLMR-small).
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Our results showed that applying vocabulary reduction reduced the model size, but we also observed a drop in performance for different languages across the downstream tasks, especially for Amharic, because it uses a non-Latin script. Hence, we compared different sampling strategies for selecting the top-k vocabulary subtokens. These include: (i) concatenating the monolingual texts, and selecting the top-70k sub-tokens (ii) the exact approach described in Section 5.1. The resulting tokenizers from the two approaches are used to tokenize the sentences in the NER test sets for Amharic, Arabic, English, and Yorùbá. Table 6 shows the number of UNKs in the respective test set after tokenization and the F1 scores obtained on the NER task for the languages. The table shows that the original AfroX-LMR tokenizer obtained the least number of UNKs for all languages, with the highest F1 scores. Note that Yorùbá has 24 UNKs, which is explained by the fact that Yorùbá was not seen during pre-training. Furthermore, using approach (i), gave 3704 UNKs for Amharic, but with approach (ii) there was a significant drop in the number of UNKs and an improvement in F1 score. We noticed a drop in the vocabulary coverage for the other languages as we increased the Amharic sub-tokens. Therefore, we concluded that there is no sweet spot in terms of the way to pick the vocabulary that covers all languages and we believe that this is an exciting area for future work.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
To demonstrate the applicability of MAFT to larger models, we applied MAFT to XLM-R-large using the same training setup as XLM-R-base. We refer to the new PLM as AfroXLMR-large. For comparison, we also trained individual LAFT models using the monolingual data 11 from Adelani et al. (2021). Table 7 shows the evaluation result on NER. Averaging over all 13 languages, AfroXLMR-large improved over XLM-Rlarge by +2.8 F1, which is very comparable to the improvement we obtained between AfroXLMR-base (81.8 F1) and XLM-R-base (79. 2 F1). Surprisingly, the improvement is quite large (+3.5 to +6.3 F1) for seven out of ten African languages: yor, luo, lug, kin, ibo, and amh. The most interesting observation is that AfroXLMR-large, on average, is either competitive or better than the individual language LAFT models, including languages not seen during the MAFT training stage like lug, luo and wol. This implies that AfroXLMR-large (a single model) provides a better alternative to XLM-R-large+LAFT (for each language) in terms of performance on downstream tasks and disk space. AfroXLMR-large is currently the largest masked language model for African languages, and achieves the state-of-the-art compared to all other multilingual PLM on the NER task. This shows that our MAFT approach is very effective and scales to larger PLMs.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
The previous section demonstrates the applicability of MAFT in the fully-supervised transfer learning setting.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Here, we demonstrate that our MAFT approach is also very effective in the zero-shot cross-lingual transfer setting using parameter-efficient fine-tuning methods. Parameter-efficient fine-tuning methods like adapters (Houlsby et al., 2019) are appealing because of their modularity, portability, and composability across languages and tasks. Often times, language adapters are trained on a general domain corpus    like Wikipedia. However, when there is a mismatch between the target domain of the task and the domain of the language adapter, it could also impact the cross-lingual performance.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Here, we investigate how we can improve the cross-lingual transfer abilities of our adapted PLM -AfroXLMR-base by training language adapters on the same domain as the target task. For our experiments, we use the MasakhaNER dataset, which is based on the news domain. We compare the performance of language adapters trained on Wikipedia and news domains. In addition to adapters, we experiment with another parameter-efficient method based on Lottery-Ticket Hypothesis (Frankle & Carbin, 2019) i.e. LT-SFT (Ansell et al., 2021).
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
For the adapter approach, we make use of the MAD-X approach (Pfeiffer et al., 2020) -an adapter-based framework that enables cross-lingual transfer to arbitrary languages by learning modular language and task representations. However, the evaluation data in the target languages should have the same task and label configuration as the source language. Specifically, we make use of MAD-X 2.0 (Pfeiffer et al., 2021) where the last adapter layers are dropped, which has been shown to improve performance. The setup is as follows: (1) We train language adapters via masked language modelling (MLM) individually on source and target languages, the corpora used are described in Appendix A.2; (2) We train a task adapter by finetuning on the target task using labelled data in a source language. (3) During inference, task and language adapters are stacked together by substituting the source language adapter with a target language adapter.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
We also make use of the Lottery Ticket Sparse
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Fine-tuning (LT-SFT) approach (Ansell et al., 2021), a parameter-efficient fine-tuning approach that has been shown to give competitive or better performance than the MAD-X 2.0 approach. The LT-SFT approach is based on the Lottery Ticket Hypothesis (LTH) that states that each neural model contains a sub-network (a "winning ticket") that, if trained again in isolation, can reach or even surpass the performance of the original model. The LTH is originally a compression approach, the authors of LT-SFT re-purposed the approach for cross-lingual adaptation by finding sparse sub-networks for tasks and languages, that will later be composed together for zero-shot adaptation, similar to Adapters. For additonal details we refer to Ansell et al. (2021).
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
For our experiments, we followed the same setting as Ansell et al. (2021) that adapted mBERT from English CoNLL03 (Tjong Kim Sang & De Meulder, 2003) to African languages (using MasakhaNER dataset) for the NER task. 12 Furthermore, we extend the experiments to XLMR-base and AfroXLMR-base. For the training of MAD-X 2.0 and sparse fine-tunings (SFT) for African languages, we make use of the monolingual texts from the news domain since it matches the domain of the evaluation data. Unlike, Ansell et al. (2021) that trained adapters and SFT on monolingual data from Wikipedia domain except for luo and pcm where the dataset is absent, we show that the domain used for training language SFT is also very important. For a fair comparison, we reproduced the result of Ansell et al. (2021) by training MAD-X 2.0 and LT-SFT on mBERT, XLM-R-base and AfroXLMR-base on target languages with the news domain corpus. But, we still make use of the pre-trained English language adapter 13 and SFT 14 for mBERT and XLM-R-base trained on the Wikipedia domain. For the AfroXLMR-base, we make use of the same English adapter and SFT as XLM-Rbase because the PLM is already good for English language. We make use of the same hyper-parameters reported in the LT-SFT paper.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
Hyper-parameters for adapters We train the task adapter using the following hyper-parameters: batch size of 8, 10 epochs, "pfeiffer" adapter config, adapter reduction factor of 8, and learning rate of 5e-5. For the language adapters, we make use of 100 epochs or maximum steps of 100K, minimum number of steps is 30K, batch size of 8, "pfeiffer+inv" adapter config, adapter reduction factor of 2, learning rate of 5e-5, and maximum sequence length of 256. For a fair comparison with adapter models trained on Wikipedia domain, we used the same hyper-parameter settings (Ansell et al., 2021) for the news domain. Table 8 shows the results of MAD-X 2.0 and LT-SFT, we compare their performance to fully supervised setting, where we fine-tune XLM-R-base on the training dataset of each of the languages, and evaluate on the test-set. We find that both MAD-X 2.0 and LT-SFT using news domain for African languages produce better performance (+4.2 on MAD-X and +3.7 on LT-SFT) than the ones trained largely on the wikipedia domain. This shows that the domain of the data matters. Also, we find that training LT-SFT on XLM-R-base gives better performance than mBERT on all languages. For MAD-X, there are a few exceptions like hau, pcm, and yor. Overall, the best performance is obtained by training LT-SFT on AfroXLMRbase, and sometimes it give better performance than the fully-supervised setting (e.g. as observed in kin and lug, wol yor languages). On both MAD-X and LT-SFT, AfroXLMR-base gives the best result since it has been firstly adapted on several African languages and secondly on the target domain of the target task. This shows that the MAFT approach is effective since the technique provides a better PLM that parameterefficient methods can benefit from.
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
In this work, we proposed and studied MAFT as an approach to adapt multilingual PLMs to many African languages with a single model. We evaluated our approach on 3 different NLP downstream tasks and additionally contribute novel news topic classification 13 https://adapterhub.ml/ 14 https://huggingface.co/cambridgeltl dataset for 4 African languages. Our results show that MAFT is competitive to LAFT while providing a single model compared to many models specialized for individual languages. We went further to show that combining vocabulary reduction and MAFT leads to a 50% reduction in the parameter size of a XLM-R while still being competitive to applying LAFT on individual languages. We hope that future work improves vocabulary reduction to provide even smaller models with strong performance on distant and low-resource languages. To further research on NLP for African languages and reproducibility, we have uploaded our language adapters, language SFTs, AfroXLMR-base, AfroXLMR-small, and AfroXLMR-mini models to the HuggingFace Model Hub 15 .   
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
It has been shown that performance on many natural language processing tasks drops dramatically on held-out data when a significant percentage of words do not appear in the training data, i.e., out-of-vocabulary (OOV) words (Søgaard and Johannsen, 2012;Madhyastha et al., 2016). A higher OOV rate (i.e., the percentage of the unseen words in the held-out data) may lead to a more severe performance drop (Kaljahi et al., 2015). OOV problems have been addressed in previous works under monolingual settings, through replacing OOV words with their semantically similar invocabulary words (Madhyastha et al., 2016;Kolachina et al., 2017) or using character/word information (Kim et al., 2016(Kim et al., , 2018Chen et al., 2018) or subword information like byte pair encoding (BPE) (Sennrich et al., 2016;Stratos, 2017).
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Recently, fine-tuning a pre-trained deep language model, such as Generative Pre-Training (GPT) (Radford et al., 2018) and Bidirectional Encoder Representations from Transformers (BERT) , has achieved remarkable success on various downstream natural language processing tasks. Instead of pre-training many monolingual models like the existing English GPT, English BERT, and Chinese BERT, a more natural choice is to develop a powerful multilingual model such as the multilingual BERT.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
However, all those pre-trained models rely on language modeling, where a common trick is to tie the weights of softmax and word embeddings (Press and Wolf, 2017). Due to the expensive computation of softmax  and data imbalance across different languages, the vocabulary size for each language in a multilingual model is relatively small compared to the monolingual BERT/GPT models, especially for lowresource languages. Even for a high-resource language like Chinese, its vocabulary size 10k in the multilingual BERT is only half the size of that in the Chinese BERT. Just as in monolingual settings, the OOV problem also hinders the performance of a multilingual model on tasks that are sensitive to token-level or sentence-level information. For ex-ample, in the POS tagging problem (Table 2), 11 out of 16 languages have significant OOV issues (OOV rate ≥ 5%) when using multilingual BERT.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
According to previous work (Radford et al., 2018;, it is time-consuming and resource-intensive to pre-train a deep language model over large-scale corpora. To address the OOV problems, instead of pre-training a deep model with a large vocabulary, we aim at enlarging the vocabulary size when we fine-tune a pretrained multilingual model on downstream tasks. We summarize our contributions as follows: (i) We investigate and compare two methods to alleviate the OOV issue. To the best of our knowledge, this is the first attempt to address the OOV problem in multilingual settings. (ii) By using English as an interlingua, we show that bilingual information helps alleviate the OOV issue, especially for low-resource languages. (iii) We conduct extensive experiments on a variety of token-level and sentence-level downstream tasks to examine the strengths and weaknesses of these methods, which may provide key insights into future directions 1 .
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We use the multilingual BERT as the pre-trained model. We first introduce the pre-training procedure of this model (Section 2.1) and then introduce two methods we investigate to alleviate the OOV issue by expanding the vocabulary (Section 2.2). Note that these approaches are not restricted to BERT but also applicable to other similar models.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Compared to GPT (Radford et al., 2018) and ELMo (Peters et al., 2018), BERT  uses a bidirectional transformer, whereas GPT pre-trains a left-to-right transformer ; ELMo (Peters et al., 2018) independently trains left-to-right and right-to-left LSTMs (Peters et al., 2017) to generate representations as additional features for end tasks.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
In the pre-training stage,  use two objectives: masked language model (LM) and next sentence prediction (NSP). In masked LM, they randomly mask some input tokens and then predict these masked tokens. Compared to unidirectional LM, masked LM enables representations to fuse the context from both directions. In the NSP task, given a certain sentence, it aims to predict the next sentence. The purpose of adding the NSP objective is that many downstream tasks such as question answering and language inference require sentence-level understanding, which is not directly captured by LM objectives.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
After pre-training on large-scale corpora like Wikipedia and BookCorpus (Zhu et al., 2015), we follow recent work (Radford et al., 2018; to fine-tune the pre-trained model on different downstream tasks with minimal architecture adaptation. We show how to adapt BERT to different downstream tasks in Figure 1 (left).  pre-train the multilingual BERT on Wikipedia in 102 languages, with a shared vocabulary that contains 110k subwords calculated from the WordPiece model (Wu et al., 2016). If we ignore the shared subwords between languages, on average, each language has a 1.1k vocabulary, which is significantly smaller than that of a monolingual pre-trained model such as GPT (40k). The OOV problem tends to be less serious for languages (e.g., French and Spanish) that belong to the same language family of English. However, this is not always true, especially for morphologically rich languages such as German (Ataman and Federico, 2018;. OOV problem is much more severe in lowresource scenarios, especially when a language (e.g., Japanese and Urdu) uses an entirely different character set from high-resource languages.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We focus on addressing the OOV issue at subword level in multilingual settings. Formally, suppose we have an embedding E bert extracted from the (non-contextualized) embedding layer in the multilingual BERT (i.e., the first layer of BERT). And suppose we have another set of (non-contextualized) sub-word embeddings {E l 1 , E l 2 , . . . , E ln } ∪ {E en }, which are pre-trained on large corpora using any standard word embedding toolkit. Specifically, E en represents the pre-trained embedding for English, and E l i represents the pre-trained embedding for non-English language l i at the subword level. We denote the vocabulary of E bert , E en , and E l i by V bert , V en , and V l i , respectively. For each subword w in V bert , we use E bert (w) to denote the pretrained embedding of word w in E bert . E l i (·) and E en (·) are defined in a similar way as E bert (·). For each non-English language l ∈ {l 1 , l 2 , . . . , l n }, we aim to enrich E bert with more subwords from the vocabulary in E l i since E l i contains a larger vocabulary of language l i compared to E bert .
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
As there is no previous work to address multilingual OOV issues, inspired by previous solutions designed for monolingual settings, we investigate the following two methods, and all of them can be applied at both word/subword level, though we find subword-level works better (Section 3). Joint Mapping For each non-English language l, we first construct a joint embedding space E l through mapping E l to E en by an orthogonal mapping matrix B l (i.e., E l = E l B l ). When a bilingual dictionary f l : V l → V en is available or can be constructed based on the shared common subwords (Section 3.1), we obtain B l by minimizing:
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
where · F denotes the Frobenius norm. Otherwise, for language pair (e.g., English-Urdu) that meets neither of the above two conditions, we obtain B l by an unsupervised word alignment method from MUSE .
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We then map E l to E bert by an orthogonal mapping matrix A l , which is obtained by minimizing
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We denote this method by M J in our discussion below, where the subscript J stands for "joint". Mixture Mapping Following the work of Gu et al. (2018) where they use English as "universal tokens" and map all other languages to English to obtain the subword embeddings, we represent each subword in E l (described in joint mapping) as a mixture of English subwords where those English subwords are already in the BERT vocab V bert . This method, denoted by M M , is also a joint mapping without the need for learning the mapping from E l to E bert . Specifically, for each w ∈ V l , we obtain its embedding e(w) in the BERT embedding space E bert as follows.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
where T (w) is a set to be defined later, and the mixture coefficient p(u|w) is defined by
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
where CSLS refers to the distance metric Crossdomain Similarity Local Scaling . We select five v ∈ V en ∪ V bert with the highest CSLS(E l (v), E en (w)) to form set T (w). In all our experiments, we set the number of nearest neighbors in CSLS to 10. We refer readers to  for details. Figure 1 (right) illustrates the joint and mixture mapping.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We obtain the pre-trained embeddings of a specific language by training fastText (Bojanowski et al., 2017) on Wikipedia articles in that language, with context window 5 and negative sampling 5. Before training, we first apply BPE (Sennrich et al., 2016) to tokenize the corpus with subword vocabulary size 50k. For joint mapping method M J , we use bilingual dictionaries provided by . For a language pair where a bilingual dictionary is not easily available, if two languages share a significant number of common subwords (this often happens when two languages belong to the same language family), we construct a bilingual dictionary based on the assumption that identical subwords have the same meaning (Søgaard et al., 2018). We add all unseen subwords from 50k vocabulary to BERT. We define a word as an OOV word once it cannot be represented as a single word. We use the multilingual BERT with default parameters in all our experiments, except that we tune the batch size and training epochs. To have a thorough examination about the pros and cons of the explored methods, we conduct experiments on a variety of token-level and sentence-level classification tasks: part of speech (POS) tagging, named entity recognition (NER), machine translation quality estimation, and machine reading comprehension. See more details in each subsection.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Previous work typically assumes a linear mapping exists between embedding spaces of different languages if their embeddings are trained using similar techniques (Xing et al., 2015;Madhyastha et al., 2016). However, it is difficult to map embeddings learned with different methods (Søgaard et al., 2018). Considering the differences between BERT and fastText: e.g., the objectives, the way to differentiate between different subwords, and the much deeper architecture of BERT, it is very unlikely that the (non-contextualized) BERT embedding and fastText embedding reside in the same geometric space. Besides, due to that BERT has a relatively smaller vocabulary for each language, when we map a pre-trained vector to its portion in BERT indirectly as previous methods, the supervision signal is relatively weak, especially for low-resource languages. In our experiment, we find that the accuracy of the mapping from our pre-trained English embedding to multilingual BERT embedding (English portion) is lower than 30%. In contrast, the accuracy of the mapping between two regular English embeddings that are pre-trained using similar methods (e.g., CBOW or SkipGram (Mikolov et al., 2013)) could be above 95% .
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Besides the joint mapping method M J (Section 2.2), another possible method that could be used for OOV problem in multilingual setting is that, for each language l, we map its pre-trained embedding space E l to embedding E bert by an orthogonal mapping matrix A l , which is obtained by minimizing
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
F . This approach is similar to (Madhyastha et al., 2016), and is referred as independent mapping method below. However, we use examples to demonstrate why these kind of methods are less promising. In Table 1, the first two rows are results obtained by mapping our pre-trained English embedding (using fastText) to the (non-contextualized) BERT embedding. In this new unified space, we align words with CSLS metric, and for each subword that appears in English Wikipedia, we seek its closest neighbor in the BERT vocabulary. Ideally, each word should find itself if it exists in the BERT vocabulary. However, this is not always true. For example, although "however" exists in the BERT vocabulary, independent mapping fails to find it as its own closest neighbor. Instead, it incorrectly maps it to irrelevant Chinese words "盘" ("plate") and "龙" ("dragon"). A similar phenomenon is observed for Chinese. For example, "册" is incorrectly aligned to Chinese words "书" and "卷".   
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
POS Tagging: We use the Universal Dependencies v1.2 data (McDonald et al., 2013). For languages with token segmentation ambiguity, we use the gold segmentation following Plank et al. (2016). We consider languages that have sufficient training data and filter out languages that have unsatisfying embedding alignments with English (accuracy is lower than 20.0% measured by word alignment accuracy or 0.25 by unsupervised metric in MUSE ). Finally, we keep 16 languages. We use the original multilingual BERT (without using CRF (Lafferty et al., 2001) on top of it for sequence labeling) to tune hyperparameters on the dev set and use the fixed hyperparameters for the expanded multilingual model. We do not tune the parameters for each model separately. As shown in Table 2, at both the word and subword level, the OOV rate in this dataset is quite high. Mixture mapping improves the accuracy on 10 out of 16 languages, leading to a 1.97% absolute gain in average. We discuss the influence of alignments in Section 3.6.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Chinese NER: We are also interested in investigating the performance gap between the expanded multilingual model and a monolingual BERT that is pre-trained on a large-scale monolingual corpus. Currently, pre-trained monolingual BERT models are available in English and Chinese. As English has been used as the interlingua, we compare the expanded multilingual BERT and the Chinese BERT on a Chinese NER task, evaluated on the Weibo NER dataset constructed from social media by Peng and Dredze (2015). In the training set, the token-level OOV rate is 2.17%, and the subwordlevel OOV rate is 0.54%. We tune the hyperparameters of each model based on the development set separately and then use the best hyperparameters of each model for evaluation on the test set.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
As shown in Table 3, the expanded model outperforms the multilingual BERT on the Weibo NER dataset. We boost the F1 score from 59.0% to 61.4%. Compared to the Chinese BERT (66.9%), there still exists a noticeable performance gap. One possible reason could be the grammatical differences between Chinese and English. As BERT uses the language model loss function for pre-training, the pre-trained Chinese BERT could better capture the language-specific information comapred to the multilingual BERT.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
As the multilingual BERT is pre-trained over 102 languages, it should be able to handle code-mixed texts. Here we examine its performance and the effectiveness of the expanded model in mixed language scenarios, using two tasks as case studies. Code-Switch Challenge: We first evaluate on the CALCS-2018 code-switched task (Aguilar et al., 2018), which contains two NER tracks on Twitter social data: mixed English&Spanish (en-es) and mixed Modern Standard Arabic&Egyptian (ar-eg). Compared to traditional NER datasets constructed from news, the dataset contains a significant portion of uncommon tokens like hashtags and abbreviations, making it quite challenging. For example, in the en-es track, the tokenlevel OOV rate is 44.6%, and the subword-level OOV rate is 3.1%; in the ar-eg track, the tokenlevel OOV rate is 64.0%, and the subword-level OOV rate is 6.0%. As shown in Table 4, on areg, we boost the F1 score from 74.7% to 77.3%. However, we do not see similar gains on the en-es dataset, probably because that English and Spanish share a large number of subwords, and adding too many new subwords might prevent the model from utilizing the well pre-trained subwords embedding. See Section 3.6 for more discussions.  Table 4: Accuracy (%) on the code-switch challenge. The top two rows are based on the test set, and the bottom three rows are based on the development set. ♣: results from Aguilar et al. (2018). ♦: results from .
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
All previous experiments are based on well-curated data. Here we evaluate the expanded model on a language generation task, where sometimes the generated sentences are out-of-control.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We choose the automatic Machine Translation Quality Estimation task and use Task 2 -wordlevel quality estimation -in WMT18 (Bojar et al., 2018). Given a source sentence and its translation (i.e., target), this task aims to estimate the translation quality ("BAD" or "OK") at each position: e.g., each token in the source and target sentence, each gap in the target sentence. We use English to German (en-de) SMT translation. On all three categories, the expanded model consistently outperforms the original multilingual BERT (Table 5) 
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Finally, we evaluate the expanded model on sequence classification in a mixed-code setting, where results are less sensitive to unseen words. Code-Mixed Machine Reading Comprehension: We consider the mixed-language machine reading comprehension task. Since there is no such public available dataset, we construct a new Chinese-English code-mixed machine reading comprehension dataset based on 37,436 unduplicated utterances obtained from the transcriptions of a Chinese and English mixed speech recognition corpus King-ASR-065-1 3 . We generate a multiple-choice machine reading comprehension problem (i.e., a question and four answer options) for each utterance. A question is an utterance with an English text span removed (we randomly pick one if there are multiple English spans) and the correct answer option is the removed English span. Distractors (i.e., wrong answer options) come from the top three closest English text spans, which appear in the corpus, based on the cosine similarity of word embeddings trained on the same corpus. For example, given a question "突然听到 21 ，那强劲的鼓 点，那一张张脸。" ("Suddenly I heard 21 , and the powerful drum beats reminded me of the players.") and four answer options { "forever", "guns", "jay", "twins" }, the task is to select the correct answer option "guns" ("21 Guns" is a song by the American rock band Green Day). We split the dataset into training, development, and testing of size 36,636, 400, 400, respectively. An-   Specia et al. (2018). MT: machine translation, e.g., target sentence, SRC: source sentence. F1-OK: F1 score for "OK" class; F1-BAD: F1 score for "BAD" class; F1-multi: multiplication of F1-OK and F1-BAD.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
notators manually clean and improve the quality problems by generating more confusing distractors in the dev and testing sets to guarantee that these problems are error-free and challenging. In this experiment, for each BERT model, we follow its default hyperparameters. As shown in Table 6, the expanded model improves the multilingual BERT (38.1%) by 1.2% in accuracy. Human performance (81.4%) indicates that this is not an easy task even for human readers.  
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
In this section, we first briefly investigate whether the performance boost indeed comes from the reduction of OOV and then discuss the strengths and weaknesses of the methods we investigate. First, we argue that it is essential to alleviate the OOV issue in multilingual settings. Taking the POS tagging task as an example, we find that most errors occur at the OOV positions (Table 7 in Section 3.3). In the original BERT, the accuracy of OOV words is much lower than that of non-OOV words, and we significantly boost the accuracy of OOV words with the expanded BERT. All these results indicate that the overall improvement mostly comes from the reduction of OOV.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We also observe that the following factors may influence the performance of the expanded model.  we add all unseen subwords from the 50k vocabulary (Section 3.1), which may be not an optimal choice. Adding too many subwords may prevent the model from utilizing the information from pretrained subword embedding in BERT, especially when there is a low word-level overlap between the training and test set. Language: We also find that languages can influence the performance of the vocabulary expansion through the following two aspects: the alignment accuracy and the closeness between a language and English. For languages that are closely related to English such as French and Dutch, it is relatively easy to align their embeddings to English as most subword units are shared (Søgaard et al., 2018;. In such case, the BERT embedding already contains sufficient information, and therefore adding additional subwords may hurt the performance. On the other hand, for a distant language such as Polish (Slavic family), which shares some subwords with English (Germanic family), adding subwords to BERT brings performance improvements. In the meantime, as Slavic and Germanic are two subdivisions of the Indo-European languages, we find that the embedding alignment methods per-form reasonably well. For these languages, vocabulary expansion is usually more effective, indicated by POS tagging accuracies for Polish, Portuguese, and Slovenian (Table 2). For more distant languages like Arabic (Semitic family) that use different character sets, it is necessary to add additional subwords. However, as the grammar of such a language is very different from that of English, how to accurately align their embeddings is the main bottleneck. Task: We see more significant performance gains on NER, POS and MT Quality Estimation, possibly because token-level understanding is more critical for these tasks, therefore alleviating OOV helps more. In comparison, for sequence level classification tasks such as machine reading comprehension (Section 3.5), OOV issue is less severe since the result is based on the entire sentence.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Most previous work address the OOV problems in monolingual settings. Before more fine-grained encoding schema such as BPE (Sennrich et al., 2016) is proposed, prior work mainly focused on OOV for token-level representations (Taylor et al., 2011;Kolachina et al., 2017). Besides simply assigning random embeddings to unseen words (Dhingra et al., 2017) or using an unique symbol to replace all these words with a shared embedding (Hermann et al., 2015), a thread of research focuses on refining the OOV representations based on word-level information, such as using similar in-vocabulary words (Luong et al., 2015;Cho et al., 2015;Tafforeau et al., 2015;Li et al., 2016), mapping initial embedding to task-specific embedding (Rothe et al., 2016;Madhyastha et al., 2016), using definitions of OOV words from auxiliary data (Long et al., 2016;Bahdanau et al., 2017), and tracking contexts to build/update representations (Henaff et al., 2016;Kobayashi et al., 2017;Ji et al., 2017;. Meanwhile, there have been efforts in representing words by utilizing character-level (Zhang et al., 2015;Ling et al., 2015a,b;Kim et al., 2016;Gimpel and Livescu, 2016) or subwordlevel representations (Sennrich et al., 2016;Bojanowski et al., 2017). To leverage the advantages in character and (sub)word level representation, some previous work combine (sub)wordand character-level representations (Santos and Zadrozny, 2014;dos Santos et al., 2015;Yu et al., 2017) or develop hybrid word/subword-character architectures (Chung et al., 2016;Luong and Manning, 2016;Pinter et al., 2017;Bahdanau et al., 2017;Matthews et al., 2018;. However, all those approaches assume monolingual setting, which is different from ours.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Addressing OOV problems in a multilingual setting is relatively under-explored, probably because most multilingual models use separate vocabularies (Jaffe, 2017;Platanios et al., 2018). While there is no direct precedent, previous work show that incorporating multilingual contexts can improve monolingual word embeddings (Zou et al., 2013;Andrew et al., 2013;Faruqui and Dyer, 2014;Lu et al., 2015;Ruder et al., 2017).
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Madhyastha and España-Bonet (2017) increase the vocabulary size for statistical machine translation (SMT). Given an OOV source word, they generate a translation list in target language, and integrate this list into SMT system. Although they also generate translation list (similar with us), their approach is still in monolingual setting with SMT. Cotterell and Heigold (2017) train charlevel taggers to predict morphological taggings for high/low resource languages jointly, alleviating OOV problems to some extent. In contrast, we focus on dealing with the OOV issue at subword level in the context of pre-trained BERT model.
Improving Pre-Trained Multilingual Models with Vocabulary Expansion
We investigated two methods (i.e., joint mapping and mixture mapping) inspired by monolingual solutions to alleviate the OOV issue in multilingual settings. Experimental results on several benchmarks demonstrate the effectiveness of mixture mapping and the usefulness of bilingual information. To the best of our knowledge, this is the first work to address and discuss OOV issues at the subword level in multilingual settings. Future work includes: investigating other embedding alignment methods such as Gromov-Wasserstein alignment (Alvarez-Melis and Jaakkola, 2018) upon more languages; investigating approaches to choose the subwords to be added dynamically.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Multilingual pretrained models (Devlin et al., 2019;Conneau and Lample, 2019;Conneau et al., 2020) have become an essential method for cross-lingual transfer on a variety of NLP tasks (Pires et al., 2019;Wu and Dredze, 2019). These models can be finetuned on annotated data of a down-stream task in a high-resource language, often English, and then the resulting model is applied to other languages. This paradigm is supposed to benefit under-represented languages that do not have annotated data. However, recent studies have found that the cross-lingual transfer performance of a language is highly contingent on the availability of monolingual data in the language during pretraining (Hu et al., 2020). Languages with more 1 Code and data are available at:
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
https: //github.com/cindyxinyiwang/ expand-via-lexicon-based-adaptation. monolingual data tend to have better performance while languages not present during pretraining significantly lag behind. Several works propose methods to adapt the pretrained multilingual models to low-resource languages, but these generally involve continued training using monolingual text from these languages (Wang et al., 2020;Chau et al., 2020;Pfeiffer et al., 2020Pfeiffer et al., , 2021. Therefore, the performance of these methods is still constrained by the amount of monolingual or parallel text available, making it difficult for languages with little or no textual data to benefit from the progress in pretrained models. Joshi et al. (2020) indeed argue that unsupervised pretraining makes the 'resource-poor poorer'. Fig. 1 plots the language coverage of multilingual BERT (mBERT; Devlin et al., 2019), a widely used pre-trained model, and several commonly used textual data sources. 2 Among the 7,000 languages in the world, mBERT only covers about 1% of the languages while Wikipedia and Com-monCrawl, the two most common resources used for pretraining and adaptation, only contain textual data from 4% of the languages (often in quite small quantities, partially because language IDs are difficult to obtain for low-resource languages (Caswell et al., 2020)). Ebrahimi and Kann (2021) show that continued pretraining of multilingual models on a small amount of Bible data can significantly improve the performance of uncovered languages. Although the Bible has much better language coverage of 23%, its relatively small data size and constrained domain limits its utility (see § 6)-and 70% of the world's languages do not even have this resource. The failure of technology to adapt to these situations raises grave concerns regarding the fairness of allocation of any benefit that may be conferred by NLP to speakers of these languages (Joshi et al., 2020;Blasi et al., 2021). On the other hand, linguists have been studying and documenting under-represented languages for years in a variety of formats (Gippert et al., 2006). Among these, bilingual lexicons or word lists are usually one of the first products of language documentation, and thus have much better coverage of the worlds' languages than easily accessible monolingual text, as shown in Fig. 1. There are also ongoing efforts to create these word lists for even more languages through methodologies such as "rapid word collection" (Boerger, 2017), which can create an extensive lexicon for a new language in a number of days. As Bird (2020) notes:
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
After centuries of colonisation, missionary endeavours, and linguistic fieldwork, all languages have been identified and classified. There is always a wordlist. . . . In short, we do not need to "discover" the language ex nihilo (L1 acquisition) but to leverage the available resources (L2 acquisition).
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
However, there are few efforts on understanding the best strategy to utilize this valuable resource for adapting pretrained language models. Bilingual lexicons have been used to synthesize bilingual data for learning cross-lingual word embeddings (Gouws and Søgaard, 2015;Ruder et al., 2019) and task data for NER via word-to-word translation (Mayhew et al., 2017), but both approaches precede the adoption of pre-trained multilingual LMs. Khemchandani et al. (2021) use lexicons to synthesize monolingual data for adapting LMs, but their experimentation is limited to several Indian languages and no attempt was made to synthesize downstream task data while Hu et al. (2021) argue that bilingual lexicons may hurt performance.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
In this paper, we conduct a systematic study of strategies to leverage this relatively under-studied resource of bilingual lexicons to adapt pretrained multilingual models to languages with little or no monolingual data. Utilizing lexicons from an open-source database, we create synthetic data for both continued pretraining and downstream Figure 2: Results for baselines and adaptation using synthetic data for both resource settings across three NLP tasks. task fine-tuning via word-to-word translation. Empirical results on 19 under-represented languages on 3 different tasks demonstrate that using synthetic data leads to significant improvements on all tasks (Fig. 2), and that the best strategy depends on the availability of monolingual data ( § 5, § 6). We further investigate methods that improve the quality of the synthetic data through a small amount of parallel data or by model distillation.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We focus on the cross-lingual transfer setting where the goal is to maximize performance on a downstream task in a target language T . Due to the frequent unavailability of labeled data in the target language, a pretrained multilingual model M is typically fine-tuned on labeled data in the down-
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
i is a textual input, y S i is the label, and N is the number of labeled examples. The fine-tuned model is then directly applied to task data D T test = {x T i , y T i } i in language T at test time. 3 The performance on the target language T can often be improved by further adaptation of the pretrained model.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
There are two widely adopted paradigms for adapting pretrained models to a target language using monolingual or parallel text.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
MLM Continued pretraining on monolingual text D T mono = {x T i } i in the target language (Howard and Ruder, 2018;Gururangan et al., 2020) using a masked language model (MLM) objective has proven effective for adapting models to the target language (Pfeiffer et al., 2020). Notably, Ebrahimi and Kann (2021) show that using as little as several thousand sentences can significantly improve the model's performance on target languages not covered during pretraining.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Trans-Train For target languages with sufficient parallel text with the source language D ST par = {(x S i , x T i )} i , one can train a machine translation (MT) system that translates data from the source language into the target language. Using such an MT system, we can translate the labeled data in the source language D S label into target language
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
, and fine-tune the pretrained multilingual model on both the source and translated labeled data D S label ∪ D T label . This method often brings significant gains to the target language, especially for languages with high-quality MT systems (Hu et al., 2020;Ruder et al., 2021).
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Both methods above require D T mono or D ST par in target language T , so they cannot be directly extended to languages without this variety of data. Joshi et al. (2020) classified the around 7,000 languages of the world into six groups based on the availability of data in each language. The two groups posing the biggest challenges for NLP are:
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
"The Left-Behinds," languages with virtually no unlabeled data. We refer to this as the No-Text setting.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
"The Scraping-Bys," languages with a small amount of monolingual data. We refer to this as the Few-Text setting.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
These languages make up 85% of languages in the world, yet they do not benefit from the development of pretrained models and adaptation methods due to the lack of monolingual and parallel text. In this paper, we conduct a systematic study of strategies directly targeted at these languages.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Since the main bottleneck of adapting to underrepresented languages is the lack of text, we adopt a data augmentation framework (illustrated in Fig. 3) that leverages bilingual lexicons, which are available for a much larger number of languages.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Given a bilingual lexicon D ST lex between the source language S and a target language T , we create synthetic sentences x T i in T using sentences x S i in S via word-to-word translation, and use this synthetic data in the following adaptation methods. Pseudo MLM Using monolingual text D S mono = {x S i } i , we generate pseudo monolingual text D T mono = { x T i } i for T by replacing the words in x S i with their translation in T based on the lexicon D ST lex . We keep the words that do not exist in the lexicon unchanged, so the pseudo text x T i can include words in both S and T . We then adapt the pretrained multilingual model on D T mono using the MLM objective. For the Few-Text setting where some gold monolingual data D T mono is available, we can train the model jointly on the pseudo and the gold monolingual data D T mono ∪ D T mono .
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo Trans-train Given the source labeled
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We keep the original word if it does not have an entry in the lexicon. We then fine-tune the model jointly on both pseudo and gold labeled data D T label ∪ D S label . Since these methods only require bilingual lexicons, we can apply them to both No-Text and Few-Text settings. We can use either of the two methods or the combination of both to adapt the model.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Our synthetic data D T could be very different from the true data D T because the lexicons do not cover all words in S or T , and we do not consider morphological or word order differences between T and S. 4 Nonetheless, we find that this approach yields significant improvements in practice (see Tab. 3). We also outline two strategies that aim to improve the quality of the synthetic data in the next section.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Label Distillation The pseudo labeled data
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Anarchism calls for the abolition of the state , which it holds to be undesirable , unnecessary , and harmful . Pseudo Mono x T ∈ D T mono Anarchism calls gal il abolition ta' il stat , lima hi holds gal tkun undesirable , bla bzonn , u harmful . eng x S ∈ D S label I suspect the streets of Baghdad will look as if a war is looming this week . Pseudo Labeled x T ∈ D T label jien iddubita il streets ta' Bagdad xewqa hares kif jekk a gwerra is looming danġimga .  Table 1: Examples of pseudo monolingual data and pseudo labeled data for POS tagging for Maltese (mlt). Words in red have different labels between the source language and the label distilled data. This is because "xewqa" in Maltese is a noun meaning "desire,will", while the word "will" is not used as a noun in the original English sentence.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
thetic examples x T i could have a different label from the original label y S i (see Tab. 1). To alleviate this issue, we propose to automatically "correct" the labels of pseudo data using a teacher model. Specifically, we fine-tune the pretrained multilingual model as a teacher model using only D S label . We use this model to generate the new pseudo la-
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We then fine-tune the pretrained model on both the new pseudo labeled data and the source labeled data D T distill ∪ D S label . Induced Lexicons with Parallel Data For the Few-Text setting, we can leverage the available parallel data D ST par to further improve the quality of the augmented data. Specifically, we use unsupervised word alignment to extract additional word pairs D ST lex from the parallel data, and use the combined lexicon D ST lex ∪ D ST lex to synthesize the pseudo data.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
In this section, we outline the tasks and data setting used by all experiments. We will then introduce the adaptation methods and results for the No-Text setting in § 5 and the Few-Text setting in § 6.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We evaluate on the gold test sets of three different tasks with relatively good coverage of underrepresented languages: named entity recognition (NER), part-of-speech (POS) tagging, and dependency parsing (DEP  We run each fine-tuning experiment with 3 random seeds and report the average performance. For NER and POS tagging, we follow the data processing and fine-tuning hyper-parameters in Hu et al. (2020). We use the Udify (Kondratyuk and Straka, 2019) codebase and configuration for parsing.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Languages For each task, we select languages that have task data but are not covered by the mBERT pretraining data. The languages we use can be found in Tab. 2. Most fall under the Few-Text setting (Joshi et al., 2020). We employ the same languages to simulate the No-Text setting as well.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Model We use the multilingual BERT model (mBERT) because it has competitive performance on under-represented languages (Pfeiffer et al., 2020). We find that our mBERT performance on WikiNER and POS is generally comparable or exceeds the XLM-R large results in Ebrahimi and Kann (2021). We additionally verify our results also hold for XLM-R in § 7.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Lexicon We extract lexicons between English and each target language from the PanLex database. 5 The number of lexicon entries varies from about 0.5k to 30k, and most of the lexicons have around 5k entries. The lexicon statistics for each language can be found in Tab. 2.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo Monolingual Data English Wikipedia articles are used to synthesize monolingual data. We first tokenize the English articles using Stanza (Qi et al., 2020) and keep the first 200k sentences. To create pseudo monolingual data for a given target language, we replace each English word with its translation if the word exists in the bilingual lexicon. We randomly sample a target word if the English word has multiple possible translations because it is difficult to estimate translation probabilities due to lack of target text.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo Labeled Data Using the English training data for each task, we simply replace each English word in the labeled training data with its corresponding translation and retain its original label. For the sake of simplicity, we only use lexicon entries with a single word.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We analyze the results of the following adaptation methods for the setting where we do not have any monolingual data.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo MLM The mBERT model is trained on the pseudo monolingual data using the MLM objective. We train the model for 5k steps for the NER tasks and 10k steps for the POS tagging and Parsing tasks.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo Trans-train We fine-tune mBERT or the model adapted with Pseudo MLM for a downstream task on the concatenation of both the English labeled data and the pseudo labeled data.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Label Distillation We use the model adapted with Pseudo MLM as the teacher model to generate new labels for the pseudo labeled data, which we use jointly with the English labeled data to finetune the final model.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
The average performance of different adaptation methods averaged across all languages in each task 5 https://panlex.org/snapshot/ can be found in Tab. 3.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo Trans-train is the best method for No-Text. Pseudo MLM and Pseudo Trans-train can both bring significant improvements over the mBERT baseline for all tasks. Pseudo Trans-train leads to the best aggregated result across all tasks, and it is also the best method or very close to the best method for each task. Adding Pseudo Transtrain on top of Pseudo MLM does not add much improvement. Label Distillation generally leads to better performance, but overall it is comparable to only using Pseudo Trans-train.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We test same adaptation methods introduced in § 5 for the Few-Text setting where we have a small amount of gold data. First we introduce the additional data and adaptation methods for this setting.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Gold Monolingual Data We use the JHU Bible Corpus (McCarthy et al., 2020) as the monolingual data. Following the setup in Ebrahimi and Kann (2021), we use the verses from the New Testament, which contain 5000 to 8000 sentences for each target language.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Gold Parallel Data We can use the parallel data between English and the target languages from the Bible to extract additional word pairs. We use an existing unsupervised word alignment tool, eflomal (Östling and Tiedemann, 2016), to generate word alignments for each sentence in the parallel Bible data. To create high quality lexicon entries, we only keep the word pairs that are aligned more than once, resulting in about 2k extra word pairs for each language. We then augment the PanLex lexicons with the induced lexicon entries.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Gold MLM The mBERT model is trained on the gold monolingual Bible data in the target language using the MLM objective. Following the setting in Ebrahimi and Kann (2021), we train for 40 epochs for the NER task, and 80 epochs for the POS and Parsing tasks.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo MLM We conduct MLM training on both the Bible monolingual data and the pseudo monolingual data in the target language. The Bible data is up-sampled to match the size of the pseudo monolingual data. We train the model for 5k steps  Table 3: Average F1 score for languages in each task. We record F1 of the LAS for Parsing. We compare three adaptation methods (∆ indicates gains over baselines): Pseudo Trans-train, Pseudo MLM, and Both. We also examine two data refinement methods: Label Distillation (∆ is gains over Both) and PanLex+Induced (∆ is gains over PanLex). Bold is the best result for each dataset, and underline indicates the best improvements among the three adaptation methods over the baselines. We test the significance of the average gains over the baselines in the last column using paired bootstrap resampling. * indicates significant gains with p < 0.001 and † indicates significant gains with p < 0.05.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
for the NER task and 10k steps for the POS tagging and Parsing tasks.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
The average performance in each task for Few-Text can be found in Tab. 3.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Pseudo MLM is the competitive strategy for Few-Text. Unlike the No-Text setting, Pseudo Trans-train only marginally improves or even decreases the performance for three out of the four datasets we consider. On the other hand, Pseudo MLM, which uses both gold and pseudo monolingual data for MLM adaptation, consistently and significantly improves over Gold MLM for all tasks. Again, using Pseudo Trans-train on top of Pseudo MLM does not help and actually leads to relatively large performance loss for the syntactic tasks, such as POS tagging and Parsing.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Label Distillation brings significant improvements for the two syntactic tasks. Notably, it is the best performing method for POS tagging, but it still lags behind Pseudo MLM for Parsing. This is likely because Parsing is a much harder task than POS tagging to generate correct labels. The effect of Label Distillation on the NER task is less consistent-it improves over Pseudo Trans-train for WikiNER but not for MasakhaNER. This is because the named entity tags of the same words in different languages likely remain the same so that the pseudo task data probably has less noise for Label Distillation to have consistent benefits. Adding Induced Lexicons We examine the effect of using the lexicons augmented by word pairs induced from the Bible parallel data. The results can be found in Tab. 3. Adding the induced lexicon significantly improves the NER performance, while it hurts the two syntactic tasks.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
To understand what might have prevented the syntactic tasks from benefiting from the extra lexicon entries, we plot the distribution of the partof-speech tags of the words in PanLex lexicons and the lexicons induced from the Bible in Fig. 4. PanLex lexicons have more nouns than the Bible lexicons while the Bible lexicons cover more verbs than PanLex. However, the higher verb coverage in induced lexicons actually leads to a larger prediction accuracy drop for verbs in the POS tagging task. We hypothesize that the pseudo monolingual data created using the induced lexicons would contain more target language verbs with the wrong word order, which could be more harmful for syntactic tasks than tasks that are less sensitive to word order such as NER.  
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Performance with XLM-R We mainly use mBERT because it has competitive performance for under-represented languages and it is more computationally efficient due to the smaller size. Here we verify our methods have the same trend when used on a different model XLM-R (Conneau et al., 2020). We focus on a subset of languages in the POS tagging task for the Few-Text setting and the results are in Tab. 4. We use the smaller XLM-R base for efficiency, and compare to the best result in prior work, which uses XLM-R large (Ebrahimi and Kann, 2021). Tab. 4 shows that our baseline is comparable or better than prior work. Similar to the conclusion in § 6, Pseudo MLM is the competitive strategy that brings significant improvements over prior work. While adding Pseudo Trans-train to Pseudo MLM does not help, using Label Distillation further improves the performance. Effect of Baseline Performance Using pseudo data might be especially effective for languages with lower performance. We plot the improvement of different languages over the baseline in Fig. 5, where languages are arranged with increasing baseline performance from left to right. We mainly plot Pseudo MLM and Pseudo Trans-train for simplicity. Fig. 5 shows that for both resource settings, lower performing languages on the left tend to have more performance improvement by using pseudo data.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Using NMT Model to Synthesize Data One problem with the pseudo data synthesized using word-to-word translation is that it cannot capture the correct word order or syntactic structure in the target language. If we have a good NMT system that translates English into the target language, we might be able to get more natural pseudo monolingual data by translating the English sentences to the target language.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Since the target languages we consider are usually not supported by popular translation services, we train our own NMT system by fine-tuning an open sourced many-to-many NMT model on the Bible parallel data from English to the target language (details in § A.2). Instead of creating pseudo monolingual data using the lexicon, we can simply use the fine-tuned NMT model to translate English monolingual data into the target language.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
The results of using NMT as opposed to lexicon for Pseudo MLM on all four tasks can be found in Tab. 5. Unfortunately, NMT is consistently worse than word-to-word translation using lexicons. We find that the translated monolingual data tend to have repeated words and phrases that are common in the Bible data, although the source sentence is from Wikipedia. This is because the NMT model overfits to the Bible data, and it fails to generate good translation for monolingual data from a different domain such as Wikipedia.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Comparison to Few-shot Learning Lauscher et al. (2020) found that using as few as 10 labeled   Table 6: Results on MasakhaNER for k-shot learning. We compare to the zero-shot mBERT baseline and our best adapted model. examples in the target language can significantly outperform the zero-shot transfer baseline for languages included in mBERT. We focus on the zeroshot setting in this paper because the languages we consider have very limited data and it could be expensive or unrealistic to annotate data in every task for thousands of languages. Nonetheless, we experiment with k-shot learning to examine its performance on low-resource languages in the MasakhaNER task. Tab. 6 shows that using 10 labeled examples brings improvements over the mBERT baseline for a subset of the languages, and it is mostly worse than our best adapted model without using any labeled data. When we have access to 100 examples, few-shot learning begins to reach or exceed our zero-shot model. In general, few-shot learning seems to require more data to consistently perform well for under-represented languages while our adaptation methods bring consistent gains without any labeled data. Combining the best adapted model with few-shot learning leads to mixed results. More research is needed to understand the annotation cost and benefit of few-shot learning for low-resource languages.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We propose a pipeline that leverages bilingual lexicons, an under-studied resource with much better language coverage than conventional data, to adapt pretrained multilingual models to underrepresented languages. Through comprehensive studies, we find that using synthetic data can significantly boost the performance of these languages while the best method depends on the data availability. Our results show that we can make concrete progress towards including under-represented languages into the development of NLP systems by utilizing alternative data sources.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Our work also has some limitations. Since we focus on different methods of using lexicons, we restrict experiments to languages in Latin script and only use English as the source language for simplicity. Future work could explore the effect of using different source languages and combining transliteration (Muller et al., 2021) or vocabulary extension (Pfeiffer et al., 2021) with lexicon-based data augmentation for languages in other scripts. We also did not test the data augmentation methods on higher-resourced languages as MLM fine-tuning and translate-train are already effective in that setting and our main goal is to support the languages with little textual data. Nonetheless, it would be interesting to examine whether our methods can deliver gains for high-resource languages, especially for test data in specialized domains.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We point to the following future directions: First, phrases instead of single word entries could be used to create pseudo data. Second, additional lexicons beyond PanLex could be leveraged. 6 Third, more effort could be spent on digitizing both existing monolingual data such as books (Gref, 2016) and lexicons into a format easily accessible by NLP practitioners. Although PanLex already covers over 5000 languages, some language varieties have only as little as 10 words in the database, while there exist many paper dictionaries that could be digitized through technologies such as OCR (Rijhwani et al., 2020). 7 Lexicon collection is also relatively fast, which could be a more cost effective strategy to significantly boost the performance of many languages without lexicons. Finally, the quality of synthetic data could be improved by incorporating morphology. However, we find that there is virtually no existing morphological analysis data or toolkits for the languages we consider. Future work could aim to improve the morphological analysis of these low-resource languages.  Figure 6: Improvements of using combined lexicons compared to PanLex lexicons for Pseudo MLM. Languages with fewer PanLex lexicons tend to benefit more from the combined lexicons.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
For all experiments using MLM training for NER tasks, we train 5000 steps, or about equivalent to 40 epochs on Bible; for MLM training for POS tagging and Parsing, we train 10000 steps, or equivalent to 80 epochs on Bible. We use learning rate of 2e − 5, batch size of 32, and maximum sequence length of 128. We did not tune these hyperparameters because we mostly follow the ones provided in (Ebrahimi and Kann, 2021). To finetune the model for a downstream task, we use learning rate of 2e − 5 and batch size of 32. We train all models for 10 epochs and pick the checkpoint with the best performance on the English development set.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We use a single GPU for all adaptation and finetuning experiments. Pseudo MLM usually takes less than 5 hours. Pseudo Trans-train and other task specific fine-tuning usually takes around 2 to 3 hours.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We use the many-to-many NMT models provided in the fairseq repoo (Ott et al., 2019). We use the model with 175M parameters and finetune the NMT model for 50 epochs on the parallel data from the Bible.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We use beam size of 5 to generate translations.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We plot the performance difference between using combined lexicons and PanLex for the Few-Text in Fig. 6. The languages are arranged from left to right based on increasing amount of PanLex entries. For MasakhaNER, the three languages with fewer entries in PanLex have much more significant gains by using the combined lexicon. While using the combined lexicons generally hurts POS tagging, the languages with fewer entries in PanLex tend to have less performance decrease. 
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Our experiments in Tab. 3 show that MasakhaNER benefits more from Pseudo Trans-train, likely because the labeled data is closer to the domain of the test data. However, this result might not hold when the amount of labeled data is limited. One advantage of Pseudo MLM over Pseudo Trans-train is that it only requires English monolingual data to synthesize pseudo training data, while Pseudo Trans-train is constrained by the availability of labeled data. We subsample the amount of English NER training data for MasakhaNER and plot the average F1 score of Pseudo Trans-train, pseudo MLM and using both. Fig. 7 shows that the advantage of Pseudo Trans-train on MasakhaNER decreases as the number of labeled data decreases, and using both methods is more competitive when the task data is small.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
We provide a list of bilingual lexicons beyond Pan-Lex:
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
• Swadesh lists in about 200 languages in Wikipedia 8
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
• Words in 3156 language varietities in CLICS 9
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
• Intercontinental Dictionary Series in about 300 languages 10
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
• 40-item wordlists in 5,000+ languages in ASJP 11
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
• Austronesian Basic Vocabulary Database in 1,700+ languages 12
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
• Diachronic Atlas of Comparative Linguistics in 500 languages 13