How good are Large Language Models at machine translation?
Finally, there is extensive work on analyzing
the translation output of LLMs, generally finding
that it is more fluent than accurate (Hendy et al.,
2023; Anonymous, 2023), good at handling doc-
ument context (Wang et al., 2023; Karpinska and
Iyyer, 2023) but also prone to problems such as
hallucination (Zhang et al., 2023; Guerreiro et al.,
2023), and frequently sub-par in low-resource set-
tings (Zhu et al., 2023; Bawden and Yvon, 2023)
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.
[{"first":"Amr","last":"Hendy","middle":[]},{"first":"Mohamed","last":"Abdelrehim","middle":[]},{"first":"Amr","last":"Sharaf","middle":[]},{"first":"Vikas","last":"Raunak","middle":[]},{"first":"Mohamed","last":"Gabr","middle":[]},{"first":"Hitokazu","last":"Matsushita","middle":[]},{"first":"Young","last":"Kim","middle":["Jin"]},{"first":"Mohamed","last":"Afify","middle":[]},{"first":"Hany","last":"Awadalla","middle":["Hassan"]}]
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating a massive number of languages? 2) Which factors affect LLMs' performance in translation? We evaluate popular LLMs, including XGLM, OPT, BLOOMZ, and ChatGPT, on 102 languages. Our empirical results show that even the best model ChatGPT still lags behind the supervised baseline NLLB in 83.33% of translation directions. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, prompt semantics can surprisingly be ignored when given in-context exemplars, where LLMs still show strong performance even with unreasonable prompts. Second, cross-lingual exemplars can provide better task instruction for low-resource translation than exemplars in the same language pairs. Third, we observe the overestimated performance of BLOOMZ on dataset Flores-101, indicating the potential risk when using public datasets for evaluation.
[{"first":"Wenhao","last":"Zhu","middle":[]},{"first":"Hongyi","last":"Liu","middle":[]},{"first":"Qingxiu","last":"Dong","middle":[]},{"first":"Jingjing","last":"Xu","middle":[]},{"first":"Shujian","last":"Huang","middle":[]},{"first":"Lingpeng","last":"Kong","middle":[]},{"first":"Jiajun","last":"Chen","middle":[]},{"first":"Lei","last":"Li","middle":[]}]
Document-Level Machine Translation with Large Language Models
Large language models (LLMs) such as Chat-GPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study fo-cuses on three aspects: 1) Effects of Discourse-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of Chat-GPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and examine the impact of training techniques on discourse modeling. By evaluating a number of benchmarks, we surprisingly find that 1) leveraging their powerful long-text mod-eling capabilities, ChatGPT outperforms commercial MT systems in terms of human evaluation. 2) GPT-4 demonstrates a strong ability to explain discourse knowledge, even through it may select incorrect translation candidates in contrastive testing. 3) ChatGPT and GPT-4 have demonstrated superior performance and show potential to become a new and promising paradigm for document-level translation. This work highlights the challenges and opportunities of discourse modeling for LLMs, which we hope can inspire the future design and evaluation of LLMs.
[{"first":"Longyue","last":"Wang","middle":[]},{"first":"Chenyang","last":"Lyu","middle":[]},{"first":"Tianbo","last":"Ji","middle":[]},{"first":"Zhirui","last":"Zhang","middle":[]},{"first":"Dian","last":"Yu","middle":[]},{"first":"Shuming","last":"Shi","middle":[]},{"first":"Zhaopeng","last":"Tu","middle":[]}]
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. However, their ability to translate paragraphs and documents remains unexplored because evaluation in these settings is costly and difficult. We show through a rigorous human evaluation that asking the Gpt-3.5 (text-davinci-003) LLM to translate an entire literary paragraph (e.g., from a novel) at once results in higher-quality translations than standard sentence-by-sentence translation across 18 linguistically-diverse language pairs (e.g., translating into and out of Japanese, Polish, and English). Our evaluation, which took approximately 350 hours of effort for annotation and analysis, is conducted by hiring translators fluent in both the source and target language and asking them to provide both span-level error annotations as well as preference judgments of which system's translations are better. We observe that discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. With that said, critical errors still abound, including occasional content omissions, and a human translator's intervention remains necessary to ensure that the author's voice remains intact. We publicly release our dataset and error annotations to spur future research on evaluation of document-level literary translation.
[{"first":"Marzena","last":"Karpinska","middle":[]},{"first":"Mohit","last":"Iyyer","middle":[]}]
Hallucinations in Large Multilingual Translation Models
Large-scale multilingual machine translation systems have demonstrated remarkable ability to translate directly between numerous languages, making them increasingly appealing for real-world applications. However, when deployed in the wild, these models may generate hallucinated translations which have the potential to severely undermine user trust and raise safety concerns. Existing research on hallucinations has primarily focused on small bilingual models trained on high-resource languages, leaving a gap in our understanding of hallucinations in massively multilingual models across diverse translation scenarios. In this work, we fill this gap by conducting a comprehensive analysis on both the M2M family of conventional neural machine translation models and ChatGPT, a general-purpose large language model~(LLM) that can be prompted for translation. Our investigation covers a broad spectrum of conditions, spanning over 100 translation directions across various resource levels and going beyond English-centric language pairs. We provide key insights regarding the prevalence, properties, and mitigation of hallucinations, paving the way towards more responsible and reliable machine translation systems.
[{"first":"Nuno","last":"Guerreiro","middle":["M."]},{"first":"Duarte","last":"Alves","middle":[]},{"first":"Jonas","last":"Waldendorf","middle":[]},{"first":"Barry","last":"Haddow","middle":[]},{"first":"Alexandra","last":"Birch","middle":[]},{"first":"Pierre","last":"Colombo","middle":[]},{"first":"Andr'e","last":"Martins","middle":["F.","T."]}]
Prompting Large Language Model for Machine Translation: A Case Study
Research on prompting has shown excellent performance with little or even no supervised training across many tasks. However, prompting for machine translation is still under-explored in the literature. We fill this gap by offering a systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection. We further explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting. Extensive experiments with GLM-130B (Zeng et al., 2022) as the testbed show that 1) the number and the quality of prompt examples matter, where using suboptimal examples degenerates translation; 2) several features of prompt examples, such as semantic similarity, show significant Spearman correlation with their prompting performance; yet, none of the correlations are strong enough; 3) using pseudo parallel prompt examples constructed from monolingual data via zero-shot prompting could improve translation; and 4) improved performance is achievable by transferring knowledge from prompt examples selected in other settings. We finally provide an analysis on the model outputs and discuss several problems that prompting still suffers from.
[{"first":"Biao","last":"Zhang","middle":[]},{"first":"Barry","last":"Haddow","middle":[]},{"first":"Alexandra","last":"Birch","middle":[]}]
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
The NLP community recently saw the release of a new large open-access multilingual language model, BLOOM (BigScience et al., 2022) covering 46 languages. We focus on BLOOM's multilingual ability by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high- and low-resourced). Our results show that 0-shot performance suffers from overgeneration and generating in the wrong language, but this is greatly improved in the few-shot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, cross-lingual transfer and the use of discursive context.
[{"first":"Rachel","last":"Bawden","middle":[]},{"first":"Franccois","last":"Yvon","middle":[]}]
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Recent advancements in natural language processing (NLP), particularly the development of largescale language modeling techniques, have brought remarkable improvements in machine translation as well as in other NLP tasks (Fan et al., 2021;Kim et al., 2021;Costa-jussà et al., 2022). The emergence of large language models with diverse capabilities, including machine translation, has opened * Corresponding author: hanyh@microsoft.com up new possibilities for building more effective translation systems (Brown et al., 2020;Chowdhery et al., 2022). Among these models, the latest Generative Pre-trained Transformer (GPT) models (Brown et al., 2020) have gained significant attention for their ability to generate coherent and context-aware text. We present a comprehensive evaluation of GPT models for machine translation, exploring their strengths and limitations, and providing insights for researchers and practitioners working in the area of machine translation.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
GPT models and the conventional Neural Machine Translation (NMT) systems are both based on the transformer architecture (Vaswani et al., 2017), but they differ in several aspects. First, GPT models are decoder-only models that use the same parameters to process the context and the source as a single input for generating the next output. On the other hand, NMT models usually have an encoderdecoder architecture that encodes the source sentence in the encoder network and decodes the target sentence conditioned on the previous outputs in the decoder network. Second, GPT models are mainly trained on monolingual data, with a strong bias towards English 1 , whereas NMT models rely on large amounts of highly curated parallel data. Third, GPT models need a much larger number of parameters to achieve multilingual in-context capabilities. We have observed that GPT models exhibit promising translation capabilities, even with these differences in architecture and training data.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
The performance of GPT models in machine translation, despite their promising potential, remains under-investigated relative to commercial and state-of-the-art research systems. This study aims to address this research gap by systematically assessing the efficacy of GPT models for machine translation, with a focus on their performance, prompts, document-level translation, domain robustness and possible advantages of integrating them with conventional NMT systems.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
To explore the potential of GPT models for translation, we perform comprehensive experiments to examine their translation abilities. Specifically, we investigate the performance of GPT models for machine translation across 18 language pairs, covering both high and low-resource languages, as well as English-centric and non-English-centric directions.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We compare the quality of three GPT models: textdavinci-002, text-davinci-003 (GPT3.5), and Chat-GPT, and show that they differ significantly in their translation capabilities.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We also explore the impact of prompting strategies on the performance of GPT models for machine translation. We examine both the content and the form of the prompts, and identify the best practices for obtaining optimal results. Furthermore, we test the hypothesis that GPT models would enhance document-level translation, as they could exploit the context and coherence of the entire document to generate more accurate and fluent translations. We evaluate this hypothesis on various language pairs using several metrics. Additionally, we evaluate the cross-domain generalization ability of GPT models for translation tasks and examine their robustness under domain shift.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Moreover, we conduct extensive human evaluations and analyses to provide valuable insights into the strengths and weaknesses of GPT models for machine translation, and to suggest directions for future work. We also perform comprehensive analyses to understand whether GPT and NMT models have complementary characteristics, and we propose several ideas to combine the advantages of the two paradigms. Finally, we touch on the effectiveness of GPT models on cross-lingual natural language tasks beyond translation, and explore their multilingual capabilities and limitations.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
To explore the research questions above, we organize the paper as follows:
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• We provide a detailed experimental setup ( §2), which includes the datasets used ( §2.1), the machine translation systems used in the comparison( §2.2), the GPT systems ( §2.3), and the evaluation methods ( §2.4).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• We present a series of experiments ( §3) that investigate different aspects of GPT models for machine translation. These experiments cover prompt selection strategies ( §3.1), zero-shot translation capabilities of GPT models ( §3.2), GPT performance on high-resource languages ( § 3.3), GPT performance on low-resource and non-English-centric languages ( § 3.4), document-level MT with GPT ( §3.5), translation robustness toward domain shift ( §3.6), and hybrid GPT and NMT translation ( §3.7).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• We then present the human evaluation and analysis that provides insights into the quality of GPT translations ( §4).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• We discuss the characteristics of GPT translations ( §5) vis-à-vis NMT and analyze the differentiating aspects of GPT translations ( §5.1) by quantitatively enumerating language modeling bias artifacts ( §5.2), the characteristics of translation across various language directions ( §5.3- §5.5), as well as parallel data bias artifacts ( §5.6).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• We explore the multilingual capabilities of GPT models beyond translation ( §6).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• We conclude by summarizing our findings and suggesting future directions for research ( §7).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We considered 18 different translation directions across a diverse set of languages for our comprehensive evaluation. The evaluation covers both high-and low-resource languages, as well as English-centric and non-English-centric direct translations. The languages considered in this study include European (English-EN, French-FR, German-DE, Czech-CS and Icelandic-IS), Asian (Chinese-ZH and Japanese-JA), Cyrillic (Russian-RU and Ukrainian-UK) and African (Hausa-HA). We use publicly available datasets to facilitate reproducibility and data sharing. We use the WMT22 testsets 2 for all languages except for Icelandic and Hausa, for which we use the WMT21 testsets 3 . We use WMT22 datasets for two reasons: first, they are recent and less likely to overlap with the GPT models training data that was collected until June 2021 4 . Second, they have natural source texts and translated target texts, which avoid the problems of using "translationese" testsets with unnatural source texts in the original language that may result in inaccurate evaluation (Zhang and Toral, 2019). Table 1 summarizes the datasets and sizes used in this paper. We focus on the most recent datasets with directional translation, as older datasets may have influenced the training data of the GPT models. We make all data and analysis in this paper publicly available to promote further research. 5 .  Table 1: Test datasets used in the evaluation and best systems used for comparison as reported in WMT by Kocmi et al. (2022b) and Barrault et al. (2021).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In this study, we compare the performance of GPT systems against both state-of-the-art (SoTA) research and commercial systems. We use the top ranked systems (WMT-Best) in the WMT evaluation campaigns for each language pair, which we use as a baseline for comparison. WMT-Best systems are a mix of top ranked commercial and research systems. We use the system outputs as provided by the evaluation campaigns (Kocmi et al., 2022b;Barrault et al., 2021). Table 1 shows the list of top ranked system for each language pair. We also utilize Microsoft Translator which we access through the public API available on Azure 4 https://help.openai.com/en/articles/ 6639781-do-the-openai-api-models-have\ -knowledge-of-current-events 5 https://github.com/microsoft/gpt-MT Cognitive Services 6 .
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We assess the latest three variants of the largest GPT models available, as listed at OpenAI's documentation 7 . These models are:
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• text-davinci-002 -an InstructGPT model (Ouyang et al., 2022) which utilizes Reinforcement Learning with reward models trained based on human comparisons.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• text-davinci-003 -an improved version of textdavinci-002.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• ChatGPT -a model that is similar to the previous two and optimized specifically for conversational purposes 8 .
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
All GPT models have been accessed through APIs on Microsoft Azure OpenAI Service 9 .
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Sentence-Level Evaluation The MT Metrics shared task  recommends the use of neural network-based metrics in machine translation evaluation, as they have demonstrated a high correlation with human evaluation and are resilient to domain shift. Following these recommendations, we employ the top-ranked metrics from Unbabel in the shared task 10 . Specifically, we use COMET-22 (wmt22-COMET-da) (Rei et al., 2022a), a reference-based metric that combines direct assessments (DA), sentence-level scores, and word-level tags from Multidimensional Quality Metrics (MQM) error annotations. For referenceless quality estimation, we adopt COMETkiwi (wmt22-COMETkiwi-da) (Rei et al., 2022b). Additionally, we report results using SacreBLEU 11 and Chrf (Popović, 2015) for completeness, although we note that these metrics are not extensively used in our analysis.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
For the experiments on document-level translation using GPT, we face a challenge in evaluating performance due to the lack of metrics that can handle the non oneto-one sentence mapping that the systems may produce. To address this challenge, we have adapted the COMET metrics to better suit document-level evaluation. The adaptation involves splitting the document into multiple segments with an overlapping sliding window and computing the average score across these segments to compare two documents. We use Doc-COMETkiwi to refer to the modified metric throughout the evaluation.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
This simple modification has three clear design advantages over a pure sentence-level evaluation. First, it allows each sentence to be evaluated within its context. Second, it enables each sentence to be evaluated across multiple contexts due to the overlapping nature of the sliding window. Lastly, it avoids the limited context window of the evaluation models that could hinder quality assessment over longer static windows on documents. We do not claim that this is an optimal metric for evaluating document-level translation, but it overcomes the limitation of the one-to-one sentence mapping and may capture the quality of translation in ambiguous contexts better than sentence-level representation. We argue that developing more robust documentlevel metrics is still essential.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
The current metrics for evaluating machine translation performance may not be adequate for measuring the performance of GPT models, and it may be necessary to develop new metrics that take into account the unique characteristics of these models.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We perform human evaluation ( § 4) using source-based sentence-level contrastive Direct Assessment + Scalar Quality Metric (contrastive DA+SQM; Akhbardeh et al. 2021, Kocmi et al. 2022a, with annotations provided by professional annotators. We also conduct thorough analysis on various characteristics of the translation ( §5).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In this section, we present various experiments. In §3.1, we describe several prompt selection strategies. In §3.2, we evaluate various GPT models in a zero-shot setup. In §3.3, we show results for highresource language pairs, followed by results for low-resource and non-English pairs in §3.4. §3.5 provides document translation results. §3.6 examines the robustness of GPT models under domain shift. Finally, in §3.7, we discuss the potential of combining the benefits of GPT and NMT models.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
It has been shown that the performance of LLMs can be enhanced through in-context learning by providing few labelled examples (prompts) in addition to the test input (Brown et al., 2020). This few-shot paradigm has demonstrated strong performance across multiple natural language processing (NLP) tasks (Ouyang et al., 2022;Goyal et al., 2022;Wei et al., 2022;Chowdhery et al., 2022). There has also been a series of recent works on incontext learning for machine translation (MT) with rather mixed results and various ways of shot selection. Recently, Zhang et al. (2023) use GLM-130B and show consistent but rather low correlation between the performance of MT and shot selection as compared to random. They use different features that show varying level of correlation to the performance. In the same spirit, Vilar et al. (2022) use different prompt selection schemes with PaLM-540B model with the main conclusion that shot selection is not necessarily better than random but they point out the importance of using high quality shots. In the same vein, Agrawal et al. (2022) use a much smaller model XGLM-7.5B and multiple selection criteria for the shots. They show that a combination of a retrieval and task metrics are consistently better than the random baseline across different translation directions.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In this paper, we explore prompt selection strategies along two dimensions: quality and relevance. Our pool to select few-shot examples is the cleaned WMT training data for each direction. This is obtained by filtering the full training data using language identification and length ratio. The size of the full and cleaned training data is shown in Table 12 for each direction. We do not use the development data from WMT shared tasks, despite its high quality, for shot selection to avoid any small chance of leaking information about the test set. In all cases, we test the performance with 0, 1 and 5 shots. In our preliminary experiments, we found that increasing beyond 5 shots did not result in any meaningful improvement. We show below how we select shots based on quality and relevance.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• Quality: To ensure high quality shots we sort our training data using LaBSE (Feng et al., 2020). We consider high quality shots that are randomly chosen from the top 1 million pairs as opposed to the full data. We also found it useful to select sentences that are longer than 50 tokens.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• Relevance: We consider relevant shots that are close to the input sentence. Based on preliminary experiments 12 we use the cosine distance between LaBSE embeddings as a measure of closeness. We always select relevant pairs from high quality ones (the top 1M pairs from LaBSE-scored training data). For computational efficiency, we adopt a two stage approach. We first use the input text and apply elastic search 13 to retrieve the top 64 pairs then return the top 1 or 5 shots based on the LaBSE distance.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In the results, we refer to full random as RR (Random) while high quality are referred to as QR (Quality Random). The high quality shots selected through relevance are referred to as QS (Quality Selected).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We compare the general zero-shot translation capabilities of the three GPT models on four language pairs, in eight distinct translation directions. The selected languages were chosen with a focus on balancing representation. The languages include 1) German, which is one of the most represented non-English languages in GPT training data, 2) Russian, a large-scale non-Latin language, 3) Chinese, which represents a large-scale language with a script distinct from that of the majority of training data languages, and 4) French-German pair as non English-centric use case.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In this experiment, we compare the performance of three GPT models text-davinci-002, text-davinci-003, and ChatGPT with the top ranked systems in WMT22, as shown in Table 2. Remarkably, text-davinci-002 shows lower performance, underperforming across all language pairs compared to the other two GPT models. On the other hand, text-davinci-003 clearly shows better translation performance across all languages in this evaluation. Its zero-shot performance is comparable to the best performing WMT DE-EN system and outperforms the best ZH-EN WMT system.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
ChatGPT shows a storng performance in the DE-EN language pair, while performing similarly to text-davinci-003 when translating to English as well as French-German pairs. In terms of translat- 12 We also tried using COMET and embeddings based on a trained NMT model. 13 Based on BM25 text retrieval.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
ing from English to other languages, text-davinci-003 shows better performance than the other two GPT models. It is noteworthy that the translation between French and German, which is not Englishcentric, exhibits surprising competitiveness with state-of-the-art systems, despite the fact that the majority of the training data used for the GPT model is English-centric. While both COMETkiwi and COMET-22 show relevant results, both lexical metrics (BLEU and ChrF) show consistent degradation with GPT models. This is consistent with similar findings from (Vilar et al., 2022) on PALM-540B model. We conducted human evaluation and more thorough analysis to further understand such results.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
From these results, we can see that the three variants of GPT models exhibit different characteristics. However, the nature and extent of these differences remain unclear and require further investigation, depending on the availability of more information regarding the models, their training data, and their training methods. This superior performance of text-davinci-003, which is achieved in a zero-shot setting, motivates further investigation into the effect of few-shot in-context learning and shot selection strategies. We investigate these questions further in the sections below.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Given the results in the previous section, we focus on evaluating text-davinci-003 model, expanding the scope of the study to 18 language pairs and comparing its performance with that of a commercial system (Microsoft Translator) in addition to WMT SoTA systems. For consistency, in all the subsequent results, we use the term "GPT" to denote the text-davinci-003 model, unless explicitly stated otherwise. We experiment with various shot selection strategies: zero-shot, random (RR), quality (QR) and relevance selected (QS) prompts as described in §3.1. We report results for 1 and 5 shots along with the best WMT systems and MS-Translator. Table 3 shows the performance of GPT textdavinci-003 with few-shot setups on high-resource languages from WMT Testsets. With both reference and reference-less COMET scores, the model achieved impressive zero-shot results for all languages when translating into English. However, the few-shot configurations did not yield signifi-  cant improvements over the zero-shot setup. GPT surpassed both the WMT-Best and MS-Translator systems for DE-EN, JA-EN and ZH-EN language pairs, and almost matched the best systems for the other three language pairs. On the other hand, when translating from English to other languages, the few-shot setup consistently improved over the zeroshot setup, with most gains obtained from a single high-quality shot. GPT outperformed both the WMT-Best and MS-Translator systems for EN-JA and EN-ZH language pairs. We experimented with high quality shots with relevance scores (QS) for three languages (German, Russian and Chinese), but we observed no improvements over quality shots alone. This result emphasises the importance of fewer quality shots especially when translating from English. This difference in behavior is consistent with the observations that the critical role of demonstrations within in-context learning is to provide specifications of the output space (Min et al., 2022;Anonymous, 2023a), with a denser in-context learning signal being preferred when translating from English to other languages. Similar to the Zero-Shot results form §3.2, we observe that the lexical metrics are showing consistent degradation with all GPT models and configurations.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We evaluated low-resource and non English-centric languages by conducting experiments with Icelandic and Hausa as two low-resource languages and French and German as direct translation languages. Table 4 presents the results. The few-shot setup yields modest gains, especially when translating out of English. Similar to the high-resource case, most of the gains were obtained from a single high-quality shot. The systems for both lowresource languages did not surpass the WMT-Best systems. The DE-FR and FR-DE language pairs show remarkable results, as a single-shot setup outperforms the zero-shot setup significantly. This is consistent with the previous finding of translating from English to other languages; a more dense in-context signal is essential for direct translation as well, as it enables the model to generate in the correct language better than the zero-shot behavior. Both direct systems surpasses their commercial counterparts in terms of COMET scores, but they slightly trail behind the WMT-Best systems on the COMET-22 reference-based metric. Similar to the high-resource language pairs, both lexical metrics (BLEU and ChrF) showed a significant and consistent degradation. To gain further insights into this, we conducted human evaluation and performed a more in-depth analysis as discussed in §4 and §5.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
This section explores the application of GPT to document-level machine translation. Previous studies on MT with LLMs have mainly concentrated on sentence-level translation, with only a brief mention of document translation for transfer learning by Zhang et al. (2023)   We hypothesize that GPT can excel at documentto-document translation, as it is trained on large contexts. Moreover, translating entire documents can reduce the number of API calls and thus improve the computational efficiency and latency. We argue that document translation improvements may need better metrics to capture its potential. Hence, in this section, we report results using doc-BLEU (Liu et al., 2020) and doc-COMET metrics as described in §2.4, in addition to our sentence-level metrics.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Evaluating Document-level Translations A document-level translation does not necessarily keep sentence-level alignments intact. We try to prompt GPT to keep sentence-level alignment intact by emphasizing sentence separation in the prompt. Our prompt template can be found in Figure 18 from the appendix. However, we find that we still need to restore sentence-level alignment with source for some of the documents in the test set. In all cases, we find two types of mismatch that we need to restore. First, sentences that were written in the source over two lines and their translation is one line. In that case, we insert a new line break to match the position of the new line break in the source sentences. Second, sentences that are skipped. In that case, we replace empty lines at the end of the documents with empty lines in place of the skipped sentences. We need to restore sentence-level alignment to calculate metrics that were mainly developed for sentence-level evaluation such as the COMET-22 and COMETkiwi models that we are using. We also follow (Liu et al., 2020) and report SacreBLEU calculated on the document level. For neural network based metrics, we extend the COMETkiwi model for document level evaluation as described in §2.4. Experiment 1: We conduct a series of experiments on zero-shot MT using News Commentary dataset, varying the window length from 1 (sentence-level) to 32 in powers of two 14 . Table 5 shows that increasing the window size leads to improvements across all metrics. However, the gains in lexical metrics (BLEU and ChrF) are larger than in neural metrics (COMET-22 and COMETkiwi). The document-based metric (doc-BLEU and doc-COMETkiwi) also exhibit similar improvements to the sentence-based metric. Remarkably, as the window size grows, the performance surpasses MS-Translator model and approaches the WMT-Best systems. This is consistent with the findings of Sun et al. (2020) for conventional MT models. The table also displays the total number of requests for each window size. We observe that the number of requests decreases dramatically as the window size increases, while the performance either improves significantly or remains relatively stable depending on the metric used. Therefore, this document-level setup achieves high efficiency without compromising the quality.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Experiment 2: The second set of experiments investigates few-shot translation for the document setting. Following the sentence experiments, we focus on using 5-shots. We also use the News Commentary dataset, which has document-level annotations. Table 6 summarizes the results. The first two rows show the best WMT22 and the MS-Translator results for reference.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
The following rows are named GPT-XX-YY where XX stands for the scope of translation (sentence or document) and YY stands for the source of the 5 shots (QR,DR,DF or DH as explained below). Rows GPT-Sent-QR and GPT-Sent-DR show results for sentence-level translation. The former uses the same quality-based shots from Table 3, while the latter uses 5 randomly selected shots from the document set, excluding the test data. We perform document translation (referred to as Doc in the table) for a window of 10 sentences in the following rows. Rows named GPT-Doc-QR and GPT-Doc-DR use the same shots as the sentence case. For row GPT-Doc-DF, we select a random document from the document data pool and use the first 5 sentences of the document as shots (i.e. document first DF). For row GPT-Doc-DH, we store GPT outputs in history for use in shots. We translate the first document 0-shot, and the subsequent documents 5-shots. For shot selection, we pick a random document from the previously translated documents and use the first 5 sentences as shots (i.e. document history DH). The results indicate that document translation outperforms sentence translation across metrics. However, while few-shots yield some consistent gains for sentence translation, this is not the case for document translation. This may be explained by the fact that document translation provides enough context, making few-shots redundant. It can be also observed from the table that the Doc-COMETKiwi shows more gain than the sentence level metrics but this might need more indepth analysis to verify.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We use the WMT datasets to examine how domain shift affects the performance of GPT models on German and Chinese, both from and to English. The WMT22 datasets span four domains: Conversational, News, e-Commerce and Social. Table 7 shows the scores of the four domains on the WMT testsets.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
GPT achieves remarkable improvements on the conversational domain for DE-EN, ZH-EN and EN-ZH, as evidenced by both COMET and lexical scores (BLEU and ChrF). This contrasts with previous observations where lexical scores consistently deteriorate with GPT models.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
GPT performs comparably to the other systems on the news domain according to COMET scores for all directions. It surpasses both other systems on DE-EN, while slightly trailing behind on EN-DE. For ZH-EN and EN-ZH, GPT exceeds MS-Translator, but falls slightly short of WMT-Best system. However, GPT scores significantly lower in terms of BLEU metric for both ZH-EN and EN-DE.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
GPT clearly outperforms both systems on ZH-EN and matches WMT-Best on DE-EN for the ecommerce domain. It slightly lags behind on other directions. In this domain, we observe consistent lower scores in BLEU metric for all directions even for ZH-EN which outperforms significantly on both COMET metrics.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
GPT outperforms both systems on DE-EN for the social domain. However, on ZH-EN and EN-ZH, GPT only surpasses them on COMETkiwi, while showing lower BLEU score for all directions with a significant difference in ZH-EN which exhibits substantial gains on COMETkiwi.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
The results demonstrate GPT's robust translation capabilities across different domains and languages. It performs well on DE-EN, ZH-EN and EN-ZH for all domains. However, we observe a discrepancy on lexical metrics for ZH-EN and DE-EN on News and Social domains even with GPT's high performance on those languages. We conduct a further examination of the ZH-EN results to gain more insights. We find that the news from Chinese outlets follows a more templatic style especially in the prefix of the news. For NMT systems that are trained   heavily on similar data, it is easier to reproduce the same patterns, e.g., WMT-Best scores 31.3 BLEU. For more general commercial scale systems that are trained on much larger and diverse data, it is harder to produce the same exact patterns, as such, MS-Translator scores 28.2 BLEU. For GPT, which is mostly trained on English, it is harder to get the lexical matches and it mostly produces the English news style, scoring 23.3 BLEU. However, COMET-22, using the same reference, seems to provide a more robust signal with all three systems being almost on the same level. This confirms the robustness of neural metrics across domains  and GPT's ability to handle diverse domains while being more robust towards parallel data biases, which we will explore further in §5. Therein, we show that in general, GPT performs better in cases where the input resonates with the noisy parts of parallel data.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
To explore the possibility of leveraging the strong performance of GPT on various languages, we propose and evaluate several hybrid approaches that combine the strengths of NMT and GPT systems. The basic idea is to use Microsoft Translator (MS-Translator) system as the primary translation system, and then use GPT as a fallback system when the quality of MS-Translator is unsatisfactory. We use COMETkiwi as the quality estimation model and COMET-22 as the performance evaluation metric. We first establish an upper bound by selecting the best translation from either systems according to COMETkiwi, which we call the "Max-Routing" approach. Then we experiment with a more practical approach where we use GPT only when the COMETkiwi score of MS-Translator falls below a predefined threshold. In this experiment, we set the threshold to the 50-th percentile of the COMETkiwi scores of MS-Translator, meaning that we use GPT for any translation that has a COMETkiwi score lower than the median MS-Translator COMETKiwi score, which can be easily estimated from previous translation requests. Figure 1 presents the results of our experiments on 12 language pairs. Firstly, we observe that in all language pairs, the "Hybrid Max-Routing" approach consistently achieves the highest COMET-22 scores, surpassing both the individual systems. "Hybrid Max-Routing" achieves a maximum gain of 1.6 Comet-22 points in the EN-UK language pair which is not among top performance language for GPT. This indicates that combining the strengths of NMT and GPT systems can lead to a significant improvement in translation quality.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Next, we compare the performance of the individual systems. In general, MS-Translator achieves higher scores than GPT on most language pairs, which is expected given that MS-Translator is an NMT system specifically optimized for translation tasks. However, GPT outperforms MS-Translator on certain language pairs, such as DE-EN, EN-JA, and EN-ZH. This suggests that GPT can be a valuable fallback system in cases where the quality of the primary system is unsatisfactory.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We also compare the performance of the two hybrid approaches. The "Hybrid Max-Routing" approach achieves slightly higher scores than the "Hybrid Threshold" approach on most language pairs, indicating that routing to GPT only when the quality of MS-Translator falls below a certain threshold may not always be the optimal strategy. However, the "Hybrid Threshold" approach still achieves comparable results to the upper bound on all language pairs, while using GPT for only 50% of the instances. This suggests that it can be a more practical approach in scenarios where computational resources are limited. Figure 2 demonstrates that hybrid approaches achieve larger and more consistent improvements than shots selection across all languages and directions. Figure 3 zooms in on the high-performing DE-EN and EN-DE systems. The hybrid system outperforms both WMT-Best and MS-Translator systems in both directions, even though the GPT system only outperforms them in DE-EN with 5shot setup.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In summary, our experiments demonstrate the potential of combining NMT and GPT systems to improve machine translation quality. The results suggest that a hybrid approach that uses GPT as a fallback system can achieve higher performance than either individual systems. Future research can explore more advanced techniques that can leverage the strengths of both systems and optimize the hybrid approach. WMT-Best systems from Table 1 and GPT with 5 shots QR as shown in Table 3. For each language pair, we randomly sample 425 non-identical translation item pairs and have them annotated with the contrastive DA+SQM annotation method by 5 distinct professional translation experts per language pair. Figure 4 and Figure 5 report aggregated human and corresponding COMETkiwi scores. Surprisingly, GPT outperforms Best-WMT systems on CS-EN, ZH-EN, EN-ZH and DE-FR, and achieves comparable results on most of the high-resource languages. On the other hand, the two low-resource languages, Hausa and Icelandic, lag behind significantly. Full details of the scores can be found in the appendix, Table 13.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We observe that the human evaluation results are highly consistent with the COMETkiwi results. This highlights the importance of neural reference-less metrics for evaluating MT in general and this family of models in particular. As we have seen in the previous results, all lexical metrics fail to capture the strong performance of GPT and exhibit lexical and reference bias. While we believe quality estimation is becoming more essential for MT in general, it is reassuring to know that COMETkiwi performs well on GPT models as well as on NMT models. Moreover, as shown on Figure 6, highly performing GPT languages pairs demonstrate higher win rate which is reflected on both Human Evaluation results and COMETkiwi scores.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We conducted a manual analysis of humanevaluated GPT translations for English-Japanese and Japanese-English directions to identify their strengths and weaknesses. In Table 14 in the appendix, we present some of the observed characteristics along with examples of GPT and WMT outputs. A notable characteristic is that GPT performs better and more robustly than WMT for source sentences that are erroneous, short, or colloquial. We found that GPT can handle misspellings or unclosed quotation marks and produce translations that do not omit any semantic information. Moreover, GPT can generate reasonable translations for partial or incomplete colloquial source sentences while WMT-Best often adds or omits content. However, GPT tends to produce unnatural translations for sentences with uncommon or complex expressions. 
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We posit that there are two key biases due to which the computation of translation done by LLMs might be different from the same computation done by NMT models, namely the Parallel Data Bias and the Language Modeling Bias.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Parallel Data Bias: Compared to NMT models trained on parallel data, which is typically webmined (and noisy), LLMs such as GPT are trained on monolingual data only with no explicit supervisory signal for the translation task. This creates interesting implications on the nature of the emergent computational abilities leveraged for our task of interest, translation. First, not using parallel data might imply that LLMs are protected against the noise associated with parallel data, which leads to problems such as the memorization of noisy/atypical (Raunak et al., 2021) or low-quality samples  and biases towards particular language characteristics predominant in the parallel data (Garcia et al., 2023). These parallel data biases can also manifest in the form of long-tailed errors such as the translations of physical units or currencies , owing to a preponderance of such incorrect token pairings in the parallel data. On the other hand, the lack of explicit supervisory signals for the task could also mean that LLM based translations might not track the desired characteristics of translations such as faithfulness to the source as well as the NMT models trained with explicit teacher-forced supervision (Anonymous, 2023b).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Language Modeling Bias: Despite the impressive performance of in-context learning, constraining LLM behavior to explicitly follow the specifications of a desired task is a non-trivial problem. Analyses of in-context learning have revealed how the implicit zero-shot performance of LLMs might be higher than their observed zero-shot performance, with the demonstrations within in-context learning themselves providing only limited learning signals (Min et al., 2022;Kojima et al., 2022;Anonymous, 2023a). A direct implication of these results for translation is that the demonstrations used for in-context learning might fail to override the underlying computational bias of language modeling which is likely to favor greater fluency at the cost of adequacy. Such language modeling bias might also introduce undesirable artifacts, e.g., punctuation insertions, acronym expansions, world knowledge insertion, etc. in the translations which could cause it to veer off from a faithful cross-lingual representation of the input.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In the next subsection, we propose properties along which finer-grained characteristics of GPT translations could be enumerated. These measures are designed to provide indirect measurements of the language modeling bias as well as the parallel data bias, which could allow a better differentiation of GPT translations against translations from  Table 8: Illustrated Examples of the Phenomena as described in Section 5. The origin of these differences between translations lie in the computational mechanism leveraged for translations: When controlled for quality, higher translation non-monotonicity suggests a more abstractive computation used for obtaining the translations. Similarly, Fluency, Punctuation Insertion, Dropped and Inserted Content measure different translation characteristics. NMT systems. We first discuss the measurements designed to elicit artifacts associated with the language modeling bias.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We propose and use five measurements over the test sets to quantitatively explore language modeling bias, in order to enumerate the differences in translations obtained from traditional NMT systems and GPT. Below, we describe the properties as well as the algorithms used for quantifying them (corresponding illustrative examples of the phenomena are presented in Table 8 tracks the source sentence. A more paraphrastic or a less literal translation is likely to deviate from a close tracking of the source word order (across language pairs). We use the nonmonotonicity metric proposed in Schioppa et al. (2021), which computes the deviation from the diagonal in the word to word alignment as the non-monotonicity measure. This measurement could also be interpreted as a normalized measure of alignment crossings, which has been shown to correlate with translation non-literalness (Schaeffer and Carl, 2014). This measurement has also been used in Anonymous (2023b) for investigating translation literalness. 2. Translation Fluency (TF): We measure translation fluency using a strong, independently trained language model ('gpt2-large', Radford et al. (2019)). We restrict this measurement to X-E direction, since GPT-2 has only been trained on English text (Radford et al., 2019).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
3. Punctuation Insertion (PI): Language modeling bias can prefer one mode of sentence completion in contrast to others. This can reveal itself in the presence of not well-formed inputs such as sentences that do not end with typical end of sentence markers (comma, period and exclamation). We measure the fraction of input sentences for which the translation contains an end of sentence marker but the source does not. The insertion of an end of sentence marker in such instances is inadequate for translation, a task which strives for bitext equivalency.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We measure the number of source words left unaligned in a word to word alignment obtained over the source and output translations. When controlled for quality, a more paraphrastic translation is likely to contain more words that do not align with the words in the source sentence. This measurement was used in Anonymous (2023b) as a measure of translation literalness and we use it similarly to obtain a measurement of content that is dropped in a translation -an untranslated word or phrase in a source sentence is likely to find no alignments in the output. For obtaining word to word alignments, we use a multilingual-bert based aligner (Devlin et al., 2019;Dou and Neubig, 2021). Translation Non-Monotonicity (E-X) MS Translator GPT Figure 15: Comparisons of Translation Non-Monotonicity for the E-X language pairs. GPT Translations score higher on translation non-monotonicity for 3 out of 4 high-resource language pairs. measure the number of unaligned words in the translation using the same word to word alignments as in the previous measurement. This indicates the presence of words that have no support in the source and is included to measure words that are potentially inserted in the translation without any basis in the input.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We collect the measurements on these properties over the test sets for all the language pairs under investigation. We compare MS Translator with GPT throughout. We report the results in the next section and present our analysis grouped by the translation direction.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Figures 7, 8, 9, 10 and 11 represent the comparisons of GPT translations against MS Translator for X-E language pairs. Figure 7 shows that GPT translations obtain lower perplexities, thereby demonstrating greater fluency. Figure 8 shows that GPT translations suffer from the problem of punctuation insertion with a much higher frequency than MS translator. We attribute this to the language modeling bias, which would prefer to generate a well-formed sentence, even if such well-formedness is unsupported in the input. Figure 9 shows that GPT translations incur slightly higher number of unaligned source words on 7 out of 8 eight language pairs. Greater unaligned source words would imply either the presence of greater paraphrasticity in the translations or greater inadequacy (dropped or inserted content). Figure 10 shows that GPT translations incur almost similar number of unaligned target words, suggesting that the GPT  Table 9: Translation Characteristics Comparisons for De-Fr and Fr-De: GPT shows a much higher tendency to add an end of sentence marker into the translation when it is absent in the source.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
translations are similarly adequate in terms of potential insertions. Another measurement, presented in Figure 11 shows that GPT translations are more non-monotonic than its NMT counterpart.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Figures 12, 13, 14 and 15 represent the comparisons of GPT translations against MS Translator for E-X language pairs. Figure 12 shows that similar to X-E translations, GPT E-X translations also suffer from a higher frequency of punctuation insertions. The magnitude of the difference however, is smaller than the X-E translations, suggesting a weaker language modeling bias for these languages. Figure 13 shows that in general, GPT translations incur greater number of unaligned source words than its NMT counterpart. Figure 14 shows that the number of unaligned translation words for GPT translations do not differ greatly from MS Translator. Similarly, Figure 15, which compares translation non-monotonicity shows no aggregate trends. As such, we find that the translation characteristics for E-X language language pairs depends heavily on the individual language pair under consideration. Table 9 reports the results of the five measurements for De-Fr and Fr-De translation directions. The results for direct translation pairs are quite different from the X-E and E-X cases, since typically non-English centric translations are done through pivoting. As such, the trends for measurements over Fluency (F), Unaligned Source Words (USW), Unaligned Translation Words (UTW) and Translation Non-Monotonicity (NM) do not show any conclusive evidence of greater paraphrasticity of GPT translations. However, GPT translations still produce greater number of punctuation insertions than the MS Translator system.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
To illustrate the parallel data bias, we analyze the translations on low-quality inputs. The intuition behind our experiment is that low-quality inputs are more likely to correspond to the noisy parts of the parallel data that underlie NMT systems trained on large-scale datasets mined from the web. As such, GPT should outperform NMT systems on such low-quality inputs.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We split the test sets into 3 buckets based on perplexity of the source sentence. We notice that the highest perplexity inputs typically correspond to ill-formatted texts, with many inputs pertaining to the e-commerce domain. Such inputs are more likely to resonate with the noisy parts of the parallel corpora on which NMT models are typically trained. For example, such high-perplexity inputs might correspond to ill-formed texts scraped from e-commerce websites usually present in parallel corpora. Since we use GPT-2 for obtaining perplexities, we conduct this experiment for E-X language pairs only.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Results: Table 10 presents the results of the experiment across different language pairs. The measurement reported is the average difference in the quality between GPT and MS Translator as measured using COMET-KIWI. We observe that on English to Chinese, English to Japanese and English to Russian language pairs, on which parallel data mining is typically harder owing to a change in script, GPT translations obtain higher performance than MS Translator on the highest perplexity bucket. For low-resource language pairs, GPT gains proportionally in even the lowest perplexity buckets.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Overall, we find that that in four out of five high resource language pairs in Table 10, GPT obtains higher improvements in the bucket with the highest input perplexity, when compared to the other lower-perplexity buckets. In the cases of English (Latin script) to Chinese, English to Japanese and English to Russian (Cyrillic script), the differences follow a monotonic order with respect to input perplexity. The results suggest that for these language pairs, GPT does obtain better performance on the lower quality inputs. We attribute this behavior to the parallel data bias. Such parallel data noise biases are likely to be correlated with input domains as well, but we leave such an exploration to future work.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Lowest -0.02 -0.68 -0.43 0.22 -0.04 -6.34 1.63  Highest -0.18 -0.14 -1.10 1.63 0.38 -6.15 1.11 Table 10: Exploring Parallel Data Bias: On English to Chinese, Japanese and Russian language pairs, GPT translations obtain higher performance than MS Translator in the highest perplexity bucket. For low-resource language pairs, GPT gains proportionally in even the lowest perplexity buckets.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We demonstrated that the computational mechanisms operating behind LLMs and NMT models produce translation artifacts that can be quantitatively differentiated. In this subsection, we summarize our comprehensive characterization of the translations produced by GPT.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Improvements Produced by GPT: For X-E translations, the translations produced by GPT are more fluent, obtaining consistently lower perplexity (as shown in Figure 7). At the same time, GPT translations for X-E language pairs generally incur higher number of unaligned source words (Figure 9) and in general, similar number of unaligned target words ( Figure 10). GPT translations are also more non-monotonic, producing translations that involve longer range reorderings ( Figure 11). The combination of these results yields an interesting conclusion: that X-E translations by GPT are more fluent and more paraphrastic than the NMT system under investigation (MS Translator), while being faithful to the source. The greater paraphrasticity is not accompanied by content that is unsupported by the source, i.e., the problem of inserted factual content is not a prominent issue in these language pairs. For E-X translations, GPT incurs a greater number of unaligned source words (USW, Figure 14), along with greater translation non-monotonicity in general (NM, Figure 15), suggesting greater paraphrasticity. However, at the same time, GPT translations incur a slightly higher number of unaligned translation words as well. This suggests that greater paraphrasticity is not the only cause behind the higher USW and NM measurements, and a less adequate translation than the NMT system under investigation is a plausible cause behind these observations as well. This is corroborated by the lower quality measurements for E-X GPT translations obtained previously. In general, we find that for deriving conclusions about GPT translation quality for E-X, it is more important to focus on the single language pair under consideration, i.e. the target non-English language is of critical importance and the effects of language modeling bias cannot be generalized as in the case of X-E translations.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Areas of Improvements: One artifact of the language modeling bias is that GPT inserts end of sentence markers not present in the source with a far greater frequency than the NMT system under investigation. This holds true across both X-E, E-X and X-Y translation directions. Such a proclivity towards greater fluency might not be appropriate for domains wherein a very literal (and faithful) translation is desired. Similarly, greater paraphrasticity might not be appropriate for certain domains. Also, a related area of improvement for future evaluations would be to conduct separate evaluations of fluency and adequacy, in addition to joint adequacy and fluency quality estimation done presently. Instituting a norm of using multi-dimensional automatic quality measurements  can provide very targeted signals on differentiating aspects of translation quality, useful especially when there are competing state-of-the-art approaches.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Our results also suggest that the greater paraphrastic nature of GPT translations could have applications in improving NMT models on the translation of figurative text. Similarly, the greater gains obtained by GPT translations in the highest perplexity buckets of multiple E-X the test sets suggest that GPT translations might be favored over NMT models when the input domain is likely to contain noisy, ill-formed sentences. These two application areas, based on the demonstrated characteristics of GPT translations, offer two avenues that could benefit from a combination of NMT models with GPT. For example, based on the results in Table 10, a hybrid English-Japanese system that could improve upon both MS Translator and GPT translations would be the one wherein the highest perplexity inputs are routed to be translated by GPT whereas the lower perplexity inputs are translated through NMT models (e.g., MS Translator). Such a composition might be able to leverage complementary strengths of NMT and LLM systems for translation.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In this section, we investigate the multilingual capabilities of GPT models beyond translation. Specifically, we aim to assess how well the models perform on emerging reasoning tasks 15 for various languages compared to English. We are interested in understanding the degree of multilingual support that GPT models can offer given their translation performance. That is, can we use the translation performance as a proxy for the multilingual performance on other tasks? We use MGSM Benchmark (Shi et al., 2022) which is a Multilingual Grade School Math (MGSM) arithmetic reasoning benchmark. The multilingual problems are human translated from the English dataset GSM8K which is Englishlanguage human-annotated grade-school math problem dataset. The dataset supports a set of ten languages other than English (EN): Bengali (BN), Chinese (ZH), French (FR), German (DE), Japanese (JA), Russian (RU), Spanish (ES), Swahili (SW), Telugu (TE), and Thai (TH). Table 11 presents the results on the MSGM benchmark. We first use Native-CoT, which uses prompts and CoT in the native language of each dataset. We observe that text-davinci-003 surpasses text-davinci-002 for all languages, highlighting the effectiveness of text-davinci-003 on multilingual tasks. The performance is especially high on EN, DE, FR and ES, while RU, JA and ZH exhibit lower scores than the Latin languages. The low-resource languages, however, achieve limited performance, indicating the need for better approaches to attain truly multilingual support.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We then use Translate-EN, which translates all prompts and CoT into English. We find that this setup enhances the performance on the non-Latin group (RU, JA and ZH) as well as the low-resource group (TH, TE, BN and SW), although the enhancements are not uniform across languages. Surprisingly, this setup shows a deterioration on the Latin languages.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Our third and final setup is Translate-EN+, which is similar to Translate-EN, but keeps the template in English for all sentences instead of translating it. Stabilizing the template improved results significantly in some languages such as French, Spanish and Russian, and gave comparable scores to Translate-EN in others. We observe that, despite the high performance of text-davinci-003 on the translation of RU, JA and ZH, the performance on MSGM is only moderate. We hypothesize that this may be due to the fact that reasoning tasks benefit greatly from training on programming languages, which is better represented in the top Latin languages, especially with the low proportion of multilingual data in the training data. In contrast, PaLM-540B results from (Shi et al., 2022) show a higher performance with the Native-CoT setup. We hypothesize that this is due to the large multilingual data proportion in its training data, which is 78% English and 22% for other languages (Shi et al., 2022), while GPT data proportion is only 7% non-English (Brown et al., 2020).
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
These results suggest that translation capability may not be sufficient for the model to exhibit more advanced multilingual reasoning capability, as shown by the poor performance on RU, ZH and JA. We hypothesize that the models acquire their reasoning capabilities through training on natural language multilingual data along with programming languages data, which may limit such capabilities for non-Latin and less represented languages. We think this area warrants more attention from the model developers to provide truly multilingual capabilities across a range of languages.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
This work presents a comprehensive and in-depth study of the machine translation capabilities of the latest GPT models. Our investigation covers 18 language pairs across four different domains, enabling a broad understanding of the models' general performance. We also conducted the multilingual reasoning task to examine the interaction between multilinguality and the emergent reasoning capabilities in GPT models. To provide a thorough evaluation of the models, we employed both human evaluations and the latest neural network-based automatic evaluation metrics together with the conventional machine translation evaluation metrics. In addition, we conducted extensive analysis, providing an indepth examination of various phenomena in GPT models' translation outputs and their comparisons to state-of-the-art NMT systems.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
As a result, our findings demonstrate that GPT systems can produce highly fluent and competitive translation outputs even in the zero-shot setting especially for the high-resource language translations. By utilizing the in-context learning capability of GPT models with few-shot examples, we were able to further improve translation quality. Additionally, we demonstrated that a hybrid approach, combining the latest NMT systems with GPT models, can achieve state-of-the-art translation quality.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
While the use of LLMs in machine translation is a rapidly developing area, there are many research directions that can be explored to improve the quality and understanding of machine translation. Below are some of the important areas that we focus on:
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• Underrepresented languages: Our study has shown that GPT models, still struggle with underrepresented languages, which makes it a critical research question to explore how to improve the translation quality for these languages.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• In-context learning: GPT models have shown great potential for in-context learning, which can be leveraged to generate different styles or nuanced translations. Future research can explore how to better utilize this capability to improve translation quality.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• Model fusion: The use of large-scale LLMs like GPT can be computationally expensive, and therefore, exploring how to more efficiently utilize them is an important research question. We are investigating more sophisticated fusion techniques that can achieve higher quality and efficiency.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
• Better metrics: The limitations of lexical matching metrics can mislead translation quality assessment. Therefore, developing metrics that can measure the contextual correctness of LLM-generated translations is essential. Future research can also explore new ways to evaluate the quality of machine translations more accurately, especially when using LLMs.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Overall, our study provides valuable insights into the strengths and weaknesses of GPT models for machine translation and opens up opportunities for future improvements and developments in this field. We investigated how GPT models can transform machine translation as it is doing with other generative tasks. We demonstrated that these models excel at translating well-represented languages in their training data, but they face challenges with less-resourced languages. We also assessed translation and reasoning tasks and detected discrepancies in the level of support of the tasks even for the same languages. One of the main benefits of training such costly models is to achieve high performance across diverse tasks and languages, but this demands more data across languages. Which poses several challenges for models scalability, diversity, and fairness. As a future research direction, we propose to tackle the challenge of enabling truly multilingual capability for such models that would enable the same capabilities across languages.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
We conducted our evaluation on 18 translation directions with reliable test sets and baselines. The conclusion of the study should be taken in this context and not generalized to other languages without further evaluation. While a more comprehensive evaluation is needed on more languages, we should be cautious about drawing conclusions from low quality testsets or weaker baselines which are usually dominating the research results for low resource languages.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
One of the limitations of this study is the inadequacy of current automatic evaluation metrics to capture the quality of GPT outputs accurately. We found that lexical comparison based metrics such as BLEU or chrF gave misleading signals, and that document-level evaluation had limited capability to realize the effect of context-based translation. These limitations stem from the inherent challenges of evaluating natural language generation systems, especially for complex tasks such as machine translation. Therefore, we complemented our automatic evaluation with a comprehensive analysis, considering all metrics together, as well as human evaluation and qualitative analysis to cover a broad range of phenomena. We recommend that readers consider the overall evaluations as a whole, rather than relying solely on a specific metric, to better understand the quality of GPT models' machine translation capabilities.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In our study, we evaluated the quality of GPT translations using various quantitative metrics. However, we acknowledge that these models may harbor language-specific biases and produce translations that perpetuate stereotypes and misinformation. One form of bias that we have identified is that the models perform better for some languages than others, as shown in §3.3 and §3.4. This may create unfairness and unequal outcomes for users paying the same cost. Furthermore, the models may amplify stereotypes from the training data, leading to inaccurate translations. For instance, the model may fail to correctly translate a sentence with a female name like "Julia" as a male. Lastly, we also observed instances of false insertions in translations (misinformation) and hallucinations, especially in E-X translation directions. We are committed to addressing these issues and mitigating biases and misinformation in our research and future work. Figure 16: Prompt template for sentence-level translation. We use the same instruction and format as recommended in the OpenAI playground for the default sentence-level translation task 16 . Keeping the prompt format the same allows us to potentially leverage the benefits of the underlying instruction finetuning protocol to the full extent.   
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
As shown in the examples for the first characteristic in Table 14, GPT is good at handling misspelled words or unclosed quotes and ends translations without missing any semantic information. Also, GPT translations are reasonable for partial or incomplete colloquial source sentences whereas WMT-Best translations have extra or missing content (the third characteristic). On the other hand, GPT tends to generate unnatural translations for sentences with unusual or complex expressions (the second characteristic). GPT also generates hallucinated words in its translations as in the fourth characteristic. For instance, the first GPT example translation uses "udon" ("noodle" in English) although the source never mentions it. In the second GPT example translation, the word "PM" (Prime Minister) is translated into a non-existing Japanese word that sound like an existing jargon. Unlike oscillatory hallucinations (i.e., repeating the same words or phrases many times) (Raunak et al., 2021), these subtle, word-level hallucinations are problematic as they are hard to be detected and may not severely affect metric scores .
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
With the increasing scale of parameters and training corpus, large language models (LLMs) have gained a universal ability to handle a variety of tasks via incontext learning (ICL, Brown et al. 2020), which allows language models to perform tasks with a few given exemplars and human-written prompts as context. One particular area where LLMs have shown outstanding potential is machine translation (MT). Previous studies have shown the surprising performance of LLMs on high-resource bilingual translation, such as English-German translation (Vilar et al., 2022;Zhang et al., 2022b), even if these models are not particularly optimized on multilingual data.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
However, the multilingual translation ability of LLMs remains under-explored. MMT is a challenging task that involves translating text among differents languages and requires semantic alignment between languages (Fan et al., 2021;Costa-jussà et al., 2022;Yuan et al., 2022). In addition, it is also unclear that which factors affect LLM's ability in machine translation, as previous research mainly investigates natural language understanding (NLU) tasks (Min et al., 2022;Kim et al., 2022;Zhang et al., 2022a;Wei et al., 2023a).
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
In this paper, we focus on studying LLMs in multilingual machine translation by answering two questions: 1) How LLMs perform MMT over massive languages? 2) Which factors affect the performance of LLMs?
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
For the first question, we evaluate and compare popular LLMs: English-centric OPT (Zhang et al., 2022b) and multilingual LLMs, including XGLM (Lin et al., 2021), BLOOMZ (Scao et al., 2022), and ChatGPT (OpenAI, 2022), on 102 languages and 202 translations directions (X-to-English translation and English-to-X translation). The results show that ChatGPT outperforms other LLMs by a large margin, especially when translating to English, which dispels the doubts that previous work had about LLM's translation ability (Wei et al., 2022a;Chowdhery et al., 2022). On the other hand, compared with a widely-used supervised baseline NLLB (Costa-jussà et al., 2022), Chat-GPT only achieves higher performance on 16.67% translation directions. We further analyze the translation cases where LLMs fail, and find three typical translation errors, namely off-target translation, hallucination and monotonic translation.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
For the second question, we find some new working patterns. First, LLMs are able to perform translation even with unreasonable prompts if in-context learning exemplars are given. However, if given mismatched translation pairs as in-context exemplars, LLMs fail to translate, which is similar to observations from concurrent studies (Wei et al., 2023b). This shows the importance of exemplars in ICL for machine translation. However, how finding better exemplars is still challenging. Random selection is a surprisingly strong baseline, and semantically-selected exemplars only bring slight improvements. Second, we also find that crosslingual translation pairs can be surprisingly good exemplars for low-resource translation, even better than exemplars in the same language. Third, we find BLOOMZ being over-estimated when evaluated on Flores-101. On the data we newly collected and human-annotated, BLOOMZ shows universal performance drops in all human-evaluated directions. It indicates the importance of closed-data evaluation to avoid data leakage problem for a fair comparison. The main contribution of this paper can be summarized below:
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• We benchmark popular LLMs on MMT in 102 languages and 202 English-centric translation directions.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• We systematically report the results of LLMs and two widely-used supervised baselines (NLLB and M2M-100).
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• We find some new ICL patterns of LLMs for machine translation.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Language modeling is a long-standing task in natural language processing (Bengio et al., 2000;Mikolov et al., 2010;Khandelwal et al., 2020), which is a task to predict the probability of the next token. Transformer (Vaswani et al., 2017) basically is the backbone of existing LLMs. LLMs show great potential as a universal multitask learner. Recently, Radford et al. (2019) find that a casual decoder-only language model can be a multi-task learner with merely unsupervised training corpus. Later, Kaplan et al. (2020) reveal the scaling law of LLM, indicating that when the scale of neural parameters and training data keeps increasing, LLM can be further strengthened. Wei et al. (2022b) show that scaling the language model also brings astonishing emergent abilities, e.g., incontext learning, which is only present in large models. Consequently, more and more efforts have been put into scaling-up language models (Brown et al., 2020;Scao et al., 2022;Vilar et al., 2022;Zeng et al., 2023;Ren et al., 2023). Among them, GPT and ChatGPT (OpenAI, 2022) is one of the most representative systems, which shows impressive results in various NLP tasks.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
In-context learning is one of the well-known emergent abilities (Brown et al., 2020;Dong et al., 2022), which enables LLM to learn target tasks according to the prompt without updating any parameters.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Specifically, the prompt is made up of in-context exemplars {(X i , Y i )} k i=1 and in-context template T . Exemplars are often selected from supervised data, where Y i is the ground truth corresponding to the input sentence X i . Template T is usually a human-written instruction related to the target task. Wrapping exemplars with the template and concatenating them together produce the final prompt:
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
where ⊕ denotes the concatenation symbol, e.g., whitespace, line-break. During inference, LLM is able to generate the corresponding output Y of the test sample X under the guidance of the prompt:  ChatGPT 3 (OpenAI, 2022).
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
ICL strategy For each model, we report its translation performance with eight randomly-picked translation pairs from the corresponding development set as in-context exemplars and "<X>=<Y>" as in-context template. "<X>" and "<Y>" are the placeholder for the source and target sentence. We use line-break as the concatenation symbol. According to our experiment analysis, this ICL strategy serves as a simple but strong recipe. We implement ICL based on OpenICL 4 (Wu et al., 2023).
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
We also report the performance of the supervised model M2M-100-12B (Fan et al., 2021) and NLLB-1.3B (Costa-jussà et al., 2022) (distillation version) as the baseline, which are widely-used many-to-many multilingual translation models. 
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
In this section, we report results on multilingual machine translation and introduce our main findings about LLMs' translation ability.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
ChatGPT is the best translator among evaluated LLMs Table 1 presents evaluation results grouped by language family. Detailed results for each translation direction are listed in Appendix A. XGLM and OPT present impressive multilingual translation ability, indicating the possibility of aligning multiple languages even with unsupervised data (Garcia et al., 2023). Instructiontuned LLMs, i.e. BLOOMZ and ChatGPT, often achieve better translation performance than pretrained models. Notably, BLOOMZ outperforms supervised baseline on seven groups of translation directions and ChatGPT achieves the highest average BLEU score on most directions among these evaluated LLMs.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
LLMs perform better on translating into English than translating from English Previous research finds that LLMs achieve impressive results when translating into English, but translating from English produces less decent results (Wei et al., 2022a;Chowdhery et al., 2022). We observe similar phenomena for XGLM, BLOOMZ and OPT. Surprisingly, ChatGPT achieves more balanced performance, but it still fails on translating English into low-resource languages.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
LLMs still lag behind the strong supervised baseline, especially on low-resource languages Figure 2 plots the translation performance of the supervised baseline (NLLB) and the best-performing LLM (ChatGPT) on each language. On the left of the figure, ChatGPT achieves similar BLEU scores to NLLB. On the right of the figure, ChatGPT lags behind NLLB significantly and these languages tend to be low-resource languages, e.g., Burmese (mya), Ukrainian (ukr). In general, ChatGPT underperforms NLLB on 83.33% translation directions. For translation cases where ChatGPT fails, we observe its translation results and find that there are three typical translation errors : 1) off-target translation 2) hallucination 3) monotonic translation. Table 2 presents an example case for each type of error. These errors also frequently occur in other LLMs. ChatGPT fails to translate English into turkish, but into Azerbaijani.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Source Huwobe illimu kimiya be fasaha jodi hefti do kujjeji ko jomirawo tagi ha asamaji do mari alaka be luttukal ko taginga ha asamaji fu bana no mari alaka be kala ko be tagi andigel pat.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Hypothesis The study of chemistry and language is important in our modern world, and we must work hard to improve our knowledge in both areas. Let us all strive to learn and improve ourselves in these fields so that we can make a positive impact on society.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
The scientists were able to conclude that the dark matter affect other dark matter in the same way regular matter does.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
ChatGPT generates highly pathological translations that are completely unrelated with the source sentence.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Source They found the Sun operated on the same basic principles as other stars: The activity of all stars in the system was found to be driven by their luminosity, their rotation, and nothing else.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Reference 他们发现，太阳和其他恒星的运行原理是一样的：星系中所有恒星的活跃度都完全决定于它们的 光度和自转
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
ChatGPT translates English sentence word-by-word, lacking effective word-reordering. Instruction-tuned LLMs can still benefit from in-context learning While instruction-tuning LLMs with supervised data can greatly improve LLMs' zero-shot performance (Wei et al., 2022a), a question naturally raises: will instruction-tuned LLMs still benefit from in-context learning? We compare the zero-shot and few-shot translation performance of BLOOMZ and ChatGPT in Figure 3. We can see that showing instruction-tuned LLM with in-context exemplars can further improve its translation performance.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Evaluating LLMs on public datasets have the risk of data leakage As LLMs are trained from a pile of data crawled from the web, it is unclear whether the evaluation dataset is covered by the training corpus. If so, the evaluation results can not precisely reflect translation ability (Elangovan et al., 2021). To examine this issue, we select 30 English sentences from recent news 6 , and ask human experts to translate them into Chinese and German, and construct a trilingual no-leakage evaluation set. Figure 4 shows that XGLM, OPT achieve similar performance on two datasets. ChatGPT shows large performance drops on English-German translation. BLOOMZ generalizes poorly on the noleakage dataset, indicating that its performance is inflated on Flores-101 dataset due to the data leakage issue 7 . To ensure the evaluation results are fair, we call the attention of the community to be aware of this issue.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
To better understand what LLM learns from incontext learning and find the best recipe to elicit its translation ability, we conduct in-depth analysis in this section. For analysis, we choose XGLM-7.5B as an example. Our findings and analysis are listed below.  Table 3: Effects of using different templates for in-context learning. "<X>" and "<Y>" denote the placeholder for source and target sentence respectively. "[SRC]" and "[TGT]" represent the placeholder for source and target language name in English. Bold text denotes the highest score along the column. The good performance of LLMs relies on carefully-selected templates The initial step of applying in-context learning for translation is determining the template. After trying a range of templates, we find that the translation performance varies greatly with different templates (Table 3), where the largest gap in the average performance is up to 16 BLEU. The best template for each direction is also different. Among these templates, "<X>=<Y>" achieves the highest average BLEU score. "[SRC]: <X> \n [TGT]: <Y>" achieves the lowest score, although it is a commonly-used tem- Even unreasonable template can instruct LLM to generate decent translation A common intuition of ICL is that the template instructs LLMs to do the target task (Brown et al., 2020), e.g., the template "<X> can be translated to <Y>" instructs the LLM to perform translation task. However, we find that wrapping translation exemplars with task-unrelated template can also serve as an effective prompt. For example, the template like "<X> can be summarized as <Y>" can also instruct LLM to generate translation, rather than guiding it to generate summarization. Given the fact that these unreasonable template are also effective, the community may misunderstand the role of incontext-template.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Semantically-selected exemplars does not brings more benefits than randomly-picked exemplars Selecting in-context exemplars is another important procedure for performing incontext learning. In this paper, we use development set for exemplar selection, which has been found to be a high-quality candidate pool (Vilar et al., 2022), and we compare four ways of selecting in-context exemplars:
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• Random: picking exemplars on a random basis.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• BM25: selecting exemplars whose source sentences are similar to the test case's source sentence according to BM25.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• TopK: selecting exemplars whose source sentences are similar to the test case's source sentence according to the similarity of sentence embedding 8 .
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• Oracle: selecting exemplars whose target sentences are similar to the test case's according to sentence embedding, which can be seen as the upper bound of selection strategy.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Effects of selecting varying number of in-context exemplars with different approaches are shown in Figure 5. The general trend in all dataset is similar. As the number of examples grows from 1 to 8, the BLEU score increases rapidly. Afterwards, the translation performance plateaus regardless of selection strategy. When more exemplars are added, e.g., 32 exemplars, the BLEU score usually starts to decline, shows an opposite phenomenon against the observation in natural language understanding tasks (Li et al., 2023). Compared to semantically-selected exemplars, randomly-picked exemplars gives comparable translation performance. Even the performance of oracle selection is on par with random selection. Based on these observations, we suggest that translation exemplars can teach LLM to translate but LLM may struggle to acquire helpful translation knowledge from semantically-selected exemplars.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
ICL exemplars teach LLM the core feature of translation task To better understand how ICL exemplars influence LLM to understand the translation task, we observe LLM's translation behaviour under abnormal in-context exemplars (Table 4).
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
We can see that LLM completely fails when mismatched translation is used as exemplars, indicating that LLM can learn from the context to keep source and target sentence semantically consistent. Word-level 9 and document-level 10 translation exemplar degenerates LLM's translation performance, which demonstrates that the translation granularity of exemplar matters as well. Another interesting phenomenon is that LLM performs worse when duplicated translation is used as the exemplar, indicating that keeping in-context exemplars diverse is also important. In general, these comparison results show that LLM learns the core feature of translation task through in-context learning.  Table 4: Translation performance of XGLM when using different contents as in-context exemplars. "Consistency" column denotes whether source and target sentence are semantically consistent. "Granularity" column denotes whether the exemplar is a sentence-level pair. "Diversity" column denotes whether exemplars in the context are different from each other. of the exemplar is a unique factor in machine translation. We find that using cross-lingual exemplars does not always causes worse performance and show two cases in Figure 6. When using crosslingual exemplars for German-English translation, the translation performance degenerates. But when using cross-lingual exemplars for low-resource Chinese-English translation, XGLM's translation performance improves significantly. This phenomenon indicates the potential usage of crosslingual exemplars in more tasks than natural language inference (Lin et al., 2022), and we will explore more about this in the future.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
The exemplar in the tail of the prompt has more impact on the LLM's behaviour While crosslingual exemplars does not always cause worse performance, reversing the translation direction of exemplars will cause LLM to fail. Relying on this observation, we conduct experiments to investigate the importance of different parts of the prompt (Table 5). We find that reversing exemplars in the tail of the prompt consistently produced worse re-   compare the importance of these factors in MT and NLU tasks ( Table 6) and find that in-context learning has some new working patterns in translation tasks, which provides more clues for the community to understand this emergent ability.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
In this paper, we evaluate the multilingual translation ability of several LLMs, including ChatGPT, on 102 languages and 202 English-centric directions, which presents the advantages and challenges of LLMs for MMT. We find that even the bestperformed LLM, i.e., ChatGPT, still lags behind the strong multilingual supervised baseline (NLLB) in 83.33% translation directions. In our analysis, we find that LLMs exhibit new working patterns when used for machine translation. For example, prompt semantics can be ignored during in-context learning and cross-lingual exemplars can provide better task instruction for low-resource translation. More importantly, we find that BLOOMZ show over-estimated results on public dataset and call people's attention for this issue when evaluating LLMs. 
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Detailed results of our evaluated models on 102 languages are shown in Table 7 and Figure 7.
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Involved languages in this paper are listed in Table  8.   
Document-Level Machine Translation with Large Language Models
In the past several years, machine translation (MT) has seen significant advancements with the introduction of pre-trained models such as BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019), and Figure 1: An example of translating a documentlevel text from English to Chinese using GPT-4 (Date: 2023.03.17). We highlight the discourse phenomena using figures and lines, which are invisible to  T5 (Raffel et al., 2020). These models have demonstrated impressive performance on MT (Zhu et al., 2020;Guo et al., 2020;Xue et al., 2021). However, most of the existing work has focused on sentencelevel translation, which can result in translations that lack coherence and context. Recent years have seen a growing interest in discourse-aware translation, which is a crucial task that involves translating entire documents (Wang et al., 2017;Bawden et al., 2018;Zhang et al., 2022) while modelling specific discourse phenomena (Voita et al., 2018;Wang et al., 2018Wang et al., , 2019Voita et al., 2019b). The most popular large language model (LLM) -Chat-GPT 2 shows the ability of maintaining long-term coherence and consistency in a conversation by conditioning on previous conversational turns. Additionally, the model is trained on a large dialogue dataset, which allows it to learn the patterns and conventions of human communication, further improving its ability to discourse awareness (as shown in Figure 1).
Document-Level Machine Translation with Large Language Models
In this paper, we are particularly interested in how LLMs such as ChatGPT perform for modeling discourse phenomena such as entity consistency, referential expressions, and coherence. Taking document-level MT as a testbed, we conduct an empirical study from three in-depth perspectives:
Document-Level Machine Translation with Large Language Models
• Effects of Discourse-Aware Prompts: Chat-GPT needs a prompt as guidance to trigger its translation ability. Thus, we enable prompts to guide ChatGPT to consider document-level contexts as long as possible. Jiao et al. (2023) has found that the candidate prompts generally work well and show minor performance differences on sentence-level translation. In this work, we further investigate the effects of prompts on the translation quality and specific discourse phenomena.
Document-Level Machine Translation with Large Language Models
ChatGPT has demonstrated remarkable abilities in long-text NLP tasks, we are specifically interested in how it performs on document-level translation. Consequently, we conduct a systematic comparison of commercial MT products and advanced document-level approaches, utilizing both automatic and human evaluations to assess their discourse awareness. • Analysis of Discourse Modelling Abilities: A more challenging question is the extent to which ChatGPT capture and utilize discourse knowledge, which can be probed through contrastive testing (Voita et al., 2019b). In addition, the impact of different training techniques on the ability of LLMs to model discourse has not been thoroughly investigated. For example, it is unclear whether long-term dependency is a side effect of code pretraining. Thus, we examine various LLM techniques including code pretraining, supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF).
Document-Level Machine Translation with Large Language Models
We conduct experiments on a variety of document-level translation benchmarks, covering three language pairs (i.e. Chinese⇒English, English⇒German and English⇒Russian) and seven domains (i.e. news, social, fiction, Q&A, TED, Europarl, and subtitle). We adopt a comprehensive set of evaluation methods to measure the performance of the models on documentlevel translation, including general-purpose met-rics, discourse-specific metrics, and human evaluation. Experimental results and extensive analysis demonstrate that: • ChatGPT has a strong ability to capture document-level contexts using simple prompts, and concatenating multiple sentences in a conversational turn further improves the consistency and cohesion of translations; • ChatGPT and GPT-4 demonstrate superior performance compared to commercial MT systems in terms of human evaluation and outperform most document-level NMT methods in terms of d-BLEU; • During contrastive testing, ChatGPT shows lower accuracy in contrastive testing when compared to conventional translation models. However, GPT-4 demonstrates a strong ability to explain discourse knowledge, even though it may select incorrect translation candidates; • A combination of training techniques such as SFT and RLHF, has been consistently shown to enhance LLMs' document translation capabilities.
Document-Level Machine Translation with Large Language Models
Step 2.1 Dataset Table 1 shows statistics of document-level datasets used in our experiments. About Group #1, we utilized the latest datasets, mZPRT  and WMT2022 (Kocmi et al., 2022), for evaluation to ensure that the testing data had not been used in commercial systems (e.g. Google Translate and ChatGPT) before. As seen, this covers four domains (i.e. news, social media, web fiction, and Q&A forum) in Chinese⇒English.
Document-Level Machine Translation with Large Language Models
Regarding Group #2, we utilized four widelyused benchmarks to compare established documentlevel methods with GPT-like applications. As seen, this covers three domains (i.e. TED talks, news commentary, European Parliament) in Chinese⇒English and English⇒German. Furthermore, we employed an English⇒Russian constructive testset (Voita et al., 2019b) that specifically targets discourse phenomena, such as deixis, ellipsis, and lexical cohesion. We use this dataset to further exploit models' capacity for modeling discourse knowledge. As seen, we also report the average length of a document (|W|/|D|), which can be considered a measure of the complexity of discourse modeling. As the length of the document increases, it becomes more challenging to model accurate cohesive devices and discourse structure. From this perspective, the mZPRT Fiction and IWSLT TED datasets pose a greater challenge compared to others.
Document-Level Machine Translation with Large Language Models
Translation Quality We evaluate different approaches and systems using both sentenceand document-level evaluation metrics. About sentence-level metrics, we employ the commonlyused sacreBLEU (Post, 2018) and TER (Snover et al., 2006). Additionally, we utilize COMET (Rei et al., 2020), which leverages pretrained language models to achieve high correlations with human quality judgments. About document-level metrics, we report document-level sacreBLEU (d-BLEU) , which is computed by matching n-grams in the whole document. Note that all evaluations are case-sensitive.
Document-Level Machine Translation with Large Language Models
Discourse Awareness To target specific discourse phenomena, we utilized two targeted metrics, namely, CTT and AZPT, which respectively evaluate the consistency of terminology translation and accuracy of zero pronoun (ZP) translation. Regarding CTT, one repeated terminology should keep the same translation throughout the whole document (Xiao et al., 2011). We adopt a lexical translation consistency metric (Lyu et al., 2021):
Document-Level Machine Translation with Large Language Models
for each terminology word w ∈ T T , the C 2 k denotes the size of the combination of translation set (t 1 , . . . , t k ), and function 1(t i = t j ) returns 1 if t i is same as t j , otherwise 0. The metric illustrates how frequently translation pairs of w is same within a document. The higher the metric value is, the more likely w is translated consistently. Regarding ZP, it is a discourse phenomenon that appears frequently in pronoun-dropping (pro-drop) languages such as Chinese and Japanese. Recovering ZPs in a target language (non-pro-drop) needs an understanding of the discourse structure. We used the AZPT score to measure the accuracy of ZP translation :
Document-Level Machine Translation with Large Language Models
where ZP is the list of zero pronouns in the source sentences, t z is the generated translation for the zero pronoun z, and A(t z |z) is a binary scorer to judge whether t z is the correct translation of z.
Document-Level Machine Translation with Large Language Models
Human Evaluation The outputs will be evaluated based on a scale from 0∼5. Each output will receive two separate scores based on the general quality (e.g. fluency and adequacy) and the discourse-aware quality (e.g. consistency, word choice, anaphora). The detailed scoring criteria are listed in Appendix § A.1.
Document-Level Machine Translation with Large Language Models
Existing document NMT methods can be mainly classified into two categories: multisentence (Wang et al., 2017;Voita et al., 2018; Tu  Table 2: Ablation study of document-level prompts (detailed in Table 3) on Chinese⇒English datasets using ChatGPT. We use BLEU and d-BLEU to measure sentence-and document-level translation quality. We also conduct two targeted evaluation metrics to measure specific discourse phenomena: accuracy of zero pronoun translation (AZPT) and consistency of terminology translation (CTT). Base is a sentence-level baseline system using InstructGPT API without any context-based chat box.
Document-Level Machine Translation with Large Language Models
P1 Please provide the TGT translation for the sentence: S P2 Translate the following SRC sentences into TGT: Table 3: The prompts suggested by ChatGPT for document-level translation. SRC and TGT denote source and target languages, respectively. Each document is orderly processed in one "Chat Box" while each prompt is fed into one "Conversational Turn". P1 represents separately translating each sentence S in a document. P2 or P3 means translating a document w/wo a sentential boundary tag "[]". et al., 2018) and whole-document (Macé and Servan, 2019;Bao et al., 2021) approaches. ChatGPT is capable of not only handling long text in a single conversational turn but also recalling the entire context in the chat box. Accordingly, we design prompts to trigger the document-level translation ability of ChatGPT. First, we follow Jiao et al. (2023) to ask Chat-GPT itself for advice and obtain three candidate translation prompts. Second, we select three representative ones as shown in Table 3. We utilize P1 to translate the document sentence by sentence, with each sentence placed in a single conversational turn and the entire document contained within one chat box. This mainly takes advantage of ChatGPT's long-term modeling ability in the chat box. P2 and P3 combine multiple continuous sentences and translate them in one conversational turn until the entire document is finished. This aims to maximize the context modeling in one conversational turn.
Document-Level Machine Translation with Large Language Models
The only difference is whether or not the sentential boundary tag "[]" is inserted into each sentence.
Document-Level Machine Translation with Large Language Models
We compare the three candidate prompts on the Zh⇒En translation task using two testsets, WMT2022 News and mZPRT Fiction. Table 2 shows the translation quality in terms of a variety of automatic evaluation metrics. In general, the candidate prompts generally work well and show minor performance differences. Out of the three prompts, P3 achieves the best scores in most evaluation metrics, except for COMET. Regarding discourse phenomena, P3 also outperforms other candidates with better consistency of terminology translation and higher accuracy of ZP translation. Upon examining the output samples, we noticed that Chat-GPT may sometimes forget the intended format of P2 and simply combine all sentences together. Takeaway: (1) Despite translating the document sentence by sentence, ChatGPT's ability to model long-term dependencies already exists within the chat box.
Document-Level Machine Translation with Large Language Models
(2) Combining multiple sentences in one conversational turn can marginally enhance both translation quality and discourse awareness.
Document-Level Machine Translation with Large Language Models
(3) A more natural way to translate a document would be to combine multiple sentences into coherent units without necessarily maintaining strict sentential boundaries.
Document-Level Machine Translation with Large Language Models
In this section, we compare various systems and methods for the document-level translation task. In the following experiments, we use the P3 prompt for ChatGPT and its window size for MT models as the default setting. All results were obtained from these systems in March 2023.   
Document-Level Machine Translation with Large Language Models
Commercial systems are known for their high accuracy and efficiency in translation, making them a strong contender for any machine translation evaluation. By comparing ChatGPT with commercial systems, we can gauge ChatGPT's performance relative to the best available machine translation technologies.
Document-Level Machine Translation with Large Language Models
We compare ChatGPT/GPT-4 with three commercial translation products, including Google Translate, 3 DeepL Translate, 4 and Tencent TranSmart. 5 We employ two evaluation methods to assess the quality of the translation outputs: (1) d-BLEU, which accounts for the coherence of the entire document and is particularly useful for evaluating document-level translation performance; (2) 3 https://translate.google.com. 4 https://www.deepl.com. 5 https://transmart.qq.com. human evaluation, which is measured based on the general quality of the discourse-aware quality (detailed in Table 10). Table 4 shows the results. When evaluated using d-BLEU, commercial MT systems generally outperform LLM-based systems, except for the Q&A domain, which involves informal spoken language. While the difference in performance is not significant in the news domain (e.g., the gap between DeepL and GPT-4 is only 0.6 points), it is considerable in the social media and web fiction domains (i.e. 3.3 and 1.9 points). A surprising finding is that GPT-4 and ChatGPT perform significantly better than MT systems in terms of both general translation quality and discourse awareness. The potential reasons may be: (1) d-BLEU only measures the similarity of the n-grams between the MT output and the reference translations. However, human takes into account additional factors such as coher-   hancing the representation of the context. • Sent2Sent: A superior sentence-level baseline that employs a transformer architecture to translate each sentence independently and then merges the translations into a document-level output. • MR-Doc2Doc and MR-Doc2Sent: Sun et al.
Document-Level Machine Translation with Large Language Models
(2022) explore to resolve document translation with the end-to-end, namely document-todocument (Doc2Doc) pattern, and utilize Multiresolutional Training, which combines documents with shorter segments like sentences or paragraphs to improve translation quality (denoted as MR-Doc2Doc). Additionally, they reproduce the document-to-sentence baseline (MR-Doc2Sent) that introduces extra model modules to capture contextual information.
Document-Level Machine Translation with Large Language Models
To enable a fair comparison with previous work, we use four widely used document-level translation benchmarks: TED (ZH-EN and EN-DE), News (EN-DE), and Europarl (EN-DE). We adopt tokenized case-insensitive BLEU and d-BLEU as the evaluation metrics. For d-BLEU, we compute the score based on either the concatenation of generated sentences or the directly generated documents. As MR-Doc2Doc and ChatGPT generate document-level translations that are difficult to separate into individual sentences, we only report d-BLEU scores for these models. Table 5 lists the results of all document-level models. As seen, the MR-Doc2Doc with extra model pre-training achieves the best documentlevel performance among previous models.  select 100 examples and ask annotators to assess whether the responses generated by the models include the correct prediction and explanation, respectively. We count the accuracy (%) of prediction and explanation for ChatGPT and GPT-4, based on which the Phi coefficient (r φ ) is calculated to measure the association between two binary variables (i.e., prediction and explanation).
Document-Level Machine Translation with Large Language Models
(EN-DE) dataset, even worse than Sent2Sent. We suspect this phenomenon may be caused by the domain distribution bias of the training data. Moreover, we find that ChatGPT is unstable, and its translation results sometimes exhibit omissions and obvious copying behaviors. Takeaway: (1) ChatGPT has exhibited superior performance in document-level translation and may become a new promising paradigm for document-level NMT.
Document-Level Machine Translation with Large Language Models
(2) It is still debatable whether these benchmarks can be considered as appropriate measures for evaluating document-level translation methods in the future.
Document-Level Machine Translation with Large Language Models
We analyze the ability of LLMs to capture discourse knowledge from two perspectives: (1) probing the discourse knowledge encoded in LLMs, and (2) examining the impact of different training techniques on discourse modeling.
Document-Level Machine Translation with Large Language Models
In order to verify whether LLMs (i.e., ChatGPT and GPT-4) truly learn to utilize the context to resolve discourse inconsistencies, we adopt the contrastive test sets proposed by Voita et al. (2019b). This dataset includes deixis, lexicon consistency, ellipsis (inflection), and ellipsis (verb phrase) for evaluating discourse phenomena in English-Russian translations. Each instance has a positive translation and a few negative ones that differ by only one specific word. The goal is to determine if a model is more likely to generate a correct translation than incorrect variations. In this experiment, we compare ChatGPT/GPT-4 with advanced methods, such as Sent2Sent, MR-Doc2Doc, CADec (Voita et al., 2019b) and DocRepair (Voita et al., 2019a), where CADec and DocRepair introduce context-aware post-editing modules to refine the sentence-level translations. For these baselines, we adopt force decoding to generate scores for all translation candidates in each instance. If the score of the positive translation is the highest, then this instance is counted as correct. For ChatGPT and GPT-4, we query them with the prompt P4 in Table 6 to obtain responses and correspondent explanations for each instance, in which D denotes the source document, T i is the translation candidate and m represents the number of candidates. Then some heuristic rules and manual verification are used to produce the model's final selection. Table 7, ChatGPT performs worse than DocRepair (discouse-enhanced method) across all discourse phenomena, with particularly significant gaps present in deixis and lexical consistency tests. These results show that it is difficult to handle deixis and lexical consistency phenomena with large-scale document-level pre-training. GPT-4 exhibits significant improvements in these areas, but it still lags behind DocRepair in deixis, lexical consistency, and ellipsis (inflection) phenomena. Takeaway: (1) ChatGPT demonstrates lower accuracy in contrastive prediction compared to conventional translation models, whereas GPT-4 exhibits significant improvement.
Document-Level Machine Translation with Large Language Models
(2) As there is no detailed technical report available for GPT-4, we argue that its significant improvements are likely due to the use of supervised data and RLHF.
Document-Level Machine Translation with Large Language Models
Evaluation on Explanation We conduct human evaluations to assess the quality of LLM-generated explanations. This provides an additional way to explore the discourse knowledge contained within LLMs. As illustrated in Table 8, we randomly select 100 examples for each contrastive test set and request native speakers to evaluate whether the models' responses contain the correct prediction and explanation, respectively. Then the Phi coefficient (r φ ) is further calculated to better measure the correlation between two binary variables (i.e., prediction and explanation). We can observe that the accuracy of explanation is often not reflective of the accuracy of prediction, indicating a mismatch in utilizing discourse knowledge for prediction and explanation. In addition, ChatGPT is not good at explaining the reason for selecting the correct translation, while GPT-4 exhibits high performance in this aspect and brings better accuracy of prediction. Takeaway: (1) GPT-4 demonstrates a strong ability to explain discourse knowledge.
Document-Level Machine Translation with Large Language Models
(2) Despite GPT-4's superior performance in prediction and explanation, the correlation between prediction and explanation does not appear to be significantly improved compared to ChatGPT.
Document-Level Machine Translation with Large Language Models
LLMs have become the foundation of natural language processing research (Brown et al., 2020), with recent advancements such as learning from source code (Chen et al., 2021) and RLHF showing promise in improving LLM performance (Ouyang et al., 2022). To further investigate the potential of these approaches, we aim to answer the following questions: 1) Can pretraining on source code improve the discourse understanding of LLMs? 2) Is RLHF helpful for LLMs in understanding discourse structure in machine translation? 3) Is contextual information useful for LLMs in documentlevel machine translation? To explore the impact of various training techniques on discourse-aware translation, we conduct   Figure 2:
Document-Level Machine Translation with Large Language Models
• GPT-3 (Brown et al., 2020): A LLM with 175B parameters pre-trained on large-scale web corpora (approximately 400B tokens). • InstructGPT (SFT) (Ouyang et al., 2022): A GPT-3 model trained with supervised fine-tuning on human demonstrations similar to Ouyang et al. (2022). 6 • InstructGPT (FeedME-1) (Ouyang et al., 2022):
Document-Level Machine Translation with Large Language Models
An improved version of GPT-3 with supervised fine-tuning on human-written demonstrations and model-generated examples rated by humans with high quality. • InstructGPT (FeedME-2) (Ouyang et al., 2022):
Document-Level Machine Translation with Large Language Models
An improved version of Codex (Chen et al., 2021) with supervised fine-tuning on human-written demonstrations and human-rated examples with high quality. 7 • InstructGPT (PPO) (Ouyang et al., 2022): An improved version of InstructGPT (FeedME-2) with extra training of RLHF, which is trained with a reward model learned from human comparisons. 8 • ChatGPT: A further improved version of In-strucGPT that can perform tasks via dialogue with users, which is able to take contextual information in dialogue into consideration.
Document-Level Machine Translation with Large Language Models
The evaluation results are shown in Table 9, which demonstrate that LLMs are capable of zeroshot document-level machine translation for fiction texts. We found that SFT greatly improves Instruct-GPT compared to GPT-3, raising its d-BLEU score from 3.3 to 7.1. InstructGPT (FeedME-1) surpasses InstructGPT (SFT), with a d-BLEU score of 14.1 and human evaluation scores of 2.2/2.5, underlining the value of high-quality examples. Instruct-GPT (FeedME-2) achieves a 2-point d-BLEU improvement over FeedME-1, indicating pre-training on source code benefits LLM performance. However, its discourse awareness lags 0.2 points behind InstructGPT (FeedME-1), suggesting room for improvement. InstructGPT (PPO) outperforms InstructGPT (FeedME-2) with a d-BLEU score of 17.2 and human evaluation scores of 2.6/2.7, showing that RLHF enhances LLM performance, particularly in understanding discourse structure (0.4 improvement over InstructGPT (FeedME-2)). Lastly, ChatGPT and GPT-4 excel in both d-BLEU and human evaluation scores, demonstrating the importance of contextual information for complex, lengthy document translation. ChatGPT's outstanding human evaluation scores highlight its superior translation capabilities and strong performance in maintaining discourse awareness. Takeaway:
Document-Level Machine Translation with Large Language Models
(1) Training techniques such as SFT and RLHF consistently enhance LLMs' document translation capabilities, as evidenced by improved d-BLEU scores.
Document-Level Machine Translation with Large Language Models
(2) The conversational ability of ChatGPT and GPT-4 significantly boosts translation quality and discourse awareness, demonstrating the value of incorporating contextual information for document-level translation.
Document-Level Machine Translation with Large Language Models
We provide a comprehensive evaluation of LLMs (such as ChatGPT and GPT-4) for documentlevel machine translation. Our evaluation covers three main aspects: (1) the effects of discourseaware prompts, (2) comparison of translation models, and (3) analysis of discourse modelling abilities. With the release of the GPT-4 model, the discourse-aware performance has been significantly improved, making it a promising paradigm for document-level translation. In our future work, we plan to explore more challenging discourseaware NLP tasks, such as the GuoFeng Bench-mark .
Document-Level Machine Translation with Large Language Models
A.1 Human Evaluation Guidelines Table 10 presents the human evaluation criteria for document-level translation, with scores ranging from 0 to 5. A score of 5 indicates excellent overall translation quality, with no grammatical errors, accurate word choice, consistent key terms, and consistent context and tone throughout the passage. A score of 0 indicates poor overall translation quality, with more than half of the translation being mistranslated or missing, inconsistent key terms, and poor fluency and clarity. In between scores reflect varying degrees of translation quality, with factors such as fluency, accuracy, consistency of key terms, and context and tone consistency affecting the score.
Document-Level Machine Translation with Large Language Models
Discourse Awareness
Document-Level Machine Translation with Large Language Models
Translation passes quality control; the overall translation is excellent. Translation is very fluent with no grammatical errors and has been localized to fit target language. Word choice is accurate with no mistranslations. The translation is a 100% true to the source text.
Document-Level Machine Translation with Large Language Models
No inconsistency relating to key terms such as names, organization, etc. Linking words or expressions between sentences keeps the logic and language of the passage clear and fluent. Context and tone are consistent throughout. The style of the text conforms to the culture and habit of the target language.
Document-Level Machine Translation with Large Language Models
4 Translation passes quality control; the overall translation is very good. Translation is fluent. Any errors that may be present does not affect the meaning or comprehension of the text. Most word choice is accurate, but some may cause ambiguity. Key terms are consistent. Inconsistency is limited to non-key terms.
Document-Level Machine Translation with Large Language Models
Logical and language is clear and fluent. Some sentences lack transition but does not affect contextual comprehension. Topic is consistent. Tone and word choice may be inconsistent, but comprehension is not affected. Translation conforms to the culture and habit.
Document-Level Machine Translation with Large Language Models
3 Translation passes quality control; the overall translation is ok. Translation is mostly fluent but there are many sections that require rereading due to language usage. Some word choice is inaccurate or errors but meaning of the sentence can be inferred from context.
Document-Level Machine Translation with Large Language Models
Some key terms may be inconsistent. Most sentences translation smoothly and logically but some sentences that may seem abrupt due to lack of linkage. Topic is consistent. Tone and word choice is inconsistent, noticeably affecting the accuracy of reading comprehension.
Document-Level Machine Translation with Large Language Models
2 Translation does not pass quality control; the overall translation is poor. Meaning is unclear or disjointed. Even with multiple rereading, passage may still be incomprehensible. Translation is not accurate to the source text or is missing in large quantities, causing the translation to deviate from the source text.
Document-Level Machine Translation with Large Language Models
Many key terms are inconsistent, needing multiple rereading to understand context of the passage. Some linkages are present but overall, the passage lacks fluency and clarity, causing trouble with comprehension. The topic or tone is different from the other passages, affecting reading comprehension.
Document-Level Machine Translation with Large Language Models
1 Translation does not pass quality control; the overall translation is very poor. More than half of the translation is mistranslated or missing.
Document-Level Machine Translation with Large Language Models
Key terms are inconsistent, causing great trouble with comprehension. Some linkages are present but overall, the passage lacks fluency and clarity, heavily interfering with comprehension. The topic or tone is different from the other passages, heavily interfering with comprehension. 0 Translation output is unrelated to the source text.
Document-Level Machine Translation with Large Language Models
Output is unrelated to previous or following sections. 
Large language models effectively leverage document-level context for literary translation, but critical errors persist
: A plot of the total number of errors annotated in sentence-level (SENT) and paragraph-level (PARA) translations produced by GPT-3.5 across 18 different language pairs. In all cases, PARA produces fewer errors than SENT, which demonstrates that GPT-3.5 takes advantage of discourse context during translation. on sentence-level benchmarks (Vilar et al., 2022;Hendy et al., 2023;Jiao et al., 2023). Furthermore, LLMs are increasingly being deployed for document-level translation (Book Maker, 2023;Pawlak, 2023), a scenario for which there are currently no reliable automatic evaluation methods. In this paper, we hire human translators to conduct a rigorous fine-grained evaluation of GPT-3.5's ability to translate paragraph-level texts from literary works across 18 different language pairs. Our results ( Figure 4) demonstrate that GPT-3.5 2 effectively leverages discourse-level context to produce higher-quality translations than when translating sentences in isolation.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Why literary texts? Translating works of literature poses unique challenges due to the intricate Figure 2: An example of paragraph-level (PARA) and sentence-level (SENT) translations of the same Japanese paragraph into English. Sentence-level translation results in a range of erroneous translations, from worse word choice ("understood" vs "right away") to incorrect pronouns ("he" vs "I"). nature of creative work and the importance of capturing the author's voice and contextual nuances. Translators thus apply a wide range of translation techniques (Chesterman, 1997;Molina and Hurtado Albir, 2004), from simple shifts in grammatical categories to more complex stylistic or content-based rearrangements that often cross sentence boundaries. Translators may also merge or split sentences, or even entire paragraphs, which renders the traditional sentence-level pipeline insufficient for capturing the full scope of the original text (Toral and Way, 2015;Taivalkoski-Shilov, 2019b). 3 Taken together, these properties make literary texts a good testbed for document-level machine translation (Thai et al., 2022); in our work, we focus on the paragraph 4 as a minimal discourselevel unit.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Why human evaluation? The absence of rigorous document-level evaluations of LLM translators is striking but also somewhat understandable given the unreliability of automatic metrics (Thai et al., 2022) and the difficulty of properly conducting human evaluations (Castilho, 2021). Furthermore, evaluations of LLM translators are especially difficult due to data contamination (Aiyappa et al., 2023), as it is unclear whether the models are pretrained on existing benchmarks (e.g., from WMT). We fill this gap by first collecting paragraphs from recently-published literary translations. Then, we 3 At least 55% of the reference target paragraphs used in our study split or merge sentences from the source text (measured with an automatic sentence tokenizer). 4 We broadly define a paragraph as a distinct passage within the novel, focusing on a single theme. provide human translators with two candidate machine translations of a given source paragraph and ask them to (1) mark error spans and categorize them based on a predefined schema inspired by MQM (Lommel et al., 2014b;Freitag et al., 2021), (2) make preference judgments of which of the two translations is of higher quality, and (3) provide free-form justifications of their preference judgments. In total, we collect such annotations on 720 pairs of translated paragraphs across 18 different language pairs (using three diverse target languages of English, Japanese, and Polish), which we then leverage for a fine-grained analysis of the behavior of different LLM translation methods.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
How do we use LLMs to translate paragraphs? We use three strategies to generate the paragraphlevel translations for our evaluations that all rely on few-shot prompting with GPT-3.5: (1) translating each sentence in the paragraph in isolation of the others (SENT); (2) translating each sentence in the paragraph when provided with the rest of the paragraph as context (PARA_SENT); and (3) translating the entire paragraph in at once (PARA), not sentence-by-sentence. Finally, we also compare these methods to Google Translate (GTR).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
LLMs produce better translations when provided with paragraph-level context: Our evaluations reveal that using GPT-3.5 to translate complete paragraphs (PARA) yields translations of significantly higher quality than both the sentenceby-sentence GPT-3.5 methods as well as Google Translate. Our detailed analysis of annotated translation errors and free-form comments show that paragraph-level translations exhibit increased coherence, better preservation of literary style, and improved handling of context-dependent expressions (see Figure 2). That said, we also observe that PARA still makes numerous critical mistranslations and other errors across different language pairs, which shows that LLM-based translators still have significant room for improvement, particularly when applied to translating contextually-rich literary texts.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Before describing our dataset and evaluation, we first contextualize our work within both the existing body of document-level 5 machine translation as well as recent papers on translation via large language models.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Before the rise of neural machine translation, several attempts were made to incorporate discourselevel phenomena into statistical machine translation systems (Hardmeier, 2012;Carpuat and Simard, 2012;Hardmeier et al., 2013;Ding et al., 2014). Neural MT systems condition sentenceby-sentence translation on discourse-level context via concatenation models (Tiedemann and Scherrer, 2017;Jean et al., 2017;Agrawal et al., 2018;Junczys-Dowmunt, 2019;Lopes et al., 2020), hierarchical models (Miculicich et al., 2018;Tan et al., 2019;Chen et al., 2020;Zheng et al., 2020), multi-pass models (Mansimov et al., 2021), dynamic context models (Kang et al., 2020), multisource models (Zhang et al., 2018;Feng et al., 2022), and transfer learning approaches (Zhang et al., 2022). Despite sometimes obtaining clear gains from discourse-level context (Voita et al., 2019), the machine translation community has not made much progress on this problem, particularly for non-English language pairs, due largely to the scarcity of parallel document-level corpora (Zhang et al., 2022). This problem has been partially addressed by introducing a pivot language (Cohn and Lapata, 2007;Utiyama and Isahara, 2007), but this approach can also lead to substantial information loss.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Many recent studies explore the potential that LLMs hold for translation, an especially attractive prospect given that training or fine-tuning on large parallel corpora is not necessary. 6 These works span paragraph-level post-editing with LLMs (Thai et al., 2022), translating sentence-level inputs (Vilar et al., 2022;Jiao et al., 2023), analyzing hallucinations in LLM-generated translations (Guerreiro et al., 2023), and employing LLMs to evaluate machine translation (Kocmi and Federmann, 2023). Studies on prompt engineering for translation conclude that simple sentence-level English prompt templates are effective for paragraph translations (Zhang et al., 2023). Other findings reveal that automatically-generated dictionaries assist translation (Ghazvininejad et al., 2023), and that example quality outweighs lexico-semantic proximity to input (Vilar et al., 2022). To the best of our knowledge, the only work other than ours that evaluates LLMs for paragraph-level translation is that of Hendy et al. (2023), which focuses on automatic evaluation of context-aware sentence-by-sentence translation. Unlike Hendy et al. (2023), we perform a fine-grained human evaluation of paragraph-level translation, which sheds more light on the concrete strengths and weaknesses of LLM translators in this setting.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Our work differs from existing research on translating with large language models in two key ways: we focus on translating literary text at the paragraph level. In this section, we describe and motivate the paragraph-level translation dataset used in our study, which covers eighteen unique language pairs (three target languages) and is sourced from recently-published novels. Then, we outline the different ways in which we leverage GPT-3.5 to translate these paragraphs at both the sentence and paragraph levels.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Literary texts (e.g., novels or short stories) pose unique challenges for translators due to their complex nature. Translators must interpret and honor the author's voice with no objective reality to measure against, which can result in several equally "Oh, poor old thing, she has a nervous bladder!" exclaimed someone's chubby mother. "Is that a Persian rug?" Whose mother was it? Unclear. No one would cop to it, of course. We canceled the performance. "Admit it, that was your mother," said a kid named Rafe to a kid named Sukey, when the parents had filed out. Some of their goblets, highball glasses, and beer bottles were completely empty. Drained. Those parents were in a hurry, then. "No way," said Sukey firmly, and shook her head. "Then who is your mother? The one with the big ass? Or the one with the clubfoot?" "Neither," said Sukey. "So fuck you."  valid translations (Sager, 1998). For machine translation systems, these challenges exacerbate the need for discourse-level context (Thai et al., 2022): an author's intended meaning or style is often unclear from just a single sentence.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Selecting paragraphs from novels: How good are machines at translating literary paragraphs? To answer this question, we extract 20 paragraphs (dialogues and narrative texts) each from 18 recentlypublished translations of novels, and we manually align these paragraphs with corresponding paragraphs in the source novel 7 (see Table 1). The target language of each translation is English, Polish, or Japanese (6 books for each), and we consider eight different source languages. Almost all of the translations were published after 2021 (see Table 2), which is important to avoid data contamination with the pretraining data of large language models. In sum, we obtain 360 aligned source-target paragraphs, which we use for all of the experiments described in the rest of the paper.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Paragraph length: All paragraphs consist of at least two sentences, and the majority of them are between four to nine sentences long (mean=7.45, std=4.14). 8 As automatic sentence tokenizers are not always reliable for all of the languages considered in our study, we manually perform sentence tokenization to enable a direct comparison of sentence and paragraph-level translation systems. For more details about the dataset statistics, including token and sentence counts, see Table 8 and Table 9.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Target language selection: We select English, Japanese, and Polish as the target languages of our study, as these languages differ considerably in many linguistic aspects. English is an analytic language that is widely spoken and extensively studied in the field of natural language processing, and it serves as the primary pretraining language of most large language models, including GPT-3.5. 9 In con-8 A paragraph with fewer sentences is not necessarily short: for example, in the German novel "An Inventory of Losses," sentences can be as long as 70 to 80 words, with the longest reaching 117 words. 9 As of 2020, the reported distribution of languages featured in the present study within the GPT-3 training data was as follows: English -92.647% (1st), French -1.818% (2nd), German -1.469% (3rd), Russian -0.188% (9th), Polish  Table 2: Details of the translated novels used in our study. In cases where the same novel is used for multiple target languages (e.g., "The Years"), identical source paragraphs are extracted to enable comparisons across language pairs. These novels exhibit distinct differences beyond just their source languages. For instance, "What Can You See From Here" presents a philosophical exploration of life and death, while "Sword of Destiny" is a fantasy story part of "The Witcher" saga.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
trast, both Japanese and Polish are comparatively under-explored. Japanese is an agglutinative language that employs three distinct writing systems: Kanji, Hiragana, and Katakana. As a high-context language, the translation of Japanese texts necessitates a profound comprehension of context and cultural nuances, rendering it a compelling choice for testing the limits of LLMs' translation capabilities. Polish, on the other hand, is a fusional language characterized by a rich morphological system. Its complex word forms, grammatical gender, conjugation, and declension make it an apt choice for testing the accuracy and robustness of LLMs. 10 Source language selection: As source languages, we select English (es), Polish (pl), Russian (ru), Czech (cs), French (fr), German (de), Japanese (ja), and Chinese (zh). These languages belong to a diverse array of language families -Indo-European (Romance, Germanic, Slavic), Sino-Tibetan, and Japonic -each with distinctive morphological traits -fusional, agglutinative, and analytic. Moreover, -0.155% (11th), Japanese -0.111% (15th), Chinese -0.099% (17th), Czech -0.071% (18th) (see https://github. com/openai/gpt-3/blob/master/dataset_ statistics/languages_by_word_count.csv). The current GPT-3.5 text-davinci-003 model is reported to incorporate data up to June 2021 and it is unclear what texts or languages were added to the original training data https://platform.openai.com/docs/ models/gpt-3-5. 10 The first author is fluent in all three target languages.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
they employ a variety of writing systems such as the Latin alphabet, the Cyrillic alphabet, Hanzi, and Kanji/Hiragana/Katakana (see Table 7 in Appendix A for details). Finally, we carefully select source-target language pairs to ensure that our study encompasses both linguistically similar and dissimilar languages. For example, we paired cs-pl, as these languages are characterized by only 10% lexical distance 11 and have similar syntactic structures (Jágrová and Avgustinova, 2023). Conversely, we also include ja-pl, as the two languages have very little lexical overlap, vastly different grammars, and utilize distinct writing systems.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
In this paper, we focus on translating the literary paragraphs in our dataset using large language models. More specifically, we use the GPT-3.5 text-davinci-003 checkpoint, which has been further tuned to follow instructions based on human feedback (Ouyang et al., 2022). Hendy et al. (2023) demonstrate that GPT-3.5 produces translations of reasonable quality, though their focus was mostly at the sentence level. Since many LLMs including GPT-3.5 are only accessible via blackbox APIs, we adapt the model for translation via in-context learning (Brown et al., 2020). Demonstration examples: We use few-shot prompting, in which a model is provided with a prompt consisting of five demonstrations. We manually curate the five demonstrations from literary texts for each of the 18 language pairs, resulting in 90 total demonstration examples. These demonstrations are sourced from novels that are not part of our translation dataset, resulting in potential differences in topic and style (see Table 10 in the Appendix A for details). We further ensure that each set of five demonstrations includes both dialogues and narrative texts.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Prompting for translation: We consider the following three prompting strategies for GPT-3.5 that allow us to compare the model's abilities to translate with and without discourse-level context (see Table 3 for templates and Appendix B for the exact prompts):
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• GPT-3.5 sentence-level translation without context (SENT): Each sentence of the paragraph is translated in isolation of the others.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
To maintain consistency, we provide the same five sentence-level examples 12 in each prompt for the given source-target language pair. 13
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• GPT-3.5 sentence-level translation with context (PARA_SENT): Each sentence of the paragraph is translated in context. The model is provided with the entire source paragraph as input, where the sentence to be translated is wrapped in <translate> and </translate> tags, in addition to a partially-translated target paragraph. The demonstrations in the prompt also contain <translate> and </translate> tags wrapped around one sentence per demonstration. For each demonstration in the prompt, a sentence in a different position was chosen (e.g., from the beginning, middle, and end of the paragraph).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• GPT-3.5 paragraph-level translation (PARA): The entire source paragraph is passed into the model, and the output target paragraph is generated conditioned on this input (i.e., without any sentence tokenization).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
12 Sentence-level demonstrations for SENT are sampled from the demonstrations for paragraph-level translation. 13 To ensure consistent quotation mark usage and enable a fair comparison with paragraph-level translations, quotation marks in sentence-level translations were manually adjusted.  Demonstrations in the prompt are also paragraphs 14 of translations from the respective source language into the target language in question. 15
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Using Google Translate (GTR) as a baseline:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
In order to compare commercial-grade translation systems to LLM translators, we also translate all paragraphs in our dataset using Google Translate. 16 We opt for an off-the-shelf commercial system instead of a state-of-the-art system from, for instance, WMT competitions for two primary reasons. First, our experiments focus on literary translations. Given that WMT systems are predominantly evaluated on the news domain, it is uncertain which system would perform best, and some language pairs may not even be supported. Second, our main research question revolves around LLMs' ability to incorporate contextual information, rather than merely comparing their perfor- 14 The examples for PARA and PARA_SENT configurations are necessarily lengthier. Due to the GPT-3.5 maximum context size, it is not always possible to include all five examples within the prompt. Consequently, around 10% of the data was translated using four or fewer examples. 15 Initially, we experimented with GPT-3 by translating between two non-English languages using English as a pivot, as it is the primary language of the model. The model had access to the source text and its English translation. After manual evaluation and comparison to translations without a pivot language, we found no significant benefit in using English as the pivot. Consequently, we directly translated paragraphs into the target language. Refer to Appendix D. for details and results of this preliminary study. 16 All paragraphs were translated in January 2023 using the GoogleTranslate API. mance with state-of-the-art translation systems. We employ GTR as a reasonably robust baseline to assess the extent to which context can enhance MT quality, rather than asserting that LLMs outperform all traditional MT systems.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
How do we compare the translation quality of the systems described above? Automatic metrics such as BLEURT and COMET are untested on document-level inputs as well as literary texts, and as such we do not consider them reliable, although we do report them in §5.1. 17 Human evaluation is equally problematic, as direct assessments of translation quality (e.g., "rate the quality of this translation from 0-100") suffer from calibration issues that are exacerbated with longer texts (Karpinska et al., 2021). Thus, we opt for a human evaluation inspired by Multidimensional Quality Metrics (Lommel et al., 2014b, MQM), in which annotators mark and classify error spans within the translation. Specifically, for each of the 18 language pairs studied in this work, we hire translators to identify all span-level errors in two competing translations. For each evaluated pair, the annotators were also asked to choose the better translation and provide a free-form rationale. For each source paragraph, the translators make three binary judgments of which translation is higher quality: SENT vs PARA, PARA_SENT vs PARA, and GTR vs PARA.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Recruiting annotators: As our task is complex and requires fluency in both the source and target language, we hire translators to provide the annotations. We recruit 13 translators, each of whom is a native speaker of English, Polish, or Japanese 18 through the Upwork freelancing platform. 19 One translator, hired directly, was a bilingual speaker of English and Polish with advanced knowledge of German; as such, she performed the pl-en, de-en, and de-pl evaluations. Evaluation of ja-pl, pl-ja, 17 Automatic metrics developed specifically for documentlevel MT are also insufficient as they either work best with one-to-one sentence level alignments (Vernikos et al., 2022;Hendy et al., 2023) or are available only for English (Jiang et al., 2022). 18 The annotators for Czech-Polish and Russian-English were both native speakers of the respective source languages and highly proficient in their respective target languages. They collaborated with native speakers of the target languages, who possessed a basic understanding of the source language, to complete their annotations.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
19 https://www.upwork.com/ 
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Why is it better? I prefer T1 to T2 as it doesn't...
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Inne zdjęcie, podpisane tym samym fotografem...
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Binary choice (yes/no)
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Binary choice (confident vs unsure)
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Was it significantly better? Figure 3: A description of the annotation process for a pair of candidate translations given a source paragraph. Note that our hired translators go through this pipeline for three different pairs per source paragraph, comparing PARA with SENT, PARA_SENT, and GTR. and pl-en texts was done by the first author in a collaboration with native speakers of Polish/Japanese to avoid any potential bias. Each translator was paid $2 per evaluated pair of candidate translations, with an additional $5 bonus to cover the time spent familiarizing themselves with the instructions. We asked them to compare three pairs of system translations (PARA vs. SENT, PARA vs. PARA_SENT, PARA vs. GTR) for 10 paragraphs per language pair; as such, 180 total source paragraphs were used in our evaluations. Altogether, we paid approximately $12 per hour, with a total cost of $955.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Annotation task: First, we tasked the hired translators 20 with annotating a subset of MQM translation errors identified through a pilot analysis and annotation of the system's outputs. Specifically, we ask them to highlight spans within the candidate translations that contain errors belonging to any of the following error categories:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• mistranslation: 21 accuracy errors that occur 20 They were presented with guidelines in their native language. The annotation task was performed using the Label-Studio annotation tool (Tkachenko et al., 2020(Tkachenko et al., -2022. See Figure 11 for the screenshot of the interface. 21 We note that mistranslations in literary text are often not as grave as, for instance, in news articles. Human translators hold poetic license, which allows them to change some details to make the text more enjoyable for the reader. Is changing "bonito" into "tuna" incorrect? Or can it be perceived as a way to accommodate an English-speaking readership that is likely more familiar with the latter? when the wrong target word or phrase is chosen to represent content from the source text. In addition to canonical mistranslations, we also include overly literal translation errors that occur when systems translate word-byword into the target language even though the result is nonsensical.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• grammar: grammatical errors, such as errors in conjugation or declension, wrong prepositions, etc.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• untranslated: words or phrases that should have been translated into the target language but were either left in the source language or just transliterated into the target language.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• inconsistency: use of different terms to refer to the same entity, or different words where the same word should be used for stylistic reasons (e.g., "Kasia" and "Kate," "coat" and "jacket," or "bad" and "awful" );
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• register: a clear violation in the use of formal and informal language within the same text, only annotated in Japanese 22
Large language models effectively leverage document-level context for literary translation, but critical errors persist
• format: incorrect usage of punctuation (e.g., "." instead of "。").
Large language models effectively leverage document-level context for literary translation, but critical errors persist
After the span-level annotation is complete, we then ask the translators to further identify if any of the candidate translations contains significant content additions or omissions in relation to the source text. 23 Finally, they are asked to choose the better translation and provide a justification for their choice in two to five sentences. We instruct them to additionally mark whether their chosen translation is significantly superior, or if the decision was difficult because both translations are of roughly comparable quality (see Figure 3 and Appendix C for details).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
In this section, we compare our different literary translation methodologies using both automatic 22 We only annotate cases where the level of formality changes abruptly within the same paragraph. It is possible that a given character would be more likely to use formal language but an informal language is being employed. As long as this is consistent we do not consider it an error as this cannot be fully determined from the paragraph context. 23 Note that this task was simplified to a binary choiceeither there were serious omissions/additions or not. We did not ask the annotators to further annotate them due to the time restrictions.  metrics and aggregate statistics from the human evaluations. Overall, we observe that the PARA configuration outperforms competing methods across all evaluations and language pairs. These results demonstrate that GPT-3.5 effectively leverages paragraph-level context to produce better translations than sentence-level methods, and also that the less efficient sentence-by-sentence translation with context is (PARA_SENT) is unnecessary to achieve high translation quality.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
We assess the translation from all four systems using the reference-based COMET (Rei et al., 2022), BLEURT (Sellam et al., 2020), and BERTSCORE (Zhang et al., 2020) metrics, as well as the reference-free COMET-QE (Rei et al., 2021) 24 metric. Although these metrics were not explicitly designed for evaluating paragraph-level outputs and their results should be interpreted with caution, they prove more reliable than string-based metrics like BLEU, especially for literary translations (Thai et al., 2022;Karpinska et al., 2022;Gehrmann et al., 2022). Table 4 shows the effectiveness of the PARA translation method: a statistical analysis with linear mixed-effects models (Baayen et al., 2008) demonstrates that PARA significantly outperforms SENT and GTR based on COMET, BLEURT, and COMET-QE scores (p<.001), and surpasses GTR based on the BERTSCORE results (p<.001). 25 Figure 5 contains human preference results comparing PARA to SENT, PARA to PARA_SENT, and PARA to GTR, aggregated across all 18 language pairs studied in this paper (i.e., 180 votes per system comparison). Table 12 breaks down these results for each language pair, and we observe the 24 We use the newest wmt22-comet-da checkpoints for COMET, Bleurt-20 checkpoints for BLEURT, wmt20-comet-qe-da checkpoints for COMET-QE, and the HuggingFace implementation which employs roberta-large for BERTSCORE. 25 We present more details of this analysis in Appendix E. Figure 4: The distribution of translator preference judgments between sentence-level translation (SENT) and paragraph-level translation (PARA). PARA is preferred (i.e., more votes) in every language pair except de-ja, fr-en and de-en.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
same trends for the vast majority of pairs. Overall, the translators significantly favored PARA translations over the alternatives (p<.001, binomial test). Table 5 contains specific information about grammar and mistranslation errors split across the three target languages (see Table 6 and Table 13 for details), which we use to discuss the three comparison settings in more detail below.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
PARA is clearly better than SENT: PARA is preferred by translators over SENT at a rate of 71.1% (p<.001, 95% CI [0.639, 0.776]). Additionally, when translators preferred PARA, they were usually confident in the decision (i.e., it was clearly better than SENT); even if we exclude all "unsure" votes, the preference for PARA translations remains significant at 78.5% (p<.001, 95% CI [0.695, 0.859]). The only language pair in which SENT is favored over PARA is de-ja (see Figure 4). This result may be attributed to the fact that the German novel An Inventory of Losses by Judith Schalansky, used for this language pair, contains the longest sentences in our dataset (on average 45 tokens per sentence), which means that the intra-sentence context is likely more informative than in other books (see Table 8). Overall, SENT translations contain 29.5% more mistranslations, 65.4% more grammatical errors, over 12 times more inconsistency errors, and three times more register errors (see Table 5).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
PARA is clearly better than GTR: PARA translations are overwhelmingly preferred over those . In the fr-ja, pl-ja, zh-ja, and cs-pl language pairs, PARA received all of the ten votes over GTR. Part of this advantage may be attributed to GTR sometimes using English as a pivot language, which can result in information loss. Our Czech translator observed that mistakes in GTR translations suggest the text was first translated into English. 26 Overall, GTR translations result in 57.7% more mistranslations, 37.3% more grammatical errors, over twice as many inconsistency errors, and ten times more register errors (see Table 5). Additionally, GTR produced 125 format errors while PARA produced perfect outputs in this regard. Finally, it is worth noting that GTR left fewer words untranslated, though this is inflated by the fact that in one German text, the word "Bauer" ("farmer") was untranslated 14 times in the PARA translation.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Our evaluations show that PARA is better than PARA_SENT, but the gap is smaller than it is for the other two methods. PARA is still preferred at a 66.1% rate ( FORMAT EN 0 n/a n/a 1 JA 0 n/a n/a 116 PL 0 n/a n/a 8 TOTAL 0 n/a n/a 125  Table 5). While PARA_SENT leaves around 22% more words untranslated, it appears to leverage the contexts and even occasionally selects better equivalents in the target language, as evidenced by translator comments. One major issue with PARA_SENT is that it occasionally repeats sentences, whereas PARA never does so.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
What do translators think about PARA? To wrap up this section, we provide a qualitative analysis of the free-form comments written by translators to justify their preference judgments. Overall, the translators praise PARA for its more skillful use of rhetoric devices, and surpas[ing] SENT as a literary rendition. They also mention that PARA uses more of a poetic license but this makes it stylistically much smoother than SENT. Furthermore, translators state that PARA clearly better reflects the content and style of the original when compared to GTR, and that it stays consistent within the paragraph. Inevitably, translations are not flawless, and there are instances where both compared systems fall short, as highlighted by one of the TRG LANG TYPE  SUBTYPE  PARA SENTS PARA_SENTS GTR   JAPANESE   PARTICLE  wrong or missing  21  22  13  12  ADJECTIVE  wrong continuative  0  2  3  0  other  0  0  2  0  VERB  tense  3  7  1  14  mood  2  1  4  5  finite/non-finite  5  2  1  3  other  2  5  6  0  ORDER  wrong order  1  6  1  16  OTHER  8  5  6 
Large language models effectively leverage document-level context for literary translation, but critical errors persist
The aggregate statistics from the previous section confirm that PARA-level translation via GPT-3.5 is the strongest literary translator of the methods that we study. Translations produced by PARA are favored by both automatic metrics and human translators, and it makes fewer errors than competing methods. In this section, we dive deeper into specific types of errors that are made within each high-level category (e.g., grammar, mistranslation), and we present examples of errors associated with lack of context understanding made by SENT and GTR that are fixed by PARA.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
We begin by analyzing the types of grammatical errors that are made by the studied translation methods in all three target languages. 27
Large language models effectively leverage document-level context for literary translation, but critical errors persist
English: Perhaps not surprisingly, translations into English contain notably fewer grammatical mistakes than Japanese or Polish (see Table 5). The most prominent mistakes in English are incorrect articles, which is most frequent in the outputs of SENT and GTR. This is to be expected, as the choice between the definite and indefinite article in English depends heavily on the context. Other mistakes include wrong or omitted prepositions, wrong parts of speech, and incorrect word order (see Table 6).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Japanese: Translations into Japanese contain considerably more mistakes. Most notably, the systems struggle with the correct choice of particle: PARA and SENT produce twice as many mistakes in this regard than PARA_SENT and GTR (see Table 6). Other mistakes include incorrect tense, verb finite form within the sentence, or incorrect word order, the latter of which is much more frequent in GTR than any of the GPT-3.5 translations.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Polish: GPT-3.5 exhibits more difficulty with Polish, as evidenced by 55 vs 42 errors for PARA, 86 vs 50 for SENT, and 64 vs 37 for PARA_SENT (see Table 5). We notice that GPT-3.5 translations frequently generate incorrect gender, case, or prepositions (see Table 6). We also observe instances in which GPT-3.5 alters the gender of a noun, such as producing grilla, a non-existent feminine form, in place of the masculine grill, while accurately modifying all adjectives and verbs to match the novel feminine noun. 28 In contrast, the performance of GTR is comparable for Polish and Japanese in terms of grammar, with 59 and 63 errors respectively. Intriguingly, GTR seems to struggle with Polish aspect, leading to 12 errors, in contrast to 1 error in both PARA and PARA_SENT, and 5 errors in SENT within the same category (see Table 6). In summary, although GPT-3.5 is primarily trained on English, it is competitive with GTR at Polish and Japanese grammar proficiency. In fact, PARA generates the fewest grammatical errors of any system, with a total of 97 for both languages. This is in contrast to 136 errors made by SENT, 101 errors by PARA_SENT, and 122 errors by GTR (see Table 5). That said, none of these systems delivers translations devoid of grammatical inaccuracies, even for English. 28 It is worth noting that grilla can also be also the genitive form of the masculine noun grill; however, the agreement of surrounding verbs and adjectives with the feminine noun suggests that the system likely treated the word as feminine. Figure 6: Quantification of mistranslations resulting from missing or misinterpreted paragraph-level context in PARA, SENT, PARA_SENT, and GTR systems, organized by the target language (Japanese, Polish, and English).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
We manually classify all annotated mistranslations (2,324 instances) into subcategories, several of which include instances where the absence of discourse-level context is clearly a contributing factor (see Table 13 for detailed classification). We also further analyze all translations in terms of content-related issues. Overall, we observe that context is indeed incorporated into the translations for both PARA and PARA_SENT outputs, which results in fewer context-dependent issues (see Figure 6).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Pronouns: Unsurprisingly, the absence of discourse context results in the incorrect translation of pronouns. Consider the following example, with English glosses of important words provided in [brackets]:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(1) И ветер [wind] то начинал шуметь в голых деревьях, то замолкал, так же как и я прислушиваясь к течению ночи. Но он [he] не уходил, он [he] был здесь.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-RUSSIAN SOURCE (from The Story of a Life) a. The wind would start to rustle in the bare trees and then fall silent, just as I listened to the flow of the night. But he didn't leave, he was here.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-GPT-3.5 SENT (ENGLISH) b. The wind would start to rustle in the bare trees, then die down, just like me, listening to the flow of the night. But it didn't go away, it was still here.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-GPT-3.5 PARA (ENGLISH)
Large language models effectively leverage document-level context for literary translation, but critical errors persist
In Russian, nouns have grammatical gender. "Wind" in the first sentence of the source text is a masculine noun, so it is later referred to as "he" in (1). Without access to the context, the SENT model incorrectly translates it as "he" into English (1a), while the PARA translation correctly modifies the pronoun to "it" (1b).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
When translating from Russian into Polish, another language with grammatical gender, we observe issues when the gender of Russian and Polish nouns differs. Consider the following example:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(2) Романы, как известно, печатались на разной бумаге [paper]. И гореть она [she] может поразному.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
- Although both Russian and Polish nouns possess grammatical gender, "Paper" in (2) is feminine in Russian and referred to as "she," whereas it is a masculine noun in Polish and should be referred to as "he," as in (2b). The absence of context in SENT leads to an incorrect translation in (2a).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Cultural nuances: Assigning appropriate pronouns without context becomes even more challenging when translating from languages like Japanese, in which speakers frequently refer to the listener (or themselves) in the third person rather than using second-person personal pronouns such as "you" in English. Consider the following example: From the context of this conversation, a Japanese listener can easily infer that "Furukura-san" or "Miss Furukura" 29 in the last source sentence (3) is used instead of the second-person "you" as per Japanese convention. Translating this sentence 29 Note that the gender of neither character is apparent from the fragment alone. without context into English, a language in which third-person reference is not common, 30 results in a confusing translation (3a) that implies that the speaker refers to some other "Furukura" rather than their listener. However, when translating the sentence in context, the model correctly changes "Furukura" into "you" (3b), which makes it clear whom the speaker refers to in English.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Ellipsis: Another example where context helps is the translation of elliptical constructions. Consider the following example: Czech uses the same collocation as English, "do the dishes" (4), which is invalid in Polish. Hence, the ellipses in the last two sentences in (4) require broader context to be translated correctly. PARA does it properly, translating both as "wash" (4b), while SENT unsurprisingly fails to choose the correct collocation (4a).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Subject ellipsis: Similarly, context may be needed to attribute a state or an action to the correct character due to subject ellipsis. This is an obvious issue for languages like Japanese, which tend to omit the subject of the sentence and do not encode any relevant information in the verb form, but it can also arise in English. Consider the following example:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(5) When we were done, the lipstick went back into some mother's Fendi handbag. We watched her apply it, unaware. From the second sentence alone it is not clear who is "unaware" (5) -the mother or the "we" (referring to children) watching her. Only from the broader context can we confidently deduce that it is in fact the mother, not the children, who is "unaware." PARA (5b) correctly attributes the state of being "unaware" to the mother, which is exhibited by its usage of the singular feminine form of the adjective. In contrast, SENT (5a) mistranslates it using the plural masculine form of the adjective "unaware," which implies that it refers to "we" rather than the "mother."
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Consistency: Context is sometimes critical for preserving the overall consistency of the text. The simplest cases include referring to the same entity -a place or a person -in the same way. More interesting cases pertain to style and can enhance the reader's experience. Consider the following example: The German source in (6) translates into English as "To forget everything is bad, certainly. Worse still is to forget nothing." 31 It is arguably important for the translation to repeat the same word which is an equivalent of the German "schlimm" ("bad"). PARA does it well, translating both as 悪い "warui," or "bad" (6b), in the exact same way as the human Japanese translator. SENT, on the other hand, uses two different words, "tragic" and "bad" (6a), which while technically correct omits the intentional repetition that is meant to introduce an unexpected conclusion.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Polysemy: The absence of context makes it difficult to interpret words or expressions that have multiple meanings in the source language. Consider the following example: 31 Excerpt taken from the official English translation by Jakie Smith (2020). The ambiguity stems here from multiple meanings of the Russian noun форма "forma" (7), which can mean either "shape" or "uniform." Since one can be "in shape" as well as "in a uniform", it is unclear from the sentence alone which meaning was intended by the author. From the preceding context, it is clear that "everything went well" for the narrator, who mastered the art of "book'n'grill," a unique form of expression exclusive to this fictional world. Based on this, we can infer that in this instance, the term "forma" signifies "shape," as in (7b), rather than "uniform," as in (7a).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Appropriateness: Finally, context may help to choose the more appropriate equivalent for the given situation. Consider the following example: The conversation above is between a clerk and a customer. The Japanese expression かしこまり ました "kashikomarimashita" (8) is an honorific that literally means "understood." However, when choosing the best equivalent, the translator needs to consider the situation at hand to best reflect its meaning in the target language. "Understood" in SENT (8a) is technically correct, but it is an unfortunate word choice for the clerk to employ. On the other hand, "right away" in PARA (8b) fits much better in the context of this conversation. Had this been a series of commands (e.g., in a military context) "understood" would be the more favorable option.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
So far, we have shown that GPT-3.5 leverages paragraph-level context to produce translations that are better than those produced by sentence-level counterparts (SENT vs PARA). However, there are still many issues with PARA's translations. From the annotations and translators' comments, we observe that PARA suffers from occasional omissions of content from the source paragraph. SENT and GTR are certainly not free of that problem either, but omission appears to be more prominent for PARA translations (see Appendix C).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Moreover, PARA still makes a sizeable number of mistranslations and grammatical errors, though fewer than SENT or GTR. We observe that PARA occasionally merges sentences with two distinctive subjects attributing all states and/or actions to one of them. Very rarely, we also find cases where context possibly confuses the model, resulting in an incorrect translation. The following example illustrates this issue: In the French text, the narrator wonders whether the brand of the desk was Ruhlman or Leleu, with both proper nouns possibly referring to a person. In the last sentence, the French text uses "il" or "he" (9), as a desk is a masculine noun in French ("le bureau"). PARA, on the other hand, appears to be confused by the two preceding names and incorrectly translates the singular pronoun as 彼ら, or "they." Furthermore, we observe (very few) cases where the paragraph-level translation disregards the context. Most representative of this class of errors is when the model struggles to translate from Japanese in cases where the subject is omitted. The following example illustrates this issue: wieku.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-  b. Miho is now married and has bought an old house in her hometown, where her friends often gather. Though she often finds it a chore to work tomorrow, it is her only connection to the world outside the convenience store, and a valuable opportunity to interact with other "normal thirty-something women" her age, so she tries to accept Miho's invitations as often as possible.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-GPT-3.5 PARA (ENGLISH)
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Both Polish (10a) and English (10b) translations of the same source text (10) share a common issue. The narrator begins the paragraph by talking about Miho and then proceeds to describe her own (the narrator's) feelings about the situation, although the gender of the narrator is never revealed in the Japanese text. The second sentence should be written from a first-person perspective, particularly since it directly references Miho towards the end (blue text). However, both the Polish and English translations produced by PARA are confused by this: by using the third-person's perspective ("she," "her"), both translations incorrectly imply that Miho is the subject of the second sentence. SENT and GTR translate this passage accurately, albeit with some clumsy phrasing.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Finally, it is important to acknowledge that the languages covered in the current study are either mid or high-resource. Performance might be much worse when translating from or into one of the lowresource languages, such as Zulu or Armenian.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
GPT-4 does not magically solve all of these issues! Our preliminary experiments indicate that GPT-4 (OpenAI, 2023) sometimes generates better paragraph-level translations than those of GPT-3.5. For instance, it seems to have a better grasp of the inverted word order in German, though no broader conclusions should be made without further testing. Nevertheless, it does not resolve all of the issues discussed in our paper. Mistranslations and grammatical errors are still abundant across many language pairs. GPT-4 produces the following translation when fed the previous example paragraph (10) as input; note that all of the issues still remain: 32 (11) Miho is now married and has bought a used singlefamily home in her hometown where her friends often gather. Although she sometimes finds it a drag to work a part-time job the next day, she makes an effort to respond to Miho's invitations because it's a valuable opportunity to interact with "normal" women in their thirties like herself, apart from her convenience store job.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-  PARA translations hold the potential to captivate readers, especially if LLMs continue to improve at their current pace. Indeed, some of our translators mentioned that they genuinely enjoyed the task, though integrating these paragraphs into a coherent novel still poses a considerable challenge. With all that said, literary translation involves more than just overall "correctness" or mere entertainment value. A translation that is perfectly "correct" and enjoyable might still fail to convey the author's intentions or meaning skillfully hidden behind a simple phrase. Our fr-en translator shares her thoughts on this matter:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Both translations [SENT and PARA] translate the words without the feeling; the original author's voice is lost.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-FRENCH TO ENGLISH TRANSLATOR
Large language models effectively leverage document-level context for literary translation, but critical errors persist
In this paper, we demonstrate that LLMs leverage paragraph-level context to produce translations that are more coherent and enjoyable than sentence-bysentence translation while containing fewer mistranslations and grammatical issues. Our evaluations reveal that professional translators prefer paragraph-level translations over both sentencelevel translations produced by the same language model, and also to those generated by an offthe-shelf commercial system (GTR). We release our dataset and error annotations to help facilitate the development of new evaluation methodologies and automatic metrics for document-level machine translation. Finally, a full-length novel extends far beyond the confines of paragraph-level translation. In future work, we will focus on integrating individual paragraphs into cohesive chapters, which can then be expanded to encompass the entire novel. 32 Although the given paragraph is already comprehensible for a human reader, we also attempt to enhance the translation by incorporating three additional preceding paragraphs for context. Intriguingly, when provided with this extended context, both GPT-3.5 and GPT-4 generated accurate translations.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Translating with LLMs: The rise of large language models has also brought many ethical concerns to the forefront of NLP research (Blodgett et al., 2020;Bender et al., 2021). LLMs encode biases and exhibit toxicity, and these behaviors can be exacerbated by unconstrained prompting (Gehman et al., 2020;Costa-jussà et al., 2022). Further ethical concerns arise in the context of machine translation, particularly literary translation, where multiple stakeholders -the author, the translator, and the audience -are involved (Taivalkoski-Shilov, 2019a). Low-quality output can influence the perception of the author's work, impair the reader's linguistic abilities, and hinder the transfer of ideas to the target language, while overrelying on machine translation can possibly threaten the role of human translators (Drugan, 2013;Ning and Domínguez, 2016;Taivalkoski-Shilov, 2019a). On the other hand, machine translation employed responsibly as an auxiliary tool holds the potential to alleviate the translator's cognitive burden (O'Brien, 2012) and make the author's work accessible to a broader audience more swiftly (Besacier, 2014). Contrary to the predictions in Eloundou et al. (2023), we do not view large language models as a substitute for human translators, but rather as a means to assist translators in their work.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Human Evaluation: The experiments involving human translators were IRB-approved, and all involved translators gave their consent to disclose their annotations, comments, and preference choices. In recognizing contributions, our acknowledgments only include the names of those translators who explicitly gave their consent to be acknowledged by their full name in this publication. Jan Wislicki, Tom Gally, Nader Akoury, Kalpesh Krishna, Simeng Sun, Katherine Thai, and the entire UMass NLP group for insightful discussion, which helped to shape this project.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Finally, we would like to thank Sergiusz Rzepkowski (pl), Paula Kurzawska (pl, en), Hiroshi Iida (ja), Grégory Fleurot (fr), Peyton Bowman (en), Simeng Sun (zh), Igor Zapala (pl, de), Marvin Hoffmann (de), Kinga Przekota (pl, ru), and Yuki Mori (ja) for further consultations on their respective native languages.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
This project was partially supported by awards IIS-1955567 and IIS-2046248 from the National Science Foundation (NSF) as well as an award from Open Philanthropy. 
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Choosing paragraphs: The selection of a particular paragraph was semi-random, with certain considerations in mind during the sampling process. We prioritized the following criteria: (1) for each source language we sample paragraphs so that there is a combination of dialogue and narrative texts;
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(2) the paragraph should be reasonably intelligible to a human translator without additional context; and (3) alignment between the source paragraph and human translation should be feasible, meaning no major content rearrangement across paragraphs. Nonetheless, meeting all these requirements was not always possible. For instance, the source text of Convenience Store Woman (ja) is mostly written in the first-person narrative. Since Japanese does not encode the speaker's gender in the verb forms, it is often impossible to determine whether the narrator is a male or a female. In cases where it was impossible to determine the gender of the character we instructed translators to accept either option, provided that the translation remained consistent within the given paragraph (i.e., the gender did not change within the paragraph).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
A note on literary translation: It is important to understand the challenges a human translator faces when translating literary texts. Even a simple sentence may lead to substantial struggles. One of the most notable examples is the first sentence of a French novel The Stranger by Albert Camus. The story begins in a seemingly trivial way:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(12) Aujourd'hui, maman est morte.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Today, mother died.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
While there is nothing particularly difficult in (12), five English translations of "The Stranger" has already been produced and there is little consensus on what the ideal translation should be (Kaplansky, 2004).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
This very first sentence is of the utmost importance as it introduces the reader to Meursault, the protagonist of the story, who will later kill an unnamed Arab without any apparent reason. Hence, it is crucial for the storyline that the reader understands, from this very beginning, who Meursault is and what affection he holds for his mother.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Stuart Gilbert (1946), Joseph Laredo (1982), and Kate Griffith (1982 all translate the begin-ning sentence as Mother died today but this translation is problematic. English word "mother," while technically correct, is too formal to fully embrace the emotions conveyed by the French "maman." Mathew Ward (1988) opts to leave the French "maman" untranslated. An understandable choice, as the English reader is likely to decipher the meaning from the surface similarity, though they may not fully grasp its sentiment. Conversely, Sandra Smith (2012) attempts to capture the intimacy of "maman" by rendering it as "my mother," which is less formal than a simple "mother" but doesn't possess the childlike connotation of the English "mom."
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Literary translation is clearly a challenge that exceeds simple word equivalence. Professional translators face choices, that current systems are unlikely to solve independently. However, they can assist translators in their tasks, in a way similar to how computer-assisted translation (CAT) tools have been doing. This approach holds the potential to make more novels accessible to a wider audience; novels that may have remained untranslated otherwise. 
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Here we present examples of prompts employed for the translation with GPT-3.5. The prompt wording for SENT, PARA_SENT, and PARA, with one demonstration each are presented in Figure 8 
Large language models effectively leverage document-level context for literary translation, but critical errors persist
In this section, we provide some further details about the human evaluation with a focus on the error annotation. First, discuss the issue of subjectivity in error annotation. Next, we explain some   choices we had to make when annotating "inconsistency" and "format" errors. Then, we discuss the issue of omissions in the produced translations. Finally, we present some details about the translators hired for the evaluation task.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Error annotation: Annotating and classifying errors in translations is inherently subjective (Lommel et al., 2014a;Han, 2020). For instance, translating French "corsage" ("bodice") as a "blouse" can be seen as either a mistranslation or a permissible deviation from the original text; this is, in fact, how the "corsage" was translated by the human translator in our data. Furthermore, sometimes there are multiple ways of annotating errors (Thomson et al., 2023). Consider the following example:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(13) We had to hide the running, though, in case our haste betrayed us, so truer to say we slipped out quietly. When one of my parents appeared, my technique was: pretend to catch sight of someone in the next room. Move in a natural manner toward this figment of my imagination, making a purposeful face.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-ENGLISH SOURCE (from A Children's Bible)
Large language models effectively leverage document-level context for literary translation, but critical errors persist
The translation of the last sentence in (13) into Polish as an imperative can be considered a mis-    Table 11: Details about the translators hired for the current annotation study. We consider a translator bilingual only if they were raised using both languages (e.g., ru-pl translator was raised in Poland while speaking Russian at home). In the broader sense of this word, all of the translators are bilingual with some of them being trilingual. For the cases where the hired translator was not a native speaker of the target language, the annotations were verified by a native speaker of the target language in consultation with the translator. Only one translator reported familiarity with the source text she was asked to evaluate. All translators were asked to evaluate each passage in isolation allowing for all possible interpretations based on the given part of the source text. Three language pairs (pl-ja, en-pl, ja-pl were annotated by the first author of this paper. translation. We would hypothesis that the system misinterpreted the source as an imperative form. However, using the infinitive form of the verb in the translation is less clear and raises questions about whether it is a mistranslation or a grammatical error. The distinction between the two lies in the point at which the mistake was made. If the original sentence was understood correctly but the resulting translation was ungrammatical, then it is a grammatical error. On the other hand, if the use of the infinitive form resulted from interpreting "move" as an infinitive, it may be considered a mistranslation as well.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Inconsistency: For marking the "inconsistency" errors we decided to the take minimal approach. For instance, is the same person is referred to in the translation as both "Piotr" and "Peter" we would mark only the one that is less frequent. If "Piotr" appears once in the paragraph, while "Peter" is used twice, "Piotr" would be annotated as being inconsistent. The same strategy was applied for "register" errors, such as when both polite and ca-sual forms were acceptable, but the translation used them randomly.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Format: We did not label "format" errors for the SENT and PARA_SENT translations, as we manually corrected the quotation marks during post-processing of the translations. This manual correction was done to ensure that SENT and PARA_SENT could be compared to PARA without relying too heavily on simple heuristic (i.e., incorrect usage of the quotation marks).
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Omissions: One thing we ought to discuss is the omission issue. Upon examining translations and annotator feedback, we observe that PARA occasionally omits details, which are crucial to the storyline. Preliminary investigation indicates that PARA translations are more prone to omissions compared to SENT and GTR. Although PARA_SENT appears to mitigate this problem to some extent, it still results in a higher number of omissions than the sentence-level approach while at the same time introducing some repetition issues. Translators: The translators in this study were hired on a freelancing platform, Upwork. All were highly proficient in the source language and most of them were native speakers of the target language. Only one translator reported familiarity with the book, which translation she evaluated. All translators were instructed to evaluate each paragraph in isolation without relying on any prior knowledge about the book. Details about the translators are reported in Table 11. Table 14 shows the results of the pilot study evaluating the effect of pivoting on English on the quality of paragraph-level translations. The evaluation was done by the first author on all 20 passages for every language pair that did not include translation from or into English, as these do not require any pivoting. A total number of 200 pairs was evaluated employing simple preference judgments.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
During the PARA_PIVOT translation process, the model utilized both the source text and its corresponding English translation (text-davinci-003, top-p=1.0, temp=0.3). This approach has the potential to mitigate the limitations associated with pivoting translations, where some information may be lost.  the past tense form of the verb. English does not, so this information is lost and will most likely result in an erroneous translation. Indeed, we notice that adding the source text helps the model to overcome this shortcoming, however, we do not observe a clear gain from using English as a pivot language. Consider the following example:
Large language models effectively leverage document-level context for literary translation, but critical errors persist
(14) Tat'ka mě zase zmlátil. Byl hrozně naštvanej, protože mamkařvala. On mě vždycky mlátí, když onǎ rve. Že prej jsem pořádně neudělala nádobí. Ale já vím, že jsem je udělala dobře. I ten hrnec jsem vydrhla pořádně. A ještě jsem to všechno utřela před koupáním. Ale možná jsem nevyždímala hadr.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-CZECH SOURCE (from Crows) a. Tata mnie znowu zbił. Był wściekły, bo mama krzyczała. On zawsze mnie bije, kiedy ona krzyczy.Że niby nie umyłam dobrze naczyń. Ale ja wiem,że umyłam je dobrze. I garnek też dokładnie wypolerowałam. I jeszcze wszystko wytrzepałam przed kąpielą. Ale może nie wyciśniłam ręcznika.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-GPT-3.5 PARA (POLISH) b. Tata mnie znów uderzył. Był wściekły, bo mama krzyczała. On zawsze mnie bije, kiedy ona krzyczy.Że niby nie umyłam dobrze naczyń. Ale ja wiem,że umyłam je dobrze. Nawet garnek dokładnie wytrzepałam. I jeszcze wszystko przed kąpielą wytrzepałam. Ale może nie wyżągnęłam mopa.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
-  In each instance, the emphasized verbs could potentially be mistranslated when translated through English as the pivot language, as the speaker's gender information would be lost. For instance, the past tense verb "washed" remains unchanged in English regardless of the gender of the speaker, with such details encoded only in the source (Czech) and target (Polish) languages. In this case, all verbs have been translated accurately with respect to grammatical gender, implying that incorporating the source language into the pivot pipeline does indeed improve the translation. However, PARA_PIVOT still selects less suitable verbs (highlighted in red) resulting in slightly more errors in this particular paragraph.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
The only pair where pivoting seems to help is pl-ja. While it is unclear why this happens, it is possible that this outcome is due to the specifics of the Polish novel employed for the translation. Sword of Destiny by Andrzej Sapkowski uses a very distinct language with many archaic expressions. It is possible that translating into English, a language the GPT models were trained on, helps the model deal with these difficult phrases.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Since we do not observe any apparent gains from performing the translation via English as a pivot language (p=0.62, 95% [0.448, 0.591]) and doing so reduces the number of examples one can fit into the prompt, we continue our experiments with a direct translation.
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Correlation with Human Judgements: We investigate the correlation of automatic metrics with human judgements in our evaluation. We consider (1) all the judgments, as well as (2) a subset of all judgments where the annotator stated that they were sure that one translation is clearly better than the other. We compute both accuracy (i.e., the percentage of cases where the metric agrees with human judgment), and a correlation coefficient Kendall's Tau which is defined as follows:           
Hallucinations in Large Multilingual Translation Models
Recent advancements in large-scale multilingual machine translation have brought us closer to realizing a universal translation system: a single model capable of handling numerous languages and translation directions (Aharoni et al., 2019;Arivazhagan et al., 2019;Fan et al., 2020;Zhang et al., 2020;Wenzek et al., 2021;Goyal et al., 2022;NLLB Team et al., 2022). Concurrently, general-purpose large language models (LLMs) have exhibited a remarkable ability to generalize to new tasks, including translation, where they are becoming increasingly stronger (Brown et al., 2020;Chowdhery et al., 2022;Hendy et al., 2023). Compared to traditional bilingual models, these systems can offer significant performance improvements and greatly simplify engineering efforts, as a single model can be used for all language pairs (Arivazhagan et al., 2019). As a result, they are an increasingly attractive choice for realworld applications. However, when deployed in the wild, these models may still generate hallucinations: highly pathological translations that can severely damage user trust and pose serious safety concerns (Perez et al., 2022;Kumar et al., 2022).
Hallucinations in Large Multilingual Translation Models
The problem of hallucinations has long been recognized by researchers (Ji et al., 2022), and recent studies have contributed towards better understanding, detection and mitigation of these pathological translations. However, these studies have been conducted on small bilingual models (<100M parameters) trained on a single English-centric high-resource language pair (Raunak et al., 2021;Ferrando et al., 2022;Guerreiro et al., 2022b,a;Dale et al., 2022;Xu et al., 2023). This leaves a knowledge gap regarding the prevalence and properties of hallucinations in largescale translation models across different translation directions, domains and data conditions.
Hallucinations in Large Multilingual Translation Models
In this work, we aim to fill this gap by investigating hallucinations on two different classes of models. The first and main class in our analysis is the de facto standard approach of massively multilingual supervised models: we use the M2M-100 family of multilingual NMT models (Fan et al., 2020), which includes the largest opensource multilingual NMT model with 12B parameters. The second class is the novel and promising approach of leveraging generative LLMs for translation. Contrary to conventional NMT models, these models are trained on massive amounts of monolingual data in many languages, with a strong bias towards English, and do not require parallel data. In our analysis, we use ChatGPT 1 , a LLM that has been shown to achieve surprisingly high translation quality over a wide range of language pairs (Hendy et al., 2023;Peng et al., 2023).
Hallucinations in Large Multilingual Translation Models
We organize our study by analyzing the two prevalent types of hallucinations in NMT considered in the literature: hallucinations under perturbation and natural hallucinations (Lee et al., 2018;Raunak et al., 2021;Guerreiro et al., 2022b). Firstly, we study hallucinations under perturbation and evaluate whether these translation systems are robust to source-side artificial perturbations. While previous studies have found that these perturbations (e.g., spelling errors and capitalization mistakes) can reliably induce hallucinations (Lee et al., 2018;Raunak et al., 2021), it is not clear whether those conclusions hold for large multilingual models. Secondly, we comprehensively investigate natural hallucinations, and evaluate their prevalence and properties in the outputs of the massively multilingual M2M models on a vast range of conditions, spanning from English-centric to non-English-centric language pairs, translation directions with little supervision, and specialized sensitive domains where hallucinations have devastating impact on user trust (e.g., medical data). Finally, we study a hybrid setup where other translation systems can be requested as fallback systems when an original system hallucinates, with the aim of mitigating hallucinations and improving overall translation quality.
Hallucinations in Large Multilingual Translation Models
Our analysis reveals several key insights on the prevalence and properties of hallucinations, including:
Hallucinations in Large Multilingual Translation Models
• multilingual models predominantly struggle with hallucinations in low-resource language pairs and translating out of English, with hallucination rates well above 10% for some translation directions;
Hallucinations in Large Multilingual Translation Models
• hallucinations in low-resource language pairs can manifest toxic patterns that can be traced back to the training data, posing serious safety issues;
Hallucinations in Large Multilingual Translation Models
• smaller distilled models can mitigate hallucinations by incorporating modeling choices that discourage them, such as leveraging less potent shallow decoders that rely more on the encoder representations, and reducing bias towards higher-resource language pairs through uniform sampling of translation directions during distillation;
Hallucinations in Large Multilingual Translation Models
• ChatGPT produces hallucinations that are qualitatively different from those of conventional translation models, mostly consisting of off-target translations, overgeneration, and even failed attempts to translate;
Hallucinations in Large Multilingual Translation Models
• hallucinations are sticky and hard to reverse with models that share the same training data and architecture, whereas employing more diverse models as fallback systems can substantially improve overall translation quality and eliminate pathologies like oscillatory hallucinations.
Hallucinations in Large Multilingual Translation Models
We release all our code and make available over a million translations in more than 100 translation directions to spur future research. 2 2 Background
Hallucinations in Large Multilingual Translation Models
Massively multilingual neural machine translation has recently emerged as a powerful paradigm for building machine translation systems that can handle numerous languages (Akhbardeh et al., 2021;Wenzek et al., 2021;NLLB Team et al., 2022;Siddhant et al., 2022;Bapna et al., 2022;Chowdhery et al., 2022). These systems aim to translate directly with a single model for multiple language pairs without relying on any pivot language.
Hallucinations in Large Multilingual Translation Models
The dominant strategy for achieving these systems is to train large multilingual models on vast amounts of parallel data often obtained through a combination of data mining and data augmentation strategies, such as backtranslation (Sennrich et al., 2016;Edunov et al., 2018). Compared to classic bilingual models, the multilinguality of these systems results in significant improvements, particularly for low-resource and non-Englishcentric language pairs, as these benefit the most from multilingual transfer (Arivazhagan et al., 2019;Fan et al., 2020).
Hallucinations in Large Multilingual Translation Models
As an alternative, a novel and promising strategy is to leverage the emergent capabilities of large language models (LLMs). These systems are pretrained on massive nonparallel corpora and can be prompted to solve arbitrary tasks (Radford et al., 2019;Brown et al., 2020). In fact, this approach has led to impressive results across a wide variety of NLP tasks (Chowdhery et al., 2022;Zhang et al., 2022). Translation is no exception: LLMs can produce fluent and adequate translations, especially for high-resource English-centric language pairs, that are competitive with those of dedicated supervised translation models (Vilar et al., 2022;Peng et al., 2023;Garcia et al., 2023;Hendy et al., 2023;Bawden and Yvon, 2023).
Hallucinations in Large Multilingual Translation Models
Hallucinations lie at the extreme end of translation pathologies and present a critical challenge in machine translation, as they can severely compromise the safety and reliability of real-world applications.
Hallucinations in Large Multilingual Translation Models
Importantly, hallucinations in machine translation are unlike hallucinations in other natural language generation tasks (e.g., abstractive summarization and generative question answering) (Ji et al., 2022). While, for these other tasks, models often produce hallucinated outputs (Falke et al., 2019;Cao et al., 2022;Manakul et al., 2023), hallucinations in machine translation, possibly attributed to the more closed-ended nature of the task, are substantially rarer and hard to observe in clean, unperturbed data. This has led several previous studies to examine their properties by creating artificial scenarios where hallucinations are more likely to occur (e.g., introducing perturbations in the source text (Lee et al., 2018) or noise in the training data (Raunak et al., 2021)). To distinguish these two scenarios, hallucinations in machine translation are categorized into two types (Raunak et al., 2021): hallucinations under perturbation and natural hallucinations.
Hallucinations in Large Multilingual Translation Models
Hallucinations under perturbation. A model generates a hallucination under perturbation when it produces a significantly lower quality translation for a slightly perturbed input compared to the original input (Lee et al., 2018). Hallucinations under perturbation explicitly reveal the lack of robustness of translation systems to perturbations in the source text (e.g., misspellings or capitalization errors) by finding translations that undergo significant negative shifts in quality due to these changes.
Hallucinations in Large Multilingual Translation Models
Natural hallucinations. Contrary to hallucinations under perturbations, these translations occur naturally without any perturbation. As a re-sult, natural hallucinations are rare and challenging to study. In this work, we follow the taxonomy introduced in Raunak et al. (2021) and later extended in Guerreiro et al. (2022b). Under this taxonomy, hallucinations are translations that contain content that is detached from the source text. To distinguish between different types of hallucinations, they can be categorized as largely fluent detached hallucinations or oscillatory hallucinations. The former refers to translations that bear minimal or no relation at all to the source, while the latter refers to inadequate translations that contain erroneous repetitions of words and phrases.
Hallucinations in Large Multilingual Translation Models
In this section, we provide an overview of the models, datasets and evaluation metrics used throughout our study.
Hallucinations in Large Multilingual Translation Models
We focus on two classes of models: (i) conventional supervised multilingual NMT models, and (ii) LLMs that can be prompted for translation.
Hallucinations in Large Multilingual Translation Models
For the supervised multilingual NMT models, we use the transformer-based (Vaswani et al., 2017) M2M-100 family of models (Fan et al., 2020), which consists of three variants with different sizes: M2M (S) with 418M parameters, M2M (M) with 1.2B parameters, and M2M (L) -the largest available open-source multilingual NMT model -with 12B parameters. These models were trained on a many-to-many parallel dataset comprising 7.5B sentences crawled from the web, and support 100 languages and thousands of translation directions. We also experiment with SMaLL100 (Mohammadshahi et al., 2022), a shallow multilingual NMT model with 330M parameters obtained via distillation of M2M (L). Unlike the M2M models, SMaLL100 was trained on a much smaller training set with uniform sampling across all language pairs to reduce the bias towards high-resource languages: only 100k parallel sentences from the original M2M training data were used for each translation direction, for a total of 456M parallel sentences. For decoding, we run beam search with a beam size of 4. All experiments were run on fairseq (Ott et al., 2019).
Hallucinations in Large Multilingual Translation Models
As for the alternative strategy using LLMs, we use ChatGPT (gpt-3.5-turbo) 3 , a vari-ant of GPT3.5 -a GPT-family (Radford and Narasimhan, 2018;Radford et al., 2019;Brown et al., 2020) large-scale model with 175B parameters -that has been fine-tuned with human feedback in the style of InstructGPT (Ouyang et al., 2022). ChatGPT has been shown to achieve impressive results for multiple multilingual NLP tasks, including translation (Kocmi and Federmann, 2023;Lu et al., 2023;Fu et al., 2023;Hendy et al., 2023;Peng et al., 2023). To generate translations, we use the zero-shot prompt template used in Hendy et al. (2023) and keep the generation parameters as the default API parameters. 4
Hallucinations in Large Multilingual Translation Models
We carefully selected datasets based on two main criteria: their familiarity to researchers and practitioners, and the avoidance of train/test overlap for the M2M models. 5 To this end, we chose to use premier translation benchmarks: FLORES-101 (Goyal et al., 2022), WMT and TICO (Anastasopoulos et al., 2020). FLORES-101 is a high-quality multi-parallel dataset that consists of Wikipedia text in 101 languages and allows for the assessment of hallucinations across a vast range of translation directions; we join the dev and devtest subsets for evaluation. For WMT, we used the same benchmarks as those used in the original M2M paper evaluation suite, as these were explicitly removed from the training data. Additionally, we selected recent WMT test sets from the WMT21 and WMT22 campaigns as they were released after the models were trained. In contrast to these general-purpose datasets, TICO is a specialized medical-domain multilingual benchmark that includes COVID-19 related data, such as medical papers and news articles; we join the dev and test sets. Full details about the datasets can be found in Appendix B.
Hallucinations in Large Multilingual Translation Models
Throughout our work, we focus mainly on sentence-level evaluation. Our main lexical met-ric is spBLEU (Goyal et al., 2022), 6 as it has been widely employed in works on massively multilingual translation (Fan et al., 2020;Wenzek et al., 2021;Mohammadshahi et al., 2022;NLLB Team et al., 2022) and offers fairer evaluation for lowresource languages compared to BLEU (Papineni et al., 2002). Moreover, we follow the most recent MT metrics shared-task recommendations (Freitag et al., 2022) and also adopt neural metrics. We use the latest reference-based and reference-free COMET variants: COMET-22 (Rei et al., 2022a) and CometKiwi (Rei et al., 2022b). Lastly, we use the cross-lingual encoder LaBSE (Feng et al., 2022) to obtain sentence similarity scores, as these have been successfully employed in prior research on detection of natural hallucinations (Guerreiro et al., 2022a;Dale et al., 2022).
Hallucinations in Large Multilingual Translation Models
We start our analysis by focusing on artificially created hallucinations. We first provide an overview of our experimental setting, focusing on the construction of the perturbed data and detection approach. Then, we present our results and analyze the properties of these hallucinations across different resource levels and models.
Hallucinations in Large Multilingual Translation Models
Perturbations. To construct the perturbed source sequences, we apply the same minimal perturbations used in Xu et al. (2023): misspeling of words, insertion of frequent tokens in the beginning of the source sequence, and capitalization errors. For full details on the construction of the perturbed data, refer to Appendix C.1.
Hallucinations in Large Multilingual Translation Models
Translation directions. We use the FLORES dataset for these experiments, and focus specifically on translation out of English. We selected all bridge languages 7 , as well as additional lowresource languages that were underrepresented among bridge languages. Overall, we generate translations for 31 different language pairs (LPs). 6 We use spBLEU as implemented in SACREBLEU (Post, 2018): nrefs:1|case:mixed|eff:yes|tok:flores101| smooth:exp|version:2.3.1. 7 In the M2M paper (Fan et al., 2020), a bridge language is defined as a language that connects languages across language groupings (e.g., Romance, Slavic languages) and is mined against all other bridge languages. It is usually one of the most resourced languages within a language grouping and its purpose is to reduce the number of bitext pairs while preserving translation directions of practical interest.  Table 1: Fraction of languages for which models produces at least one hallucination under perturbation, and average hallucination rate (and median, in subscript) across all languages at each resource level. We present the language pairs and more details on our choice of languages in Appendix C.1.
Hallucinations in Large Multilingual Translation Models
Detection. Our detection approach is inspired by that of previous works on hallucinations under perturbation (Lee et al., 2018;Raunak et al., 2021;Ferrando et al., 2022;Xu et al., 2023). The algorithm is a simple 2-rule process: we fix (i) a minimum threshold quality score for the original translations, and (ii) an extremely low maximum quality score for the perturbed translations. A model generates a hallucination under perturbation when both translations meet the thresholds. Crucially, rule (i) ensures that low-quality translations for unperturbed sources are not considered as candidates for hallucinations under perturbation. 8 We extend this algorithm to handle multiple models and language pairs by adapting rule (i). We first obtain source sentences for which all models produce translations that meet a minimum quality threshold (spBLEU > 9). Then, we sort them according to average quality across the different models, and select the top 20% as candidates. Finally, we apply rule (ii) and set the threshold to spBLEU < 3. We selected both thresholds based on the choices made in previous works (Raunak et al., 2021;Ferrando et al., 2022;Xu et al., 2023).
Hallucinations in Large Multilingual Translation Models
This approach ensures a fixed sample size across different language pairs, and that the sentences analyzed for each language pair are consistent across all models. Moreover, it allows us to effectively detect hallucinations under perturbation across multiple models in a multilingual scenario in a scalable manner, while accounting for the unique quality trends observed across different models and languages. 9
Hallucinations in Large Multilingual Translation Models
We show aggregated results in Table 1 and language-specific results in Figure 1. Overall, they reveal that perturbations have the potential to trigger hallucinations under perturbation, even in larger models. In what follows, we highlight several noteworthy trends found in our results.
Hallucinations in Large Multilingual Translation Models
Average hallucination rates generally decrease with increasing resource levels. Table 1 shows that all models, with the exception of ChatGPT that we analyze separately below, exhibit lower hallucination rates as resource levels increase. This is expected and suggests that models are better equipped to handle source-side perturbations for language pairs with more parallel data during training. In fact, hallucinations under perturbation for high-resource languages are almost non-existent. However, Figure 1 reveals variability across languages, and even within the models in the same family that have been trained on the same data. For instance, when translating to Asturian (ast), M2M (L) and its distilled version SMaLL100 have significantly higher hallucination rates than the smaller M2M (S). Thus, hallucinations under perturbation may emerge in other nontrivial ways unrelated to the training data.
Hallucinations in Large Multilingual Translation Models
SMaLL100 exhibits lower hallucination rates than its teacher model M2M (L). Recall that SMaLL100 was trained using uniform sampling across all language pairs to prevent bias towards higher resourced language pairs. The results in Table 1 may reflect one positive outcome from such approach: despite being much smaller than M2M (L), SMaLL100 hallucinates less and for fewer languages than its teacher model for lowand mid-resource language pairs.
Hallucinations in Large Multilingual Translation Models
Hallucinations under perturbation are not correlated with the quality of original translations. The common approach for detection of hallucinations under perturbation (see Section 4.1) raises an interesting question: are the original source sentences for which models produce higher quality translations less likely to lead to hallucinations when perturbed? Our analysis found a very weak correlation (according to Pearson correlation; see Appendix C.2) between hallucinations under perturbation and spBLEU scores for the original unperturbed sources across all models. This indicates that even minimal perturbations in the source text can cause models to undergo significant shifts in translation quality.
Hallucinations in Large Multilingual Translation Models
ChatGPT exhibits different hallucination patterns from conventional translation models. Table 1 shows that, contrary to traditional models, ChatGPT generates more hallucinations for mid-resource languages than for low-resource languages. In fact, it surprisingly produces fewer hallucinations for low-resource languages than any other model. Moreover, ChatGPT's hallucinations are qualitatively different from those of other mod-els: they often consist of off-target translations, 10 overgeneration, or even failed attempts to translate (e.g., "This is an English sentence, so there is no way to translate it to Vietnamese"; we provide further examples in Appendix C.2). Furthermore, unlike traditional NMT models that frequently produce oscillatory character hallucinations, ChatGPT does not generate any such hallucinations under perturbation. This is further evidence that translation errors, even severely critical ones, obtained via prompting a LLM are different from those produced by traditional machine translation models (Vilar et al., 2022;Garcia et al., 2023;Hendy et al., 2023;Bawden and Yvon, 2023).
Hallucinations in Large Multilingual Translation Models
Interestingly, we also found that the vast majority of the hallucinations can be reversed with further sampling from the model. 11 This connects to findings in Guerreiro et al. (2022b);Manakul et al. (2023): as with traditional NMT models, hallucinations with a LLM may not necessarily indicate model defect or incapacity to generate adequate translations, and may just result from "bad luck" during generation.
Hallucinations in Large Multilingual Translation Models
Let us now turn to investigating natural hallucinations. 12 We first provide an in-depth overview of our evaluation setting, focusing on the scenarios and detection methodology. Subsequently, we present a thorough analysis, exploring diverse properties of natural hallucinations such as their different types, the influence of translation direction, and prevalence of toxicity.
Hallucinations in Large Multilingual Translation Models
Evaluation scenarios. Analyzing massively multilingual translation models opens up several research scenarios that have not been studied in previous works that focused solely on bilingual models. We will take advantage of this opportunity and investigate natural hallucinations in three different evaluation scenarios, studying more than 100 translation directions in the main text alone.
Hallucinations in Large Multilingual Translation Models
We start with an English-centric scenario where we pair 32 different languages with English for a total of 64 translation directions. Then, we study a non-English-centric scenario inspired by Fan et al. (2020), where we explore 25 language pairs corresponding to real-world use cases of translation not involving English (e.g., translating Greek directly to Turkish). Finally, we assess the prevalence of hallucinations on sensitive medical data where they can have a devastating impact on user trust. We pair 9 different languages with English for a total of 18 directions. We present all the translation directions investigated in these setups in Appendix D.1. We report results for the first two setups using the FLORES dataset in the main text and WMT in Appendix D.2. For the final setup, we use the medical-domain TICO dataset.
Hallucinations in Large Multilingual Translation Models
Detection. We integrate key findings from recent research on detection of hallucinations and focus on two main detectors: ALTI+ (Ferrando et al., 2022) for detached hallucinations, and top n-gram (TNG) (Raunak et al., 2021(Raunak et al., , 2022Guerreiro et al., 2022b) for oscillatory hallucinations.
Hallucinations in Large Multilingual Translation Models
ALTI+ evaluates the relative contributions of both source and target prefixes to model predictions. As hallucinations are translations detached from the source sequence, ALTI+ can effectively detect them by identifying sentences with minimal source contribution. Notably, it faithfully reflects model behavior and explicitly signals model detachment from the source text in any translation direction (Ferrando et al., 2022). In previous works, this method has been successfully employed to detect hallucinated toxicity in a multilingual context in NLLB Team et al. (2022), and it has been validated on human-annotated hallucinations in Dale et al. (2022), where it was demonstrated that ALTI+ scores easily separate detached hallucinations from other translations. 13 TNG, on the other hand, is a straightforward, lightweight black-box heuristic targeting oscillatory hallucinations. It works by comparing the count of the top repeated translation n-gram to the 13 We followed the recommendations in Guerreiro et al. (2022b) and set model-based ALTI+ thresholds based on validation data where the models are expected to perform well. Specifically, we obtained the lowest 0.02% -in line with natural hallucination rates reported in the literature (Raunak et al., 2022) -of the ALTI+ score distributions for highresource WMT benchmarks. Additionally, to ensure further trustworthy, high-precision measurements, we excluded detected candidates with LaBSE or CometKiwi scores -as these have been also been validated for detection of humanannotated detached hallucinations (Dale et al., 2022; Guerreiro et al., 2022a) -exceeding the top 10% of scores on translations from the same WMT benchmarks. count of the top repeated source n-gram, ensuring the difference is at least t. This approach has been validated on human-annotated hallucinations and found to identify oscillatory hallucinations with perfect precision (Guerreiro et al., 2022b). We follow previous work by using n = 4 and t = 2 (Raunak et al., 2021;Guerreiro et al., 2022b) and excluding translations that meet the reasonable quality threshold outlined in Section 4.1. 14 Remark on Model Selection. We rely on ALTI+, a model-based detector, for reliable detection of detached hallucinations. Since we lack access to glass-box internal features from ChatGPT, we exclude it from our model selection to ensure consistency in our analysis. It is important to note that using alternative detectors could lead to misleading results and create discrepancies between the evaluation scenarios for ChatGPT and other models. Nonetheless, we will further examine ChatGPT in Section 6, exploring various aspects such as the generation of oscillatory hallucinations and translation quality in scenarios where other models produce hallucinations.
Hallucinations in Large Multilingual Translation Models
We start by investigating natural hallucinations on English-centric language pairs. We reveal key insights on how properties of hallucinations change across resource levels, models and translation directions. We present language-pair specific results in Appendix D.2.
Hallucinations in Large Multilingual Translation Models
Hallucinations in low-resource language pairs are not only more frequent, but also distinct. Table 2 shows that hallucinations occur frequently for low-resource directions, with all M2M models exhibiting average hallucination rates exceeding 10%. Furthermore, all models generate hallucinations for the vast majority of low-resource language pairs. On what comes to the type of hallucinations, Figure 2 demonstrates that, in contrast to mid-and high-resource language pairs, oscillatory hallucinations are less prevalent, while detached hallucinations occur more frequently in low-resource languages. This reveals that models tend to rely less on the source context when translating to or from low-resource languages. Importantly, although massive multilingual models have significantly improved translation quality for low-   resource languages, these findings not only suggest that there is considerable room for improvement, but also highlight potential safety concerns arising from translations in these directions.
Hallucinations in Large Multilingual Translation Models
SMaLL100 consistently relies more on the source text than other models. Despite having the smallest number of parameters, SMaLL100 shows remarkable hallucination rates across low-and mid-resource language pairs, hallucinating significantly less than its larger counterparts in lowresource settings. These improved rates may be attributed not only to the uniform sampling of language pairs during training, but also to architectural decisions. While SMaLL100 shares a 12layer encoder with the other models to process source representations, it diverges by employing a shallow 3-layer decoder-instead of a 12-layer decoder-and placing the target language code on the encoder side. We hypothesize that this de-sign encourages greater reliance on the more complex encoder representations, reducing the likelihood of detachment from the source. In fact, distinct patterns in ALTI+ scores (shown in Appendix D.2) support this hypothesis: SMaLL100 consistently demonstrates higher source contributions and similar patterns across all resource levels. In contrast, M2M models show a greater tendency to rely less on the source, especially in low-resource language pairs. Importantly, however, SMaLL100's reduced hallucination rates do not necessarily imply superior translation quality compared to the other M2M models: we observed a strong correlation between M2M models' corpus-level COMET-22 scores and their respective hallucination rates for low-resource languages, whereas, contrastingly, for SMaLL100 the correlation is weak. This indicates that despite detaching less from the source content, SMaLL100's translations are not necessarily of higher quality to those of other M2M models. This and other statistics can be found in the Appendix D.2.
Hallucinations in Large Multilingual Translation Models
Scaling up models within the same model family leads to reduced hallucination rates. As shown in  Table 4: Fraction of LPs on the non-English-centric setup for which models produce at least one hallucination, and average hallucination rate (and median, in subscript) across all LPs at each resource level.
Hallucinations in Large Multilingual Translation Models
lucinations and hallucinating for fewer languages than all other models.
Hallucinations in Large Multilingual Translation Models
Hallucinations are more frequent when translating out of English. Table 3 demonstrates that models are significantly more prone to hallucinate when translating out of English. In fact, in line with the observations of Ferrando et al. (2022), we found that models tend to detach more from the source text when translating out of English. This is evidenced by ALTI+ source contributions being lower across all language pairs in this direction compared to translating into English. Interestingly, we discovered that the translation direction can also influence the properties of hallucinations: (i) over 90% of off-target hallucinations occur when translating out of English, and (ii) nearly all hallucinations into English for mid-and highresource language pairs are oscillatory.
Hallucinations in Large Multilingual Translation Models
Toxic hallucinations pose substantial safety risks. Toxic text in translations can emerge in the form of hallucinations (NLLB Team et al., 2022). To assess the prevalence of toxic text in detected hallucinations, we utilized the toxicity wordlists provided by NLLB Team et al. (2022). We found that toxic text primarily appears in translations out of English and almost exclusively affects low-resource language pairs. For instance, over 1 in 8 hallucinations in Tamil contain toxic text. Interestingly, these toxic hallucinations not only exhibit high lexical overlap among them, but are repeated across models for multiple unique source sentences. Moreover, they are not necessarily reduced by scaling up the model size. These observations suggest that these hallucinations are likely to be traced back to toxic patterns in the training data, 15 aligning with observations in Rau-nak et al. (2021); Guerreiro et al. (2022b). Moreover, we also found that these hallucinations can be propagated through model distillation, as evidenced by SMaLL100 generating toxic hallucinations that are copies of those of its teacher model. This underlines the necessity of rigorously filtering training data to ensure safe and responsible use of these models in real-world applications.
Hallucinations in Large Multilingual Translation Models
We shift our focus to translation directions that do not involve English, typically corresponding to directions with less supervision during training. We present language-pair specific results in Appendix D.3.
Hallucinations in Large Multilingual Translation Models
Trends are largely similar to English-centric directions. Table 4 reveals trends that largely mirror those observed in the English-centric setup: 16 (i) hallucinations are more frequent in low-resource settings; (ii) SMaLL100 significantly outperforms the M2M models in low-resource language pairs; and (iii) scaling up to M2M (L) consistently yields substantial improvements over the smaller M2M models in low-and mid-resource directions. Additionally, the trends related to hallucination types also hold across the two setups: detached hallucinations are more prevalent in low-resource settings, while oscillatory hallucinations overwhelmingly dominate in mid-and high-resource directions (see Appendix D.3).
Hallucinations in Large Multilingual Translation Models
Less supervised language pairs exhibit extremely high hallucination rates. As expected, models struggle more with hallucinations for directions with less or even no supervision during training, such as ro-hy and af-zu. For instance,  M2M (M) hallucinates for nearly half of the translations in these directions.
Hallucinations in Large Multilingual Translation Models
We now turn to investigating hallucinations in data from the medical domain, where they can have devastating consequences. Using the TICO dataset, we compare hallucination rates with the FLORES dataset for 18 translation directions. We present language-pair specific results in Appendix D.4.
Hallucinations in Large Multilingual Translation Models
Hallucinations are not exacerbated under medical domain data. Table 5 reveals that hallucination rates for the TICO medical data do not consistently exceed those observed for the FLORES Wikipedia data. This finding diverges from previous works that investigated hallucinations for specialized domain data (Wang and Sennrich, 2020;Müller et al., 2020). We hypothesize that, in contrast with the smaller models typically trained on limited datasets from a single domain used in those works, the concept of "domain shift" may not be as pronounced for M2M models. These models are not only much larger but, crucially, they are trained on a dataset containing over 7 billion parallel sentences gathered from the web, which encompasses a broad array of domains. This massive training set potentially mitigates the impact of domain shift and, consequently, reduces its influence on hallucinations.
Hallucinations in Large Multilingual Translation Models
Building upon our analysis on natural hallucinations in the previous section, we now explore the potential of reducing hallucinations and enhancing overall translation quality by employing a simple hybrid setup that can take advantage of multiple systems with possible complementary strengths. Put simply, we leverage an alternative system as a fallback when the primary original model produces hallucinations. Our analysis in the main text is focused on the more extensive English-centric setup. We provide results on the non-Englishcentric setup in Appendix E.
Hallucinations in Large Multilingual Translation Models
6.1 Employing models of the same family as fallback systems
Hallucinations in Large Multilingual Translation Models
We begin by analyzing the performance of samefamily models when employed as fallback systems for one another (e.g., using SMaLL100, M2M (M), and M2M (L) as fallbacks for M2M (S)). 17
Hallucinations in Large Multilingual Translation Models
Detached hallucinations are particularly sticky across M2M models. Figure 3 reveals that when employing M2M models as fallback systems, reversal rates-percentage of hallucinations from the original system that are corrected by the fallback system-are consistently higher for oscillatory hallucinations than for detached hallucinations. These findings not only corroborate those in Guerreiro et al. (2022b), where oscillatory hallucinations were found to be less related to model  defects, but also further emphasize the close connection between detached hallucinations and training data. This connection can help explain their stickiness: since the M2M models share the same training data, reversing these hallucinations using other same-family models as fallbacks is more challenging. Interestingly, we also observe that M2M (L) particularly struggles to reverse the detached hallucinations generated by its distilled counterpart SMaLL100, suggesting that model defects can persist and be shared during distillation.
Hallucinations in Large Multilingual Translation Models
Scaling up within the model family is not an effective strategy for mitigating hallucinations.
Hallucinations in Large Multilingual Translation Models
In line with our analysis in Section 5.2, Figure 3 shows that reversal rates using SMaLL100 as a fallback system are higher for detached hallucinations than for oscillatory hallucinations. In fact, although SMaLL100 is a distilled M2M model, its training data, training procedure, and architecture differ from those of the M2M models. This distinction may make it more complementary as a fallback system to other M2M models than simply scaling up within the same model family. This suggests that merely increasing the scale of models within the same family is not an effective strategy for mitigating hallucinations, and exploring alternative models with different architectures and trained on different data could yield more substantial improvements. In the next section, we will an-alyze this alternative strategy.
Hallucinations in Large Multilingual Translation Models
Motivated by the findings from the previous section, we will now study how models that are not from the M2M family can be employed to further mitigate hallucinations and improve translation quality. We will test this approach with two different models: (i) we will prompt ChatGPT as detailed in Section 3, and (ii) we will use a highquality 3.3B parameter model from the NLLB family of multilingual NMT models (NLLB) proposed in NLLB Team et al. (2022).
Hallucinations in Large Multilingual Translation Models
Translation quality can be significantly improved with external fallback systems. Figure 4a demonstrates that external fallback systems, particularly NLLB, can significantly enhance translation quality of originally hallucinated translations compared to same-family models. 18 This improvement is especially notable for low-resource languages, where both ChatGPT and NLLB consistently boost translation quality. Remarkably, NLLB generally outperforms ChatGPT as a fallback system for low-and mid-resource languages, aligning with the findings in Hendy et al. (2023), which revealed that GPT models have limited capabilities for lower-resourced languages and lag behind dedicated translation models in those settings. Nonetheless, ChatGPT still surpasses dedicated M2M translation systems in these resource levels when used as a fallback system, underscoring the limitations of relying on same-family models as fallback systems.
Hallucinations in Large Multilingual Translation Models
Oscillatory hallucinations are practically nonexistent when ChatGPT is the fallback system. From Figure 4b, we see another benefit of employing external fallback systems: oscillatory hallucinations are almost entirely eliminated. Interestingly, consistent with our findings in Section 4, we observe that ChatGPT produces very few, if any, oscillations, slightly improving the rates obtained with NLLB. This provides further evidence that, although hallucinations obtained via prompting LLMs may still occur, they exhibit different properties and surface forms. Investigating and understanding these differences in hallucination properties presents an interesting research path for future work.
Hallucinations in Large Multilingual Translation Models
We have comprehensively investigated the phenomenon of hallucinations in massively multilingual translation models. By departing from the settings studied in previous work that focused on bilingual models trained on high-resource language pairs, we were able to explore a wide range of research scenarios that remained overlooked. Our analysis revealed several key insights on the prevalence and properties of hallucinations across various models of different scale, translation directions, and data conditions, including: the prevalence of hallucinations across multiple translation directions across different resource levels and beyond English-centric translation; the emergence of toxicity in hallucinations; and the effect of scaling up within the same model family on the prevalence of hallucinations. Additionally, we explored how fallback systems can mitigate hallucinations and improve overall translation quality. We found that hallucinations can be sticky and difficult to reverse when using models that share the same training data and architecture. However, by leveraging other external models, we can significantly improve translation performance and virtually eliminate pathologies such as oscillatory hallucinations.
Hallucinations in Large Multilingual Translation Models
To support future research on this topic, we are open-sourcing our code and releasing over a million translations and detection results across several models and language pairs.
Hallucinations in Large Multilingual Translation Models
Our study mainly focuses on the M2M family of multilingual models. We chose this family of models as it includes several models at different sizes and the largest open-source multilingual NMT model. It is unclear how our findings generalize to other families of multilingual models (e.g., the NLLB family of models).
Hallucinations in Large Multilingual Translation Models
Our detection approaches inherit the limitations that carry over with the metrics that are leveraged in them. For instance, following all of previous work, we adopt a BLEU metric to detect hallucinations under perturbation. However, this and other lexical metrics ranked worst than referencebased neural metrics in last year's WMT22 Metrics Shared Task (Freitag et al., 2022).
Hallucinations in Large Multilingual Translation Models
We analyzed ChatGPT as it has demonstrated impressive capabilities for translation and other multilingual tasks, such as MT evaluation. Unfortunately, the model remains behind API walls and documentation is scarce. As such, we could not ensure that ChatGPT was not trained on our evaluation sets, nor could we evaluate the contribution of the source text to ChatGPT's translations, which would have enabled detection of detached hallucinations. Despite these limitations, we believe our findings provide relevant insights into the properties of translations generated by the model. B Dataset Statistics FLORES. The FLORES benchmark (Goyal et al., 2022) consists of a multi-parallel dataset in 101 languages with sentences extracted from Wikipedia. The sentences in the dataset are translated from original English sentences. We join the dev and devtest splits for a total of 2009 parallel sentences for each translation direction.
Hallucinations in Large Multilingual Translation Models
TICO. The TICO benchmark (Anastasopoulos et al., 2020) is a multilingual dataset specialized in the medical domain obtained by combining English open-source data from various sources, such as scientific articles, government health announcements, and others. We join the dev and test set for a total of 2100 parallel sentences for each translation direction.
Hallucinations in Large Multilingual Translation Models
WMT benchmarks. We selected the same English-centric WMT benchmarks that were used as evaluation sets in the original M2M paper (Fan et al., 2020). Additionally, we selected recent WMT test sets from the WMT21 and WMT22 campaigns as they were released after the models were trained. A summary of the datasets used for each translation direction can be found in Table 7.  Perturbations. We randomly misspell words by changing the characters with a probability of 0.01; 20 insert a token chosen randomly from the 50 most frequent tokens or punctuation in the test set; and, randomly title-case words with a probability of 0.1, guaranteeing that at least one word's capitalization is perturbed.
Hallucinations in Large Multilingual Translation Models
Languages. We pair the following languages with English: af, ar, ast, bn, cs, cy, de, el, es, fa, fi, fr, he, hi, hr, hu, id, ja, ko, lt, nl, oc, pl, pt, ru, sv, sw, tl, tr, vi, zh. We selected all bridge languages (see Table 6) and additional low-resource languages, as they are underrepresented among the bridge languages. Although Tamil is a bridge language, we did not include it in our analysis, as we could not find enough candidates with reasonable quality for hallucinations under perturbation after applying rule (i) of the detection approach (see Section 4.1).
Hallucinations in Large Multilingual Translation Models
Correlation between original translation quality and hallucination under perturbation. We present the statistics in Table 8. As mentioned in the main text, the Pearson correlation between the original translation quality and the detection outputs of hallucinations under perturbation is very weak across all models.  Discussion on hallucinations under perturbation and natural hallucinations. As we have mentioned in the main text, hallucinations under perturbation, in contrast to natural hallucinations, explicitly reveal the lack of robustness of translation systems. Put simply, hallucinations under perturbation are cases where the model is not robust and, as a result, undergoes significant negative shifts in translation quality due to perturbations. This means that they do not necessarily entail detachment from the source. Hallucinations under perturbation have been studied as such in previous work, and detection of these translations with the 2-step method that we adopted in our work (see Section 4.1) has been consistently used in all of previous research. We decided to keep the same designations, definitions and detection approach so as to be consistent with previous work. Nevertheless, we assess here whether the detected hallucinations under perturbation are, in fact, detached or oscillatory hallucinations. We found that more than 85% of hallucinations under perturbation would be detected with our detection approach for natural hallucinations. Inspection of the non-detected translations revealed that they usually contain critical mistranslation errors, but do not necessarily exhibit detachment from the source content (e.g., off-target translations or copies of the source).
Hallucinations in Large Multilingual Translation Models
We provide examples of hallucinations under perturbation in Figure 5. 
Hallucinations in Large Multilingual Translation Models
Hereinafter, we will use the terms natural hallucinations and hallucinations interchangeably.
Hallucinations in Large Multilingual Translation Models
English-Centric setup. We pair the following languages with English: ar, ast, az, bn, cs, cy, de, el, es, fa, fi, fr, he, hi, hr, hu, id, ja, ko, lt, nl, oc, pl, ps, pt, ru, sv, sw, ta, tr, vi, zh. Similarly to the choice for the experiments on hallucinations under perturbation, we selected all the bridge languages and additional low-resource languages.
Hallucinations in Large Multilingual Translation Models
Non-English-Centric setup. We analyzed the following translation directions: hi-bn, it-fr, de-hu, it-de, cs-sk, nl-fr, fr-sw, ro-ru, ro-uk, de-hr, hr-sr, be-ru, hr-hu, hr-cs, el-tr, hr-sk, nl-de, af-zu, ro-hu, hi-mr, ro-tr, uk-ru, ro-hy, ar-fr, ro-de. We selected language pairs that were used in the analysis from the original M2M paper on real-world settings for many-to-many translation (Fan et al., 2020). They represent translation directions that are commonly used in countries that have official and regional languages that do not include English.
Hallucinations in Large Multilingual Translation Models
Specialized Domains setup. We pair the following languages with English: ar, fr, hi, id, ps, pt, ru, sw, zh. We selected languages from the TICO benchmark that are supported by the M2M models.
Hallucinations in Large Multilingual Translation Models
Evaluation scores. We present spBLEU and COMET-22 corpus-level scores for the FLORES setup in Table 10.
Hallucinations in Large Multilingual Translation Models
ALTI+ scores. We present the distribution of ALTI+ scores for all translation directions in Figure 6. For the M2M models, we find the same trends presented in Ferrando et al. (2022): smaller source contributions when translating out of English and for low-resource language pairs. Remarkably, it is also clear from the distributions that ALTI+ scores for SMaLL100 are not only significantly higher than those of the M2M models, but they are also more consistent across different resource levels.
Hallucinations in Large Multilingual Translation Models
Correlation between COMET-22 scores and hallucination rates. We present the statistics in Table 9.
Hallucinations in Large Multilingual Translation Models
As we have remarked in the main text, SMaLL100's reduced hallucination rates for low-resource language pairs do not necessarily equate to superior translation quality compared to the other M2M models. We observed a strong correlation between M2M models' corpus-level COMET-22 scores and their respective hallucination rates for low-resource languages, whereas, contrastingly, for SMaLL100 the correlation is weak. This indicates that despite detaching less from the source content, SMaLL100's translations are not necessarily of higher quality to those of other M2M models. Interestingly, the correlations for the M2M models are significantly weakened on high-resource language pairs, mirroring observations from previous work (Lee et al., 2018).    Hallucination rates for all language pairs. We present a heatmap with hallucination rates for each model for all translation directions in Figure 7.   Hallucination rates for all language pairs. We present aggregated results in Table 12 and a heatmap with hallucination rates for each model for all translation directions in Figure 8. 21 Importantly, the trends regarding the prevalence of hallucinations analyzed in-depth in Section 5 hold for the WMT language pairs as well.   Prevalence of oscillatory hallucinations. We present a heatmap with the relative prevalence of hallucinations detected only by TNG (oscillatory hallucinations) in Figure 9. 
Hallucinations in Large Multilingual Translation Models
Evaluation scores. We present spBLEU and COMET-22 corpus-level scores in Table 13.
Hallucinations in Large Multilingual Translation Models
Hallucination rates for all language pairs. We present a heatmap with hallucination rates for each model for all translation directions in Figure 10.
Hallucinations in Large Multilingual Translation Models
Prevalence of oscillatory hallucinations. We present a heatmap with the relative prevalence of hallucinations detected only by TNG (oscillatory hallucinations) in Figure 11. The trends follow those presented in the analysis in the main text (see Section 5.2).      Hallucination rates for all language pairs. We present a heatmap with hallucination rates for each model for all translation directions in Figure 12. Prevalence of oscillatory hallucinations. We present a heatmap with the relative prevalence of hallucinations detected only by TNG (oscillatory hallucinations) in Figure 13. Similarly to the non-Englishcentric setup, the trends follow those presented in the analysis in the main text (see Section 5.2). 
Hallucinations in Large Multilingual Translation Models
English-Centric directions. We present COMET-22 scores on the original model hallucinated translations for every translation direction in Figure 14.
Hallucinations in Large Multilingual Translation Models
Non-English-Centric directions. We present the results on employing external models as fallback systems in Figure 15, and COMET-22 scores on the original model hallucinated translations for each translation direction in the non-English-centric setup in Figure 16. Importantly, the trends analyzed in-depth in the main text largely hold in this setup as well. In contrast with the analysis for the English-centric setup, ChatGPT struggles to outperform all other models in Non-English-Centric directions, especially in low and mid-resource levels. This is expected: as ChatGPT was trained on a non-parallel heavily English-centric corpus, it is likely that it struggles more with translation directions that do not include English.    
Hallucinations in Large Multilingual Translation Models
We provide several examples of English-centric natural hallucinations generated by the M2M models in Figure 17, which spans 2 pages. Figure 17: Examples of hallucinations generated by the M2M models in English-centric directions. We present an oracle English translation of the generated hallucination with Bing Microsoft Translator (ORACLE TO en) when necessary.
Prompting Large Language Model for Machine Translation: A Case Study
Large language models (LLMs) pretrained on massive unlabeled corpora have shown impressive emergent abilities under model scaling which enable prompting for downstream applications (Brown et al., 2020;Kaplan et al., 2020;Wei et al., 2022b;Zhang et al., 2022a;Chowdhery et al., 2022). Different from task-specific finetuning, prompting constructs task-specific prompts by rephrasing test examples with descriptive task instructions and executes the task by feeding prompts to LLMs directly. It can be further enhanced through in-context learning by providing a few labeled examples (or prompt examples) as a demonstration (Brown et al., 2020). As a new paradigm, prompting LLMs has achieved state-of-the-art performance over a range of natural language processing (NLP) tasks (Chung et al., 2022;Goyal et al., 2022;Wei et al., 2022c;Chowdhery et al., 2022).
Prompting Large Language Model for Machine Translation: A Case Study
In this paper, we focus on prompting LLMs for machine translation (MT). MT represents a complex task requiring transforming a source input into its semantically equivalent target output in a different language, which combines sequence understanding and generation. It offers a unique platform to assess the cross-lingual generation capability of LLMs, and the assessment may shed light on pretraining/finetuning algorithm design for achieving universal LLMs (Chowdhery et al., 2022). While a few studies have reported translation results (Brown et al., 2020;Chowdhery et al., 2022), a systematic study on how prompting works for MT is still missing in the literature.
Prompting Large Language Model for Machine Translation: A Case Study
We aim at filling this gap by thoroughly examining different prompting setups using the recently released GLM-130B (Zeng et al., 2022), particularly concerning three aspects: the prompting strategy, the use of unlabeled/monolingual data, and the feasibility of transfer learning. Prompting has shown varying sensitivity to the choice of prompt templates and examples (Zhao et al., 2021). For MT, prior studies adopted different templates (Brown et al., 2020;Wei et al., 2022a;Chowdhery et al., 2022), and we reevaluate them to figure out the optimal one. We further design a set of features for prompt examples and explore which one(s) could explain the prompting performance, according to which we develop the example selection strategy.
Prompting Large Language Model for Machine Translation: A Case Study
Since leveraging monolingual data to improve MT has long been of interest, we would like to determine whether and how such data can be used in prompt example construction. We make a step in this direction by studying the effect of data aug-mentation using back-/forward-translation (Sennrich et al., 2016b;Zhang and Zong, 2016) via zero-shot prompting. In addition, neural MT and pretrained LLMs have shown encouraging transfer abilities (Devlin et al., 2019;Arivazhagan et al., 2019;Zhang et al., 2020;Xue et al., 2021) but transfer learning for prompting has received little attention. Whether prompt examples are transferable across different settings, such as from one domain/language pair to another and from sentencelevel examples to document-level translation, is yet to be addressed.
Prompting Large Language Model for Machine Translation: A Case Study
We address the above concerns with GLM-130B as the testbed and conduct extensive experiments on FLORES and WMT evaluation sets. We mainly study translation for three languages: English, German and Chinese. We also provide a quantitative and qualitative analysis to disclose problems when prompting for MT, which might offer insights for future study. Our main findings are listed as below:
Prompting Large Language Model for Machine Translation: A Case Study
• Prompting performance varies greatly across templates, and language-specific templates mainly work when translating into languages LLMs are pretrained on. An English template in a simple form works best for MT.
Prompting Large Language Model for Machine Translation: A Case Study
• Several features of prompt examples, such as sequence length, language model score, and semantic similarity, correlate significantly with its prompting performance while the correlation strength is weak in general. Selecting examples based on these features can outperform the random strategy, but not consistently.
Prompting Large Language Model for Machine Translation: A Case Study
• Using monolingual examples for prompting hurts translation. By contrast, constructing pseudo parallel examples via back-/forwardtranslation is a good option. Back-translation performs better and is more robust.
Prompting Large Language Model for Machine Translation: A Case Study
• Prompting shows some degree of transferability. Using demonstrations from other settings can improve translation over the zero-shot counterpart, while the superiority of a demonstration in one setting can hardly generalize to another.
Prompting Large Language Model for Machine Translation: A Case Study
• Prompting for MT still suffers from copying, mistranslation of entities, hallucination, inferior direct non-English translation, and prompt trap where translating the prompt itself via prompting becomes non-trivial.
Prompting Large Language Model for Machine Translation: A Case Study
Prompting for MT Given a pretrained and fixed LLM L, MT prompting first converts each test input X to a prompt according to a template T and then generate the translation Y by feeding the prompt to L. In this study, we consider zero-shot and few-shot prompting for translation. Zero-shot prompting only has access to the test input X, while few-shot prompting assumes that a few extra labeled examples (or prompt/demonstration examples)
Prompting Large Language Model for Machine Translation: A Case Study
are available and can be used as a demonstration. Particularly, we adopt the following template for zero-shot prompting based on the results in Section 3:
Prompting Large Language Model for Machine Translation: A Case Study
[src]: X [tgt]:
Prompting Large Language Model for Machine Translation: A Case Study
(1)
Prompting Large Language Model for Machine Translation: A Case Study
where We also explore template language, which denotes the language in which the template is expressed. For example, the Chinese template " 中 文：X 英文：" represents the Chinese counterpart of the following English template "Chinese: X English: ".
Prompting Large Language Model for Machine Translation: A Case Study
Setting We experiment with GLM-130B, a LLM with 130B parameters pretrained on Chinese and English monolingual corpora, which was reported to outperform GPT-3 and OPT-175B on several NLP tasks (Zeng et al., 2022). Note GLM-130B is a raw LLM without any further finetuning. We use its INT4-quantized version, which is more affordable and suffers little performance degradation. We adopt beam search for MT with a beam size of 2, and perform experiments with 4 RTX 3090 and A100-40G GPUs.
Prompting Large Language Model for Machine Translation: A Case Study
We work on three languages: English (En), German (De), and Chinese (Zh). We perform major   and their prompting performance, we construct an Ablation set for Wiki, WMT and Multi-Domain (IT and Medical) based on the dev set of FLORES, WMT21 and Multi-Domain, separately, where we randomly sample 100 instances as the ablation test set and use the rest as the default example selection pool. To distinguish, we will refer to the official dev and test set as Full set. Detailed statistics are listed in Table 9, Appendix.
Prompting Large Language Model for Machine Translation: A Case Study
We evaluate translation performance using both a surface-based metric, detokenized BLEU↑ from SacreBLEU (Post, 2018), and a model-based metric, COMET↑ from unbabel-comet with wmt20comet-da (Rei et al., 2020).
Prompting Large Language Model for Machine Translation: A Case Study
To perform MT, prompting needs to cast the translation problem into a language modeling problem via the prompt. Thus, the format of the prompt, including its wording, directly affects how LLM understands the task and its behavior. For MT, we are interested in the following research questions:
Prompting Large Language Model for Machine Translation: A Case Study
• Which template should we use for MT prompting? And what language for the template?
Prompting Large Language Model for Machine Translation: A Case Study
• Does demonstration matter for MT prompting? How to select optimal prompt examples?
Prompting Large Language Model for Machine Translation: A Case Study
We address them through extensive experiments on Wiki Ablation sets.
Prompting Large Language Model for Machine Translation: A Case Study
Zero-shot prompting performance varies greatly across templates. We start with zeroshot prompting and explore the effect of different templates. Depending on how to describe MT and partially inspired by prior studies (Brown et al., 2020;Chowdhery et al., 2022;Wei et al., 2022a), we compare 6 templates and evaluate them on the Wiki Ablation sets covering 6 language pairs (En↔De, En↔Zh, De↔Zh). Table 1 shows the results (we list detailed results in    Language-specific template delivers mixed results. Table 1 also shows the prompting results of German and Chinese templates, which often largely underperform their English counterparts. Since German is not a major pretraining language in GLM-130B, a German template degenerates the translation substantially. By contrast, a Chinese template yields improved performance when translating into Chinese (see Table 10). Still, an English template works best on average.
Prompting Large Language Model for Machine Translation: A Case Study
The preference of GLM-130B to English template also shows that the level of language understanding and cross-lingual ability in GLM-130B varies across languages, even though it's pretrained on the same amount of monolingual Chinese and English tokens. This might be caused by the fact that English is used more globally than Chinese, but might also suggest that improving the language understanding of LLM requires more advanced training algorithms beyond scaling training data.
Prompting Large Language Model for Machine Translation: A Case Study
Using more prompt examples for demonstration improves translation significantly on aver- age. We next study few-shot prompting following the template A but in format (2) with K varying from 1 to 20. We evaluate multiple demonstrations for each K via random sampling to reduce data biases. Figure 1 shows that the more examples used, the better average performance (more results are shown in Figure 5, Appendix), albeit at the cost of using more GPU memory and increasing the inference time per token as in Figure 3.
Prompting Large Language Model for Machine Translation: A Case Study
The performance of demonstration is not stable. However, we also see high performance variance under the same K. It's possible that a demonstration with 5 examples outperforms its 10 or 20 counterpart. Figure 1 also shows that 1-shot prompting underperforms zero-shot prompting in many cases, even on average. This echoes with previous findings on other NLP tasks (Zhao et al., 2021;Liu et al., 2022) and also highlights the significance of developing effective example selection strategies.
Prompting Large Language Model for Machine Translation: A Case Study
Note that few-shot prompting greatly improves translation into Chinese. The reason based on our manual analysis is that the zero-shot baseline tends to translate into traditional Chinese with messy codes, where prompt examples help (the reference text is always simplified Chinese).
Prompting Large Language Model for Machine Translation: A Case Study
Several features correlate with prompting performance significantly yet weakly. We thus turn to explore example selection for prompting. Our idea is to extract a couple of diverse features from demonstration and examine whether any of them are informative enough to be used as an indicator for the selection. In this study, we simplify our analysis by focusing on 1-shot prompting, which ignores the ordering of prompt examples (we leave few-shot analysis to future). Particularly, we extract and analyze 7 features of a demonstration:  LMScore GLM-130B-based, length-normalized log likelihood of the demonstration;
Prompting Large Language Model for Machine Translation: A Case Study
MTScore translation quality of the prompt example from COMET QE model wmt20-cometqe-da (Rei et al., 2020);
Prompting Large Language Model for Machine Translation: A Case Study
SemScore semantic score based on the cosine similarity of the demonstration's source and target sentence embeddings from LASER2 (Heffernan et al., 2022);
Prompting Large Language Model for Machine Translation: A Case Study
CaseSemScore-Src similarity to the input that averages over SemScores between the test input and the demonstration's source;
Prompting Large Language Model for Machine Translation: A Case Study
CaseSemScore-Tgt similar to CaseSemScore-Src but compares to demonstration's target;
Prompting Large Language Model for Machine Translation: A Case Study
We sample multiple demonstrations randomly and inspect the Spearman's correlation between feature values and prompting performance. We consider high-quality and low-quality pool for sampling. Table 2 summarizes the results and Figure 2 illustrates the relation between COMET and LMScore (more results are given in Table 11 and Figures 6, 7, Appendix). With the high-quality pool, different demonstrations yield similar translation results (see blue points) despite their feature values varying greatly. Several features show insignificant and inconsistent correlation, particularly for De→En and Zh→En. This suggests developing selection policy for high-quality example pool is non-trivial.
Prompting Large Language Model for Machine Translation: A Case Study
After mixing with demonstrations from the lowquality pool, the significance gets strengthened.
Prompting Large Language Model for Machine Translation: A Case Study
LMScore and CaseSemScore-Tgt shows the highest correlation on average followed by TLength and SemScore. MTScore behaves much worse which might be caused by its instability on sentence-level evaluation (Moghe et al., 2022). However, we didn't see significant difference in terms of Spearman's ρ between input-relevant and input-agnostic features (Agrawal et al., 2022), neither among surface-based, LLM-based or semantic-based features. Surprisingly, the simple feature, S/TLength, yields reasonably high correlation. We argue that long examples could offer LLM with more signals about the task's input and output space. This finding suggests that researchers should select long unlabeled sentences for annotation to improve prompting. Yet, most Spearman's ρs are much smaller than 0.5, indicating a weak/fragile relation.
Prompting Large Language Model for Machine Translation: A Case Study
In general, selecting prompt examples of high translation quality, high semantic similarity, high LLM likelihood, long sequence length and high similarity to test inputs are all preferable strategies. Unfortunately, none of them can guarantee optimal translation performance.
Prompting Large Language Model for Machine Translation: A Case Study
Using prompt examples selected based on the proposed features yields improved performance. We next verify the above findings on the Full sets. We explore selection strategies based on SemScore, LMScore and TLength (i.e. use topranked examples) as they show high average correlation. We didn't analyze CaseSemScore-Tgt as it's more complicated and doesn't make significant difference. Note we excluded too long (more than 100 tokens) or too short (less than 10 tokens) examples during selection. We also consider 5-shot prompting, where we concatenate top-ranked 5 examples in an ascending order (Liu et al., 2022). Table 3 shows that, with high-quality pool, adopting the feature-based strategy is likely to outperform the random baseline, and the SemScore-based strategy performs well across different settings (detailed results are available in Table 13 and 14, Appendix). These strategies also generalize to 5-shot prompting to some extent. For selection from lowquality pool, we propose a combined strategy: we  first choose top-11K examples according to Sem-Score to filter out poor examples, the top-1K of which are also dropped as they tend to be uninformative (see Table 12 in Appendix); then we re-rank the rest with LMScore and retain top-1K examples, upon which we further apply the TLength-based strategy. In Table 3, this combined strategy outperforms the random one by varying degrees.
Prompting Large Language Model for Machine Translation: A Case Study
A longstanding concern in MT is how to utilize unlabeled data to improve translation. While prompting enables few-shot learning reducing the data requirement, exploring whether demonstration could benefit from monolingual examples is still valuable, both for MT study and for understanding of the role of demonstration in prompting. Min et al. (2022) argue that the key role of demonstration lies in its support of the input space, the label space and the prompt format, rather than the genuineness of the examples. They found that randomly replacing labels in demonstration barely hurts performance on classification tasks. We reexamine this argument in the context of MT by studying the following three prompting settings: 1) random examples constructing sentence pairs from monolingual sources and targets randomly; 2) source/target example only using monolingual source/target alone for prompting.
Prompting Large Language Model for Machine Translation: A Case Study
Directly using monolingual data for demonstration doesn't work. Figure 4 (top) shows a totally different story (see Figures 8 and 9 in Ap-pendix for more results): monolingual examplebased demonstration almost always hurts translation, and the more examples used, the more degeneration yielded. Using random examples misleads the prompting and performs the worst in general; compared to target-only examples, using source examples yields slightly better results except translating into Chinese. This indicates that the genuine source-target mapping should be retained in the demonstration, and also indicates that MT features unique challenges which deserves more attention when studying prompting.
Prompting Large Language Model for Machine Translation: A Case Study
Pseudo parallel examples by forward-/backtranslation benefits prompting. Inspired by data augmentation in MT (Sennrich et al., 2016b;Zhang and Zong, 2016), we next resort to constructing pseudo parallel data. We first adopt GLM-130B to translate the source or target examples via zero-shot prompting, and then use the generated parallel examples as demonstration. Despite low quality, Figure 4 (bottom) shows that this is an effective way to improve prompting, and using more examples often produces better results. We also observe that back-translation (i.e. translating target monolingual examples) performs better and behaves more robustly than forward-translation (i.e. translating source examples instead), which even approaches prompting with real parallel examples.  Table 4: Spearman's ρ and relative performance for crosslingual transfer under 1-shot prompting on Wiki Ablation sets (among En, De and Zh). When studying transfer from language pair S1 to S2, we randomly sample 300 demonstrations from the default pool of S1, and then evaluate them on the Ablation test sets for S1 and S2 respectively, based on which we compute the correlation. The performance is also averaged. ∆ Quality: relative quality against the zero-shot baseline. Blue cells indicate positive gains. Source/Target Shared: average result for transfer settings where the source/target language is shared; Reversed: average result for the same language pair but in different directions.
Prompting Large Language Model for Machine Translation: A Case Study
could be transferred across different settings, especially from one domain/language pair to another and from sentence-level to document-level translation. While previous studies demonstrate the feasibility with continuous prompts on classification tasks , transfer for hard prompting on MT has never been investigated. Assume that demonstrations D 1 and D 2 are selected in setting S 1 and that D 1 performs better (i.e. D 1 > D 2 ), We have the following research questions:
Prompting Large Language Model for Machine Translation: A Case Study
• Could we also expect D 1 > D 2 in setting S 2 ?
Prompting Large Language Model for Machine Translation: A Case Study
• Whether using demonstrations from S 1 could outperform zero-shot prompting in S 2 ?
Prompting Large Language Model for Machine Translation: A Case Study
We next study these questions through experiments with 1-shot prompting.
Prompting Large Language Model for Machine Translation: A Case Study
The superiority of a demonstration doesn't generalize across settings. If the ranking D 1 > D 2 holds across settings, the results of the same set of demonstrations in different settings should show high and significant Spearman's correlation. Unfortunately, the correlations in Table 4 and 5 are very weak and often insignificant (more results are given in Table 15, 16, and 17), even for the same language pairs in different directions (Reversed) and for similar domains (Wiki⇒WMT). This suggests that we will need setting-specific demonstration to get the optimal translation quality.
Prompting Large Language Model for Machine Translation: A Case Study
Using out-of-setting demonstrations can benefit translation. However, we can still gain from using out-of-setting demonstrations as demonstrated by the positive gains in Table 4 and 5, where we   find that transfer in target-shared and reversed settings is relatively easier, and that transfer across distant domains can be successful particularly when insetting example pool is of low quality. This is also supported by the transfer to document-level translation, where both BLEU and document-specific evaluation get improved as shown in Table 6. Results in Table 19 show that the transfer is unstable and could deliver negative results, i.e. worse than zero-shot prompting, partially resonating with previous findings (Lin et al., 2021). We leave the question of how to select prompt examples in transfer learning setups to future.
Prompting Large Language Model for Machine Translation: A Case Study
Although prompting enables translation with decent performance, it still suffers from many (wellknown) problems. Here, we briefly explain the problems we observed from the model's outputs. Prompting sometimes rejects translating the input. Instead, it emits either empty or off-target outputs, i.e. translating in a wrong target language. This occurs frequently when translating into Chinese, where the model often translates into traditional Chinese with messy codes, causing unstable performance. Besides overly relying on a language Source 根据三江源国家公园管理局长江源园区可可西里管理处统计，藏羚羊回迁数量总体呈 逐年上升态势，2019年藏羚羊回迁数量为4860只，比2018年增加338只。
Prompting Large Language Model for Machine Translation: A Case Study
Statistics from the Sanjiangyuan National Park Administration Yangtze River Origin Park Hoh Xil Management Office show that the number of Tibetan antelopes on the return migration route has been increasing each year, with 4,860 counted in 2019, an increase of 338 over 2018.
Prompting Large Language Model for Machine Translation: A Case Study
According to the三江源国家公园管理局长江源园区可可西里管理处, the total number of re-migration of the Tibetan antelope ::: has ::: been ::: on :: the ::: rise ::::: since :::: 2018, with 4,860 re-migrating in 2109, an increase of 338 compared to 2808.
Prompting Large Language Model for Machine Translation: A Case Study
English: Dominic Raab has defended the Government's decision to re-introduce quarantine measures on Spain at short notice. Translate from English to Chinese: Chinese:
Prompting Large Language Model for Machine Translation: A Case Study
持。Translate from English to Chinese:  model, prompting tends to under-translate the input, copy source phrases, produce code-switched output, mistranslate entities (e.g. dates) and generate hallucination, as illustrated in Table 7.
Prompting Large Language Model for Machine Translation: A Case Study
We also observe a phenomenon specific to prompting: prompt trap where prompting behaves unpredictable when its input is mixed with prompt template phrases. In the second case in Table  7, the model copies the template phrases, rather than translating them into Chinese. This means that translating prompt itself (not just the input) becomes non-trivial, and that users may attack prompting-based translation systems by manipulating the input format.
Prompting Large Language Model for Machine Translation: A Case Study
We find that the translation quality between German and Chinese is very poor (see Table 13). We argue that the cross-lingual ability of GLM-130B mainly centers around English (although GLM-130B was pretrained on Chinese as well), and thus explore pivoting translation instead. Table 8 shows that pivoting through English greatly improves non-English translation. It's still unclear whether the current LLM pretraining recipe could achieve promising non-English-centric cross-lingual ability. We might need to consider adding parallel data into the LLM pretraining or finetuning.
Prompting Large Language Model for Machine Translation: A Case Study
In this paper, we presented a systematic study on prompting for MT, exploring topics ranging from prompting strategy, the use of unlabelled monolingual data, to transfer learning. We found that prompt template and demonstration example selection both have substantial impact on translation. Some prompt example features correlate significantly with prompting performance; treating them as criteria for example selection benefits translation to some extent but not consistently as the correlations are not strong enough.
Prompting Large Language Model for Machine Translation: A Case Study
Prompting for MT requires retaining the sourcetarget mapping signals in the demonstration. Directly applying monolingual data for prompting sounds interesting but doesn't work. Constructing pseudo parallel prompt examples by back-/forwardtranslation via zero-shot prompting is a simple yet effective solution. Regarding transfer learning, we saw positive results when applying a (sentencelevel) demonstration to other domains, other language pairs or document-level translation. Unfortunately, the optimality of the demonstration doesn't generalize across settings and the transfer perfor-mance is also unstable. We argue that MT provides a set of unique challenges and call for more efforts on evaluating prompting LLMs for MT.
Prompting Large Language Model for Machine Translation: A Case Study
Prompting also faces a number of other issues, like off-target generation and prompt traps, which we plan to address in the future. We are also interested in examining whether our findings can generalize to other LLMs, like GPT-3, OPT and PaLM. We would also like to explore further how to improve the cross-lingual ability in LLM.
Prompting Large Language Model for Machine Translation: A Case Study
Our study heavily depends on the INT-4 quantized GLM-130B, which, unlike GPT and PaLM, was pretrained with both bidirectional and unidirectional training objectives. The quantization might weaken the model's capability and deteriorate some unknown aspects. It's unclear how our findings generalize to other pretrained LLMs. In addition, we mainly work on three languages due to resource constraints, and in experiments, results vary greatly across language pairs. Increasing the coverage of experimental languages would make the results more reliable.   72 -20.65 -81.28 -125.21 -137.02 -125.31 -108.45 -99.65 C 15.40 5.70 5.70 3.00 6.00 6.70 7. 08 -23.46 -80.15 -86.27 -104.10 -87.18 -58.23 -73 02 -6.41 -90.63 -12.10 -159.66 -9.24 -121.29 -66.55 C 11.40 3.20 14.30 0.40 20.80 5.00 9.18 -32.55 -114.57 -9.91 -140.54 2.89 -85.58 -63.38 D 17.10 6.40 15.90 0.20 19.60 1.90 10.18 -34.15 -101.69 -24.36 -166.15 -9.20 -125.20 -76.79 E 29.00 50 -11.65 -102.50 -63.95 -161.96 -46.84 -128.12 -85.84 C 12.10 3.00 13.80 0.80 21.20 9.90 10.13 -36.39 -105.55 -42.16 -151.06 -15.41 -74.90 -70.91 D 14.10 3.20 15.10 0.20 20.00 2.50 9.18 -19.15 -106.69 -19.34 -154.73 -11.51 -94.82 -67.71 E 28.60 8.00 26.50 0.90 32.30 21.40 19.62 8.71 -118.14 15.34 -124.30 21.18 14.91 -30.38 F 26.90 3.40 26.10 0.20 25.80 16.00 16.40 11.58 -120.31 10.33 -129.61 -21.19 -20.52 -44.95 
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Large language models (LLMs) trained at scale with simple objectives have been found to achieve results that match dedicated systems on numerous NLP tasks (Radford et al., 2019), as long as tasks are formulated as text generation though "prompting" (Liu et al., 2023). LLMs' multitask performance can even be improved with "instruction" fine-tuning (Sanh et al., 2022;Muennighoff et al., 2022), few-shot priming, and better strategies to select or learn prompts (Petroni et al., 2019;Shin et al., 2020;Schick and Schütze, 2021;Lester et al., 2021;Wei et al., 2022). In multilingual settings, their performance on machine translation (MT) tasks, as measured by automatic scores, is often close to state of the art, even when mostly trained on monolingual data (Brown et al., 2020). Moreover, prompting-based MT offers the prospect of better control of outputs, e.g. in terms of quality, style and dialect (Garcia and Firat, 2022). However, these abilities remain poorly understood, as LLM analyses primarily focus on their multitask rather than multilingual ability (see however (Vilar et al., 2022;Zhang et al., 2023;Moslem et al., 2023), which we discuss in Section 2).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
In this work, we focus on the MT performance of BLOOM (BigScience et al., 2022), a (family of) open-access multilingual LLM(s), designed and trained by the collaborative BigScience project. 1 Our main aims are to (i) evaluate BLOOM's zeroand multi-shot behaviour, (ii) study the effect of prompt design, (iii) evaluate a diverse set of language pairs and (iv) assess its ability to use linguistic context. Our main conclusions, which extend those in (BigScience et al., 2022), are (i) 0shot ability is blighted by overgeneration and generating in the wrong language, (ii) using few-shot improves both issues, with results much closer to state of the art across datasets and language pairs, (iii) there are clear transfer effects, with high scores for languages not officially seen in training, and successful transfer across language pairs via few-shot examples and (iv) although linguistic context does not lead to higher scores, there is evidence that BLOOM's translations are influenced by it. We release our code and translation outputs. 2
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Since the early attempts to use language models (LMs) as multi-task learners (McCann et al., 2018), MT has been a task of choice to gauge LMs' multilingual ability. Results for the zero-and fewshot ability of LMs were discussed for both GPT-2 and GPT-3 (Radford et al., 2019;Brown et al., 2020), which is especially intriguing as they were trained primarily on monolingual (English) data. These results have since been confirmed for other monolingual LMs such as T5 (Raffel et al., 2020) and multilingual LMs such as XGLM (Lin et al., 2022), PALM (Chowdhery et al., 2022), and ALEX-ATM (Soltan et al., 2022). However, the focus has mainly been on global multi-task performance; often only a small part of the discussion is devoted to MT. Moreover, results are often only reported for a few well-resourced language pairs (e.g. English-French and English-German), and the scores reported (mostly BLEU), are hard to compare due to a non-systematic use of standardised evaluation protocols and metrics. 3 There are however some in-depth analyses of MT performance of LLMs, each focusing on a specific LM's performance in a true multilingual setting with respect to prompt design and number of few-shots. For instance, Vilar et al. (2022) reevaluate the MT performance of the multilingual PALM (Chowdhery et al., 2022), focusing notably on the selection of few-shot examples. Consistent with our findings, they determine that prompt choice becomes unimportant in few-shot settings and that using few-shot examples increases performance with diminishing returns for k > 5 examples, using BLEURT and BLEU scores, as well as the results of a human evaluation. They find that the quality of few-shot examples has a large impact on performance. However, even with good prompts, PALM lags a couple of points behind state-of-the-art MT systems, especially when translating from English, notable due to adequacy problems. Zhang et al. (2023) focus on the evaluation of GLM-130B, a bilingual (Chinese and English) LLM (Zeng et al., 2022). Their main conclusions are also consistent with ours: (a) zeroshot performance varies greatly across different prompts, (b) increasing the number of prompts from 0 to 20 yields consistent improvements in performance, again with variance across instructions, and (c) finding the best few-shot example selection policy is difficult. It seems that having good and long examples, for instance, may help, even though none of the criteria explored in this study seem to provide any systematic improvement. A last point worth mentioning is that prompting with monolingual data hurts performance, but that using pseudo-parallel data obtained with back-translation (Bojar and Tamchyna, 2011) is an effective workaround. Moslem et al. (2023)  decoding and a prompt resembling our own xglm-source+target prompt. They report strong zero-shot behaviour using multiple metrics, plus clear improvements with an increased number of shots for the well-resourced languages, less so for the only low-resource language in their lot (Kinyarwanda). The main novelty of this study is to use prompting as a vehicle to perform local adaptation and to ensure terminological consistency. For this, they use fuzzy matches from a translation memory as well as MT outputs to build their prompts, yielding results that both outperform their zero-shot system, but also their initial MT engine. Additionally inserting terms and their translation in the instruction yields supplementary improvements. Finally note the preliminary evaluation of CHATGPT in (Jiao et al., 2023), which reports interesting insights regarding the multilingual abilities of this model, as well as proposing innovative techniques to generate (artificial) prompts and to use pivoting in prompting. Similar to ours, this study considers multiple test domains such as news (WMT) and Wikipedia (Flores). A more in-depth analysis of the same model can be found in (Hendy et al., 2023), which confirms CHATGPT's strong translation abilities, at least for "well-resourced" 5 language pairs. Documentlevel scores are also reported, as well as human evaluations and qualitative analyses.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Multilingual MT is also the subject of dedicated (monotask) architectures and training regimes. Originally introduced in (Dong et al., 2015;Firat et al., 2016;Luong et al., 2016) with limited language coverage, the latest versions of these approaches are able to handle hundreds of languages, including very low-resource language pairs (Fan et al., 2021;Bapna et al., 2022;Costa-jussà et al., 2022). Although we found that BLOOM is able to match this performance, given sufficient training data, we also see that it still lags behind for many languages pairs that are under-represented in its training data.   (Bojar et al., 2014). All prompts specify the target language (L2). Each prompt exists in a 'target-only' version (-target), where only the target language is specified, and two prompts also exist in a second -source+target version, where the source language (in red and in brackets) is explicit in the instruction.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
It is an auto-regressive language model designed to generate text to complete a user-entered text prefix, known as a prompt.  (Toral et al., 2018)) and between other languages may mostly reflect their similarity with the original English. We use it to test and compare BLOOM's multilinguality, including for low- 6 The ROOTS corpus can now be queried using the dedicated search tool https://hf.co/spaces/ bigscience-data/roots-search. 7 https://hf.co/bigscience/bloom 8 English, French and Hindi make up 30%, 12.9% and 0.7% of the training data respectively (Laurençon et al., 2022). resource languages. 9 DiaBLa is a bilingual test set of spontaneous written dialogues between English and French speakers, mediated by MT. We use this as a test of MT in an informal domain and the impact of (cross-lingual) linguistic context in MT.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
We evaluate and compare BLOOM (and its variants) using the Language Model Evaluation Harness (Gao et al., 2021) in 0-shot and few-shot settings. For few-shot, k examples are prefixed to the prompt and separated with ### as shown in Example 1 (1-shot example is underlined).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
(1) Input: French: je m'ennuie = English: I'm bored. ### English: Is that your dog that's just wandered in over there? = French: Reference: Est-ce que c'est votre chien qui vient de rentrer par là ?
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Results are reported on the datasets' test splits. Few-shot examples are randomly taken from the data splits according to availability (train for WMT, dev for Flores-101 and test for DiaBLa). We evaluate using BLEU (Papineni et al., 2002) as implemented in SacreBLEU (Post, 2018), using as tokenisation 13a for WMT and DiaBLa and spm for Flores-101 as recommended (Costa-jussà et al., 2022). 10 BLEU has many shortcomings but is good enough to provide quantitative comparisons for most systems used in this study. We additionally use COMET (Rei et al., 2020) for finer grained comparisons when the scores are closer.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
In our cross-dataset comparison (Section 5.1), we compare BLOOM to other LLMs: (i) two taskfine-tuned models: T0 11 (Sanh et al., 2022), trained on English texts, and MT0-XXL 12 (Muennighoff et al., 2022), the multilingual version, and (ii) OPT 13 (Zhang et al., 2022), an English generative LM. We evaluate all models on the same prompt xglm-source+target. To evaluate multiple language pairs with Flores-101, we compare (as a topline) to the supervised 615Mparameter MT model M2M-100 (Fan et al., 2021), using the scores computed by Goyal et al. (2022).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
We use several prompts, designed to illustrate different sources of variation: (i) the inclusion (or not) of the source language name, (ii) the relative order of source and target language names, (iii) the position of the source sentence (beginning or end of the prompt) and (iv) the prompt's verbosity. These prompts, available in PromptSource (Bach et al., 2022), are shown in  (Wei et al., 2022;Garcia and Firat, 2022).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Considering the entries in Table 1, we can see that "prompting" in fact refers to two distinct aspects of the input: (i) the formulation of the task in natural language and (ii) the presentation of related examples (for few-shot setups) interleaved with language tags (perhaps more clearly referred to as priming by Pham et al. (2020)). As illustrated by the xglm prompt for example, the instruction part can reduced to one single word. As our results below suggest, the instruction mostly matters in 0-shot setups, but can almost be dispensed with in few-shot scenarios. The authors of (Brown et al., 2020) and (Hendy et al., 2023) also use a verbose, instruction-like prompt in their zero-shot setup, and a much more compact one for few shots experiments. Also note that InstructGPT's prompt combines both an instruction and language tags (Ouyang et al., 2022, p. 49). 12 https://hf.co/bigscience/mt0-xxl 13 https://hf.co/facebook/opt-66b 14 This was not always straightforward due to incomplete documentation concerning (a) prompts tested, and (b) those actually used in each experiment (e.g. different ones for 0shot and few-shot runs (Chowdhery et al., 2022)). 15 Used only it seems, for zero-shot learning in the form "Q: what is the L2 translation of sentence [source sentence]. A:", where special tokens Q and A are the query and the answer texts (cf. Figure G.36, pp 59).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Our evaluation of BLOOM starts with a comparison across the three datasets and detection of major MT errors with a focus on WMT (Section 5.1) and then we present more in-depth analyses of particular aspects: (i) using WMT, a comparative study of BLOOM model sizes (Section 5.2) and prompts (Section 5.3), (ii) using Flores-101 an evaluation of more language pairs and crosslingual few-shot transfer (Section 5.4), and (ii) using DiaBLa, a study of the use of linguistic context (Section 5.5).  We first prompt BLOOM and the comparative models using the same prompt across datasets, restricting the directions tested to en↔fr and to en↔hi. We choose to systematically use the xglm-source+target prompt (Table 1), which corresponds to the following template:
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
(2) L1: [source sentence] = L2:
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
where L1 and L2 refer to the source and target languages respectively (e.g. English and French for en→fr) and [source sentence] is replaced by a given source sentence. BLEU scores are in Table 2a for both 0-shot and 1-shot (results with COMET are given in Appendix A). There are issues for 0-shot MT for all directions, particularly when translating into non-English languages, (BLEU scores are systematically lower than into English). Even into English, the scores remain low with respect to state of the art (e.g. 2.7 BLEU for Flores-101 fr→en BLOOM vs. 60.1 for MT0-XXL). 16 MT0-XXL's good results illustrate the benefits of prompt finetuning. Results are considerably improved in the 1-shot setting (except for T0 and MT0-XXL, both fine-tuned) and are comparable to other LLM MT results reported in the literature; they are markedly better for BLOOM than for OPT, which was mostly trained on English data. These general observations apply for WMT and Flores-101, while issues remain for DiaBLa.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
What are the main issues? We identify two main issues (particularly for 0-shot): (i) overgeneration and (ii) generating in the wrong language (an obvious problem for MT). Examples 3 and 4 illustrate some instances of these problems. The first example shows a very common problem, with repetition of the prompt in multiple languages until the maximum generation length is reached.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
(3) Input: French: Il dit avoir conçu une sonnette de porte Wi-Fi. = English: BLOOM: He says he conceived a Wi-Fi doorbell. = Spanish: Dice que concibió un timbre de puerta Wi-Fi. = German: Er sagt, er hat einen WLAN-Türglocken gegründet. = Italian: Dice di aver concepito un chiuso Wi-Fi [. . . ] (4) Input: French: Le tigre fait partie de la même famille (genre Panthera) que les lions, les léopards et les jaguars. Ces quatre félins sont les seuls capables de rugir. = English: BLOOM: The tiger is a member of the same family (genus Panthera) as lions, leopards and jaguars. These four cats are the only ones capable of roaring. Le tigre est un grand félin qui vit en Asie. Il est le plus grand des félins. Il est aussi le plus gros carnivore terrestre. Il est très rapide et peut courirà plus de 60 km/h. [. . . ] Separating MT quality from overgeneration Overgeneration as seen in Example 3 is a separate issue from BLOOM's capacity to translate into another language. We therefore devise a custom truncating method for this type of overgeneration such that only the first translation in a prediction is kept, i.e. anything after a newline or the regular expression pattern = .+?: is discarded.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Results after truncation (Table 2b) show that for all three datasets, 0-shot and 1-shot scores are significantly improved (e.g. 1-shot DiaBLa fr→en increases from 12.05 to 41.36 and 0-shot Flores-101 hi→en increases from 3.40 to 30.19). BLOOM is capable of performing good MT but has a problem knowing when to stop generating. We use the same truncation elsewhere too and indicate when we show results for original or truncated outputs.  
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
We automatically detect the language of predictions using fasttext langid 17 (Joulin et al., 2017). Table 3 shows the number of translations identified as being in the correct target language, or alternatively in the source or another language for 0-shot and 1-shot setups after truncation. 18 ,19 The number of sentences in the correct target language increases from 0-to 1-shot, particularly for the two non-English target languages. When translating into Hindi (0-shot), 1/5 (509) of predictions are not detected as Hindi; the 1-shot largely mitigates the issue (only 76 outputs are in the wrong language).  
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Several versions of BLOOM exist, with differing numbers of parameters. To test how size impacts performance, we report average scores and ranges for WMT across the seven prompts. Table 4 shows that as the size decreases (from 176B to 560M parameters), the performance also decreases significantly. We see substantial gains for all models when moving from 0-shot to 1-shot, the smaller models (e.g. BLOOM-7b1, BLOOM-3b) slightly closing the gap with the largest one. As the ranges in Table 4 are computed across prompts, we see that different prompts yield markedly different BLEU scores in the 0-shot setup; for 1shot, we still see variations of 6-8 BLEU points between the best and the worst prompt. Similar analyses performed with post-processing and also for English↔Hindi (Appendix C) confirm that (i) truncation improves scores for all model sizes and prompts and (ii) the choice of a bad prompt can result in catastrophic MT performance as compared to a good one.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Looking at average WMT results computed with respect to prompt choice (using the prompts in Table 1) allows us to further investigate cross-prompt variability.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Which prompt works best? This variability is illustrated in Tables 5 and 6 report  performance across prompts   get the highest average (and maximum) scores. Both prompts are more verbose (instruction-like), but the performance gap in the 1-shot setting between these prompts and the simpler, 'primingstyle' prompts (e.g. xglm) narrows. The worst results are seen for gpt3. With this prompt, translating into French after a text that only contains English seems particularly difficult: half of the 0-shot translations for gpt3 are classified as non-French by langid (most of them are English). When translating into Hindi, only 10 outputs are detected as being in Hindi.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Does it help to specify the source language in the prompt? We compare the two versions (-target and -source+target) of a good translation and xglm. Results in Tables 5 and 6 are inconclusive. For these language directions and prompts, we see small differences for 1-shot, which may be due to variance between runs. For 0-shot, it clearly helps xglm to indicate the source language, but for the more verbose a good translation, it helps one direction and hurts the other. This question would need to be further explored to draw more solid conclusions, including with non-English prompts.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
We further explore more language directions in the 1-shot setting using Flores-101. As in Section 5.    
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
To optimise computational resources, instead of running all language combinations, we concentrate on: (i) high-resource language pairs, (ii) high→mid-resource language pairs, (iii) lowresource language pairs and (iv) related languages (specifically Romance languages). Results are shown in Tables 7 and 8 for original outputs, given that overgeneration is less problematic for 1-shot.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
High-resource and high→mid-resource The results for high-resource and high→mid-resource language directions are generally good, surpassing M2M scores for high-resource, except for es→fr. 22 This suggests that BLOOM a has good multilingual capacity, even across scripts (between (extended) Latin, Chinese, Arabic and Devanagari scripts).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Low-resource For low-resource languages, the results are more variable; some language directions see better results than M2M, notably most into-English directions, but others are less good (e.g. into Hindi and Swahili). Results for the lowest-resourced languages tested (sw↔yo 22 French and Spanish, although related and comparably represented in ROOTS, have very different scores. Our preliminary analysis suggests that this is due to the Spanish references being less literal than the French and structurally more different from the original English. See Appendix E for some examples. and en↔yo) are particularly disappointing because the scores indicate that the resulting translations are meaningless, even though Yoruba and Swahili are present (although under-represented) in BLOOM's training data (<50k tokens each).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Romance languages This contrasts with the results between Romance languages, where results are good across-the-board, including from and into Italian (it) and Galician (gl), which are not officially in the training data. Note that Galician shares many similarities with the other Romance languages, in particular with Portuguese (pt). These contrasted results show the performance of an LLM not only depends on the amount of training data, but also largely on the similarity with seen languages. To be complete, these analyses should also take into account the possibility of mislabellings in the training data, 23 which have been found to explain a great deal of cross-lingual abilities of LLMs (Blevins and Zettlemoyer, 2022).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
1-shot results are positive for many of the language directions tested (including low-resource), provided they are sufficiently represented in the ROOTS corpus. To better understand how cross-     (en→bn), (iii) a language direction whereby the source languages are related (hi→en), (iv) the same related direction but from a different dataset (the WMT dev set) (v) a high-resource direction into the same target language (fr→en) and (vi) a high-resource unrelated language direction (fr→ar). The results (Table 9) show that cross-lingual transfer is possible, but using a different language direction can impact overgeneration and translation quality. The unrelated direction fr→ar gives the worst results, with most overgeneration (see the score difference between original and truncated), but also the worst quality after truncation, suggesting that language relatedness does play a role. Overgeneration is still a problem (although less so) when using the opposite direction (en→bn) or the same target language (fr→en). Using a related (higher-resource) source language (hi→en) reduces overgeneration and also gives the best MT results. However, better results are seen when using Flores-101 rather than WMT examples, suggesting that in-domain examples are best.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
There has been a considerable amount of research on linguistic context in MT, e.g. to disambiguate lexically ambiguous texts or when additional information is necessary for the output to be wellformed (e.g. translating anaphoric pronouns into a language that requires agreement with a coreferent) (Hardmeier, 2012;Libovický and Helcl, 2017;Bawden et al., 2018;Voita et al., 2018;Lopes et al., 2020;Nayak et al., 2022).  Table 10: Comparison of 1-shot results (BLEU) for Di-aBLa when using the previous/random sentence for the 1-shot example (using the xglm-source+target prompt). In bold are the best results for each language direction.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
We test the usefulness of linguistic context in DiaBLa in the 1-shot setting (again using xglm-source+target) by changing the origin of 1-shot examples: (i) a random example vs. (ii) the previous dialogue utterance. If linguistic context is useful, we would expect there to be an improvement for (ii). We also vary the language direction of the 1-shot example. By default, given that the dataset is bilingual, the direction of 1-shot examples is en→fr or fr→en, independent of the current example's direction. Given the results in Section 5.4.2 and the poor 0-shot results in Table 2a, it is important to account for this to provide a fair comparison. We therefore compare each type of context (random/previous) with (i) the same random directions, and (ii-iii) the same (and opposite) language directions as the current example. We show results for original and truncated outputs.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
Results are shown in Table 10. Truncation helps considerably; even for 1-shot, BLOOM struggles not to overgenerate and this is considerably reduced when the same rather than the opposite language direction is used for the 1-shot example. It is unclear whether using previous rather than random context helps: BLEU is higher (38.5 vs. 37.6), whereas COMET is lower (0.328 vs. 0.342). These differences could be the result of randomness in 1-shot example selection, and different results could be obtained with a different random seed. Despite these inconclusive results, it is clear that using previous context influences the translation, for better or worse. For evidence of this, see Table 19 in Appendix F, which provides three such examples: (i) an unlucky negative influence on the translation of an ambiguous word glace 'ice cream or mirror' from the previ-ous context, resulting in the wrong sense being chosen, (ii) the use of a coreferent instrument 'instrument' from the previous sentence and (iii) the correct gender agreement of the pronoun they into French (elles 'they (fem.)' as opposed to ils 'they (masc.)') to correspond to the feminine coreferent filles 'girls'.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
We have evaluated BLOOM's MT performance across three datasets and multiple language pairs. While there remain problems of overgeneration and generating in the wrong language (particularly for 0-shot MT), MT quality is significantly improved in few-shot settings, closer to state-ofthe-art results. Low-resource MT remains challenging for some language pairs, despite the languages being in the training data, questioning what it means to be a BLOOM language. However, we see evidence for cross-lingual transfer for non-BLOOM languages and when using few-shot examples from other language pairs. Finally, although using linguistic context does not give improvements with automatic metrics, there is evidence that discursive phenomena are taken into account.   B Wrong language prediction and over-generation
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
As described in Section 5.1, one problem identified with BLOOM, particularly for 0-shot translation, is generating in the wrong language. Tables 12 and 13 give the full analysis including raw figures for language identification for WMT14 fr↔en and hi↔en translation directions. For 0-5 few-shot examples, we indicate the number of truncated outputs identified as being from each language (indicated by the rows), the correct language (the target) being indicated in green, and the source language (therefore incorrect) being indicated in red. We also provide the average length difference (∆) between BLOOM's outputs and the reference translations (negative numbers indicate that the prediction is longer than the reference).      Table 13: Raw figures for language identification and length differences of outputs compared to the reference translation for WMT2014 en→hi using the xglm-source+target prompt. For 0-5 few-shot examples, N is the number of sentences identified as being in each language (the target language's row (correct) is indicated in green and the source language's row (one of the many incorrect options) in red) and ∆ is the length difference in number of characters (N.B. it is negative when the prediction is longer than the reference).
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
with the large BLOOM; for this language pair, we even observe a large variation across prompts (looking at the range of scores) in the 1-shot setting for all models.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
In this section, we replicate the analysis of Section 5.3 and report results per prompt with truncated outputs in Tables 16 and 17. The conclusions are overall consistent with what we report for non-truncated outputs in the main text. We note that after truncating the outputs, xglm-source+target yields very good results across the board, outperforming its closest contenders a good translation-source+target and version-target in almost all configurations. However, the choice of the prompt seems to matter more (a) in the zero-shot setting, (b) when translating out of English. Conversely our more stable results are for fr-en, 1-shot.
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
A striking observation reported in the main text (Section 5.4.1) is the difference between French and Spanish for the Flores-101 experiments. This is unexpected, as both languages are well represented in the training data. Yet, when translating from and into English the difference in spBLEU score is huge; and there is a clear gap with the other Romance languages as well. A related question is the poor translation between French and Spanish, not much better than for French→Arabic. Looking at some sample outputs, this seems to be due to the peculiarities of the Spanish translations, which appear to be less literal than their French counterparts, but which yield equally good translations into English. This can be seen when we compare translations back into English for these languages (see a random subset in Table 18). The last example illustrates this very clearly: we see "34 percent" in both the original English and in the translation from French, while translation from Spanish starts with "one third". Table 19 contains examples where the preceding context in 1-shot examples has a positive, negative or neutral influence on the current prediction, showing that the choice of the 1-shot example is important and is taken into account by the model.    