LSTM  machine translation text
5033497
Phrase Based Neural Unsupervised Machine Translation
Machine translation systems achieve near human level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This work investigates how to learn to translate when having access to only large monolingual corpora in each language. We propose two model variants, a neural and a phrase based model. Both versions leverage a careful initialization of the parameters, the denoising effect of language models and automatic generation of parallel data by iterative back translation. These models are significantly better than methods from the literature, while being simpler and having fewer hyper parameters. On the widely used WMT'14 English French and WMT'16 German English benchmarks, our models respectively obtain 28.1 and 25.2 BLEU points without using a single parallel sentence, outperforming the state of the art by more than 11 BLEU points. On low resource languages like English Urdu and English Romanian, our methods achieve even better results than semi supervised and supervised approaches leveraging the paucity of available bitexts. Our code for NMT and PBSMT is publicly available.
EMNLP
Guillaume Lample Myle Ott Alexis Conneau Ludovic Denoyer Marc'Aurelio Ranzato
19169084
Asynchronous Bidirectional Decoding for Neural Machine Translation
The dominant neural machine translation (NMT) models apply unified attentional encoder decoder neural networks for translation. Traditionally, the NMT decoders adopt recurrent neural networks (RNNs) to perform translation in a left toright manner, leaving the target side contexts generated from right to left unexploited during translation. In this paper, we equip the conventional attentional encoder decoder NMT framework with a backward decoder, in order to explore bidirectional decoding for NMT. Attending to the hidden state sequence produced by the encoder, our backward decoder first learns to generate the target side hidden state sequence from right to left. Then, the forward decoder performs translation in the forward direction, while in each translation prediction timestep, it simultaneously applies two attention models to consider the source side and reverse target side hidden states, respectively. With this new architecture, our model is able to fully exploit source and target side contexts to improve translation quality altogether. Experimental results on NIST Chinese English and WMT English German translation tasks demonstrate that our model achieves substantial improvements over the conventional NMT by 3.14 and 1.38 BLEU points, respectively. The source code of this work can be obtained from this https URL.
AAAI
Xiangwen Zhang Jinsong Su Yue Qin Yang P Liu R Ji Hongji Wang
44172616
Dense Information Flow for Neural Machine Translation
Recently, neural machine translation has achieved remarkable progress by introducing well designed deep neural networks into its encoder decoder framework. From the optimization perspective, residual connections are adopted to improve learning performance for both encoder and decoder in most of these deep architectures, and advanced attention connections are applied as well. Inspired by the success of the DenseNet model in computer vision problems, in this paper, we propose a densely connected NMT architecture (DenseNMT) that is able to train more efficiently for NMT. The proposed DenseNMT not only allows dense connection in creating new features for both encoder and decoder, but also uses the dense attention structure to improve attention quality. Our experiments on multiple datasets show that DenseNMT structure is more competitive and efficient.
NAACL
Yanyao Shen Xu Tan Di He Tao Qin Tie-Yan Liu
76654723
Getting Gender Right in Neural Machine Translation
Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996) One such difference is related to the way gender is expressed in a language. Saying "I am happy" in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either "Je suis heureux" for a male speaker or "Je suis heureuse" for a female one. Apart from morphological agreement, demographic factors (gender, age, etc. also influence our use of language in terms of word choices or syntactic constructions (Tannen, 1991; Pennebaker et al. 2003) We integrate gender information into NMT systems. Our contribution is two fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.
EMNLP
Eva Vanmassenhove Christian Hardmeier Andy Way
53247198
Findings of the 2018 Conference on Machine Translation (WMT18)
This paper presents the results of the premier shared task organized alongside the Conference on Machine Translation (WMT) 2018. Participants were asked to build machine translation systems for any of 7 language pairs in both directions, to be evaluated on a test set of news stories. The main metric for this task is human judgment of translation quality. This year, we also opened up the task to additional test sets to probe specific aspects of translation.
WMT
Ondrej Bojar Christian Federmann Mark Fishel Yvette Graham Barry Haddow Philipp Koehn Christof Monz
44062236
Context Aware Neural Machine Translation Learns Anaphora Resolution
Standard machine translation systems process sentences in isolation and hence ignore extra sentential information, even though extended context can both prevent mistakes in ambiguous cases and improve translation coherence. We introduce a context aware neural machine translation model designed in such way that the flow of information from the extended context to the translation model can be controlled and analyzed. We experiment with an English Russian subtitles dataset, and observe that much of what is captured by our model deals with improving pronoun translation. We measure correspondences between induced attention distributions and coreference relations and observe that the model implicitly captures anaphora. It is consistent with gains for sentences where pronouns need to be gendered in translation. Beside improvements in anaphoric cases, the model also improves in overall BLEU, both over its context agnostic version +0.7) and over simple concatenation of the context and source sentences +0.6)
ACL
Elena Voita Pavel Serdyukov Rico Sennrich Ivan Titov
3957063
Fine grained attention mechanism for neural machine translation
Neural machine translation (NMT) has been a new paradigm in machine translation, and the attention mechanism has become the dominant approach with the state of the art records in many language pairs. While there are variants of the attention mechanism, all of them use only temporal attention where one scalar value is assigned to one context vector corresponding to a source word. In this paper, we propose a fine grained (or 2D) attention mechanism where each dimension of a context vector will receive a separate attention score. In experiments with the task of En De and En Fi translation, the fine grained attention method improves the translation quality in terms of BLEU score. In addition, our alignment analysis reveals how the fine grained attention mechanism exploits the internal structure of context vectors.
Neurocomputing
Heeyoul Choi Kyunghyun Cho Yoshua Bengio
52074346
Training Deeper Neural Machine Translation Models with Transparent Attention
While current state of the art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2 3x) deeper Transformer and Bi RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7 1.1 BLEU on the benchmark WMT'14 English German and WMT'15 Czech English tasks for both architectures.
EMNLP
Ankur Bapna Mia Xu Chen Orhan Firat Yuan Cao Yonghui Wu
53249630
Findings of the Third Shared Task on Multimodal Machine Translation
We present the results from the third shared task on multimodal machine translation. In this task a source sentence in English is supplemented by an image and participating systems are required to generate a translation for such a sentence into German, French or Czech. The image can be used in addition to (or instead of) the source sentence. This year the task was extended with a third target language (Czech) and a new test set. In addition, a variant of this task was introduced with its own test set where the source sentence is given in multiple languages: English, French and German, and participating systems are required to generate a translation in Czech. Seven teams submitted 45 different systems to the two variants of the task. Compared to last year, the performance of the multimodal submissions improved, but text only systems remain competitive.
WMT
Loic Barrault Fethi Bougares Lucia Specia Chiraag Lala Desmond Elliott Stella Frank
49413369
On Adversarial Examples for Character Level Neural Machine Translation
Evaluating on adversarial examples has become a standard procedure to measure robustness of deep learning models. Due to the difficulty of creating white box adversarial examples for discrete text input, most analyses of the robustness of NLP models have been done through black box adversarial examples. We investigate adversarial examples for character level neural machine translation (NMT) and contrast black box adversaries with a novel white box adversary, which employs differentiable string edit operations to rank adversarial changes. We propose two novel types of attacks which aim to remove or change a word in a translation, rather than simply break the NMT. We demonstrate that white box adversarial examples are significantly stronger than their black box counterparts in different attack scenarios, which show more serious vulnerabilities than previously known. In addition, after performing adversarial training, which takes only 3 times longer than regular training, we can improve the model's robustness significantly.
COLING
J Ebrahimi Daniel Lowd Dejing Dou