Summarize this abstract: While BERT is an effective method for learn-
ing monolingual sentence embeddings for se-
mantic similarity NAME_1 embedding based trans-
fer learning (NAME_2 NAME_1 NAME_3, 2019),
BERT based cross-lingual sentence embed-
dings have yet to be explored. We sys-
tematically investigate methods for learning
multilingual sentence embeddings by combin-
ing the best methods for learning monolin-
gual NAME_1 cross-lingual representations includ-
ing: masked language modeling (MLM), trans-
lation language modeling (TLM) (NAME_4
NAME_1 NAME_5, 2019), dual encoder translation
ranking (NAME_6 et al., 2018), NAME_1 additive margin
softmax (NAME_7 et al., 2019a). We show that in-
troducing a pre-trained multilingual language
model dramatically reduces the amount of par-
allel training data required to achieve good
performance by 80%. Composing the best of
these methods produces a model that achieves
83.7% bi-text retrieval accuracy over 112 lan-
guages on Tatoeba, well above the 65.5%
achieved by Artetxe NAME_1 NAME_8 (2019b),
while still performing competitively on mono-
lingual transfer learning benchmarks (Con-
NAME_9 NAME_1 NAME_10, 2018). Parallel data mined
from CommonCrawl using our best model is
shown to train competitive NMT models for
en-zh NAME_1 en-de. We publicly release our best
multilingual sentence embedding model for
109+ languages at https://tfhub.dev/
google/LaBSE.