@inproceedings{li-etal-2023-dual,
    title = "Dual-Alignment Pre-training for Cross-lingual Sentence Embedding",
    author = "Li, Ziheng  and
      Huang, Shaohan  and
      Zhang, Zihan  and
      Deng, Zhi-Hong  and
      Lou, Qiang  and
      Huang, Haizhen  and
      Jiao, Jian  and
      Wei, Furu  and
      Deng, Weiwei  and
      Zhang, Qi",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.191",
    doi = "10.18653/v1/2023.acl-long.191",
    pages = "3466--3478",
    abstract = "Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at \url{https://github.com/ChillingDream/DAP}.",
}

Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at \url{https://github.com/ChillingDream/DAP}.
Li, Ziheng Huang, Shaohan Zhang, Zihan Deng, Zhi-Hong Lou, Qiang Huang, Haizhen Jiao, Jian Wei, Furu Deng, Weiwei Zhang, Qi
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding