@inproceedings{jain-etal-2023-contraclm,
    title = "{C}ontra{CLM}: Contrastive Learning For Causal Language Model",
    author = "Jain, Nihal  and
      Zhang, Dejiao  and
      Ahmad, Wasi Uddin  and
      Wang, Zijian  and
      Nan, Feng  and
      Li, Xiaopeng  and
      Tan, Ming  and
      Nallapati, Ramesh  and
      Ray, Baishakhi  and
      Bhatia, Parminder  and
      Ma, Xiaofei  and
      Xiang, Bing",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.355",
    doi = "10.18653/v1/2023.acl-long.355",
    pages = "6436--6459",
    abstract = "Despite exciting progress in causal language models, the expressiveness of their representations is largely limited due to poor discrimination ability. To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level. We assess CONTRACLM on a variety of downstream tasks. We show that CONTRACLM enhances the discrimination of representations and bridges the gap with encoder-only models, which makes causal language models better suited for tasks beyond language generation. Specifically, we attain 44{\%} relative improvement on the Semantic Textual Similarity tasks and 34{\%} on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of representations, CONTRACLM also boosts the source code generation capability with 9{\%} relative improvement on execution accuracy on the HumanEval benchmark.",
}

Despite exciting progress in causal language models, the expressiveness of their representations is largely limited due to poor discrimination ability. To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level. We assess CONTRACLM on a variety of downstream tasks. We show that CONTRACLM enhances the discrimination of representations and bridges the gap with encoder-only models, which makes causal language models better suited for tasks beyond language generation. Specifically, we attain 44{\%} relative improvement on the Semantic Textual Similarity tasks and 34{\%} on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of representations, CONTRACLM also boosts the source code generation capability with 9{\%} relative improvement on execution accuracy on the HumanEval benchmark.
Jain, Nihal Zhang, Dejiao Ahmad, Wasi Uddin Wang, Zijian Nan, Feng Li, Xiaopeng Tan, Ming Nallapati, Ramesh Ray, Baishakhi Bhatia, Parminder Ma, Xiaofei Xiang, Bing
ContraCLM: Contrastive Learning For Causal Language Model