@inproceedings{adolphs-etal-2023-cringe,
    title = "The {CRINGE} Loss: Learning what language not to model",
    author = "Adolphs, Leonard  and
      Gao, Tianyu  and
      Xu, Jing  and
      Shuster, Kurt  and
      Sukhbaatar, Sainbayar  and
      Weston, Jason",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.493",
    doi = "10.18653/v1/2023.acl-long.493",
    pages = "8854--8874",
    abstract = "Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data {--} examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the {``}CRINGE{''} loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.",
}

Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data {--} examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the {``}CRINGE{''} loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.
Adolphs, Leonard Gao, Tianyu Xu, Jing Shuster, Kurt Sukhbaatar, Sainbayar Weston, Jason
The CRINGE Loss: Learning what language not to model