@inproceedings{zhang-etal-2023-mixce,
    title = "{M}ix{CE}: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies",
    author = "Zhang, Shiyue  and
      Wu, Shijie  and
      Irsoy, Ozan  and
      Lu, Steven  and
      Bansal, Mohit  and
      Dredze, Mark  and
      Rosenberg, David",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.502",
    doi = "10.18653/v1/2023.acl-long.502",
    pages = "9027--9050",
    abstract = "Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P {--} that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE). We have observed that models trained in this way may {``}over-generalize{''}, in the sense that they produce non-human-like text. Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P relative to Q, is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MixCE, an objective that mixes the forward and reverse cross-entropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies.",
}

Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P {--} that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE). We have observed that models trained in this way may {``}over-generalize{''}, in the sense that they produce non-human-like text. Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P relative to Q, is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MixCE, an objective that mixes the forward and reverse cross-entropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies.
Zhang, Shiyue Wu, Shijie Irsoy, Ozan Lu, Steven Bansal, Mohit Dredze, Mark Rosenberg, David
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies