@inproceedings{du-etal-2023-measure,
    title = "A Measure-Theoretic Characterization of Tight Language Models",
    author = "Du, Li  and
      Torroba Hennigen, Lucas  and
      Pimentel, Tiago  and
      Meister, Clara  and
      Eisner, Jason  and
      Cotterell, Ryan",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.543",
    doi = "10.18653/v1/2023.acl-long.543",
    pages = "9744--9770",
    abstract = "Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 over all finite strings. However, in some pathological cases, probability mass can {``}leak{''} onto the set of infinite sequences. In order to characterize the notion of leakage more precisely, this paper offers a measure-theoretic treatment of language modeling. We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense. We also generalize characterizations of tightness proposed in previous works.",
}

Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 over all finite strings. However, in some pathological cases, probability mass can {``}leak{''} onto the set of infinite sequences. In order to characterize the notion of leakage more precisely, this paper offers a measure-theoretic treatment of language modeling. We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense. We also generalize characterizations of tightness proposed in previous works.
Du, Li Torroba Hennigen, Lucas Pimentel, Tiago Meister, Clara Eisner, Jason Cotterell, Ryan
A Measure-Theoretic Characterization of Tight Language Models