@inproceedings{gale-etal-2023-mixed,
    title = "Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers ({BORT})",
    author = "Gale, Robert  and
      Salem, Alexandra  and
      Fergadiotis, Gerasimos  and
      Bedrick, Steven",
    editor = "Can, Burcu  and
      Mozes, Maximilian  and
      Cahyawijaya, Samuel  and
      Saphra, Naomi  and
      Kassner, Nora  and
      Ravfogel, Shauli  and
      Ravichander, Abhilasha  and
      Zhao, Chen  and
      Augenstein, Isabelle  and
      Rogers, Anna  and
      Cho, Kyunghyun  and
      Grefenstette, Edward  and
      Voita, Lena",
    booktitle = "Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.repl4nlp-1.18",
    doi = "10.18653/v1/2023.repl4nlp-1.18",
    pages = "212--225",
    abstract = "Speech language pathologists rely on information spanning the layers of language, often drawing from multiple layers (e.g. phonology {\&} semantics) at once. Recent innovations in large language models (LLMs) have been shown to build powerful representations for many complex language structures, especially syntax and semantics, unlocking the potential of large datasets through self-supervised learning techniques. However, these datasets are overwhelmingly orthographic, favoring writing systems like the English alphabet, a natural but phonetically imprecise choice. Meanwhile, LLM support for the international phonetic alphabet (IPA) ranges from poor to absent. Further, LLMs encode text at a word- or near-word level, and pre-training tasks have little to gain from phonetic/phonemic representations. In this paper, we introduce BORT, an LLM for mixed orthography/IPA meant to overcome these limitations. To this end, we extend the pre-training of an existing LLM with our own self-supervised pronunciation tasks. We then fine-tune for a clinical task that requires simultaneous phonological and semantic analysis. For an {``}easy{''} and {``}hard{''} version of these tasks, we show that fine-tuning from our models is more accurate by a relative 24{\%} and 29{\%}, and improved on character error rates by a relative 75{\%} and 31{\%}, respectively, than those starting from the original model.",
}

Speech language pathologists rely on information spanning the layers of language, often drawing from multiple layers (e.g. phonology {\&} semantics) at once. Recent innovations in large language models (LLMs) have been shown to build powerful representations for many complex language structures, especially syntax and semantics, unlocking the potential of large datasets through self-supervised learning techniques. However, these datasets are overwhelmingly orthographic, favoring writing systems like the English alphabet, a natural but phonetically imprecise choice. Meanwhile, LLM support for the international phonetic alphabet (IPA) ranges from poor to absent. Further, LLMs encode text at a word- or near-word level, and pre-training tasks have little to gain from phonetic/phonemic representations. In this paper, we introduce BORT, an LLM for mixed orthography/IPA meant to overcome these limitations. To this end, we extend the pre-training of an existing LLM with our own self-supervised pronunciation tasks. We then fine-tune for a clinical task that requires simultaneous phonological and semantic analysis. For an {``}easy{''} and {``}hard{''} version of these tasks, we show that fine-tuning from our models is more accurate by a relative 24{\%} and 29{\%}, and improved on character error rates by a relative 75{\%} and 31{\%}, respectively, than those starting from the original model.
Gale, Robert Salem, Alex ra Fergadiotis, Gerasimos Bedrick, Steven
Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers (BORT)