@inproceedings{xu-etal-2023-introducing,
    title = "Introducing Semantics into Speech Encoders",
    author = "Xu, Derek  and
      Dong, Shuyan  and
      Wang, Changhan  and
      Kim, Suyoun  and
      Lin, Zhaojiang  and
      Liu, Bing  and
      Shrivastava, Akshat  and
      Li, Shang-Wen  and
      Tseng, Liang-Hsuan  and
      Lin, Guan-Ting  and
      Baevski, Alexei  and
      Lee, Hung-yi  and
      Sun, Yizhou  and
      Wang, Wei",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.639",
    doi = "10.18653/v1/2023.acl-long.639",
    pages = "11413--11429",
    abstract = "Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding (SLU) performance by over 5{\%} on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) FF1 score by over 2{\%}. Our approach, which uses no ASR data, achieves similar performance as methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.",
}

Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding (SLU) performance by over 5{\%} on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) FF1 score by over 2{\%}. Our approach, which uses no ASR data, achieves similar performance as methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.
Xu, Derek Dong, Shuyan Wang, Changhan Kim, Suyoun Lin, Zhaojiang Liu, Bing Shrivastava, Akshat Li, Shang-Wen Tseng, Liang-Hsuan Lin, Guan-Ting Baevski, Alexei Lee, Hung-yi Sun, Yizhou Wang, Wei
Introducing Semantics into Speech Encoders