@inproceedings{shon-etal-2023-slue,
    title = "{SLUE} Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks",
    author = "Shon, Suwon  and
      Arora, Siddhant  and
      Lin, Chyi-Jiunn  and
      Pasad, Ankita  and
      Wu, Felix  and
      Sharma, Roshan  and
      Wu, Wei-Lun  and
      Lee, Hung-yi  and
      Livescu, Karen  and
      Watanabe, Shinji",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.496",
    doi = "10.18653/v1/2023.acl-long.496",
    pages = "8906--8937",
    abstract = "Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will release a new benchmark suite, including for each task (i) curated annotations for a relatively small fine-tuning set, (ii) reproducible pipeline (speech recognizer + text model) and end-to-end baseline models and evaluation metrics, (iii) baseline model performance in various types of systems for easy comparisons. We present the details of data collection and annotation and the performance of the baseline models. We also analyze the sensitivity of pipeline models{'} performance to the speech recognition accuracy, using more than 20 publicly availablespeech recognition models.",
}

Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will release a new benchmark suite, including for each task (i) curated annotations for a relatively small fine-tuning set, (ii) reproducible pipeline (speech recognizer + text model) and end-to-end baseline models and evaluation metrics, (iii) baseline model performance in various types of systems for easy comparisons. We present the details of data collection and annotation and the performance of the baseline models. We also analyze the sensitivity of pipeline models{'} performance to the speech recognition accuracy, using more than 20 publicly availablespeech recognition models.
Shon, Suwon Arora, Siddhant Lin, Chyi-Jiunn Pasad, Ankita Wu, Felix Sharma, Roshan Wu, Wei-Lun Lee, Hung-yi Livescu, Karen Watanabe, Shinji
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks