@inproceedings{li-etal-2023-toward,
    title = "Toward Interactive Dictation",
    author = "Li, Belinda Z.  and
      Eisner, Jason  and
      Pauls, Adam  and
      Thomson, Sam",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.854",
    doi = "10.18653/v1/2023.acl-long.854",
    pages = "15319--15338",
    abstract = "Voice dictation is an increasingly important text input modality. Existing systems that allow both dictation and editing-by-voice restrict their command language to flat templates invoked by trigger words. In this work, we study the feasibility of allowing users to interrupt their dictation with spoken editing commands in open-ended natural language. We introduce a new task and dataset, TERTiUS, to experiment with such systems. To support this flexibility in real-time, a system must incrementally segment and classify spans of speech as either dictation or command, and interpret the spans that are commands. We experiment with using large pre-trained language models to predict the edited text, or alternatively, to predict a small text-editing program. Experiments show a natural trade-off between model accuracy and latency: a smaller model achieves 30{\%} end-state accuracy with 1.3 seconds of latency, while a larger model achieves 55{\%} end-state accuracy with 7 seconds of latency.",
}

Voice dictation is an increasingly important text input modality. Existing systems that allow both dictation and editing-by-voice restrict their command language to flat templates invoked by trigger words. In this work, we study the feasibility of allowing users to interrupt their dictation with spoken editing commands in open-ended natural language. We introduce a new task and dataset, TERTiUS, to experiment with such systems. To support this flexibility in real-time, a system must incrementally segment and classify spans of speech as either dictation or command, and interpret the spans that are commands. We experiment with using large pre-trained language models to predict the edited text, or alternatively, to predict a small text-editing program. Experiments show a natural trade-off between model accuracy and latency: a smaller model achieves 30{\%} end-state accuracy with 1.3 seconds of latency, while a larger model achieves 55{\%} end-state accuracy with 7 seconds of latency.
Li, Belinda Z. Eisner, Jason Pauls, Adam Thomson, Sam
Toward Interactive Dictation