- Roshan Sharma, B. Raj. 2022. Cross-utterance context for multimodal video transcription. Abstract: In multimodal speech transcription of videos, audio and video are segmented using the concept of utterances, and transcription is performed independently on these segments. However, for real-world video transcription, multimodal information from past utterances can be leveraged to contextualize and improve video transcription. In this work, we first build multimodal speech recognition systems on instructional YouTube videos using the How2 corpus. We examine different visual representations and multimodal fusion techniques to infuse visual information into audio-only models. Then we explore methods to embed past context by training on longer input sequences. Experiments on the How2- 300h data demonstrate the importance of multi-modality and long-term context, which result in a 1 % WER absolute improvement over audio-only speech recognizers.
