@inproceedings{gheini-etal-2023-joint,
    title = "Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data",
    author = "Gheini, Mozhdeh  and
      Likhomanenko, Tatiana  and
      Sperber, Matthias  and
      Setiawan, Hendra",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.483",
    doi = "10.18653/v1/2023.findings-acl.483",
    pages = "7637--7650",
    abstract = "Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient parallel data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing in this way results in additional gains on top of the vanilla pseudo-labeling setup providing a total improvement of up to 0.4{\%} absolute WER and 2.1 BLEU points for En{--}De and 0.6{\%} absolute WER and 2.2 BLEU points for En{--}Zh.",
}

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient parallel data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing in this way results in additional gains on top of the vanilla pseudo-labeling setup providing a total improvement of up to 0.4{\%} absolute WER and 2.1 BLEU points for En{--}De and 0.6{\%} absolute WER and 2.2 BLEU points for En{--}Zh.
Gheini, Mozhdeh Likhomanenko, Tatiana Sperber, Matthias Setiawan, Hendra
Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data