@inproceedings{wang-etal-2023-listen,
    title = "Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition",
    author = "Wang, Liming  and
      Ni, Junrui  and
      Gao, Heting  and
      Li, Jialu  and
      Chang, Kai Chieh  and
      Fan, Xulin  and
      Wu, Junkai  and
      Hasegawa-Johnson, Mark  and
      Yoo, Chang",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.424",
    doi = "10.18653/v1/2023.findings-acl.424",
    pages = "6785--6800",
    abstract = "Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2sign-U, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50{\%} recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at \url{https://github.com/cactuswiththoughts/UnsupSpeech2Sign.gitcactuswiththoughts/UnsupSpeech2Sign.git}.",
}

Existing supervised sign language recognition systems rely on an abundance of well-annotated data. Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and sign-language corpora. We propose speech2sign-U, a neural network-based approach capable of both character-level and word-level SSR-U. Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50{\%} recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability. The code is available at \url{https://github.com/cactuswiththoughts/UnsupSpeech2Sign.gitcactuswiththoughts/UnsupSpeech2Sign.git}.
Wang, Liming Ni, Junrui Gao, Heting Li, Jialu Chang, Kai Chieh Fan, Xulin Wu, Junkai Hasegawa-Johnson, Mark Yoo, Chang
Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition