@inproceedings{xu-etal-2023-bridging,
    title = "Bridging the Granularity Gap for Acoustic Modeling",
    author = "Xu, Chen  and
      Zhang, Yuhao  and
      Jiao, Chengbo  and
      Liu, Xiaoqian  and
      Hu, Chi  and
      Zeng, Xin  and
      Xiao, Tong  and
      Ma, Anxiang  and
      Wang, Huizhen  and
      Zhu, Jingbo",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.688",
    doi = "10.18653/v1/2023.findings-acl.688",
    pages = "10816--10833",
    abstract = "While Transformer has become the de-facto standard for speech, modeling upon the fine-grained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights. We propose Progressive Down-Sampling (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like text-level representation. In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression. In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task. And as a bonus, it yields inference speedups ranging from 1.20x to 1.47x.By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task.",
}

While Transformer has become the de-facto standard for speech, modeling upon the fine-grained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights. We propose Progressive Down-Sampling (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like text-level representation. In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression. In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task. And as a bonus, it yields inference speedups ranging from 1.20x to 1.47x.By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task.
Xu, Chen Zhang, Yuhao Jiao, Chengbo Liu, Xiaoqian Hu, Chi Zeng, Xin Xiao, Tong Ma, Anxiang Wang, Huizhen Zhu, Jingbo
Bridging the Granularity Gap for Acoustic Modeling