@inproceedings{li-etal-2023-qap,
    title = "{QAP}: A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition",
    author = "Li, Ziming  and
      Zhou, Yan  and
      Liu, Yaxin  and
      Zhu, Fuqing  and
      Yang, Chuanpeng  and
      Hu, Songlin",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.772",
    doi = "10.18653/v1/2023.findings-acl.772",
    pages = "12191--12204",
    abstract = "Multimodal emotion recognition for video has gained considerable attention in recent years, in which three modalities (\textit{i.e.,} textual, visual and acoustic) are involved. Due to the diverse levels of informational content related to emotion, three modalities typically possess varying degrees of contribution to emotion recognition. More seriously, there might be inconsistencies between the emotion of individual modality and the video. The challenges mentioned above are caused by the inherent uncertainty of emotion. Inspired by the recent advances of quantum theory in modeling uncertainty, we make an initial attempt to design a quantum-inspired adaptive-priority-learning model (QAP) to address the challenges. Specifically, the quantum state is introduced to model modal features, which allows each modality to retain all emotional tendencies until the final classification. Additionally, we design Q-attention to orderly integrate three modalities, and then QAP learns modal priority adaptively so that modalities can provide different amounts of information based on priority. Experimental results on the IEMOCAP and MOSEI datasets show that QAP establishes new state-of-the-art results.",
}

Multimodal emotion recognition for video has gained considerable attention in recent years, in which three modalities (\textit{i.e.,} textual, visual and acoustic) are involved. Due to the diverse levels of informational content related to emotion, three modalities typically possess varying degrees of contribution to emotion recognition. More seriously, there might be inconsistencies between the emotion of individual modality and the video. The challenges mentioned above are caused by the inherent uncertainty of emotion. Inspired by the recent advances of quantum theory in modeling uncertainty, we make an initial attempt to design a quantum-inspired adaptive-priority-learning model (QAP) to address the challenges. Specifically, the quantum state is introduced to model modal features, which allows each modality to retain all emotional tendencies until the final classification. Additionally, we design Q-attention to orderly integrate three modalities, and then QAP learns modal priority adaptively so that modalities can provide different amounts of information based on priority. Experimental results on the IEMOCAP and MOSEI datasets show that QAP establishes new state-of-the-art results.
Li, Ziming Zhou, Yan Liu, Yaxin Zhu, Fuqing Yang, Chuanpeng Hu, Songlin
QAP: A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition