@inproceedings{li-etal-2023-multi,
    title = "Multi-modal Action Chain Abductive Reasoning",
    author = "Li, Mengze  and
      Wang, Tianbao  and
      Xu, Jiahe  and
      Han, Kairong  and
      Zhang, Shengyu  and
      Zhao, Zhou  and
      Miao, Jiaxu  and
      Zhang, Wenqiao  and
      Pu, Shiliang  and
      Wu, Fei",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.254",
    doi = "10.18653/v1/2023.acl-long.254",
    pages = "4617--4628",
    abstract = "\textit{Abductive Reasoning}, has long been considered to be at the core ability of humans, which enables us to infer the most plausible explanation of incomplete known phenomena in daily life. However, such critical reasoning capability is rarely investigated for contemporary AI systems under such limited observations. To facilitate this research community, this paper sheds new light on \textit{Abductive Reasoning} by studying a new vision-language task, \textbf{M}ulti-modal \textbf{A}ction chain abductive \textbf{R}easoning (\textbf{MAR}), together with a large-scale \textit{Abductive Reasoning} dataset: Given an incomplete set of language described events, MAR aims to imagine the most plausible event by spatio-temporal grounding in past video and then infer the hypothesis of subsequent action chain that can best explain the language premise. To solve this task, we propose a strong baseline model that realizes MAR from two perspectives: (i) we first introduce the transformer, which learns to encode the observation to imagine the plausible event with explicitly interpretable event grounding in the video based on the commonsense knowledge recognition ability. (ii) To complete the assumption of a follow-up action chain, we design a novel symbolic module that can complete strict derivation of the progressive action chain layer by layer. We conducted extensive experiments on the proposed dataset, and the experimental study shows that the proposed model significantly outperforms existing video-language models in terms of effectiveness on our newly created MAR dataset.",
}

\textit{Abductive Reasoning}, has long been considered to be at the core ability of humans, which enables us to infer the most plausible explanation of incomplete known phenomena in daily life. However, such critical reasoning capability is rarely investigated for contemporary AI systems under such limited observations. To facilitate this research community, this paper sheds new light on \textit{Abductive Reasoning} by studying a new vision-language task, \textbf{M}ulti-modal \textbf{A}ction chain abductive \textbf{R}easoning (\textbf{MAR}), together with a large-scale \textit{Abductive Reasoning} dataset: Given an incomplete set of language described events, MAR aims to imagine the most plausible event by spatio-temporal grounding in past video and then infer the hypothesis of subsequent action chain that can best explain the language premise. To solve this task, we propose a strong baseline model that realizes MAR from two perspectives: (i) we first introduce the transformer, which learns to encode the observation to imagine the plausible event with explicitly interpretable event grounding in the video based on the commonsense knowledge recognition ability. (ii) To complete the assumption of a follow-up action chain, we design a novel symbolic module that can complete strict derivation of the progressive action chain layer by layer. We conducted extensive experiments on the proposed dataset, and the experimental study shows that the proposed model significantly outperforms existing video-language models in terms of effectiveness on our newly created MAR dataset.
Li, Mengze Wang, Tianbao Xu, Jiahe Han, Kairong Zhang, Shengyu Zhao, Zhou Miao, Jiaxu Zhang, Wenqiao Pu, Shiliang Wu, Fei
Multi-modal Action Chain Abductive Reasoning