@inproceedings{qiu-etal-2023-sccs,
    title = "{SCCS}: Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment",
    author = "Qiu, Jielin  and
      Zhu, Jiacheng  and
      Xu, Mengdi  and
      Dernoncourt, Franck  and
      Bui, Trung  and
      Wang, Zhaowen  and
      Li, Bo  and
      Zhao, Ding  and
      Jin, Hailin",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.101",
    doi = "10.18653/v1/2023.findings-acl.101",
    pages = "1584--1601",
    abstract = "Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics with video/document. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. Our method first decomposes both videos and articles into segments in order to capture the structural semantics, and then follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three MSMO datasets, and achieved performance improvement by 8{\%} {\&} 6{\%} of textual and 6.6{\%} {\&}5.7{\%} of video summarization, respectively, which demonstrated the effectiveness of our method in producing high-quality multimodal summaries.",
}

Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics with video/document. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. Our method first decomposes both videos and articles into segments in order to capture the structural semantics, and then follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three MSMO datasets, and achieved performance improvement by 8{\%} {\&} 6{\%} of textual and 6.6{\%} {\&}5.7{\%} of video summarization, respectively, which demonstrated the effectiveness of our method in producing high-quality multimodal summaries.
Qiu, Jielin Zhu, Jiacheng Xu, Mengdi Dernoncourt, Franck Bui, Trung Wang, Zhaowen Li, Bo Zhao, Ding Jin, Hailin
SCCS: Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment