How are features of text and images fused in multimodal summarization models?
Multimodal Summarization Tasks. In the field
of multimodal summarization, there are usually
three steps. First, different feature extractor mod-
ules are adopted to extract the features of the text
and the image, respectively. Second, the different
features are fused at the fusion layer. Finally, the
fused context features are fed into the text decoder
to generate a summary. Existing studies focus on
multimodal fusion. Specifically, the fusion meth-
ods consist of concatenate, attention-based, and
gate-based. The concatenate fusion directly con-
catenates multimodal features into a fusion context
(Li et al., 2018b, 2020a). It can fully extract high-
level features of different modalities, but there is a
large gap between high-dimensional spaces. The
attention-based methods fuse all multimodal fea-
tures with attention mechanism (Atri et al., 2021;
Palaskar et al., 2019; Kitada et al., 2022), which
can get the correlations between each unit of text
and image. Gate-based methods take text as the
central modality (Jangra et al., 2021) and exploit
images to help focus on the core information (Liu
et al., 2020; Li et al., 2020b). In summary, (1) all
fusion methods do not explicitly model the image
complementarity for the summary, which leads to
inefficient use of the image. (2) concatenate and
attention-based cannot eliminate the influence of
useless images in the fusion layer
Multimodal Sentence Summarization via Multimodal Selective Encoding
This paper studies the problem of generating a summary for a given sentence-image pair. Existing multimodal sequence-to-sequence approaches mainly focus on enhancing the decoder by visual signals, while ignoring that the image can improve the ability of the encoder to identify highlights of a news event or a document. Thus, we propose a multimodal selective gate network that considers reciprocal relationships between textual and multi-level visual features, including global image descriptor, activation grids, and object proposals, to select highlights of the event when encoding the source sentence. In addition, we introduce a modality regularization to encourage the summary to capture the highlights embedded in the image more accurately. To verify the generalization of our model, we adopt the multimodal selective gate to the text-based decoder and multimodal-based decoder. Experimental results on a public multimodal sentence summarization dataset demonstrate the advantage of our models over baselines. Further analysis suggests that our proposed multimodal selective gate network can effectively select important information in the input sentence.
[{"first":"Haoran","last":"Li","middle":[]},{"first":"Junnan","last":"Zhu","middle":[]},{"first":"Jiajun","last":"Zhang","middle":[]},{"first":"Xiaodong","last":"He","middle":[]},{"first":"Chengqing","last":"Zong","middle":[]}]
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
In this paper, we introduce a multi-modal sentence summarization task that produces a short summary from a pair of sentence and image. This task is more challenging than sentence summarization. It not only needs to effectively incorporate visual features into standard text summarization framework, but also requires to avoid noise of image. To this end, we propose a modality-based attention mechanism to pay different attention to image patches and text units, and we design image ﬁlters to selectively use visual information to enhance the semantics of the input sentence. We construct a multi-modal sentence summarization dataset and extensive experiments on this dataset demonstrate that our models signiﬁcantly outperform conventional models which only employ text as input. Further analyses suggest that sentence summarization task can beneﬁt from visually grounded representations from a variety of aspects.
[{"first":"Haoran","last":"Li","middle":[]},{"first":"Junnan","last":"Zhu","middle":[]},{"first":"Tianshang","last":"Liu","middle":[]},{"first":"Jiajun","last":"Zhang","middle":[]},{"first":"Chengqing","last":"Zong","middle":[]}]
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In recent years, abstractive text summarization with multimodal inputs has started drawing attention due to its ability to accumulate information from different source modalities and generate a fluent textual summary. However, existing methods use short videos as the visual modality and short summary as the ground-truth, therefore, perform poorly on lengthy videos and long ground-truth summary. Additionally, there exists no benchmark dataset to generalize this task on videos of varying lengths. In this paper, we introduce AVIATE, the first large-scale dataset for abstractive text summarization with videos of diverse duration, compiled from presentations in well-known academic conferences like NDSS, ICML, NeurIPS, etc. We use the abstract of corresponding research papers as the reference summaries, which ensure adequate quality and uniformity of the ground-truth. We then propose FLORAL, a factorized multi-modal Transformer based decoder-only language model, which inherently captures the intra-modal and inter-modal dynamics within various input modalities for the text summarization task. FLORAL utilizes an increasing number of self-attentions to capture multimodality and performs significantly better than traditional encoder-decoder based networks. Extensive experiments illustrate that FLORAL achieves significant improvement over the baselines in both qualitative and quantitative evaluations on the existing How2 dataset for short videos and newly introduced AVIATE dataset for videos with diverse duration, beating the best baseline on the two datasets by $1.39$ and $2.74$ ROUGE-L points respectively.
[{"first":"Yash","last":"Atri","middle":["Kumar"]},{"first":"Shraman","last":"Pramanick","middle":[]},{"first":"Vikram","last":"Goyal","middle":[]},{"first":"Tanmoy","last":"Chakraborty","middle":[]}]
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
There is increasing interest in the use of multimodal data in various web applications, such as digital advertising and e-commerce. Typical methods for extracting important information from multimodal data rely on a mid-fusion architecture that combines the feature representations from multiple encoders. However, as the number of modalities increases, several potential problems with the mid-fusion model structure arise, such as an increase in the dimensionality of the concatenated multimodal features and missing modalities. To address these problems, we propose a new concept that considers multimodal inputs as a set of sequences, namely, deep multimodal sequence sets (DM$^2$S$^2$). Our set-aware concept consists of three components that capture the relationships among multiple modalities: (a) a BERT-based encoder to handle the inter- and intra-order of elements in the sequences, (b) intra-modality residual attention (IntraMRA) to capture the importance of the elements in a modality, and (c) inter-modality residual attention (InterMRA) to enhance the importance of elements with modality-level granularity further. Our concept exhibits performance that is comparable to or better than the previous set-aware models. Furthermore, we demonstrate that the visualization of the learned InterMRA and IntraMRA weights can provide an interpretation of the prediction results.
[{"first":"Shunsuke","last":"Kitada","middle":[]},{"first":"Yuki","last":"Iwazaki","middle":[]},{"first":"Riku","last":"Togashi","middle":[]},{"first":"Hitoshi","last":"Iyatomi","middle":[]}]
Multimodal Abstractive Summarization for How2 Videos
In this paper, we study abstractive summarization for open-domain videos. Unlike the traditional text news summarization, the goal is less to “compress” text information but rather to provide a fluent textual summary of information that has been collected and fused from different source modalities, in our case video and audio transcripts (or text). We show how a multi-source sequence-to-sequence model with hierarchical attention can integrate information from different modalities into a coherent output, compare various models trained with different modalities and present pilot experiments on the How2 corpus of instructional videos. We also propose a new evaluation metric (Content F1) for abstractive summarization task that measures semantic adequacy rather than fluency of the summaries, which is covered by metrics like ROUGE and BLEU.
[{"first":"Shruti","last":"Palaskar","middle":[]},{"first":"Jindřich","last":"Libovický","middle":[]},{"first":"Spandana","last":"Gella","middle":[]},{"first":"Florian","last":"Metze","middle":[]}]
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Multimodal summarization for open-domain videos is an emerging task, aiming to generate a summary from multisource information (video, audio, transcript). Despite the success of recent multiencoder-decoder frameworks on this task, existing methods lack ﬁne-grained multimodality interactions of multi-source inputs. Besides, unlike other multi-modal tasks, this task has longer multimodal sequences with more redundancy and noise. To address these two issues, we propose a multistage fusion network with the fusion forget gate module, which builds upon this approach by modeling ﬁne-grained interactions between the multisource modalities through a multistep fusion schema and controlling the ﬂow of redundant information between multimodal long sequences via a forgetting module. Experimental results on the How2 dataset show that our proposed model achieves a new state-of-the-art performance. Comprehensive analysis empirically veriﬁes the effectiveness of our fusion schema and forgetting module on multiple encoder-decoder architectures. Spe-cially, when using high noise ASR transcripts ( WER> 30% ), our model still achieves performance close to the ground-truth transcript model, which reduces manual annotation cost.
[{"first":"Nayu","last":"Liu","middle":[]},{"first":"Xian","last":"Sun","middle":[]},{"first":"Hongfeng","last":"Yu","middle":[]},{"first":"Wenkai","last":"Zhang","middle":[]},{"first":"Guangluan","last":"Xu","middle":[]}]
Multimodal Sentence Summarization via Multimodal Selective Encoding
Text summarization is a task that condenses a long sentence to a short version. Existing researches (Rush et al., 2015;Chopra et al., 2016;Zeng et al., 2016;Tan et al., 2017;Zhou et al., 2017;Li et al., 2020b) produce summary only from the text. However, it has been proved that human understands the text relying on multimodal information (Waltz, 1980;Srihari, 1994;He and Deng, 2017), such as linguistic and visual signals. In this paper, we focus on the multimodal summarization task (Li et al., 2018a) that generates summary simultaneously drawing knowledge from coupled text and image, which can facilitate other applications such as image captioning Vinyals et al., 2015), multimodal news summarization (Narayan et al., 2017;Chen and Zhuge, 2018), and electronic commerce (e-commerce) product description generation (Chen et al., 2019;Elad et al., 2019;Li et al., 2020a), etc.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Intuitively, it is easier for a reader to grasp the highlight of a news event by viewing an image than by reading a long text. Hence we believe that the image will benefit text summarization system. Figure 1 illustrates this phenomenon. For a given source sentence, a paired image visualizes a set of event highlight words, which highly correlates with the reference summary.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Multimodal sequence-to-sequence (seq2seq) learning has been widely explored in machine translation (MT) Caglayan et al., 2017;Helcl et al., 2018;Grönroos et al., 2018) in recent years, and the performances of their models surpass text-only models (Barrault et al., 2018). The main difference between multimodal MT and multimodal text summarization is: the model for MT is required to convert the same semantics from the input sentence and the paired image to the output, while for summarization, the model is expected to select the important information from the input. Li et al. (2018a) propose a hierarchical attention model for the multimodal sentence summarization task, while the image is not involved in the process of text encoding. Obviously, it will be easier for the decoder to generate Source sentence: british prime minister tony blair has rescued a danish swimmer from the shark infested sea during his holiday in seychelles in africa , a government spokesman said friday . Reference summary: uk pm rescues swimmer from sea . an accurate summary if the encoder can filter out trivial information when encoding the input sentence. Based on this idea, we propose a multimodal selective mechanism which aims to select the highlights from the input text using visual signals, and then the decoder generates the summary using the filtered encoding information. Concretely, an encoder reads the input text and generates the hidden representations. Then, multimodal selective gates measure the relevance between the input words and the image to construct the selected hidden representation. Finally, a decoder generates the summary using the selected hidden representation. We explore the capacity of the image to select salient content from the input text at different levels including global image descriptor, activation grids, and object proposals in the image. Accordingly, as shown in Figure 2, we design three visual selective gates: global-level, grid-level, and object-level gates. Furthermore, some abstract concepts, such as "guilty" and "freedom", can hardly be well represented by the image, and thus we combine textual and visual selective gates to construct multimodal selective gates. In addition, we argue that a good summarizer should adequately capture the highlight words. Thus, we impose a modality regularization on our model that encourages the semantic similarity between (image, generated summary) higher than that between (image, source text).
Multimodal Sentence Summarization via Multimodal Selective Encoding
Our main contributions are as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
• We propose a novel multimodal selective mechanism that can use both the textual and visual signals to select the important information from the source text.
Multimodal Sentence Summarization via Multimodal Selective Encoding
• We propose a visual-guided modality regularization module to encourage the model focus on the key information in the source.
Multimodal Sentence Summarization via Multimodal Selective Encoding
• The experimental results on a multimodal sentence summarization dataset demonstrate that our proposed system can take advantage of multimodal information and outperform baseline methods.
Multimodal Sentence Summarization via Multimodal Selective Encoding
2 Related Work 2.1 Abstractive Sentence Summarization Rush et al. (2015) first propose a seq2seq model to generate the summary for a sentence. Chopra et al. (2016) and    ..  Figure 2: The framework of our model. We design visual selective gates including global-level, gridlevel and object-level visual gates to select salient encoding information. We also integrate the textual and visual selective gates to construct multimodal selective gates. In this figure, the summary is generated with the text-based decoder, and we also apply multimodal selective gates to the multimodal-based decoder (Li et al., 2018a) in our work.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Libovický and Helcl (2017) propose multi-source seq2seq learning with hierarchical attention.  use images as source words to improve translation quality. Delbrouck and Dupont (2017) adjust various attention for the visual modality.  propose attention mechanisms for textual and visual modalities and combine them to decode target words. Narayan et al. (2017) develop extractive summarization with side information including images and captions. , Chen and Zhuge (2018) and Zhu et al. (2020) propose to generate multimodal summary for multimodal news document. Li et al. (2018a) first introduce the multimodal sentence summarization task, and they propose a hierarchical attention model, which can pay different attention to image patches, words, and different modalities when decoding target words. Li et al. (2020a) propose a aspect-aware multimodal summarization model for e-commerce products. We use visual signals to enhance the encoder for the multimodal sentence summarization task, aiming to discriminate the important source information when encoding the input sentence.
Multimodal Sentence Summarization via Multimodal Selective Encoding
The input of the multimodal text summarization task is a pair of text and image, and the output is a textual summary. We propose visual selective gates to encourage important source information encoded into the second-level hidden sequence. We argue that the text can be pertinent to visual information at the level of the whole image, the image parts, and the object proposals in the image. Thus, as shown in Figure 2, we design three visual selective gates: global-level, grid-level, and object-level gates. Next, the summary decoder produces the summary based on the second-level hidden states. We apply two types of decoders including the text-based decoder and multimodal-based decoder.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Given a source text x = (x 1 , · · · , x n ), an encoder builds the first-level hidden state sequence h = (h 1 , · · · , h n ). We apply a Bidirectional GRU to encode the source sentence forwardly and backwardly into two sequences of the hidden states:
Multimodal Sentence Summarization via Multimodal Selective Encoding
Given an image, we apply a pre-trained VGG (VGG19) and a Faster R-CNN (Ren et al., 2017) model to extract visual features at multiple levels including the whole image, the image parts, and the object proposals in the image. We extract the 4096-dimensional fully-connected layer of VGG19 (fc7 layer) as the image global feature v g . We extract a 7 × 7 × 512 feature map of the last pooling layer (pool5 layer) of VGG19 as the image grid-level feature v r . For object proposal features v o of an image, we use Faster R- CNN (Ren et al., 2017) initialized with ResNet-101 (He et al., 2016) pretrained for classification on ImageNet (Deng et al., 2009), and then we retrain it on Visual Genome Dataset (Krishna et al., 2017). v o j ∈ R 2048 is obtained from the ROI pooling layer in the Region Proposal Network. We choose the top 36 object proposals after non-maximum suppression (Neubeck and Gool, 2006).
Multimodal Sentence Summarization via Multimodal Selective Encoding
In this section, we briefly describe the selective mechanism (Zhou et al., 2017). The selective encoding (Zhou et al., 2017) extends the seq2seq model by constructing a second-level hidden state h i to control the information flow from the encoder to the decoder as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
where σ denotes the sigmoid function, W and U are parameter matrices. r is the overall representation for the text x.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Then, h i is computed as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
Finally, the decoder generates the summary depending on the second-level hidden sequence h as a standard seq2seq model does.
Multimodal Sentence Summarization via Multimodal Selective Encoding
We design three kinds of visual selective gates to select salient encoding information: global-level, gridlevel, and object-level gates. We further combine textual and visual selective gates to construct multimodal selective gates.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Global-Level Visual Selective Gate To conduct a global-level visual selective gate sGate g , we explore the association between text hidden state h i and global visual feature v g :
Multimodal Sentence Summarization via Multimodal Selective Encoding
Grid-level image features retain the spatial information of an image, which can be used to bridge the gap between text and image patch-by-patch as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
where v r j ∈ R 512 (j ∈ [1, 49]) is a grid-level image feature. Specifically, for each h i , we compute grid-level visual selective gate with each grid-level image patch v r j , and then we apply a max-pooling over the gates to take the maximum value of each dimension as the final selective gate for h i . The idea behind this is that different dimensions of h i represent different semantics, which correspond to different image patches, and the max operation can capture the most related image regions for each h i .
Multimodal Sentence Summarization via Multimodal Selective Encoding
Object-Level Visual Selective Gate Grid-level image features correspond to a uniform grid of equally sized and shaped neural receptive fields for the images. In fact, salient image region, such as object proposal, is a much more natural basis for human attention (Egly et al., 1994). Thus, we compute the selective gate based on object-level image features as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
where v o j ∈ R 2048 (j ∈ [1, 36]) is an object-level image feature.
Multimodal Sentence Summarization via Multimodal Selective Encoding
To avoid missing necessary information which can hardly be well represented by the image, such as abstract concepts, we combine textual and visual selective gates to construct multimodal selective gates.
Multimodal Sentence Summarization via Multimodal Selective Encoding
To jointly select important encoding information by linguistics and different levels of visual signals, we propose three kinds of multimodal selective gates as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
is the overall representation for the source text. Then, the second-level encoding states h i is computed by one of the sGate ∈ {sGate g , sGate r , sGate o , sGate tg , sGate tr , sGate to } as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
Next, the decoder generates summaries based on h .
Multimodal Sentence Summarization via Multimodal Selective Encoding
To verify the generalization of our model, we apply multimodal selective gates to both the text-based decoder and multimodal-based decoder.
Multimodal Sentence Summarization via Multimodal Selective Encoding
The decoder for a standard text-only seq2seq model calculates the decoding state s t as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
where
Multimodal Sentence Summarization via Multimodal Selective Encoding
. Context vector c t is computed as a weighted sum of the source annotations as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
The probability for the next target word y t is computed using the decoder state s t and the previous emitted word y t−1 as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
The loss function L t for each time t is as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
Multimodal-Based Summary Decoder Li et al. (2018a) propose a multimodal-based summarization method. In this work, we adopt the summary decoder with weighted image local features initialization, and the hierarchical attention mechanism which can pay different attention to the input words and image patches, producing textual and visual context vectors. The textual context vector c txt is calculated with Equation 11. The image context vector c img t is as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
The second-level modality attention is:
Multimodal Sentence Summarization via Multimodal Selective Encoding
where β txt t is attention weight for textual context and β img t is attention weight for visual. The probability for the next target word y t is computed based on Equation 10 and 13.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Given a source text, we argue that the paired image, especially for the foreground of an image, contains more centralized information that should be presented in the summary. To guarantee the summarizer can better capture the highlights expressed by the image, we attempt to constrain the image more related to the generated summary than to the source. To achieve this goal, we design a Modality Regularization (MR) module that scores (image, target) higher than (image, source).
Multimodal Sentence Summarization via Multimodal Selective Encoding
We first learn representations for the source and summary by a cross-modal attention, which emphasize the words related to the image:
Multimodal Sentence Summarization via Multimodal Selective Encoding
Then we project the source, the target, and the image into the shared space: e src = tanh(W h c src ), e tgt = tanh(W s c tgt ), e v = tanh(U v v g ). A pairwise ranking loss function to encourage similarity score between (image, summary) higher than (image, source):
Multimodal Sentence Summarization via Multimodal Selective Encoding
L r = max{0, δ − cos(e tgt , e v ) + cos(e src , e v )}
Multimodal Sentence Summarization via Multimodal Selective Encoding
where δ = 2, and cos is a cosine similarity function. The final loss function is:
Multimodal Sentence Summarization via Multimodal Selective Encoding
In addition, inspired by Zhou et al. (2018a), we calculate s 0 with e src as follows:
Multimodal Sentence Summarization via Multimodal Selective Encoding
4 Experiments
Multimodal Sentence Summarization via Multimodal Selective Encoding
We evaluated our methods on the public multimodal sentence summarization dataset (Li et al., 2018a). Each sample in this dataset is a triplet (sentence, image, summary). The dataset consists of 62,000 training samples, 2,000 test samples and 2,000 validation samples.
Multimodal Sentence Summarization via Multimodal Selective Encoding
We set word embedding size to 300 and GRU hidden state size to 512. We use the full source and target vocabularies collected from the training data, which have 36,916 source words and 26,168 target words, respectively. The mini-batch size is 64, and beam search size is 10. Adam optimizer is applied with the learning rate of 0.0005, momentum parameters β 1 = 0.9 and β 1 = 0.999, and = 10 −8 . We use dropout (Srivastava et al., 2014) with probability of 0.2 and gradient clipping (Pascanu et al., 2013) with range [−1, 1]. During training, we test the ROUGE-2 (Lin, 2004) F1-score on the validation set for every 5,000 batches, and we halve the learning rate if the score drops for 5 consecutive testings.  Table 1: Experimental results of our models with the text-based decoder. "T-Selective", "V-Selective" and "MR" denote Textual Selective, Visual Selective, and Modality Regularization, respectively. Our "T + Grid V-Selective + MR" model performs significantly better than the baselines by the 95% confidence interval in the official ROUGE script. Best result among different "V-Selective" is underlined.
Multimodal Sentence Summarization via Multimodal Selective Encoding
Lead. A baseline uses the first eight words as the summary according to the reference length. Compress. Clarke (2008) propose to compress sentence based on syntactic structure. ABS. Rush et al. (2015) use an attentive CNN encoder and a neural network language model decoder to summarize the sentence. SEASS. Zhou et al. (2017) propose a summarization model with textual selective encoding. Multi-Source. Libovický and Helcl (2017) propose a hierarchical attention model. Doubly-Attentive.  propose a doubly-attentive mechanism for multimodal machine translation. Seq2seq. Bahdanau et al. (2015) propose a standard seq2seq model for machine translation. 
Multimodal Sentence Summarization via Multimodal Selective Encoding
We report ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) F1-scores 1 . We conduct the experiments with different initializations for eight times, and we report the mean and standard deviation. Table 1 shows the results for text-based decoders with different selective encoding mechanisms and modality regularization (MR). Generally, our models with visual selective gates perform better than the model with textual selective gate, and global-level feature constructs the most effective visual selective gate, while the object-level feature performs worst. We argue that some background regions, such as "sea" in Figure 1 and "highway" in Figure 3, are also necessary for our task, while object proposals may ignore these background. Recognition errors caused by Region Proposal Network may also influence the summarization performance. Multimodal Selective gates lead to further improvements, and the model with "Textual + Grid-level Visual Selective" achieves the highest ROUGE score. We hypothesize the reason may be that similar to the global-level visual selective gate, the textual gate is also a global-level selective gate. Thus, compared with the global-level visual selective gate, the local-level grid-level visual gate contribute more to the textual gate. In addition, we test with combining all of three different levels of visual information and textual information to compute the selective gate, while the result is a little worse than that of "Textual + Grid-level Visual Selective". Table 2 show comparisons with other work. We can conclude that Multimodal Selective (Textual + Grid-Level Visual Selective, TGS) and Modality Regularization (MR) strategies exhibit good generalization ability for both the text-based decoder (Seq2seq) and multimodal-based decoder (MAtt), and the model "MAtt + PGNet + TGSMR" outperforms other baselines.  Table 2: Comparisons with other work. "TGSMR" denotes "Textual + Grid-Level Visual Selective + Modality Regularization". The model of "MAtt + PGNet + TGSMR" performs significantly better than other baselines by the 95% confidence interval in the official ROUGE script. "*" marks the results from Li et al. (2018a).
Multimodal Sentence Summarization via Multimodal Selective Encoding
Precision Recall F1-score  Table 3: Results (%, averaged precision, recall and F1-score) for keyword extraction.
Multimodal Sentence Summarization via Multimodal Selective Encoding
In this section, we explore whether the selective gates can correctly discriminate the important words from the input. To this end, we evaluate how many activated words by the selective gates are groundtruth keywords (we take the overlapping words, except for stop-words, between the input sentence and the reference summary as the ground-truth keywords). Following Zhou et al. (2017), we use the method of  to calculate the contribution of the selective gates to the final summary. Considering that the average word count of the summaries in the validation set is eight, we take the words with top-8 contribution values as the activated keywords by the selective mechanisms. Table 3 shows the results for keyword extraction. Our models with selective encoding perform better than unsupervised TextRank algorithm (Mihalcea and Tarau, 2004) (we also take the top-8 scored words as the keywords), and Multimodal Selective show advantages over Textual Selective. We further train a BiLSTM-CRF model (Huang et al., 2015) using sentence-keyword samples of the multimodal sentence summarization dataset, and the keyword extraction result for BiLSTM-CRF is better than our model, indicating a promising prospect for further development of the selective mechanism. In the future, we will dedicate our efforts to explore whether the selective gate can benefit from supervision signals of a special keyword extractor. A feasible research direction may be to explore whether the selective gate can benefit from supervision signals of a special keyword extractor.
Multimodal Sentence Summarization via Multimodal Selective Encoding
We perform human evaluation with "Seq2seq + TGSMR" to avoid the influence of other modules such as multimodal decoder (MAtt) and copying (PGNet). We randomly choose 100 samples from the test set, and three postgraduates are involved to annotate whether "Seq2seq + TGSMR" outperforms "Seq2seq" model with respect to the informativeness and readability. The results are shown in Table 4. For example, "win" denotes that the results of "Seq2seq + TGSMR" is better than the "Seq2seq" model. We can see  Table 4: Human evaluation results (%). "Win" denotes that the generated summaries of "Seq2seq + TGSMR" is better than "Seq2seq" model. that the percentage of "win" is larger than "lose" (45.33% vs. 13.33% for informativeness, and 45% vs. 13.67% for readability. Kappa (Fleiss, 1971) is used to verified the consistency for different annotators.), demonstrating that the improvements of our multimodal summarizer are significant (p-value < 0.01, paired t-test).
Multimodal Sentence Summarization via Multimodal Selective Encoding
We show a sample from the test set, with comparisons of the reference summary and the summaries generated by the Seq2seq model, the text-based decoder with "Textual Selective", and "Textual + Gridlevel Visual Selective". In Figure 3, the image shows the "highway" where the event occurs, and "Textual + Grid-level visual selective" model successfully spells out the place while other models fail.
Multimodal Sentence Summarization via Multimodal Selective Encoding
We visualize the gate values in a salience heat map for each source word. We observe that our multimodal selective gates accurately determine the salience of the source words, while textual selective gates fail, especially for the word "highway". We also show visual selective gate values for "Textual + Gridlevel Visual Selective" model in Figure 3 (b) for the target word "highway", which demonstrates that our visual grounded selective mechanism successfully captures the relationship between the image and the word that has corresponding visual semantics.  . Note that we simplify the input sentence to show the heat map.
Multimodal Sentence Summarization via Multimodal Selective Encoding
This paper addresses the multimodal text summarization task, namely, transforming a pair of text and image into a summary. We propose multimodal selective encoding gates that can effectively control the information flow from the encoder to the decoder. We adopt our multimodal selective encoding strategies to the text-based and multimodal-based decoders. The experimental results on the public multimodal sentence summarization corpus prove that our proposed model significantly outperforms other comparative methods. Note that our method is not specifically designed for RNN-based models, and it can be easily applied to Transformer-based models, which is left for our future work.
Multimodal Sentence Summarization via Multimodal Selective Encoding
This work is partially supported by National Key R&D Program of China (2020AAA0105200), National Key R&D Program of China (2018YFB2100802), and Beijing Academy of Artificial Intelligence (BAAI).
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Sentence summarization is a well-studied task that creates a condensed version of a long sentence. Sequence-to-sequence (seq2seq) model that encodes a source sequence into a latent representation and outputs another sequence is the dominating framework for sentence summarization [Rush et al., 2015;Takase et al., 2016;Li et al., 2017b]. Intuitively, readers can easier grasp the gist of the event by scanning the image than by reading long sentences, and thus we believe that the image will also reduce the difficulty for machine to understand a news event [Li et al., 2017a]. As shown in Figure 1, the two source sentences contain complex details and multiple event objects, leading to unsatisfactory summaries using the text-only model. However, it is easy to see that the paired images visualize the event highlights which can help to produce better summaries. To explore the effectiveness of images, we introduce in this work a new multimodal sentence summarization (MMSS) task that generates a short summary based on a pair of sentence and image.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Source sentence: a house explosion rocked a neighborhood in eastern maryland , killing a gas utility worker and injuring four residents and ## firefighters . Reference summary: house explosion in maryland kills gas worker injures ## Text-only model: gas explosion in us kills gas explosion Multi-modal model: house explosion rocks maryland killing ## Source sentence: the flood death toll in southern malaysia has risen to ## , an official said thursday . Reference summary: flood death toll rises to ## in southern malaysia Text-only model: southern malaysia death toll rises to ## Multi-modal model: death toll from heavy floods rises to ## Multi-modal approaches are explored in machine translation (MT) recently and outperform text-only models [Calixto et al., 2017;Helcl and Libovický, 2017;Caglayan et al., ]. The multi-modal MT requires that the paired image should describe the whole information covered by the source sentence and this requirement is hard to meet when the sentence is very long. In contrast to multi-modal MT, images which usually indicate event highlights are more suitable for summarization that aims at abstracting the important information from the original text.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
To tackle the MMSS task, we design a novel seq2seq model with hierarchical attention mechanisms. The bottom attention layer focuses on the inner parts of image and sentence. The top attention layer balances the distribution of different modalities. Since some abstract concepts, such as guilty and freedom, can only be well encoded in textual modality [Paivio, 1990;Hill et al., 2013;Kiela et al., 2014], we design image filter modules to block out visual noises. In addition, we incorporate different visual features into our model to initialize target language decoder and present a visual coverage mechanism to handle the word repetition problem.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Our main contributions are as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
• We introduce an MMSS task that can generate a textual summary from a pair of sentence and image.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
• We propose an inner-modality and inter-modality atten-Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence  tion model to focus on image patches or text units for summary generation.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
• We propose image filters to selectively use visual information to benefit summarization.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
• We construct a multi-modal sentence summarization corpus that consists of 66,000 summary triples (sentence, image, summary). The experimental results on this dataset demonstrate that our proposed system can take advantage of multi-modal information and outperform other baseline methods.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
2 Background: Seq2seq Learning
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
In this section, we describe the basic seq2seq learning framework. Given a source sequence x, the seq2seq model maximizes the conditional probability of a target sequence y: p(y|x). The encoder and the decoder are the two basic components in the seq2seq model. Recurrent neural networks (RNN) encoder ] converts x into a context representation c as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where h t ∈ R n is a hidden state at time t, and c t is a context vector generated from the sequence of the hidden states. f enc and f c are nonlinear activation functions. The decoder generates word y t given the context vector c t and the previously generated words {y 1 , · · · , y t−1 }:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where s t is the hidden state of the decoder and f dec is a nonlinear activation function that computes the probability vector for output words at time t. The loss function L t for each time t is the negative log likelihood of the target word y t :
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
L t = −log p(y t |{y 1 , · · · , y t−1 }, c t ) (4)
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
We begin by defining the multi-modal sentence summarization task. The input of the task is a pair of sentence and image, and the output is a condensed summary. As shown in Figure 2, our proposed model consists of four modules: sentence encoder, image encoder, summary decoder and image filter. The sentence encoder is a bidirectional GRU (Bi-GRU) [Chung et al., 2014], and pre-trained CNN (Oxford VGGnet [Simonyan and Zisserman, 2014]) is used to encode images. Our summary decoder is a uni-directional GRU with a hierarchical attention mechanism and a softmax layer over the target vocabulary to generate words. Specifically, beyond the attention over regions of the image and words of the input sentence, we explore the modality-based attention to navigate our model to pay different attention to textual and visual information when generating different words. We design two types of image filters, namely, image attention filter and image context filter, to remove visual noises and enrich the semantic meaning of the corresponding sentence.  
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Given a source sentence x = (x 1 , · · · , x n ), we employ a BiGRU to build its hidden representation (h 1 , · · · , h n ).
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
The BiGRU encodes source sentence forwardly and backwardly to generate two sequences of the hidden states:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
E[x i ] is the embedding for word x i . The final sentence representation h i is the concatenation of the forward and backward vectors:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Given an image, we apply a pre-trained VGG (VGG19) model to extract visual features. We extract a 7 × 7 × 512 feature map of the last pooling layer (pool5 layer) of the network as the local feature, and we extract the 4096-dimensional fullyconnected layer (fc7 layer) as the global feature q. We flatten the local patch feature into matrix A = (a 1 , · · · , a L ) (L = 49) and a l ∈ R 512 corresponds to a patch of the image.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
At each time step t, the state of the decoder s t is calculated as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
In text-only seq2seq model, the decoder's hidden state s 0 is initialized as follow:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence  Intuitively, s 0 is important because the first few words to be decoded significantly influence the remainder of the generated summary. We incorporate three different visual features into our model to initialize target language decoder: image global features (Equation 9), the average of image local features (Equation 10) and weighted image local features (Equation 11).
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where α i is the weight scalar for a i :
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where σ denotes sigmoid activation function. The underlying intuition for Equation 11 is that different image patches do not make equal contributions aligning to the source sentence; for instance, background regions often contain trivial information, which would have less contribution to the model than foreground regions. To do so, the image patch and the sentence are matched to determine informative patches. Context vector c t in Equation 7 is obtained by a modalitybased attention mechanism as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where β txt t is attention weight for text context vector c txt and β img t is attention weight for image context vector c img , respectively, calculated as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
We compute the text context vector c txt as a weighted sum of the source annotations as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where each vector is weighted by the attention weight α txt t,i , as calculated in Equations 17 and 18 as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
We compute the image context vector c img as a weighted sum of the source image patch representation A using the "soft" attention strategy [Xu et al., 2015]. This process is very similar to the text attention strategy assuming that decoder needs to attend to different image patches to generate different words.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where each visual vector a l is weighted by the attention weight α img t,i , as calculated in Equations 20 and 21 as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
The probability for the next target word y t is computed using the multi-modal hidden state s t , and the previously emitted word y t−1 as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
where L y , L s and L e are model parameters.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
In addition, to solve the problem of repetition, we adopt the textual coverage mechanism [See et al., 2017] which aims to avoid generating repetitive target words by penalizing repetitive attention to the same position of the source sequence. For coverage model, the attention distributions over all previous decoder time are maintained, which record the attention history for all source words. Then the decoder is not expected to pay current attention to the words which have received enough attention before. For our task, we apply a multi-modal coverage mechanism including textual and visual coverage.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
In our multi-modal coverage model, we maintain two coverage vector c txt t,i and c img t,i , which is the sum of attention distributions over all previous decoder steps:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Intuitively, c txt t,i and c img t,i denote the degree of coverage that the source words and the image patches have been covered by the target before time t, respectively. Note that c txt 0,i and c img 0,i are zero vectors, because at that step, none of the source word or the image patch is covered by the target. The coverage vectors are used to calculate the attention mechanism, changing Equation 17 and 20 to:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
The coverage losses for text and image at time t are defined as follows respectively:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Finally, we add the multi-modal coverage loss to the original loss function in Equation 4 to yield new loss functions as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18)
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Compared to text summarization task, the key point for multimodal summarization is effectively making use of image. Meanwhile, we should avoid introducing noises in the case that (1) the image fails to represent some semantic meanings of words, such as abstract concepts, or (2) the image is insufficiently exact to capture the keypoint of the source sentence.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
To this end, we introduce an image filtering mechanism and propose two types of image filters, i.e., an image attention filter and an image context filter.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
The image attention filter is a scalar applied to the image attention β img t (in Equation 15), which aims to re-balance the attention of the image and sentence based on the following aspects: (1) The correlation between the source image and sentence, which is measured by the image global feature q and the initial state of summary decoder s 0 . (2) The relation to the next target word, which relates to the state of the decoder s t−1 . The image attention filter I a ∈ [0, 1] is calculated as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Then, the image attention β img t in Equation 15 is updated as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Xu et al. [2015] and Calixto et al. [2017] adopt gating scalars to image context. The difference is that our scalar not only relates to the current decoder state for generating the next target word but also associate different modalities, which aims to evaluate the effect of the image.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
The image context filter is an element-wise filtering vector applied to the image context c img (in Equation 19) that is expected to filter out the image noises inside the image context. The image context filter is calculated as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
The image context in Equation 19 is updated as follows:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Image context filter is partially inspired by gating mechanism which has gained great popularity in neural network models [Hochreiter and Schmidhuber, 1997;Srivastava et al., 2015], and this filter is expected to give fine tune for each dimension, and the information can only flow on the dimensions where the gate is "open". In conclusion, image attention filter is directly applied to change the attention scale between image and text, and image context filter is designed to select the most salient visual features. 
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Multi-modal MT has recently attracted much attention from researchers. Libovický and Helcl [2017] propose multisource seq2seq learning with hierarchical attention combination. Calixto and Liu [2017] use images as words in the source sentences to improve translation quality. They use visual representation to initialize the encoder or decoder. Our work is partially inspired by the models of Libovický and Helcl [2017] and Calixto et al. [2017] with some differences:
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
(1) Libovický et al. [2017] use the hierarchical attention but do not pay attention to image patches. In addition, they do not apply image filters. (2) Calixto et al. [2017] propose two separate attention mechanisms to decode target words, while our model can hierarchically pay attention to source words, image patches and different modalities. (3) Calixto et al. [2017] apply a gating scalar which only relates to decoder state, but our image filters not only relate to the current decoder state for generating the next target word but also consider the correlations between different modalities, which is peculiar for sentence summarization tasks. Our motivation is that (1) Different image patches relate to different words and image filters enable our model to selectively use the image. (2) Hierarchical attention mechanism can decide how much attention the decoder pay to different modalities when generate different words. (3) Our image filters explore association between textual and global visual information, which can measure the extent to which the image can capture the gist of the source sentence.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
To the best of our knowledge, there is no benchmark dataset for the MMSS task, and we construct a corpus. Each sample in our corpus is a triple (sentence, image, headline) in which the sentence-headline pair is from the annotated Gigaword corpus [Rush et al., 2015] 1 and the image is crawled from Yahoo! Image Search. The Gigaword corpus provides 3.8 million first-sentence-headline pairs. For each first-sentence, we search Yahoo! Image Search, and crawl the top-5 ranked images. Next, we delete the explicit trivial images such as portraits, thumbnails and advertisements.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
As a result, we collect 123,839 sentences paired with five images each. Then we employ 10 graduate students to select the best-match image for each sentence. If there is no matching image for the sentence, the annotators label '0'. Each five-image-sentence sample is annotated by two students and the samples with accordance annotations are preserved. Finally, we collect 66,000 samples. Following the settings of the Gigaword summarization corpus, we randomly split our corpus as a training set with 62,000 samples, a test set with 2,000 samples and a development set with 2,000 samples.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
We compare a set of sentence summarization and multimodal MT baselines. Lead baseline uses the first 8 words as the summary. Compress model [Clarke and Lapata, 2008] produces a compressed result based on the syntactic structure of the original sentence. ABS [Rush et al., 2015] uses an attentive CNN encoder and a neural network language model decoder to summarize the sentence. SEASS is a state-of-theart sentence summarization systems, which employs a selective encoding model to control the information flow from the encoder to the decoder. Multi-Source [Libovický and Helcl, 2017] and Doubly-Attentive [Calixto et al., 2017] are multimodal MT baselines (introduced in Section 4). Text-only model is a seq2seq model with attention only using a sentence to generate summary. We report performances of our multi-modal (MM) models with different decoder initializations, image attention filter, image context filter, and coverage for different modalities.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
We set word embedding size to 300 and GRU hidden state sizes to 512. We use the full source and target vocabularies collected from the training data, which have 36,916 and 26,168 words, respectively. We use dropout [Srivastava et al., 2014] with probability of 0.2. We set the initial learning rate for Adam [Kingma and Ba, 2015] to 5 × 10 −4 . At training time, we test ROUGE-2 [Lin, 2004] F1 score on the development set for every 2,000 batches, and we halve the learning rate if model performance drops. Our models typically converge within 50 epochs using an early stopping strategy for our MM model without coverage. To obtain our final coverage model, we add the coverage loss to the objective function for further training with the initial learning 5 × 10 −5 . At test time, we use beam search with beam size 10 to generate the summary. We report ROUGE F1 score including ROUGE-1 (R-1), ROUGE-2 (R-2) and ROUGE-L (R-L). Table 1 shows the results of the comparative methods and our proposed methods. The state-of-the-art sentence summarization systems SEASS performs better than our Text-only model. For our MM models, the model without an image filter performs only slightly better than the text-only model, and the models with image filters outperform the text-only model and other baseline models. This proves that image filter plays a significant role in the MMSS task. Initializing decoder with image features is effective for our task. Compared to global and average of local visual features, the models with weighted average of local visual feature achieve greater improvements over the text-only model, which suggests that different image patches do not make equal contributions for our task. Coverage mechanism leads to further improvement, which achieves +2.70% R-1, +2.17% R-2, +2.57% R-L improvement over the Text-only model) for our best model. In addition, we have tested an image captioning model [Vinyals et al., 2015] and get poor results (R-1 is lower than 10%), which indicates that image captioning model is not suitable for this task.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
In this section, we discuss the effectiveness of visual information for sentence summarization task (results are reported on the test set). 1. Effectiveness of multi-modal initialization of decoder. From Table 1 we observe that incorporating images to initialize hidden state of decoder improves the performance. We conjecture that image features are effective as these features can capture the highlights of the source sentences. Table 2 shows that our image attention filter model with multimodal decoder initialization can predict the first words that appear in the reference summary in much higher accuracies than that with text-only initialization. Example summaries can be found in Figure 1. In part at least, this behavior can be ascribed to better initialization of decoder. Furthermore, we observe the model which predicts the first word correctly achieves 0.235 higher ROUGE-1 score on average than the model which predicts wrong in our experiment.   Table 3: The average count of correct novel n-grams (i.e., n-grams that don't appear in the source but appear in the reference summary).
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
2. Are images helpful to produce more abstractive summary? Abstractive summarization can produce novel words that do not appear in source sentence. This phenomenon is common for human-written summaries, but is challenging for a machine. We believe that attention mechanism allows a model to focus on the specific positions of source representation that will help to predict the next word, but it adds a constraint on the context vectors for summarization system, which tends to predict a word obtaining the greatest attention in the source. Table 3 shows that our multi-modal model is more abstractive than text-only model. This is expected because the attention to the image can provide abundant information beyond source text. An example is shown in Figure 3.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
3. Which kind of images are more useful?
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
To answer this question, we ask 3 graduate students to annotate the matching scores for 300 sentence-image pairs in the test set. The score ranges from 1 to 3: 1 denotes "partially match", 2 denotes "basically match" and 3 denotes "completely match". The results in Table 4 show that the samples with better matched images achieve higher ROUGE score and also attract higher attention for image, indicating that better matched images are more useful for the MMSS task.
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
4. Effectiveness of multi-modal coverage. To explicitly demonstrate how the repetition problem is eliminated by coverage mechanism, we calculate the average count of repeated words for each summary with coverage for different modalities. Figure 4 shows the results for (MM Decoder wconv ) model with different image filters, which indicates that textual and visual coverage model can reduce repeated words.  Figure 3: Multi-modal model successfully predicts the novel words (in blue and italic). 
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
This paper addresses a multi-modal sentence summarization task, namely, how to transform a sentence-image pair into a short-length summary. Our proposed model can simultaneously focus on image patches and text units to generate summaries with a modality-based attention mechanism. We design image filters to selectively use visual information to enrich the semantics for the source sentence. We introduce a visual coverage mechanism which prove that multi-modal coverage is effective for our task. We provide a publicly available multi-modal sentence summarization corpus and the experiments on this dataset show our proposed model significantly outperforms the baseline models. Our dataset is released to the public 2 .
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Abstractive text summarization focuses on generating a summary of a text from its main ideas, not by replicating the most salient sentences, rather by generating new sentences and/or rephrasing existing sentences so that the key semantics and overall meaning of the original text remain intact in summary. It has a wide range of applications in our daily life − from media monitoring, search marketing, email filtering, newsletter production to question-answering chatbots, we are frequently exposed to abstractive summarization. Due to its ability to generate fluent and coherent summaries, abstractive summarization is an important and challenging area of research in Natural Language Processing and has got enough attention in the last few years. While most of the previous studies (See et al., 2017;Lebanoff et al., 2018;Gehrmann et al., 2018;Chowdhury et al., 2020;Chung et al., 2020) on abstractive summarization use only textual data as the input modality, some studies (Shah et al., 2016;Palaskar et al., 2019) have recently focused on incorporating multimodal signals as the input to enhance the quality of text summary. Intuitively, humans can comprehend the gist of an occurrence more quickly by watching relevant images or videos than by only reading a text document, and therefore we believe that multimodal data can also reduce the difficulties for machines to interpret the context.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Motivation: With the emergence of multimedia technology and the rapid growth of social media video-sharing platforms such as Youtube and Vimeo, multimedia data (including text, image, audio, and video) have increased dramatically. Specifically, during the COVID-19 outbreak in the last six months, there has been a steep rise in various e-learning platforms, resulting in a drastic increase of online video tutorials and academic presentation videos. However, such videos often do not have the text meta-data associated with them, or the existing ones fail to capture the subtle differences with related videos (Wang et al., 2012). Additionally, different modalities in most of these videos are asynchronous with each other, leading to the unavailability of subtitles. In this work, we address the task of generating an abstractive text summary of a given academic presentation video so that the viewers can acquire the gist of the presentation in a short time, without watching the video from the beginning to the end. For this purpose, we incorporate automatic speech recognition (ASR) and optical character recognition (OCR) generated text transcripts and capture tonal-specific details of the speaker in addition to extracting semantics and sentics from the video, which are jointly optimized to produce a rich and informative textual summary of the entire presentation. We also show the generalizability of our model on the non-academic dataset (instructional videos).
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
State-of-the-art and Limitations: The existing studies on abstractive text summarization with multimodal signals include multimodal news summarization (Li et al., , 2016Chen and Zhuge, 2018) and summarization of instructional videos (Palaskar et al., 2019). However, all of them use images and/or short videos as the visual modality, which do not generalize on long videos. The generated summaries by these systems are also one or two lines long, and therefore, not suitable for longer academic videos (such as course lecture, conference tutorials). Some other closely related studies include image and video captioning (Mun et al., 2017;Iashin and Rahtu, 2020;Shen et al., 2020), video story generation (Gella et al., 2018), video title generation (Zeng et al., 2016) and multimodal sentence summarization ; but all of them deal with short videos or images which are not appropriate for our application. The lack of previous studies on this task can be attributed to the absence of a suitable benchmark dataset. In a very recent work, Palaskar et al. (2019) studied the task of summarization of instructional videos on the How2 dataset (Sanabria et al., 2018), which is the only existing dataset for abstractive text summarization with multimodality. However, the How2 dataset consists of short videos, with an average duration of 90 seconds only. The ground-truth text summaries of this dataset have an average length of 33 words, which are very small as well.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Our Contributions: In this paper, we explore the role of multimodality in abstractive text summarization for academic presentation videos of diverse duration and introduce a new resource to further enable research in this area. More specifically, our main contributions in this work are as follows:
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
1. We curate the first large-scale dataset, Audio VIdeo lAnguage daTasEt (AVIATE), for abstractive text summarization using multimodal inputs for academic presentation videos of diverse duration. To collect the videos for this dataset, we scraped 6 publicly available websites and accumulated paper presentation videos from 28 well-known international conferences in computer science and social science. To obtain the transcripts of these videos, we apply Deep Speech (Hannun et al., 2014), a pre-trained endto-end automatic speech recognition (ASR) system. We use the abstracts of corresponding research papers as the ground-truth summaries, which ensure adequate quality and uniformity. In contrast to How2, AVIATE contains longer videos and larger ground-truth summaries, which help the deep learning models trained on AVIATE to generalize the performance on other datasets.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
2. We introduce several baselines to show that multimodal frameworks are substantially more effective when compared to their unimodal variants for abstractive text summarization.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
3. We propose Factorized Multimodal Transformer based decoderonly Language Model (FLORAL), which uses an increasing number of self-attentions to inherently capture inter-modal and intra-modal dynamics within the asynchronous multimodal input sequences. FLORAL demonstrates the utility of pre-trained language model (LM) for summary generation in relatively low-resource setups over traditional encoder-decoder based networks.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
4. For the videos of AVIATE, we show the importance of OCR generated text transcripts, which contain keywords and informative phrases displayed on slides in academic presentations. To fuse ASR and OCR generated texts, we introduce a novel guided attention based fusion mechanism which attends the complementary features in both the sources and filters out repetitive and redundant words. After the incorporation of OCR transcript, the baselines and FLORAL yield [0.7 − 3.6] ROUGE-L points performance improvement on AVIATE.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
5. FLORAL reports benchmark results in terms of both automatic and manual evaluation metrics on How2 for short videos. It beats the best baseline by 1.39 ROUGE-L points. On AVIATE, FLORAL also turns out to be highly effective − it beats the best baseline by 2.74 ROUGE-L points.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
6. Finally, we report the transferability of FLORAL between How2 and AVIATE.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
When trained on AVIATE and tested on How2, FLORAL yields 49.9 ROUGE-L score, which is only 6.9 points less than ROUGE-L obtained when both trained and tested on How2. The diverse-length videos of AVIATE make the model transferable, which is tremendously effective for practical applications.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Reproducibility: To reproduce our results, we present detailed hyperparameter configurations in Table 1 and Section 5.1. Moreover, we also supplement our submission with full AVIATE dataset and source code of FLORAL 1 .
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
To enable the exploration of abstractive text summarization using multimodal signals and to generalize the task for videos of different lengths, we introduce AVIATE, the first large-scale multimodal text summarization dataset with videos of diverse duration, compiled from academic paper presentations. Currently, the only existing benchmark dataset relevant to our task is the How2 dataset (Sanabria et al., 2018), which includes short instructional videos on different topics like cooking, sports, indoor/outdoor activities, music, etc. Our study reveals that deep neural models trained on such short videos fail to produce satisfactory results on longer videos. Moreover, the facial expression of the speakers in academic talks and presentations often plays an important role to preserve the most informative frames, which is not always the case for the How2 dataset.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
The How2 dataset consists of 2, 000 hours of short instructional videos, where the training, validation, and test set contain 73, 993, 2, 965, and 2, 156 videos respectively, with an average length of 90 seconds. Each video in this dataset is accompanied by a human-generated transcript and a 2 − 3 sentence ground-truth summary. The average length of transcripts and summaries is 291 and 33 words respectively.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
To collect conference presentation videos, we identified 28 academic conferences in computer science and social science, spanning over various domains such as Machine Learning, Natural Language Processing, Data Mining, Computer Vision, Computational Linguistics, Semantic Web, and Complex Systems. Most of the videos of our dataset come from conferences like NDSS, ICML, NeurIPS, ACL, NAACL, CVPR, EMNLP, ISWC, KDD, etc. To collect the oral and spotlight presentations of these conferences, we scrapped six different academic online video repositories, namely Videolectures.NET 2 , ACL Anthology 3 , CVF Open Access 4 , ICML 5 , NeurIPS 6 , and NDSS Symposium 7 websites. All the paper presentation videos are accompanied by an abstract, which we use as the ground-truth summary. Thus, unlike the How2 dataset, we did not annotate the summaries ourselves, which significantly improved the quality of ground truth summaries, and hence of the entire dataset.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
AVIATE consists of a total of 8, 201 videos, which spans over almost 2, 300 hours. Among them, we use 6, 680 videos for training, 662 for validation, and 859 for testing. The length of summaries is mostly between 100 − 300 words, with an average of 168 words. A brief source and duration statistics of AVIATE is presented in Figure 1.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Transcription: Since we collected all the videos from six different sources, not all of them had subtitles or transcripts readily available. This is particularly the case for videos from ACL Anthology and Videolectures, which contribute the majority of the AVIATE dataset. In the case of videos from NDSS, ICML, NeurIPS, and CVF Open Access corpus, subtitles are available for those videos which are present on Youtube. To maintain uniformity in the quality of transcripts, we apply Deep Speech (Hannun et al., 2014), a pre-trained end-to-end speech recognition algorithm, to extract transcripts for all the videos. To ensure the quality of Deep Speech generated transcripts, we manually transcribe 300 randomly selected videos from our dataset. A low word error rate (24.19%) of the Deep Speech model for those videos indicates the satisfactory standard of the transcripts. An additional normalization step, which includes formatting 8 entities like numbers, dates, times, and addresses, helps us to further reduce the error rate to 20.12%. Figure 2 shows a comparison of video length and ground-truth summary length distribution for AVIATE and How2. For both datasets, longer videos 2 http://videolectures.net/ 3 https://www.aclweb.org/anthology/ 4 https://openaccess.thecvf.com/ 5 https://icml.cc/ 6 https://nips.cc/ 7 https://www.ndss-symposium.org/ 8 For example, labeling 'September 16, 2017' as 'september sixteenth twenty seventeen'.  generally have a longer ground-truth summary, which leads to an overall positive correlation between video duration and ground-truth summary length. The average length of AVIATE videos is almost 12 times more than that of How2 videos. The longer videos and lengthier summaries in AVIATE make it harder than How2 to train on, which is explained in Section 6.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In this section, we describe our proposed system, Factorized Multimodal Transformer (Zadeh et al., 2020) based Language Model (FLORAL) for abstractive text summarization using multimodal signals. Figure  3 shows the overall architecture of FLORAL. It takes a video, its corresponding audio and text transcript as input and generates an abstractive textual summary. A video generally has three distinct modalities -visual, textual, and acoustic, which supplement each other by providing complementary information, and thus when fused, separately contribute to generating richer and more fluent summaries. The first part of FLORAL extracts unimodal features using respective unimodal feature extraction networks. This phase does not  consider the contextual relationship between the three different modalities. In the next part, unimodal features are processed using the Factorized Multimodal Transformer (FMT) based decoder-only network over multiple steps, which in turn generates one summary word in each step. After every step, the generated word is appended to the source text with a delimiter. Therefore, FLORAL considers the entire summarization problem as a language modeling task, simplifying traditional encoder-decoder architecture. The remaining part of this section discusses individual modules of FLORAL in detail.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
The visual features are extracted using a pre-trained action recognition model, ResN eXt − 152 3D Convolutional Neural Network (Hara et al., 2018) trained on the Kinetics dataset (Kay et al., 2017) to recognize 400 different human actions. All the frames, computed at a rate of 5 FPS, are first preprocessed by resizing, center-cropping, and normalization to have a resolution of 112 × 112. For every 16 non-overlapping frames in a video, ResN eXt − 152 extracts a 2048 dimensional feature vector. Therefore, the result is a sequence of feature vectors per video rather than a global one. The sequential feature vector,
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
, is then used as the visual embedding input to the FMT.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
The acoustic modality is expected to contribute information related to tonal-specific details of the speaker (Tepperman et al., 2006). To achieve this, we obtain low-level features from the audio stream for each video. Similar to Castro et al. (2019), we use the popular speech processing library, Librosa (McFee et al., 2018) and perform the steps mentioned next. First, the audio sample for a video is stacked as a time-series signal with a sampling rate of 16000 Hz. Next, we remove the echos and background noise from the audio signals by integrating it with Audacity instance 9 , which is a free and open-source audio editor. Then, we segment the audio signals into d w nonoverlapping windows with a window size of 25 ms and successive window shift of 10 ms to extract low-level features that include Mel Frequency Cepstral Coefficients (MFCCs) with hamming window and the related temporal derivatives. Padding and segmentation are performed to achieve a fixed-length representation of the audio sources which are otherwise variable in length. At last, we concatenate all the extracted features to compose a d a = 512 dimensional joint representation for each window. Final MFCC features are obtained by applying a log Mel frequency filter bank over 0 to 8000 Hz and applying discrete cosine transformation (DCT). Similar to the visual features, the audio features, u a = {u a i } la i=1 , are also sequential for every video sample and are then used as the acoustic embedding input of FMT.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Guided Attention: At first, we represent the text in both ASR and OCR-generated transcripts using pre-trained BERT (Devlin et al., 2019), which provides dynamic embedding for every word. In particular, we use the sequence of 768-dimensional hidden states at the output of the last layer of the BERT model. Let F ∈ R n×768 and H ∈ R m×768 be the BERT representations for ASR and OCR texts respectively, where n and m are the respective token counts. The guided attention mechanism begins with defining an affinity matrix C ∈ R n×m , whose element c ij denotes the similarity between the feature vector pairs, h i ∈ R 768 and f i ∈ R 768 :
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
where W b ∈ R 768×768 is a correlation matrix to be learned during training. Subsequently, we compute a normalized weight α h ij to denote the relevance of the i th ASR-generated word to j th OCR-generated word. Therefore, the weighted summation of the ASR transcript, a h j , can be represented as,
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Since our goal is to emphasize the dissimilar features between the ASR and OCR transcripts, we define the relevance matrix R(f i , a h j ) as cosine distance between the attended ASR sentence vector a h j and OCR word embedding f i -
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Now, the weighted summation of all word embeddings produces the modified ASR representation U computed as,
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
where R(f i , a h j ) acts as a filter for the ASR encoding f i . Finally, we concatenate the attended ASR word representations with OCR word embeddings to get the sequential textual features u t = {u t i } lt i=1 , which is used as the textual embedding input of FMT.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
The pre-trained Language Model (LM) has recently been shown to have superior performance in abstractive summarization, particularly to enhance sample efficiency (Khandelwal et al., 2019). This decoder-only network, known as Transformer LM, takes a pre-trained transformer  as its base module and treats summarization as a language modeling task where each generated summary word in every step is appended to its source article. We extend the concept of Transformer LM to a multimodal setting, where we use Factorized Multimodal Transformer (Zadeh et al., 2020) based Language Model (FLORAL) for multimodal sequential learning. After each step of summary generation, we append the generated summary word to its source text transcript, along with a delimiter, and train the transformer on this reformulated data. FLORAL has three crucial advantages over traditional encoder-decoder based summarization networks:
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
1. In contrast to encoder-decoder architecture, FLORAL uses a single network to encode the source and generate the target, and thus, avoids the loading of same pre-trained weights into separate encoder and decoder.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
2. Compared to the encoder-decoder network, FLORAL has fewer number of parameters.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
3. Most critically, all the parameters of FLORAL can be pre-trained.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Since there is no available large-scale multimodal corpus, we pre-train FLORAL on the text-only 2-billion word corpus 11 based on Wikipedia, called WikiLM (Khandelwal et al., 2019), and fine-tune on the AVIATE and How2 datasets.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Factorized Multimodal Transformer (FMT) (Zadeh et al., 2020), which is the current state-of-the-art model for multimodal emotion recognition and multimodal speaker traits recognition on well-studied IEMOCAP (Busso et al., 2008) and POM (Park et al., 2014) datasets, applies seven distinct self-attention mechanisms to simultaneously capture all possible uni-modal, bi-modal, and tri-modal interactions across its multimodal input. We use 11 https://github.com/tensorflow/tensor2tensor   layers, which inherently capture inter-modal and intra-modal dynamics within the asynchronous multimodal input sequence. Blue, red and green colors are used to illustrate the propagation of visual, acoustic and textual features within FMS.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
the FMT architecture as the backbone of our decoder-only Transformer LM. Before feeding into FMT, the unimodal embeddings are resampled using a reference clock so that modalities can follow the same frequency (Chen et al., 2017). Additionally, zero paddings are used to unify the length of all samples of the entire dataset to a desired fixed length L. Hence, the i th data point consists of three distinct sequences of embeddings corresponding to three modalities -visual, acoustic, and language:
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
where x i ∈ R L×dx and tar i ∈ R M ×dy are the inputs and target summaries respectively, M is the length of the summary; d x , d y denote the input and output dimensionality at each time step respectively; N is the total number of samples within the dataset. Positional embeddings are also added to the input. The FMT consists of a stack of Multimodal Transformer Layers (MTL), which captures factorized dynamics within multimodal data and aligns the time asynchronous information both within and across modalities using multiple Factorized Multimodal Self-attentions (FMS), each of which has 7 distinct self-attention layers. Each attention has a unique receptive field with respect to modalities f ∈ F = {L, V, A, LV, LA, VA, LVA}. The high dimensional output of FMS is controlled by a summarization network S 1 to have a reduced dimension R L×dx which goes through feedforward and normalization layers. If there are a total of P number of FMS units inside MTL, the dimensionality of the normalization layer is R P ×L×dx which is again mapped to R L×dx using a secondary summarization network S 2 . The output of the last MTL of FMT, thus computed, is fed into a Gated Recurrent Unit (GRU) to have a d y dimensional predicted summary word embedding, Summ i . An overview of a single Factorized Multimodal Self-attention (FMS) block in MTL is presented in the 4.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
The summary word predicted by the FMT in the first step, Summ 1 , is appended to the text transcript and fed into the same FMT in the next step to predict the second summary word, Summ 2 . This process is continued until the model generates a stop-word or a predetermined summary length is reached. We only compute loss over the target sequence, as suggested by Khandelwal et al. (2019).
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
To explore the role of multimodality in abstractive text summarization, we conduct multiple experiments evaluating textual and visual modalities separately and jointly on both How2 and AVIATE datasets. Additionally, we investigate the role of OCR-generated text for the academic presentation videos in AVIATE for improving summary generation.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
We train FLORAL using Pytorch framework on NVIDIA Tesla V100 GPU, with 32 GB dedicated memory, with CUDA-10 and cuDNN-7 installed. We pre-train all the parameters of FLORAL using WikiLM (Khandelwal et al., 2019), and fine-tune on the summarization datasets (How2 and AVIATE). Similar to encoder-decoder models, we only compute loss over the target sequence. In Table 1, we present the details of hyper-parameters used in the baselines and in FLORAL.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
We compare the performance of the following extractive and abstractive unimodal and multimodel text summarization models both on How2 and AVIATE datasets. 
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• Lead3 is the most common baseline which simply selects the leading three sentences of the document as its summary.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• KLSumm (Haghighi and Vanderwende, 2009) is a greedy algorithm that minimizes the Kullback-Lieber (KL) divergence between the original document and the ground-truth summary.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• TextRank (Mihalcea and Tarau, 2004) runs a modified version of PageRank on a weighted graph, consisting of nodes as sentences and edges as similarities between sentences.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• LexRank (Erkan and Radev, 2004) is a graph-based algorithm that represents sentences as vertices, and edges represent the similarity.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• Pointer Generator (PG) (See et al., 2017) network is one of the most popular sequence to sequence (seq2seq) summarization architectures. PG allows both generating words from the vocabulary or copying from the source document.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• Pointer Generator-MMR (Lebanoff et al., 2018) uses MMR along with PG for better coverage and redundancy mitigation. Here MMR computes a similarity score of sentences with the source text and modifies the attention weights for a better summary generation. • CopyTransformer (Bottom-up Abstractive Summarization) (Gehrmann et al., 2018) uses the transformer parameters proposed by . It uses a content selection module that over-determine phrases in the source document.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• Multimodel Hierarchical Attention (Palaskar et al., 2019) extends the work of Libovický and Helcl (2017), which was originally proposed for multimodal machine translation. This model fuses visual and textual modalities and captures the context of visual and textual features along with hierarchical attention to generate summaries.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• MulT Encoder-Decoder is an encoder-decoder based summarization architecture, which uses MulT (Multimodal Transformer for Unaligned Multimodal Language Sequences) model (Tsai et al., 2019) as its encoder and decoder unit.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• FMT Encoder-Decoder is an encoder-decoder network, similar to MulT encoder-decoder. This baseline uses Factorized Multimodal Transformer (Zadeh et al., 2020) as the encoder and decoder units.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
• MulT LM is MulT-based architecture. This is most akin to our proposed FLORAL model; only the FMT module of FLORAL is replaced by MulT (Tsai et al., 2019) to have this multimodal summarization baseline.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
We present a quantitative analysis of the summaries using the standard metrics for abstractive summarization -ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) (Lin, 2004;Graham, 2015) that measure the unigrams, bigrams, and longest common sequence between the ground-truth and the generated summaries, respectively. Additionally, we perform extensive qualitative analysis using human experts to primarily understand the fluency and informativeness of the summaries. We also analyze the word distributions in the transcriptions and summaries.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
At first, we evaluate the performance of commonly used extractive and abstractive text summarization models both on the How2 and AVIATE  Table 2: Ablation results after incorporating OCR generated text into the ASR generated text transcript using guided-attention for different extractive and abstractive unimodal and multimodal text summarization systems on AVIATE.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
datasets. Note that the average length of text transcripts in How2 is much less than that of AVIATE. Following our intuition, PG-based text summarization networks perform relatively well on How2 as shown in Table 3; but their performance drastically drops on AVIATE. This result can be attributed to the fact that attention-based encoder-decoder networks often fail to capture long-term dependencies when the source text is long and noisy. Hence, we decide to use transformer-based pre-trained BERT (Devlin et al., 2019) as the text-embedding layer in our model.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In addition to text-only models, we train two video-only models -the first one uses a single convolutional and pooling layer for feature extraction from the entire video, while the second one applies a single layer RNN over these vectors in time. We observe in Table 3 that even using only action features in the videos leads to almost competitive R-1, R-2, and R-L scores compared to text-only models, in some cases often better than extractive text-only systems. This result demonstrates the importance of both modalities for summarization.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Model  
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Since the AVIATE dataset is composed of conference presentation videos, we observe that in almost 94.8% videos in the entire dataset, the speaker shows slides during the presentation. The text in these slides is succinct and contains the most important key-phrases which are crucial for summary generation. Table 2 shows the performance improvement for every summarization model when the OCR is fused with ASR transcript using our guided attention mechanism. We also consider direct concatenation of OCR transcript with the ASR-transcript; however, it resulted in lower performance as compared to guided attention fusion. The guided attention ensures the filtering of redundant and repetitive words in OCR and ASR transcripts. For every summarization model, we only use the first 500 tokens of the OCR transcript. We did not consider incorporating OCR for How2 as this dataset only contains instructional videos, and there is no text shown in the frames of instructional videos. Table 2 shows that unimodal extractive text summarization models, namely KLSumm, TextRank and LexRank, yield an improvement of [2.1−2.3] R-1 points, [0.1 − 0.3] R-2 points and [0.6 − 6.5] R-L points after incorporating OCR-generated text. Similarly for abstractive summarization, the very popular PG-MMR network produces 0.5, 1.25, and 2.3 points performance improvement in terms of R-1, R-2, and R-L scores, respectively. The other abstractive summarization networks, namely PG, Hi-MAP, CopyTransformer, also support our hypothesis and show an improvement of [0.5 − 4] points in terms of all three evaluation metrics.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Influenced by the performance of unimodal summarization models, we incorporate the OCR transcripts into all of our multimodal baselines. Supporting our intuition, the multimodal systems obtain significant performance enhancement with OCR transcripts as shown in Table 2. The multimodal hierarchical attention model, MulT, and FMT-based encoder-decoder models show [0.8 − 1.5] points improvement in the R-L score. Our proposed FLORAL model yields the highest performance boost with OCR among all the multimodal systems, showing 3.87, 4.66, and 2.95 point enhancement in R-1, R-2, and R-L scores respectively. The performance boost can be easily attributed to the keywords in the OCR-generated transcript, which guides the text-embeddings to attend the most important portions in a very long ASR transcript. Hence, in the rest of our discussion, we always report results with (ASR + OCR) transcript, fused with guided attention, as the textual modality. Table 3 shows the ROUGE scores for different unimodal and multimodal text summarization systems on the How2 and AVIATE datasets. Among the unimodal variants, the abstractive text summarization systems generally perform much better than the extractive systems, especially on AVIATE. Note that despite being a very strong extractive baseline, Lead3 does not perform well on AVIATE, as the text transcripts of academic presentation videos do not tend to be structured with the most important information at the beginning. The two video-only models, simple conv-pool action features and action features with RNN perform very close to the abstractive textonly baselines, which clearly indicates the necessity of visual modality in addition to the textual modality. 12 As presented in Table 3, the MulT, and FMT multimodal baselines and the proposed FLORAL model beat most of the unimodal systems by a large margin, on both the datasets. This result is expected because of the inherent ability of MulT and FMT to capture the intra-model and inter-modal dynamics within asynchronous multimodal sequences and incorporate diverse information in a single network. Overall, the combination of visual, acoustic, and textual signals significantly improves over the unimodal variants, with an improvement of 1.57, 3.04, and 3 R-1, R-2, and R-L points on How2 and 6.86, 7.1 and 4.41 on AVIATE.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
We manually investigate some video samples of AVIATE where the multimodal system generates a better summary than the unimodal system. In most of these samples, the textual transcript is very noisy and contains many irrelevant words that are not much required for the summary generation. Figure 6 shows an example training instance of the AVIATE dataset with three different modalities. A closer look into the ASR and OCR transcripts reveals the presence of irrelevant and noisy words. For example, the very first sentence of the ASR transcript "hi i'm lisa ann hendricks and today" does not contribute to the summary generation. As a result, these samples require additional cues for performance improvement, which are availed from the multimodal signals. The variation of outputs from various unimodal and multimodal summarization networks for a single video sample is shown in Table 9. Table 3 shows that the performance of unimodal and multimodal summarization systems on How2 as compared to AVIATE. In contrast to prior work on news-domain summarization (Nallapati et al., 2016), the seq2seq model performs the best among all unimodal systems on How2, achieving 55.32, 23.06, and 53.9 R-1, R-2, and R-L scores, respectively. As indicated by Palaskar et al. (2019), the PG model performs lower than seq2seq on How2 due to the lack of overlaps between input and output, which is the important feature of PG networks. Among the multimodal systems, our proposed FLORAL model yields the best results; however, the other multimodal baselines reach almost competitive ROUGE scores compared to FLORAL on this dataset. Noticeably, despite having a simple structure, the multimodal hierarchical attention model performs very well on How2. On this dataset, FLORAL achieves 56.89, 26.93, and 56.80 R-1, R-2, and R-L scores, respectively, which are [0.1 − 2] points higher than the scores achieved by other multimodal baselines.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
AVIATE contains longer videos than How2, resulting in longer transcripts and ground-truth summaries. As shown in Table 3, the best performing unimodal summarization model on AVIATE is CopyTransformer. As the ASR and OCR generated summaries are very long, the extractive systems do not perform well on this dataset. While PG and seq2seq yield 23.32 and 23.81 R-L scores respectively, CopyTransformer produces a 27.06 R-L score, outperforming all other unimodal systems. The superior performance of CopyTransformer over PG and seq2seq can be attributed to the self-attention mechanism of transformers which helps to capture long-term dependencies. The incorporation of visual and acoustic modalities significantly improves the ROUGE scores on this dataset. FLORAL beats all the transformer-based encoder-decoder networks and language models. FLORAL produces 37.13, 11.04 and 31.47 R-1, R-2 and R-L scores respectively, where the second-ranked model on AVIATE, MulT-LM obtains 33.47 R-1, 4.12 R-2, 28.73 R-L scores, which are almost [2.7 − 7] points lower than that of FLORAL. Since AVIATE contains 6, 680 training samples, which may not be enough for today's deep neural models, the factorization mechanism on FMT, which allows an increasing number of self-attention to better model the multimodal phenomena, results in its superior performance, without encountering difficulties even on the relatively low-resource setup of AVIATE. Pre-training of all the parameters of FLORAL also has an immense impact, which helps in beating all other baselines by a significant margin. Table 3 also shows that all the unimodal and multimodal summarization models obtain almost [18 − 25] points higher R-1 and R-L scores and [3 − 6] points higher R-2 score on How2 over AVIATE. For example, FLORAL yields 56.89, 26.93, and 56.80 R-1, R-2, and R-L scores on How2 and 37.13, 11.04 and 31.47 R-1, R-2 and R-L scores on AVIATE. We can observe from Table 3 that all the baseline models as well as FLORAL yield higher R-1, R-2 and R-L scores on How2 than AVIATE. The overall better performance of every system on How2 than AVIATE can be attributed to two factors -firstly, the text transcripts of How2 are manually annotated. In contrast, we use ASR and OCR outputs as the transcripts for AVIATE. The large margin of ASR and OCR errors in some of the train and test samples significantly affect the model performance. Secondly, since the video length, transcript length, and reference summary length are much longer in AVIATE than How2, the summarization task becomes more challenging in AVIATE. Furthermore, since AVIATE comprises many scientific presentation videos, the audio transcript contains complex academic words, leading to a larger dictionary for the language generation task. Overall, the results in Table 3 conclude that the AVIATE dataset is more exacting than How2, indicating room for further research with fine-grained and sophisticated multimodal models for long videos.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In our next experiment, we demonstrate how the summarization task becomes more challenging with longer videos. We divide the AVIATE dataset into three portions -short videos (duration less than 10 minutes), medium videos (duration between 10 minutes and 30 minutes) and long videos (duration more than 30 minutes). We split each portion in 4 : 1 ratio and train and test all the multimodal systems on each segment. Table 4 shows the performance reduction of each model with the increase in video length. In general, we observe that all four multimodal baselines yield R-L score in the range of [27.03 − 34.09] on the short videos. However, the score reduces to [23.19 − 28.69] for the long videos. The performance of FLORAL also decreases from short to medium and long videos; however, the span of reduction of R-L score is only [0.39 − 0.64], which is relatively less than all other baselines. We also notice that the LM-based systems generally capture long-term dependencies better than traditional encoder-decoder based systems.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
The complementarity of multiple modalities in the performance of FLORAL is shown in Table 5. To understand the importance of visual modality,   Table 5: Significance of multimodal cues in FLORAL. The combination of visual, textual, and acoustic signals significantly improves over the unimodal variants, with a relative improvement of R-1, R-2 and R-L scores of 9.99%, 8.11% and 11.80% respectively over the best unimodal variant.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
we feed zero input in other two modality channels of FLORAL and continue the process for all three modalities. We observe that the textual modality provides the best performance among unimodal variants. The addition of visual and acoustic features improves significantly over the unimodal baselines and achieves the best performance -with an increase in R-1, R-2 and R-L score of 9.99, 8.11 and 11.80 respectively over the best unimodal variant.  Table 6: Transferability of the proposed FLORAL model on the two available multimodal abstractive text summarization datasets -How2 and AVIATE. The network is trained on the dataset in each row, and is tested on the dataset shown in each column. The second row indicates the performance of FLORAL on the How2 videos whose transcripts are generated from ASR.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Medium Videos Long Videos R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L  6.1.5. Transferability of FLORAL Table 6 shows the transferability property of FLORAL between How2 and AVIATE. When trained and tested on the same dataset, FLORAL produces the best ROUGE scores, which is expected. However, when trained on AVIATE and tested on How2, FLORAL yields an R-L score of 49.90, which is just 6.9 decrease in R-L score (11.83% reduction in performance) than the one when trained and tested on How2. The vice-versa is not true, i.e., when trained on How2 and tested on AVIATE, the performance drop is drastic (26.56% reduction in performance). As the videos in How2 have human-annotated transcripts and those in AVIATE have ASR-generated transcripts, for fair comparison of transferability, we extract the ASR transcripts of the How2 videos and train FLORAL. The results of this experiment are shown in the second row of Table 6. We observe that the ASR transcript reduces the test performance on How2, which is expected due to the noise in the ASR output. The transferability score on AVIATE improves a bit, but the performance drop is still heavy (25.51% reduction in performance). From all these experiments, we can conclude that since the videos of How2 are very short, the learned weights do not perform well for longer videos. However, AVIATE consists of diverse-length videos, and thus, the trained model on AVIATE yields good results on How2 as well. Table 7 shows the transferability of FLORAL across short, medium and long videos of AVIATE. When trained on the long videos, FLORAL performs the best across all three portions. However, when trained on short videos, the model can not learn long-term dependency for lengthier videos. The same property supports the results on Table 6. Since the How2 dataset contains only short videos, the model does not perform well when trained on How2 and tested on AVIATE. The longer videos in the training set helps the model to generalize well across videos of various lengths.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In addition to ROUGE scores, we conduct a qualitative analysis by performing a human evaluation to understand the standard of the summary outputs. Following the abstractive summarization human annotation work of Grusky et al. (2018)    to rate the generated summaries on a scale of [1 − 5] on four parametersinformativeness, relevance, coherence, and fluency. For the evaluation, we randomly sampled 300 videos from the test sets of How2 and AVIATE. Table  8 shows the average human evaluation scores for 4 text-only, 1 video-only and 3 multimodal models. In general, we observe that PG has difficulty in summarizing articles with repetitive information and tends to assign a lower priority to less occurring important keywords. The extractive summarization systems sometimes pick sentences extraneous to the summary. For example, we notice some summaries generated by KLSumm starting with "Good afternoon everyone, I am ·", which is the very first line of the transcript. In contrast, the multimodal summarization models generate summaries with greater relevance and informativeness. Our proposed FLORAL model obtains high scores on informativeness, relevance, and coherence on AVIATE, but sometimes seems to generate less fluent summaries. This fluency problem mostly stems from errors in ASR and OCR generated text. Some of these phenomena are illustrated with instances from AVIATE in Table 9.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
We also analyze the word distributions of the ground-truth summaries and different system-generated summaries. The density curves in Figure 5 shows that for both How2 and AVIATE, the abstractive unimodal and multimodal summarization models generate summaries shorter than the ground-truth summary. The average length of summaries is highest for CopyTransformer. Interestingly, FLORAL and PG generated summaries are similar in length. However, FLORAL outperforms PG by a large margin, which illustrates that for improvements in ROUGE scores, an informative summary is more crucial than a lengthier summary.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In this paper, we explore the role of multimodality in abstractive text summarization. All the previous studies in this direction have used either images or short videos as the visual modality and generate one or two lines long summary, and thus, fail to perform on longer videos. Moreover, there exists no benchmark dataset for abstractive text summarization of medium and long videos. In this work, we introduce AVIATE, the first large-scale dataset for abstractive text summarization with videos of diverse duration, compiled from paper presentation videos in renowned academic conferences. We then propose FLORAL, a Factorized Multimodal Transformer based decoder-only Language Model, which uses an increasing number of self-attentions to inherently capture inter-modal and intra-modal dynamics within the asynchronous multimodal sequences, without encountering difficulties during training even on relatively low-resource setups. To evaluate FLORAL, we perform extensive experiments on How2 and AVIATE datasets and compare them against several unimodal and multimodal baselines. Overall, FLORAL achieves superior performance over previously proposed models across two datasets.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
Ground-truth -
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
We study the problem of semi-supervised question answering-utilizing unlabeled text to boost the performance of question answering models. We propose a novel training framework, the Generative Domain-Adaptive Nets. In this framework, we train a generative model to generate questions based on the unlabeled text, and combine model-generated questions with human-generated questions for training question answering models. We develop novel domain adaptation algorithms, based on to learn richer context-aware model on the insight and the human-generated data distribution. Experiments show that our proposed framework obtains substantial improvement from unlabeled text.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
FLORAL (ours) 31. 47 We study the problem of semi-supervised concepts unlabeled text to boost the performance of without without models. We propose a novel training framework, the Generative Domain-Adaptive Nets. In this framework, we train a database model and generate concepts based on procedure on how partitioning in the levels between t, and combine model-generated attention with human-generated types for training associations directed-generation models. We develop novel algorithms, based on model, to understand the discrepancy between the generated data results and the human-generated data distribution. Experiments show model obtains improvement from text.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
MultT LM 28.73 In learn paper are neural paper, we propose a new model and the paper, we study demonstrated the character score We investigate the problem answering utilizing unlabeled text to understand trained models. we train a understanding model to generate understanding based on the text, and combine trained understanding with human generated understanding to training data trained we assume like, on a public unsupervised Subsequently, prediction trained to learn richer context-aware model on the insight In trained facilitates smaller annotations previously reported understanding understanding on approach and questions along five perform two then performance The the semantic best dataset and the proposed
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
MultT En-De 27.2 We present the problem of supervised new utilizing text of new new models. We propose a problem model review a new model In this paper, we propose a simple model Experimental resultsprove the standard model achieves method to then the proposed among review model to also performance based on the proposed CoNLL-2012 dataset. In this we train a new model to generate a model and combine model. of machine morphology into more prediction on the drawback of the system classes. using error-correcting codes collection. We evaluate investigation of NMT and induced Experiments errors. shed light affect vs. scores on a word its dataset with the proposed based
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
CopyTrans. 27.06 We propose a novel method for novel method for state-of-the-art question answering we show that the method achieves state-of-the-art performance on the state-of-the-art performance of the proposed method on the systems over sequential text tasks or as independent, parallel tasks. In this framework model generate text trained benefits of the new semantic high a image and at them as HA 26. 12 We propose the novel model has the paper, we propose a generative model In both learning can can the benchmark model To capable to also performance of existing the pipelined of the model-generated model: distribution for the human-generated data distribution. Experiments show that our proposed framework to the consistency of the art data distribution and event feedforward network. The an unsupervised dependency framework performance and the human data distruction .
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
PG 23.81 In this paper we consider the problem of learning a deep neural network, we propose a novel neural network architecture based on novel neural network architecture that is trained end-toend trainable convolutional neural network (CNN) architecture is able to train a convolutional neural network architecture that is capable of achieving state-of-the-art performance compared to state-of-the-art performance on three benchmark datasets.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
PG-MMR 22.63 We propose a novel method for object detection based on a novel object detection method that uses a novel model based on the posterior distribution of the posterior distribution over the parameters of the number of claims and We show that models can be used as well as compared to In this paper, we study the a algorithm that the proposed method can outperforms the state-of-the-art methods on a large number of Table 9: Comparison of ground-truth summary and outputs of 7 different unimodal and multimodel abstractive text summarization systems -FLORAL, MultT LM, MultT Encoder-Decoder, CopyTransformer, multimodal hierarchical attention (HA), Pointer Generator (PG) and Pointer Generator with MMR (PG-MMR) -arranged in the order of best to worst ROUGE-L scores in this table. Red highlighted text indicates a positive correlation of context w.r.t. ground-truth summary while blue color represents a negative correlation with ground-truth summary. Figure 6: Example of AVIATE dataset with three different modalities. To obtain the text transcripts from the acoustic modality, we apply Deep Speech (Hannun et al., 2014), a pre-trained end-to-end automatic speech recognition (ASR) system. We extract the text shown in the slides in the presentation videos using Google OCR Vision API. We use the abstracts of corresponding research papers as the ground-truth summaries.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
In industry (e.g., digital advertising [1], [2] and ecommerce [3], [4]), huge amounts of multimodal data are often obtained, in which each instance is described as a set of information from multiple modalities. Thus, handling multimodal information is a fundamental problem in most web applications. Deep neural networks (DNNs) also experience technical challenges in handling multimodal data [5].
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
The mainstream approaches of multimodal models are based on a mid-fusion architecture, which encodes the data from each modality and then concatenates multiple features [6], [7]. Previous studies have explored various midfusion models, such as the gated multimodal units (GMU) [6] and neural architecture search (NAS)-based models [7]. However, mid-fusion models often face several challenges. First, as the number of modalities increases, the dimensionality of the concatenated features generally increases as well. Moreover, mid-fusion models require the impractical assumption that each instance in a dataset must have the same number of modalities without missing information.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
To address the above multimodal problems, Reiter et al. [8] proposed deep multimodal sets (DMMS) based on Deep Sets [9], which is a class of models that is defined according to sets. The underlying concept of DMMS is that a function defined on sets should satisfy the invariance to the number and order of elements in a set (i.e., modalities). However, as DMMS encodes/compresses each modality and then further compresses the features using a pooling operation, the architecture may omit the importance of signals in a single modality.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Although set-aware architectures such as DMMS [8] can deal with "unordered" sets of modalities, the data in multimodal information takes various forms; certain types of data are inherently sequential (e.g., text tokens), whereas others are not (e.g., categorical data). Recent BERT [10]-based multimodal methods often convert an image into textual tokens using optical character recognition (OCR), and thus, obtain a textual sequence [11]- [13]. In particular, such a preprocessing technique is effective in extracting rich semantic information that is difficult to capture by encoding raw modality data. Therefore, these situations motivate us to handle sets of sequences.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
As the interpretability and explainability of the model predictions are indispensable factors in industry, it is necessary to provide an interpretation of the importance of each modality. The DMMS [8] enables an explainable prediction in terms of the importance of the modalities. However, conventional mid-fusion models may ignore the higher-order interactions between elements across sequences, as they only focus on the fusion of encoded modality representations. Furthermore, owing to the black-box nature of DNN-based models, such "encode-and-concatenate" models do not provide a breakdown of the importance of factors in a modality. This is an insufficient diagnosis in a practical setting; for example, it is often desirable to know the element-level importance to investigate the problem in text tokenization.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
In this study, we propose a new concept for handling multimodal data known as deep multimodal sequence sets (DM 2 S 2 ). Our concept consists of three components that capture the relationship between multiple modalities: (1) a BERT-based encoder, (2) intra-modality residual attention (IntraMRA), and (3) inter-modality residual attention (In-terMRA). For (1), the BERT-based encoder outputs a rich representation that handles the inter-and intra-orders of the elements in a modality. In (2), IntraMRA captures the importance of the elements in a modality. In (3), InterMRA further enhances the importance of the elements with modality-level granularity. We employ OCR methods to extract semantic features from multimodal data (especially for the visual modality) that may not be captured by image recognition. The model based on our concept empirically demonstrates that it is preferable to use semantic features that are derived from OCR tokens over features that are encoded directly from visual information. Our two-step MRA can capture fine granularity representations that are not explicitly modeled by the DMMS [8].
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Empirical experiments on various real-world datasets reveal that our sequence set-aware concept achieved performance that is comparable to or better than previous multimodal and set-aware models. In particular, the model based on our concept outperformed those deployed in the production environment on a large-scale production advertising (Ad) dataset. Furthermore, the model enables the visualization of the contribution of elements in modalities to model predictions based on the importance that is learned by the MRA.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
The contributions of this study are summarized as follows:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
• We propose a new model for capturing higher-order interactions, and the inter-and intra-relationships of each modality, considering that multimodal data often include set structured contents. • Our set-aware model with a multimodal structure achieves equivalent or better prediction performance compared to previous multimodal and set-aware models. • An evaluation on datasets, including a real-world production dataset, demonstrates that our DM 2 S 2 can be reasonably applied in a real environment. The visualization of the learned MRA weights can be used to explain predictions that are important in industrial fields.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
In general, DNN-based multimodal models include multiple streams of networks for modalities [14]. The models often have a component for fusing the features [2], [6], [15] to make a prediction based on intermediate features from the streams. Hence, by considering the fusion stage in the model architecture, multimodal architectures can be categorized into early fusion [15], mid-fusion [6], [7], [14], [16], and late fusion [2], [17]. The early fusion approach integrates the features of different modalities as inputs and uses a unified feature for the downstream task [15]. The mid-fusion approach involves the concatenation of features that are encoded from the raw data of each modality into a single feature [6], [7], [14], [16]. Typical multimodal architectures rely on the midfusion approach to combine multimodal information. However, mid-fusion may suffer from the increased dimensionality of concatenated features and the impractical assumption of complete modality information for each sample. The late fusion approach combines the prediction scores of different predictors for multiple modalities [2], [17]. The late fusion method often exhibits sub-optimality of the trained models owing to the dependency between modalities being ignored.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
To solve this problem of mid-fusion architecture, which is the most common architecture in multimodal models, Reiter et al. [8] proposed the DMMS, which is a set-aware model based on Deep Sets [9] that represents a collection of features as an unordered set rather than an increasing fixedsize vector. The DMMS can deal with the set input by pooling an arbitrary number of modality features into a constantsize vector without dependence on the number of modalities. Nevertheless, the DMMS may ignore the importance of signals in a single modality by compressing the modality information through the leading encoder. Several midfusion models in the earlier stage of architecture introduced techniques to capture the higher-order interactions between modalities to construct each modality representation [18], [19]. Owing to the set-aware architecture, the DMMS can handle an arbitrary number of modality features without any unnatural workarounds (e.g., padding for missing modalities).
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
This study presents an architecture for retaining the finegrained signals in each modality while capturing the im-portance between the modalities. Our proposed mid-fusion architecture operates similarly to models in the class of Deep Sets with modality-level granularity as in the DMMS while leveraging the hierarchical structure of multimodal data, namely, sets of sequences. As recent BERT-based encoders output a sequence of embeddings rather than a single representation for an input sequence, we leverage it as a rich modality representation through our two-step attention mechanism.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
According to Bahdanau et al. [20], various modality-aware attention mechanisms have been proposed [21]- [23] to achieve effective learning in multimodal tasks. Similar concepts have been explored for named entity recognition [21], audio-visual speech recognition [22], and sentence summarization [24].
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
This study employs an attention mechanism to capture the intra-modality signals and inter-modality importance based on the representation of sequence sets. Whereas our proposed MRA is closely related to the modality attention proposed by Moon et al. [21], our framework clearly differs in that it considers the hierarchical structure, such as the inter-and intra-relationships of modalities. Furthermore, we introduce a residual connection to preserve the information of the source feature representation.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
First, we briefly introduce Deep Sets. Our DM 2 S 2 is based on Deep Sets, which is a class of models defined according to sets. Subsequently, we describe the proposed DM 2 S 2 and its key component, namely MRA.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
For each modality m ∈ M, we extract a sequence x (m) ∈ V lm of length l m from the tokens that are contained in vocabulary set V. Each token is expected to be extracted appropriately depending on the modality type; for example, a word or character for the textual modality and an OCRdetected word for the visual modality. Moreover, tokens can be obtained from images, audio sources, and videos; that is, a visual patch such as ViT [25] from an image, transcription from audio, and the results of the same procedure for images from each frame of video.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We wish to handle the set of sequences X = {x (m) | m ∈ M} as the input of a model; the loss functions should also be computed on such sets. To this end, we design our proposed architecture based on Deep Sets [9] as in the DMMS [8]. A model in the class of Deep Sets can be expressed as follows:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where φ is the neural network that encodes the input set X, and ρ is the prediction network for a downstream task.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
As described above, we use the sets of sequences that are extracted from each modality as inputs for the model. Therefore, in contrast to the conventional models [8], [9], we aim to handle the order between the tokens {x (2) IntraMRA, and (3) InterMRA. The input is the set of sequences from multiple modalities. Each set contains token representations, as described previously. Even if this unimodalized (tokenized) information is input, we consider this to be a multimodal setting.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
First, to handle the sets of sequences appropriately, a BERT-based encoder is used as a building block of φ. We construct a unified sequence by concatenating the sequences of the modalities: 
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We assume that each sample has the same number of modalities (i.e., M ). When a modality is missing for a sample, the corresponding entry x (m) can be filed with an empty sequence. Hence, by ensuring a fixed order of modalities in x, the BERT encoder can be viewed as permutation-invariant in terms of the order of the modalities, and permutationsensitive in terms of that of the tokens in a single modality.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
As the BERT encoder can handle the order of tokens appropriately, we further enhance both the intra-and intermodality relationships. Intuitively, the importance of each token from multiple modalities can be decomposed into inter-and intra-modality factors; a token may be important because it is an essential modality for a prediction, and/or it is representative of its modality. We introduce a two-step MRA into the representation of H (m) to model this hierarchical structure of token importance.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We apply IntraMRA : R lm×d → R lm×d to quantify the token importance in each modality m ∈ M (described in detail in Section III-C2).
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We denote the concatenated matrix of the IntraMRA outputs for each modality as H Intra = (H
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Subsequently, we apply
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
The input is the sets of sequence from multiple modalities. The sequence is encoded into the hidden states by the BERT-based encoder. Our Intra-and InterMRA learn each relationships and their contribution (as indicated by the red field) of the modalities. The output is a hidden vector that is obtained through the encoder and the mechanisms.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
InterMRA : R L×d → R L×d to capture the importance of each modality (described in detail in Section III-C3):
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
The representation H Inter is aggregated as the final hidden vector h ∈ R d through a pooling function ψ : R L×d → R d . In this study, we use the following mean pooling as ψ:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where H Inter,i is the i-th token representation of H Inter . The projection that is defined by the above procedure can be considered as a permutation-agnostic encoder φ. This is because the Intra/InterMRAs and ψ are evidently permutation invariant in terms of the order of the modalities. Based on the representation of a sequence set φ(X) = h, we obtain the predictionŷ = f (X) through a multi-layer perceptron (MLP), ρ:ŷ = ρ(h).
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Our concept of MRA originates from the attention mechanism [20]. The attention mechanism estimates the contribution of the input to the prediction. Using these properties, we propose an MRA that can learn the intra-and interrelationships for multiple modalities. A possible extension is available to replace our Intra-and InterMRA mechanisms by adding a Transformer module [26] to the BERT encoder; however, we will leave this as future work. We compute the attention score from the alignment function A : R l → [0, 1] l by passing the hidden state H ∈ R l×d to the attention mechanism, which consists of a similarity function S : R l → R l followed by the softmax function:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where W ∈ R d×d is a trainable weight matrix, and q ∈ R d is a trainable self-attention vector [27]. The similarity function may be the additive attention [20] or the scaled dot-product attention [26]. In this study, we considered the additive attention, as follows:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We demonstrate that additive attention is preferable to scaled dot-product attention in Section V-B2.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
IntraMRA learns the token-level attention score for the modality m. Although it is possible to learn the global attention (i.e., the attention of the entire token sequence) for the token sequence that is input into the encoder, the attention score for each token will be relatively small. IntraMRA helps to learn the modality inter-relationships by calculating the attention for the token sequence in the modality. We ensure that the value of the feature representation does not become excessively small by employing a residual connection [28] that preserves the information of the source feature representation. For the hidden state H (m) t that is obtained by the encoder, the IntraMRA score a (m) Intra,t is calculated as follows:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where W (m) ∈ R d×d and q (m) ∈ R d are trainable parameters for modality m. Using the attention score, we obtain the intra-attended representation H 
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
. IntraMRA outputs the concatenated matrix for each element:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
3) InterMRA InterMRA learns the modality-level (e.g., visual-, textual-, and categorical-level) attention scores. The mechanism captures the modalities that contribute to the prediction, which is important for model training. We employ a residual connection for InterMRA with the expectation that the connection will provide the same effect as in IntraMRA. We first compute the summation of each modality representation for H ∈ R M ×d that contains the hidden representation for each modality H Intra :
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
(12) Thereafter, we calculate the InterMRA score a 
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where W ∈ R d×d and q ∈ R d are trainable parameters for InterMRA. We obtain the following inter-attended representation: H Inter for modality m with residual function F :
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Intra Finally, InterMRA outputs the resulting concatenated matrix for each modality, as follows:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We used three multimodal datasets, namely, MM-IMDB [6], Ads-Parallelity [2], and Production Ad-LP datasets for the empirical experiments. The MM-IMDB [6] dataset contains 25,925 movies with multiple labels (genres). We used the original split provided in the dataset and reported the F1 scores (micro, macro, and samples) of the test set. The Ads-Parallelity [2] dataset contains 670 images and slogans from persuasive advertisements to understand the implicit relationship (parallel and non-parallel) between these two modalities.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
A binary classification task is used to predict whether the text and image in the same ad convey the same message. We reported the overall and per-class average accuracy and ROC-AUC scores of five-fold cross-validation [2], [8].
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
The Ad-LP dataset contains 257,235 search engine ads and landing pages (LPs) with search keywords, ad titles, descriptions, and LP text. The dataset was collected from CyberAgent Inc. 1 from August 1, 2020, to November 30, 2020. The task is to regress the conversion rate from the LP after clicking on the ad; the conversion rate expresses the number of clicks/total number of impressions of the ad. We divided the dataset into training, validation, and testing sets. The resulting splits comprised 179,922, 45,138, and 32,175 Ad-LP pairs, respectively.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We transformed the following multimodal information (i.e., visual, textual, and categorical data) into textual tokens and fed these into our proposed model. We used the Google Cloud Vision API 2 for the visual features to obtain the following four pieces of information as tokens 3 : (1) text from the OCR, (2) category labels from the label detection, (3) object tags from the object detection, and (4) the number of faces from the facial detection. We input the labels and object detection results as a sequence in order of confidence, as obtained from the API. We describe the visual, textual, and categorical features of each dataset below.  We used the following features: (1) as visual features, we used the result of applying the API to the screenshot for the first view of the LPs; (2) as textual features, we used the search words, titles, and descriptions of the ads, and the URL path for the LPs; and (3) as categorical features, we used the match type to search keywords as tokens. Refer to Appendix A for details on the production dataset.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We implemented our proposed concept using AllenNLP version 2.5.0 [29] with Hugging Face transformer version 4.5.1 [30]. We evaluated the test set only once in all experiments. The experiments were conducted on an Ubuntu 20.04 PC with an NVIDIA RTX A6000 GPU.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Following the DMMS of Reiter et al. [8], we used a pretrained RoBERTa large encoder [23] 4 in MM-IMDB and Ads-Parallelity as the BERT-based encoder. The encoder outputs the hidden state with dimension d = 1024. We froze the parameters of the encoder and used the output of the hidden representation. We used a pre-trained Japanese BERT 5 as the encoder in the Production Ad-LP dataset. This encoder outputs a hidden state with dimension d = 768.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We followed the settings of Reiter et al. [8] for the optimizer. Specifically, we used Adam [31] with a decoupled weight decay [32]. The learning rate was warmed up linearly from 0 to 0.001 during the first five epochs and then decayed with a cosine annealing schedule over the remaining epochs. For the MM-IMDB and Ads-Parallelity datasets, which are designed for single-and multi-label classification tasks, we used the sigmoid cross-entropy with class weights [33] to train the model on N training samples:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where K is the number of classes and w k = 1/N k is the class weight for class k that is calculated with the class frequency N k . In this case, Y ∈ {0, 1} N ×K is the target label matrix in which the (j, k)-entry indicates whether the training sample j is in class k, andŶ ∈ [0, 1] N ×K is the predicted label matrix. For the Production Ad-LP dataset, which is a regression task, we used the root mean squared error (RMSE) to train the model, as follows:
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
where y i is the i-th ground truth andŷ i is the i-th predicted value. Refer to Appendix B for the implementation details.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
For MM-IMDB and Ads-Parallelity, we compared our model, DM 2 S 2 , and the state-of-the-art baselines based on NAS, Transformer, and set-aware models with a BERT-based encoder. Tables 1 and 2 present the evaluation results for  MM-IMDB and Ads-Parallelity, respectively. We refer to the scores provided in each of the papers as a comparison and report them along with the DM 2 S 2 scores. For MM-IMDB, we compared our DM 2 S 2 with five stateof-the-art models: GMU [6], Bilinear-Gated [34], multimodal fusion architecture search (MFAS) [7], multimodal bitransformer (MMBT) [35], and DMMS [8]. In addition to the baseline above baselines, we compared our model with a pure BERT-based encoder (indicated in the table as RoBERTa encoder only) that uses token type IDs (also known as segment IDs) to differentiate tokens that belong to different modalities. This is a common approach in the multimodal literature [38].
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
As indicated in Table 1, DM 2 S 2 consistently outperformed the baselines on all performance measures. The comparison between DM 2 S 2 and the NAS-based MFAS reveals that our DM 2 S 2 exhibits the advantage of token-and modality-level fusion based on the proposed two-step MRA, in contrast to MFAS, which searches for the best mid-fusion architecture based on the NAS algorithm. As the DMMS is closely related to our DM 2 S 2 in terms of the set-aware fusion technique and BERT-based encoder, the performance gain of our DM 2 S 2 compared to the DMMS was remarkable. However, it should be noted that the DMMS uses a proprietary OCR method, and, thus, a fair comparison between DM 2 S 2 and the DMMS is difficult owing to the difference in the OCR performances. Therefore, we reproduced the DMMS as effectively as possible and trained the model on a dataset built with tesseract [37], EasyOCR [39] and the Cloud Vision API.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
According to Table 1, DM 2 S 2 with the Cloud Vision API exhibited better results than the other OCR methods. The performance of DM 2 S 2 with EasyOCR, which is a neuralbased model based on CRAFT [40] and CRNN [41], was comparable to that of the DMMS. Although we conducted experiments using the same OCR methods (Cloud Vision API), our DM 2 S 2 outperformed the DMMS. We also obtained good results in all performance measures for the RoBERTa encoder only model with token-type IDs. Based on these results, we conclude that our MRA makes a more important contribution to learning multimodal data than the pure BERT-based model.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
For Ads-Parallelity as indicated in Table 2, we compared DM 2 S 2 with Combined Classifiers [2] and the DMMS [8]. Furthermore, we compared the RoBERTa encoder only model, as described above. In terms of all measures, the DMMS and DM 2 S 2 (with the Cloud Vision API) substantially outperformed Combined Classifiers, which uses features that are carefully designed for this dataset; moreover, a visual feature in Combined Classifiers is based on Cloud Vision API. This demonstrates the effectiveness of the BERTbased encoder and set-aware architectures. Despite the underperformance of Combined Classifiers with visual features from the Cloud Vision API compared to DM 2 S 2 and the DMMS, our DM 2 S 2 outperformed the DMMS when using TABLE 1. Comparison of GMU, Bilinear-Gated, MFAS, MMBT, DMMS, and DM 2 S 2 on the MM-IMDB dataset. Compared to state-of-the-art models that handle multiple input formats (i.e., images and text encoded by their respective encoders), the proposed method that handles a single input format (i.e., multiple modalities as text) achieves comparable or better prediction performance. In summary, our DM 2 S 2 significantly outperformed the conventional methods on public datasets. Compared with the baseline models that encode images directly, our model, which tokenizes even images, consistently exhibited superior results. We believe that our model can capture more semantic features by employing the OCR method to extract the features explicitly, instead of implicitly, from images. Our model is expected to be effective for modalities other than images, such as audio and video, in which text tokens can be obtained via transcription. Compared to the DMMS, which is based on a set-aware architecture and a BERTbased encoder, our DM 2 S 2 outperformed the baselines on all evaluation measures. Considering that the DMMS uses proprietary visual features, we also examined the effect of the OCR methods in DM 2 S 2 as far as possible. Table 3 displays the comparison of the prediction performance of MRA when using both Inter-and IntraMRA. Although InterMRA and IntraMRA are beneficial, even when they are used independently, DM 2 S 2 achieved the best performance by leveraging them simultaneously. In contrast, the performance gain of IntraMRA was slightly larger than that of InterMRA. This is possibly because inter-modality importance is essential for capturing several important tokens in a sequence when the number of tokens is large. However, InterMRA can quantify the importance of modalities rather than that of tokens, and is thus complementary to IntraMRA. We further examined the effectiveness of our concept on a real-world dataset, namely Production Ad-LP. As a baseline model for the Production Ad-LP dataset, we considered a gradient boosting decision tree [42] model that provides daily predictions of conversion rates from ads and LPs in the production environment at CyberAgent Inc. The model was implemented by using LightGBM [43] and used the same input features as those described in Section IV-B. Moreover, as ablated variants of DM 2 S 2 , we considered DM 2 S 2 with or without Intra-and InterMRA. Table 4 presents the results of the models in terms of the RMSE, MAPE, and AUC. To avoid exposing the raw measurements of the production model, the performance of each model was divided by that of the LightGBM model, as indicated at the top of the table. We considered the RMSE as the main metric for the production scenario, and we were interested in the difference between the true and predicted probabilities because the prediction of the exact probability (i.e., conversion rate) is crucial for a cost-effective ad. DM 2 S 2 with Intra-and InterMRA exhibited a substantial performance gain in terms of the RMSE compared to the LightGBM model. All measures revealed the performance gains of the Inter-and IntraMRA. We observed the same trend for this dataset as those for the MM-IMDB and Ads-Parallelity datasets, whereas the gain of IntraMRA was larger than that of InterMRA, and DM 2 S 2 achieved the best performance with both of them. Table 5 presents a comparison of the DM 2 S 2 prediction performance with different variations of the similarity function. We observed an average performance difference of approximately 1% between the additive attention and scaled dotproduct attention on both the MM-IMDB and Ads-Parallelity datasets. This result confirms that the additive attention in our DM 2 S 2 is effective. An extension that replaces MRAs can be considered by adding a Transformer module on top of the BERT encoder. However, the module is based on scaled dot-product attention, the results suggest that the replacement would exhibit limited performance gains. Table 6 displays a performance comparison of DM 2 S 2 with and without a residual connection in the MRA. In both the MM-IMDB and Ads-Parallelity datasets, the residual connection contributed to an average performance improvement of approximately 2%. Therefore, the residual connection plays an important role in MRA.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We compared the prediction performance when one of the modalities was erased to identify the modalities that contribute to the prediction in DM 2 S 2 . Fig. 2 depicts a visualization of the prediction performance of our model when one modality was erased during inference. We confirmed that erasing any modality feature impaired the performance.  Visualization of the change in performance when each modality was erased during inference. The performance was degraded regardless of which modality feature is erased, and this was particularly significant when erasing OCR text.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
Remarkably, a significant performance degradation was observed when the OCR text was erased. This result implies that the OCR text that is obtained from the visual modality contributes to the model's prediction.  Fig. 3 (a) and 3 (b), where the predictions were correct, the MRA captured the corresponding words (or subwords) that contributed to the prediction. For example, in Fig. 3 (a), we can observe a strong response to words relating to the documentary film in IntraMRA (e.g., "documentary" in movie plot). Furthermore, InterMRA could select important modalities; for example, it strongly focused on movie plot and OCR text features that contained words directly relating to prediction. Remarkably, it can be observed from Fig. 3 (a) that high IntraMRA weights were assigned to the split words "document" and "ary" in the OCR text. This is possibly because the BERT-based encoder encoded the two words (i.e., "document" and "ary") by considering the context, and IntraMRA could assign importance to them based on the encoded representations. Consequently, the errors in the OCR could be compensated. This can be viewed as evidence for that, even with EasyOCR, DM 2 S 2 can achieve comparable performance to that of the DMMS with a proprietary OCR method (see Tables 1 and 2). In the samples in which the model made an incorrect prediction (as illustrated in Fig. 3  (c)), although the model could capture important words such as "friends" in the movie plot, the overall importance was  dominated by the other modalities.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We have proposed a new concept for a multimodal set-input DNN architecture, namely DM 2 S 2 . Our proposed concept consists of three projections that capture the global and local relationships among multiple modalities. DM 2 S 2 is specifically designed for sequence sets, which are the structured data that are often collected in real multimodal applications. Whereas DM 2 S 2 is compatible with the sequential embeddings that are provided by recent BERT-based encoders, the Intra-and InterMRA capture the hierarchical importance of the elements in the modality sequences. We empirically demonstrated the effectiveness of our concept compared to state-of-the-art multimodal models. Furthermore, we examined its applicability to real applications by confirming the substantial performance gain from production-running models using real-world Ad data.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
In future work, we would like to explore attention mechanisms for various structures of multimodal data, such as grouped modalities (i.e., sets of sets) and hierarchical fields (i.e., sets of trees), further. Moreover, we are planning to investigate the theoretical relationship between our attentionbased fusion and conventional multimodal methods, such as gradient blending [17] and factorization machines [44]. .
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
The statistics for the Production Ad-LP dataset are listed in Table 4. The dataset was collected from CyberAgent Inc. 6 from August 1, 200, to November 30, 2020. This Japanese dataset contains 257,235 search engine advertisements (Ad) and landing pages (LPs), and these two modalities are called high-level modalities. Each high-level modality includes some low-level modalities. For example, the Ad modality contains search keywords, multiple titles (title_1, title_2, and title_3, where title_2 and title_3 are optional values), multiple descriptions of the ads (description_1 and description_2, where description_2 is an optional value), an LP URL path (path_1 and path_2, where both are optional), and a match type. 7 Several low-level modalities exist in the LP, such as the LP text that is obtained by applying optical character recognition (OCR) from LP screenshot images and safe search results (five types: adult, spoof, medical, violence, and racy), labels, object face detection results, and page performance scores with the lighthouse tool. 8 The statistics of the features of each row are shown in the Details column. We used a screenshot of the first view of the landing page at iPhone 8 resolution (1334 × 750). The lighthouse performance has a value between 0 and 100. We divided the dataset in an Ad group-based manner to train and evaluate the models. The Ad group 9 contained one or more ads that shared similar targets. In most advertising systems, ads are served in campaign units, and multiple creatives with similar trends are developed during a campaign. Each ad campaign consists of one or more ad groups. Thus, to avoid data leakage owing to potential similarity, we divided the dataset into a training set, a validation set, and test sets in an ad group-based manner. Note that, as our experiments were based on open datasets that are widely available to the public for performance comparison and results from analysis, no privacy handling issues would occur. In addition to the advertising data received from the company, information that could identify individuals was also discarded. Moreover, we took great care to ensure that the results of our analysis would not be disclosed in any manner that would violate privacy or ethics.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We used the OCR 10 , label detection 11 , object detection 12 , and facial detection 13 services to obtain visual features as tokens in the Google Cloud Vision API. We converted the obtained features, including numeric ones, as strings and treated them as tokens.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
We used the pre-trained RoBERTa [23] large encoder 14 in MM-IMDB and Ads-Parallelity following Reiter et al. [8] as 10 https://cloud.google.com/vision/docs/ocr 11 https://cloud.google.com/vision/docs/labels 12 https://cloud.google.com/vision/docs/object-localizer 13 https://cloud.google.com/vision/docs/detecting-faces 14 https://huggingface.co/roberta-large the BERT-based encoder. This encoder outputs a hidden state with dimension d = 1024. In the Production Ad-LP dataset, we used a pre-trained Japanese BERT 15 as the encoder. This encoder outputs a hidden state with dimension d = 768.
DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
To segment the Japanese text of the ads into words in the Ad-LP dataset, we used MeCab [45], which is a type of morphological analyzer. We used the analysis dictionary for the morphological analyzer MeCab (UniDicMA), as the custom dictionary. These are in accordance with the standard methods of the Japanese BERT.
Multimodal Abstractive Summarization for How2 Videos
In recent years, with the growing popularity of video sharing platforms, there has been a steep rise in the number of user-generated instructional videos shared online. With the abundance of videos online, there has been an increase in demand for efficient ways to search and retrieve relevant videos (Song et al., 2011;Wang et al., 2012;Otani et al., 2016;Torabi et al., 2016). Many cross-modal search applications rely on text associated with the video such as description or title to find relevant content. However, often videos do not have text meta-data associated with them or the existing ones do not provide clear information of the video content and fail to capture subtle differences between related videos (Wang et al., 2012). We address this by aiming to generate a short text summary of the video that describes the most salient content of the * *Work done while SG was at University of Edinburgh video. Our work benefits users through better contextual information and user experience, and video sharing platforms with increased user engagement by retrieving or suggesting relevant videos to users and capturing their attention.
Multimodal Abstractive Summarization for How2 Videos
Summarization is a task of producing a shorter version of the content in the document while preserving its information and has been studied for both textual documents (automatic text summarization) and visual documents such as images and videos (video summarization). Automatic text summarization is a widely studied topic in natural language processing (Luhn, 1958;Kupiec et al., 1995;Mani, 1999); given a text document the task is to generate a textual summary for applications that can assist users to understand large documents. Most of the work on text summarization has focused on single-document summarization for domains such as news (Rush et al., 2015;Nallapati et al., 2016;See et al., 2017;Narayan et al., 2018) and some on multi-document summarization (Goldstein et al., 2000;Lin and Hovy, 2002;Woodsend and Lapata, 2012;Cao et al., 2015;Yasunaga et al., 2017). Video summarization is the task of producing a compact version of the video (visual summary) by encapsulating the most informative parts (Money and Agius, 2008;Lu and Grauman, 2013;Gygli et al., 2014;Song et al., 2015;Sah et al., 2017). Multimodal summarization is the combination of textual and visual modalities by summarizing a video document with a text summary that summarizes the content of the video. Multimodal summarization is a more recent challenge with no benchmarking datasets yet. Li et al. (2017) collected a multimodal corpus of 500 English news videos and articles paired with manually annotated summaries.
Multimodal Abstractive Summarization for How2 Videos
The dataset is small-scale and has news articles with audio, video, and text summaries, but there are no human annotated audio-transcripts. today we are going to show you how to make spanish omelet . i 'm going to dice a little bit of peppers here . i 'm not going to use a lot , i 'm going to use very very little . a little bit more then this maybe . you can use red peppers if you like to get a little bit color in your omelet . some people do and some people do n't …. t is the way they make there spanish omelets that is what she says . i loved it , it actually tasted really good . you are going to take the onion also and dice it really small . you do n't want big chunks of onion in there cause it is just pops out of the omelet . so we are going to dice the up also very very small . so we have small pieces of onions and peppers ready to go . how to cut peppers to make a spanish omelette; get expert tips and advice on making cuban breakfast recipes in this free cooking video .
Multimodal Abstractive Summarization for How2 Videos
Figure 1: How2 dataset example with different modalities. "Cuban breakfast" and "free cooking video" is not mentioned in the transcript, and has to be derived from other sources.
Multimodal Abstractive Summarization for How2 Videos
Related tasks include image or video captioning and description generation, video story generation, procedure learning from instructional videos and title generation which focus on events or activities in the video and generating descriptions at various levels of granularity from single sentence to multiple sentences (Das et al., 2013;Regneri et al., 2013;Rohrbach et al., 2014;Zeng et al., 2016;Zhou et al., 2018;Zhang et al., 2018;Gella et al., 2018). A closely related task to ours is video title generation where the task is to describe the most salient event in the video in a compact title that is aimed at capturing users attention (Zeng et al., 2016). Zhou et al. (2018) present the YouCookII dataset containing instructional videos, specifically cooking recipes, with temporally localized annotations for the procedure which could be viewed as a summarization task as well although localized with time alignments between video segments and procedures.
Multimodal Abstractive Summarization for How2 Videos
In this work, we study multimodal summarization with various methods to summarize the intent of open-domain instructional videos stating the exclusive and unique features of the video, irrespective of modality. We study this task in detail using the new How2 dataset (Sanabria et al., 2018) which contains human annotated video summaries for a varied range of topics. Our models generate natural language descriptions for video content using the transcriptions (both user-generated and output of automatic speech recognition systems) as well as visual features extracted from the video. We also introduce a new evaluation metric (Content F1) that suits this task and present detailed results to understand the task better.
Multimodal Abstractive Summarization for How2 Videos
The How2 dataset (Sanabria et al., 2018) contains about 2,000 hours of short instructional videos, spanning different domains such as cooking, sports, indoor/outdoor activities, music, etc. Each video is accompanied by a human-generated transcript and a 2 to 3 sentence summary is available for every video written to generate interest in a potential viewer.
Multimodal Abstractive Summarization for How2 Videos
The example in Figure 1 shows the transcript describes instructions in detail, while the summary is a high-level overview of the entire video, mentioning that the peppers are being "cut", and that this is a "Cuban breakfast recipe", which is not mentioned in the transcript. We observe that text and vision modalities both contain complementary information, thereby when fused, helps in generating richer and more fluent summaries. Additionally, we can also leverage the speech modality by using the output of a speech recognizer as input to a summarization model instead of a human-annotated transcript. The How2 corpus contains 73,993 videos for training, 2,965 for validation and 2,156 for testing. The average length of transcripts is 291 words and of summaries is 33 words. A more general comparison of the How2 dataset for summarization as compared with certain common datasets is given in (Sanabria et al., 2018).
Multimodal Abstractive Summarization for How2 Videos
Video-based Summarization. We represent videos by features extracted from a pre-trained action recognition model: a ResNeXt-101 3D Convolutional Neural Network (Hara et al., 2018) trained to recognize 400 different human actions in the Kinetics dataset (Kay et al., 2017). These features are 2048 dimensional, extracted for every 16 non-overlapping frames in the video. This results in a sequence of feature vectors per video rather than a single/global one. We use these sequential features in our models described in Section 3.
Multimodal Abstractive Summarization for How2 Videos
Speech-based Summarization. We leverage the speech modality by using the outputs from a pretrained speech recognizer that is trained with other data, as inputs to a text summarization model. We use the state-of-the-art models for distantmicrophone conversational speech recognition, AS-pIRE (Peddinti et al., 2015) and EESEN (Miao et al., 2015;Le Franc et al., 2018). The word error rate of these models on the How2 test data is 35.4%. This high error mostly stems from normalization issues in the data. For example, recognizing and labeling "20" as "twenty" etc. Handling these effectively will reduce the word error rates significantly. We accept these as is for this task.
Multimodal Abstractive Summarization for How2 Videos
Transfer Learning. Our parallel work Sanabria et al. (2019) demonstrates the use of summarization models trained in this paper for a transfer learning based summarization task on the Charades dataset (Sigurdsson et al., 2016) that has audio, video, and text (summary, caption and question-answer pairs) modalities similar to the How2 dataset. Sanabria et al. (2019) observe that pre-training and transfer learning with the How2 dataset led to significant improvements in unimodal and multimodal adaptation tasks on the Charades dataset.
Multimodal Abstractive Summarization for How2 Videos
We study various summarization models. First, we use a Recurrent Neural Network (RNN) Sequenceto-Sequence (S2S) model (Sutskever et al., 2014) consisting of an encoder RNN to encode (text or video features) with the attention mechanism (Bahdanau et al., 2014) and a decoder RNN to generate summaries. Our second model is a Pointer-Generator (PG) model (Vinyals et al., 2015;Gülçehre et al., 2016) that has shown strong performance for abstractive summarization (Nallapati et al., 2016;See et al., 2017). As our third model, we use hierarchical attention approach of Libovický and Helcl 2017 originally proposed for multimodal machine translation to combine textual and visual modalities to generate text. The model first computes the context vector independently for each of videoframes············ResNeXtfeatures(w/RNN:7;w/oRNN:6,8,8,9)attention⊕hier.attn.(8,9)w...RNNdecoder video frames · · · · · · · · · · · · ResNeXt features (w/ RNN: 7; w/o RNN: 6,8,9) attention RNN over transcript (3-5, 8, 9) attention ⊕ hier. attn. the input modalities (text and video). In the next step, the context vectors are treated as states of another encoder, and a new vector is computed. When using a sequence of action features instead of a single averaged vector for a video, the RNN layer helps capture context. In Figure 2 we present the building block of our models.
Multimodal Abstractive Summarization for How2 Videos
We evaluate the summaries using the standard metric for abstractive summarization ROUGE-L (Lin and Och, 2004) that measures the longest common sequence between the reference and the generated summary. Additionally, we introduce the Content F1 metric that fits the template-like structure of the summaries. We analyze the most frequently occurring words in the transcription and summary. The words in transcript reflect the conversational and spontaneous speech while the words in the summaries reflect their descriptive nature. For examples, see Table A1 in Appendix A.2.
Multimodal Abstractive Summarization for How2 Videos
Content F1. This metric is the F1 score of the content words in the summaries based over a monolingual alignment, similar to metrics used to evaluate quality of monolingual alignment (Sultan et al., 2014). We use the METEOR toolkit (Banerjee and Lavie, 2005; Denkowski and Lavie, 2014) to obtain the alignment. Then, we remove function words and task-specific stop words that appear in most of the summaries (see Appendix A.2) from the reference and the hypothesis. The stop words are easy to predict and thus increase the ROUGE score. We treat remaining content words from the reference and the hypothesis as two bags of words and compute the F1 score over the alignment. Note that the  (2b), different text-only (3,4,5a), pointer-generator (5b), ASR output transcript (5c), video-only (6-7) and text-and-video models (8-9).
Multimodal Abstractive Summarization for How2 Videos
Text-only (5a) 3.86 3.78 3.78 3.92 Video-only (7) 3.58 3.30 3.71 3.80 Text-and-Video (8) 3.89 3.74 3.85 3.94 score ignores the fluency of output.
Multimodal Abstractive Summarization for How2 Videos
Human Evaluation. In addition to automatic evaluation, we perform a human evaluation to understand the outputs of this task better. Following the abstractive summarization human annotation work of Grusky et al. (2018), we ask our annotators to label the generated output on a scale of 1 − 5 on informativeness, relevance, coherence, and fluency. We perform this on randomly sampled 500 videos from the test set. We evaluate three models: two unimodal (text-only (5a), video-only (7)) and one multimodal (text-and-video (8)). Three workers annotated each video on Amazon Mechanical Turk. More details about human evaluation are in the Appendix A.5.
Multimodal Abstractive Summarization for How2 Videos
As a baseline, we train an RNN language model (Sutskever et al., 2011) on all the summaries and randomly sample tokens from it. The output obtained is fluent in English leading to a high ROUGE score, but the content is unrelated which leads to a low Content F1 score in Table 1. As another baseline, we replace the target summary with a rule-based extracted summary from the transcription itself. We used the sentence containing words "how to" with predicates learn, tell, show, discuss or explain, usually the second sentence in the transcript. Our final baseline was a model trained with the summary of the nearest neighbor of each video in the Latent Dirichlet Allocation (LDA; Blei et al., 2003) based topic space as a target. This model achieves a similar Content F1 score as the rulebased model which shows the similarity of content and further demonstrates the utility of the Content F1 score.
Multimodal Abstractive Summarization for How2 Videos
We use the transcript (either ground-truth transcript or speech recognition output) and the video action features to train various models with different combinations of modalities. The text-only model performs best when using the complete transcript in the input (650 tokens). This is in contrast to prior work with news-domain summarization (Nallapati et al., 2016). We also observe that PG networks do not perform better than S2S models on this data which could be attributed to the abstractive nature of our summaries and also the lack of common n-gram overlap between input and output which is the important feature of PG networks. We also use the automatic transcriptions obtained from a pretrained automatic speech recognizer as input to the summarization model. This model achieves competitive performance with the video-only models (described below) but degrades noticeably than Ground-truth Transcript (5a) avg:30.0, std:8.9 ASR output+Action Feat (9) avg:29.2, std:7.9 ASR output (5c) avg:29.2, std:7.9 First 200 (4) avg:29.3, std:7.6 Action only (6) avg:29.0, std:7.3 ground-truth transcription summarization model. This is as expected due to the large margin of ASR errors in distant-microphone open-domain speech recognition.
Multimodal Abstractive Summarization for How2 Videos
We trained two video-only models: the first one uses a single mean-pooled feature vector representation for the entire video, while the second one applies a single layer RNN over the vectors in time. Note that using only the action features in input reaches almost competitive ROUGE and Content F1 scores compared to the text-only model showing the importance of both modalities in this task. Finally, the hierarchical attention model that combines both modalities obtains the highest score.
Multimodal Abstractive Summarization for How2 Videos
In Table 2, we report human evaluation scores on our best text-only, video-only and multimodal models. In three evaluation measures, the multimodal models with the hierarchical attention reach the best scores. Model hyperparameter settings, attention analysis and example outputs for the models described above are available in the Appendix.
Multimodal Abstractive Summarization for How2 Videos
In Figure 3, we analyze the word distributions of different system generated summaries with the human annotated reference. The density curves show that most model outputs are shorter than human annotations with the action-only model (6) being the shortest as expected. Interestingly, the two different uni-modal and multimodal systems with groundtruth text and ASR output text features are very similar in length showing that the improvements in Rouge-L and Content-F1 scores stem from the difference in content rather than length. Example presented in Table A2 Section A.3 shows how the outputs vary.
Multimodal Abstractive Summarization for How2 Videos
We present several baseline models for generating abstractive text summaries for the open-domain videos in How2 data. Our presented models include a video-only summarization model that performs competitively with a text-only model. In the future, we would like to extend this work to generate multidocument (multi-video) summaries and also build end-to-end models directly from audio in the video instead of text-based output from pretrained ASR. We define and show the quality of a new metric, Content F1, for evaluation of the video summaries that are designed as teasers or highlights for viewers, instead of a condensed version of the input like traditional text summaries. We restrict the input length to 600 tokens for all experiments except the best text-only model in the section Experiments and Results. We use vocabulary the 20,000 most frequently occurring words which showed best results in our experiments, largely outperforming models using subword-based vocabularies. We ran all experiments with the nmtpytorch toolkit (Caglayan et al., 2017).
Multimodal Abstractive Summarization for How2 Videos
Transcript the, to, and, you, a, it, that, of, is, i, going, we, in, your, this, 's, so, on Summary in, a, this, to, free, the, video, and, learn, from, on, with, how, tips, for, of, expert, an Table A1 shows the frequent words in transcripts (input) and summaries (output). The words in transcripts reflect conversational and spontaneous speech while words in the summary reflect their descriptive nature. Table A2 shows example outputs from our different text-only and text-and-video models. The text-only model produces a fluent output which is close to the reference. The action features with the RNN model, which sees no text in the input, produces an in-domain ("fly tying"' and "fishing") abstractive summary that involves more details like "equipment" which is missing from the text-based models but is relevant. The action features without RNN model belongs to the relevant domain but contains fewer details. The nearest neighbor model is related to "knot tying" but not related to "fishing". The scores for each of these models reflect their respective properties. The random baseline output shows the output of sampling from the random language model based baseline. Although it is a fluent output, the content is incorrect. Observing other outputs of the model we noticed that although predictions were usually fluent leading to high scores, there is scope to improve them by predicting all details from the ground truth summary, like the subtle selling point phrases, or by using the visual features in a different adaptation model. Figure A1 shows an analysis of the attention distributions using the hierarchical attention model in an example video of painting. The vertical axis denotes the output summary of the model, and the horizontal axis denotes the input time-steps (from the transcript). We observe less attention in the first 1 Random Baseline 27.5 8.3 learn tips on how to play the bass drum beat variation on the guitar in this free video clip on music theory and guitar lesson . Table A2: Example outputs of ground-truth text-and-video with hierarchical attention (8), text-only with groundtruth (5a), text-only with ASR output (5c), ASR output text-andv-video with hierarchical attention (9), action features with RNN (7) and action features only (6) models compared with the reference, the topic-based next neighbor (2b) and random baseline (1). Arranged in the order of best to worst summary in this  part of the video where the speaker is introducing the task and preparing the brush. In the middle half, the camera focuses on the close-up of brush strokes with hand, to which the model pays higher attention over consecutive frames. Towards the end, the close up does not contain the hand but only the paper and brush, where the model again pays less attention which could be due to unrecognized ac-tions in the close-up. There are black frames in the very end of the video where the model learns not to pay any attention. In the middle of the video, there are two places with a cut in the video when the camera shifts angle. The model has learned to identify these areas and uses it effectively. From this particular example, we see the model using both modalities very effectively in this task of the summarization of open-domain videos.
Multimodal Abstractive Summarization for How2 Videos
To understand the outputs generated for this task better, we ask workers on Amazon Mechanical Turk to compare outputs of unimodal and multimodal models with the ground-truth summary and assign a score between 1 (lowest) and 5 (highest) for four metrics: informativeness, relevance, coherence and fluency of generated summary. The annotators were shown the ground-truth summary and a candidate summary (without knowledge of the type of modality used to generate it). Each example was annotated by three workers. Annotation was restricted to English speaking countries. 129 annotators participated in this task.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
With the popularity of video platforms, personal videos abound on the Internet. Multimodal summarization for open-domain videos, first organized as a track of the How2 Challenge at the ICML 2019 workshop, aims to integrate multisource information of videos (video, audio, transcript) into a fluent textual summary. An example can be seen in Figure 1. This study, which uses compressed text description to reflect the salient parts of videos, * Corresponding author. is of considerable significance for helping users better retrieve and recommend videos.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Existing approaches have obtained promising results. For example, Libovickỳ et al. (2018) and Palaskar et al. (2019) utilize multiple encoders to encode videos and audio transcripts and a joint decoder to decode the multisource encodings, which acquire better performance than single modality structures. Despite the effectiveness of these approaches, they only perform multimodal fusion during the decoding stage to generate a target sequence, lacking fine-grained interactions between multisource inputs to complete the missing information of each modality. For example, as shown in Figure 1, text context representations containing birds should be associated with visual semantic information containing parrots to build thorough multimodal representations.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Besides, unlike other multimodal tasks such as visual question answering (Antol et al., 2015;Gao et al., 2015) and multimodal machine translation (Elliott et al., 2015;Specia et al., 2016), a major challenge is that this task has longer input sequences with more noise and redundancy. The flow of noise information during multimodal fusion, such as redundant frames in video and noisy words in transcription, interferes with the interaction and complementarity of the effective information between modalities, which leads to a significant negative effect on the model. Moreover, when using an automatic speech recognition (ASR) system to transform audio to transcription instead of ground-truth transcription, high noise ASR-output transcripts further reduce model performance.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
To address these two issues, we propose a multistage fusion network with the fusion forget gate module for multimodal summarization in videos. The model involves multiple information fusion processes to capture the correlation between multisource modalities spontaneously, and a fusion for-in this clip we 're going to file allister 's nails down with a drill . that will help smooth out the nails once again making it comfortable for you to hold your bird as well as comfortable for him . you do n't want any sharp edges on there . it just files it down . you want to use a medium speed on the drill so that you have control over it and it 's not going too fast . and all we 're doing with this is taking off the very tip of the nail after we 've trimmed it just to smooth it out . we 're not going to need to do much with it other than to make it smooth so it 's comfortable to hold him . the bird does not get hurt by the drill but you do want to make sure that the other toes are out of the way of the drill so that the drill piece is not going against their skin . once again , that can be difficult to do , you have to pry their toes apart to get them opened to drill them .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Summary after trimming your parrot 's nails , file them with a dremel to make the nail smooth ; learn more pet parrot care in this free pet care video about parrots . Figure 1: The audio transcript does not mention "parrot", only "bird" or "allister". The complete summary has to be derived from multi-source. This example is taken from the How2 dataset.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
get gate is proposed to effectively suppress the flow of unnecessary multimodal noise. As illustrated in Figure 2, our proposed multistage fusion model mainly consists of four modules: 1) multisource encoders to build representations for video and audio (ground-truth or ASR-output transcript); 2) cross fusion block in which cross fusion generator (CFG) and a feature-level fusion layer are designed to generate and fuse latent adaptive streams from one modality to another at low levels of granularity; 3) hierarchical fusion decoder (HFD) in which hierchical attention networks are designed to progressively fuse multisource features carrying adaptive streams from other modalities to generate a target sequence; 4) fusion forget gate (FFG) (detailed in Figure 3) in which a memory vector and a forget vector are created for the information streams in the cross fusion block to alleviate interference from long-range redundant multimodal information.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We build our proposed model on both RNNbased (Sutskever et al., 2014) and transformerbased (Vaswani et al., 2017) encoder-decoder architectures and evaluate our approach on the largescale public multimodal summarization dataset, How2 (Sanabria et al., 2018). Experiments show that our model achieves a new state-of-the-art performance. Comprehensive ablation experiments and visualization analysis demonstrate the effectiveness of our multistage fusion schema and forgetting module.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Specially, we also evaluate the model performances under the ASR-output transcript. We use an automatic speech recognition (ASR) system (Google-Speech-V2) to generate audio transcripts (word error rate>30%) to replace the ground-truth transcripts provided by the How2 dataset. Exper-iments show that our model still achieves performance close to the model trained with ground-truth transcripts, and significantly outperforms the stateof-the-art system, which indicates the advantage of our model in the absence of ground-truth transcript annotation.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The extracted ASR-output transcripts and code will be released on https://github.com/ forkarinda/MFN.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
...  which we focus on in this paper. A similar task is video captioning (Venugopalan et al., 2015a,b), which mainly places emphasis on the use of visual information to generate descriptions, but this task focuses on how to make full use of multisource and multimodal long inputs to obtain a summary and additionally needs ground-truth transcripts. Recent methods use multiencoder-decoder RNNs to process multisource inputs but lack the interaction and complementarity between multisource modalities and the ability to resist the flow of multimodal noise. To handle above two challenges, our multistage fusion model is introduced.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
In this section, we will explain our model in detail. The overall architecture of our proposed model is shown in Figure 2, and the fusion forget gate inside is illustrated in Figure 3. Specifically, multistage fusion consists of the cross fusion block and hierarchical fusion decoder, which aims to model the correlation and complementarity between modalities spontaneously. In addition, the fusion forget gate is applied in the cross fusion block to filter the flow of redundant information streams. We build our model based on the RNN and transformer encoder-decoder architectures, respectively.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Our multimodal summarization system takes a video and a ground-truth or ASR-output audio transcription as input and generates a textual summary that describes the most salient part of the video. Formally, the transcript is a sequence of word tokens T = (t 1 , ..., t n ) and the video representation is denoted by V = (v 1 , ..., v m ), where v m is the feature vector extracted by a pretrained model. The output summary is denoted as a sequence of word tokens S = (s 1 , ..., s l ) consisting of several sentences. The task aims to predict the best summary sequence S by finding:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
where θ is the set of trainable parameters.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Encoding Video. The video encoding features are obtained by a pretrained action recognition model: a ResNeXt-101 3D convolutional neural network (Hara et al., 2018) trained for recognizing 400 different human actions in the Kinetics dataset (Kay et al., 2017).
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The video representation features denoted by V = (v 1 , ..., v m ) are extracted every 16 nonoverlapping frames, where v m is the 2048-dimensional vector.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We add learnable position embeddings for video features.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Encoding Transcript. For the RNN encoder, we use a bidirectional GRU (Cho et al., 2014) to encode the text to obtain a contextualized representation for each word:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
For the transformer encoder, we employ an universal bidirectional transformer encoder (Vaswani et al., 2017) in which each layer is composed of a multihead self-attention layer followed by a feedforward sublayer with residual connections (He et al., 2016) and layer normalizations (Ba et al., 2016), and denoted by the following equation:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We use learnable position embedding instead of sinusoidal position embedding.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The cross fusion generator (CFG) is used to correlate meaningful elements across modalities. We apply the CFG to generate the adaptive fusion information from one modality encoding to another. The CFG learns two cross-modal attention maps, one is from text to video, and the other is from video to text. It is inspired by parallel co-attention (Lu et al., 2016), which computes an affinity matrix between two sequences, while we apply two unidirectional matrices instead of assigning shared parameters to both directions, and use scaled dot-product attention (Vaswani et al., 2017). At each of the crossmodal attention maps, the low-level signals from the source modality are transformed to key and value pairs to interact with the target modality as a query. Following the two maps, CFG is divided into the video-to-text fusion generator (V2TFG) and text-to-video fusion generator (T2VFG), which are detailed as follows:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Text-to-video Fusion Generator (T2VFG). The T2VFG generates the most relevant video information to low-level text features by a text-to-video cross-modal attention map. The cross-modal attention consists of text queries Q T = T W Q T , video key and value pairs K T = V W K T , V T = V . The contextual video vector derived from the cross-modal attention map is calculated by
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
where the common spatial parameter W α is used to simplify the calculations. Video-to-Text Fusion Generator (V2TFG). Similar to the T2VFG, the V2TFG aims to generate the latent adaptive text information stream for video modality. The difference between the V2TFG and T2VFG is that they flow in opposite directions. We transform the low-level video features to queries Q V = V W Q V and the text to key and value pairs
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
where W β is a mapping of text flowing to video.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Although the CFG builds an unsupervised lowlevel signal alignment between original multisource features, noise modality information generated by CFG is hard to be suppressed. In particular, when the whole modality cannot guide the task at all, the forced normalization of the softmax function in the attention structure makes the calculated fusion vector generated by the noise modality hard to be suppressed. For this reason, we propose a fusion forget gate (FFG) to filter low-level cross-modal adaptation information of each modality generated by the CFG. The FFG reads the original modal signals as well as the adaptation information derived from other modalities, and determines whether the adaptation information is noise and matches the original modality. As shown in Figure 3, we assign a video FFG and a text FFG to receive bidirectional adaption information that originated from the CFG.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Specifically, it creates a memory vector and a forget gate to control the flow of noise and mismatched information. First, we project the connected source and target modality embeddings and activate them with a sigmoid function to obtain a forget vector:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Then the adaptation information passes a linear mapping to obtain a memory vector, which prevents essential information from being weighted down due to the scaling limit of the sigmoid function ranging from 0 to 1. We apply the dot-product to the memory vector and the forget vector to represent the cross-modal adaptive stream after FFG filtering, which is finally calculated as follows:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
where represents elementwise dot production and W V , W T , W 1 , W 2 , b V , b T , b 1 and b 2 are trainable parameters.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
This module combines the low-level signal T /V of the original modality with the matching adaptive stream V /T of other modalities. The fusion vector flowing through CFG and FFG has the same sequence length as the original modality so that we apply a concat&forward layer with a ReLU activation function. In addition, we specially add a residual connection inside the fusion layer to deepen the neural network's memory of the original modality. The calculation formulas are below:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
where W 1 , W 2 , b 1 , b 2 are trainable parameters.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The HFD receives multimodal information of different granularity from multisource inputs and generates a target sequence. Inspired by hierarchical attention (Libovickỳ and Helcl, 2017), HFD transforms the decoder hidden states and multisource encodings into a context vector by three attention maps: video attention, text attention, and attention over multimodal attention (AoMA). At each decoding time step t, the decoder hidden state h t attends to video/text encodings V F /T F carrying aligned multimodal information separately via video/text attention to calculate the video/text context vector:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Then, a second attention mechanism is constructed over the two context vectors, and a higher-level context vector is computed. We concatenate the two contexts and apply a new MLP attention:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The context vector of hierarchical multimodal fusion is finally obtained and combined with the decoder hidden state vector to compute an output for attending the next decoder layer or caculating the vocabulary distribution.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Corresponding to the two encoders introduced in section 3.2, we design RNN-based and transformerbased decoding strategies. The formula expression and model diagram of the two structures are detailed in Appendix A.1.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We evaluate our method on the How2 dataset (Sanabria et al., 2018). The How2 dataset is a large-scale dataset of open-domain videos, spanning different topics, such as cooking, sports, indoor/outdoor activities, and music. It consists of 79,114 how-to instructional videos with an average length of 1.5 minutes and a total of 2,000 hours, accompanied by corresponding ground-truth English transcripts with an average length of 291 words, crowdsourced Portuguese translations of transcripts and user-generated summaries with an average length of 33 words. The statistics are shown in Figure 4 and Table 1.  
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We also extract audio transcripts by a speech recognition system (Google-Speech-V2). The word error rate (WER) of the speech-recognition output on the How2 test data is 32.9%.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We compare our model with the following baseline models of single or multiple modalities: S2S (Luong et al., 2015): a standard sequenceto-sequence architecture using an RNN encoderdecoder with a global attention mechanism.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
PG (See et al., 2017): a commonly used encoderdecoder summarization model with attention (Bahdanau et al., 2015), which combines copying words from source documents and outputting words from a vocabulary.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
FT: a strong baseline that applies a transformerbased encoder-decoder model to a flat sequence.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
VideoRNN ( MT : a transformer-based encoder-decoder architecture receiving sequence features of video for end-to-end dense video captions.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
HA (RNN/Transformer) (Palaskar et al., 2019): a multisource sequence-to-sequence model with a hierarchical attention approach to combine textual and visual modalities, which is currently the stateof-the-art method for the multimodal summarization task on the How2 dataset.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
For the RNN-based models, we uniformly use a 2-layer GRU with 128-dimensional word embeddings and 256-dimensional hidden states for each direction. We truncate the maximum text sequence length to 600.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
For the transformer-based models, we uniformly use a 4-layer transformer of 512 dimensions with 8 heads. We truncate the maximum text sequence length to 800, and the maximum video sequence length to 1024.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
For both the two architectures, we use the crossentropy loss and Adam optimizer (Kingma and Ba, 2015). The initial learning rate is set to 1.5e −4 . All trainable parameters are randomly initialized with the Kaiming initialization (He et al., 2015). The training of the proposed models are conducted on {1, 2} GeForce RTX 2080 Ti GPUs for 50 epochs with a batch size of {4, 16}. During decoding for prediction, we use beam search with a beam size of 6 and a length penalty with α = 1 (Wu et al., 2016).
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
For a fair comparison, following Palaskar et al. (2019), all the methods take the same 2048-dimensional video features extracted from a ResNeXt-101 3D convolutional neural network (Hara et al., 2018) as input; the vocabulary is built based on the How2 data, and do not use pre-trained word embeddings.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We adopt multiple automatic metrics to comprehensively evaluate model performance: BLEU (1,2,3,4) (Papineni et al., 2002), ROUGE (1,2,L) (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and CIDEr (Vedantam et al., 2015). Table 2 shows the results for different models on the How2 dataset. Table 3 shows the model performances of using automatic transcripts obtained from a speech recognition system instead of ground-truth transcripts provided by the dataset. The results show that our proposed model achieves the state-of-the-art performance in each evaluation metric on both the RNN-based and transformer-based models. It can also be seen that the performances of the pure video modality models are modest because of the frozen video features extracted from a task-independent pretraining model.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
In particular, Table 3 shows that when the performances of all the prior models trained with ASR-   Table 3: Results on the How2 test set. The ASR-output transcripts is used to replace the provided ground-truth transcripts. The down arrow (↓) indicates the performance degradation when using ASR-output transcript to replace ground-truth transcript under the same model.   Table 5: Ablation analysis on RNN-based models. The ASR-output transcripts is used to replace the provided ground-truth transcripts.  Table 6: Ablation analysis on the How2 test set.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
ASR-output Transcript: first thing you have to do is attach the thread to the hook . what do you want to do as security . i suggest . lacrosse and fatherhood . and then wrap . backwards that way . just enough to catch . that . standing . piece of the . trader . therefore raps is usually good . and then . you can just depends on what you doing you can leave it hanging out you can clip it off close there . but you're now . my thread is not good . come loose . some radio star attachment other materials . sometimes you can go in wrap it all the way back . no just make sure you get it on there secure . rabbits back the other way . and then start retiring or if you want to start . start time back here grab it back and keep it back . but the trick is a just make those first couple of laps trap that . then go back this way few times . and then either continue back to the back of the head . turn up to the front . and um . the . gives a good song . foundation to start time Summary: watch and learn how to tie thread to a hook to help with fly tying as explained by out expert in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Ground-truth Transcript: alvin dedeux : first thing you have to do is attach the thread to the hook , and what you want to do is secure it . i usually just lay it across in front of the hook and then wrap backwards that way just enough to catch that standing piece of the thread there . three or four wraps is usually good and then you can just , depending on what you 're doing , you can leave it hanging or you could clip it off close there . but now , my thread is not going to come loose so i 'm ready to start attaching my other materials . sometimes you can go ahead and wrap it all the way back , just make sure you got it on there secure , wrap it back the other way and then start your tying . or if you want to start tying back here , you 'd wrap it back here and keep it back here . but the trick is to just make those first couple of wraps , trap that thread and then go back this way a few times . and then either continue back to the back of the hook or up to the front . and that gives you a good solid foundation to start tying your fly . Figure 5: A example taken from How2 test set. For the extracted ASR-output transcripts, we use the period "." as the separator of the automatically segmented audio clips.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
output transcripts drop sharply due to the high error rate (W ER = 32.9%) of speech recognition, our model still has good performance close to the models trained with ground-truth transcripts. In using ASR-output transcripts, our framework outperforms the HA 8.3 BLEU-4 points, 7.4 ROUGE-L points, 3.9 METEOR points, and 71.8 CIDEr points on the RNN-based architecture, and 6.9 BLEU-4 points, 4.9 ROUGE-L points, 2.7 ME-TEOR points, and 42.8 CIDEr points on the transformer-based architecture, which fully shows the effectiveness of our approach.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The purpose of this study is to examine the role of the proposed multistage fusion and fusion forget gate (FFG). We divide the fusion process into transcript-to-video-fusion (T2VF) and video-totranscript fusion (V2TF) in the cross fusion block, the following FFG, and the final HFD, and retrain our approach by ablating one or more of them.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
• We retrain only T2VF and only V2TF and replace HFD with a standard decoder to handle single-source multimodal encodings.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
• We add the FFG to the above T2VF and V2TF models separately.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
• We retain T2VF, V2TF, HFD, and remove all the FFG of the full model. Table 4 lists the results on the How2 dataset. We can observe that: 1) except that the V2TF's per-formance is weaker than the single-text modality on RNN {1a}, the performances of all the V2TF and T2VF models {3a, 1b, 3b} exceed the performances of the single-modality models. 2) Compared with using only V2TF or T2VF, using V2TF and T2VF together with HFD {5a, 5b} further improves the model effect. 3) When FFG is added, the performances of all the fusion structures improve, which is particularly evident in the RNNbased models. 4) Only one-way fusion structures with FFG {2a,4a,2b,4b} can achieve comparable and even better performance compared to the HA. These results demonstrate the effectiveness of the multistage fusion and inside FFG. Table 5 lists the results of using the ASR-output transcript instead of the provided ground-truth transcript. The observation results are similar to those observed in Table 4. In particular, we can see a greater increase in the performance of the FFG when using high noise ASR-output trancript compared to using the ground-truth transcript. This further verifies the ability of FFG to the resist the flow of multimodal noise.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Additionally, we also evaluate 1) the effect of model depth and 2) the effect of FFG on HFD. We deepen the model depth, and apply FFG to the multimodal context representation generated by the AoMA in HFD. The results in Table 6 indicate that the two measures do not improve model performance.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Method R-L Output -Reference -watch and learn how to tie thread to a hook to help with fly tying as explained by out expert in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Ground-truth transcript FT 0.543 learn about attaching the thread in fly tying and other fly fishing tips in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Video MT 0.468 learn how to attach a backing tail to fly fishing backing in this free how-to video on fly tying and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Ground-truth transcript+Video HA (RNN) 0.557 learn from our expert how to attach a hook to fly tying in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
HA (Trm) 0.559 learn about using a bobbin in fly tying from our expert in this free how-to video on techniques for and making fly tying nymphs .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Proposed (RNN) 0.582 watch and learn from an expert how to attach the thread to fly tying in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Proposed (Trm) 0.574 learn some great tips on attaching the thread to the fly fishing in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
ASR-output transcript+Video HA (RNN) 0.487 tying a knot for fly fishing is easy with these tips , get expert advice on woodworking in this free video .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
HA (Trm) 0.501 tying a knot onto a knot , make sure the snap is secure and connected to the hoop knot . attach a french braid to a knot with tips from an experienced handyman in this free video on fly tying .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Proposed (RNN) 0.561 watch and learn from our expert on fly fishing tips in this free how-to video on fly tying tips and techniques .
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Proposed (Trm) 0.550 learn how to use a wrapped knot to wrap a fly fishing knot in this free how-to video on fly tying tips and techniques . in this clip we 're going to file allister 's nails down with a drill . that will help smooth out the nails once again making it comfortable for ... 
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We provide some example outputs from trained models. The example is taken from the How2 test set, and we show its ground-truth transcript and the extracted ASR-output transcript in Figure 5. Table  7 lists the generated results. We can observe that: 1) compared to single-modality models, the multimodality models can generate more accurate and fluent contents. 2) In using ground-truth transcript, both HA and our proposed model generate accurate and fluent summaries. 3) In using ASR-output transcripts, our proposed model still generates a relatively accurate summary while the content generated by HA is not accurate enough, which intuitively illustrates the advantage of our model in the absence of ground-truth transcripts.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
To better understand what our model has learned, we take the sample shown in Figure 1 to visualize the FFG and cross-attention in CFG. We sum the FFG weights and use the color depth of the word to represent the intensity of the FFG of controlling the flow of video to text, and demonstrate the interaction between video and text by displaying the video frame with the highest transcript-to-video attention when generating adaptive video streams. As shown in Figure 6, in the input segment, we can observe the following: 1) For some words related to the summary such as "file", "nails", the FFG retains video streams for it, in contrast, for words such as "once again", the FFG forgets most of the video information. 2) For the words that FFG remembers deeply, the corresponding video frame has a certain correlation with it, for example, "file allister's nails" point to a close-up of manicuring the parrot's nails.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We introduce a multistage fusion network with fusion forget gate for generating text summaries for the open-domain videos. We propose a multistep fusion schema to model fine-grained interactions between multisource modalities and a fusion forget gate module to handle the flow of multimodal noise of multisource long sequences. Experiments on the How2 dataset show the effectiveness of the proposed models. Furthermore, when using high noise speech recognition transcription, our model still achieves the effect of being close to the groundtruth transcription model, which reduces the manual annotation cost of transcripts. compute a new hidden state h t , which is defined as:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The context vectors of each modality are firstly calculated by:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We adopt an MLP attention for RNN-based methods. Then the second attention AoMA over the video context vectors C V and text context vectors C T are implemented as:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The context vector C C of multimodal fusion and the decoder state h t are merged to get the output state y t+1 :
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
where W 1 , W 2 , W 3 , W and b are trainable parameters.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Transformer-based HFD. Transformer-based HFD has a similar strategy as RNN-based. We mainly introduce how it absorbs multimodal information. It firstly receives target token embeddings x t through the masked multi-head self-attention and residual connection to obtain the hidden state vector h t , denoted as:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Then h t is transformed into a query, separately attends to a set of key and value pairs mapped by previous encodings of each modality by the multihead encoder-decoder attention, denoted as:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
Similarly, the generated multimodal context vectors are fused by AoMA:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The final output state reaches through the feedforward and add&norm layer like the general transformer, calculated as the following equation:
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
y t+1 = W 2 ReLu(W 1 (C c +h t )+b 1 )+b 2 +C c +h t (26) where W 1 , W 2 , b 1 and b 2 are trainable parameters.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
We use the nmtpytorch evaluation library https: //github.com/lium-lst/nmtpytorch suggested by the How2 Challenge, which includes BLEU (1, 2, 3, 4), ROUGE-L, METEOR, and CIDEr evaluation metrics. As an alternative, nlg-eval https: //github.com/Maluuba/nlg-eval can obtain the same evaluation score as nmtpytorch.
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
In addition, we also use a ROUGE evaluation library https://github.com/ neural-dialogue-metrics/rouge, which supports the evaluation of ROUGE series metrics (ROUGE-N, ROUGE-L and ROUGE-W).
Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos
The extracted ASR-output transcript data is available on https://github.com/forkarinda/MFN.