Text-dominated multimodal text classification with quadruplet attention

Visual-linguistic interaction is a problem of addressing the obstacles of Information Deficiency and Weight Deviation in text classification. Information deficiency usually occurs in vision-dominated or modality-balanced tasks, and many multimodal fusion approaches have been proposed (e.g. Gated-based and Contextualized-based methods). However, there is still no remarkable solution to the weight deviation of irrelevant descriptions in textdominated tasks. To solve it, we introduce a novel Quadruplet Attention to adjust the textual-visual weight distribution, where visual-linguistic information interacts with each other and dot product can be represented as a 2 × 2 matrix, referred to as quadruplet. Then, a Multimodal Architecture is further proposed to enhance text-dominated classification. Extensive experiments on Daily Mail have proved the effectiveness of our method, which achieves significant improvements of 1.42 and 3.4 respectively in ROUGE-L F1 and Marco F1.