How has GAN been applied to multimodal tasks?
Adversarial Network. The concept of adversarial
network starts from GAN (Goodfellow et al., 2014), which has attracted a surge of research interests
due to its strong ability of generating high-quality
novel samples according to existing data. The best-
known applications include image-to-image trans-
lation (Isola et al., 2017) and image synthesis (Den-
ton et al., 2015; Radford et al., 2015). Recently,
GAN is further applied to multimodal tasks such
as text-to-image synthesis (Reed et al., 2016; Tan
et al., 2020), video captioning (Yang et al., 2018;
Bai et al., 2021) and cross-modal retrieval (Qian
et al., 2021). In this work, we leverage the strong
distinguishing ability of adversarial network to
strengthen the modality agnosticism of the learned
modality-invariant representations.
Video Captioning by Adversarial LSTM
In this paper, we propose a novel approach to video captioning based on adversarial learning and long short-term memory (LSTM). With this solution concept, we aim at compensating for the deficiencies of LSTM-based video captioning methods that generally show potential to effectively handle temporal nature of video data when generating captions but also typically suffer from exponential error accumulation. Specifically, we adopt a standard generative adversarial network (GAN) architecture, characterized by an interplay of two competing processes: a “generator” that generates textual sentences given the visual content of a video and a “discriminator” that controls the accuracy of the generated sentences. The discriminator acts as an “adversary” toward the generator, and with its controlling mechanism, it helps the generator to become more accurate. For the generator module, we take an existing video captioning concept using LSTM network. For the discriminator, we propose a novel realization specifically tuned for the video captioning problem and taking both the sentences and video features as input. This leads to our proposed LSTM–GAN system architecture, for which we show experimentally to significantly outperform the existing methods on standard public datasets.
[{"first":"Yang","last":"Yang","middle":[]},{"first":"Jie","last":"Zhou","middle":[]},{"first":"Jiangbo","last":"Ai","middle":[]},{"first":"Yi","last":"Bin","middle":[]},{"first":"Alan","last":"Hanjalic","middle":[]},{"first":"Heng","last":"Shen","middle":["Tao"]},{"first":"Yanli","last":"Ji","middle":[]}]
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
This paper presents a new framework, Knowledge-Transfer Generative Adversarial Network (KT-GAN), for fine-grained text-to-image generation. We introduce two novel mechanisms: an Alternate Attention-Transfer Mechanism (AATM) and a Semantic Distillation Mechanism (SDM), to help generator better bridge the cross-domain gap between text and image. The AATM updates word attention weights and attention weights of image sub-regions alternately, to progressively highlight important word information and enrich details of synthesized images. The SDM uses the image encoder trained in the Image-to-Image task to guide training of the text encoder in the Text-to-Image task, for generating better text features and higher-quality images. With extensive experimental validation on two public datasets, our KT-GAN outperforms the baseline method significantly, and also achieves the competive results over different evaluation metrics.
[{"middle":[],"last":"Tan","first":"Hongchen"},{"middle":[],"last":"Liu","first":"Xiuping"},{"middle":[],"last":"Liu","first":"Meng"},{"middle":[],"last":"Yin","first":"Baocai"},{"middle":[],"last":"Li","first":"Xin"}]
Discriminative Latent Semantic Graph for Video Captioning
Video captioning aims to automatically generate natural language sentences that can describe the visual contents of a given video. Existing generative models like encoder-decoder frameworks cannot explicitly explore the object-level interactions and frame-level information from complex spatio-temporal data to generate semantic-rich captions. Our main contribution is to identify three key problems in a joint framework for future video summarization tasks. 1) Enhanced Object Proposal: we propose a novel Conditional Graph that can fuse spatio-temporal information into latent object proposal. 2) Visual Knowledge: Latent Proposal Aggregation is proposed to dynamically extract visual words with higher semantic levels. 3) Sentence Validation: A novel Discriminative Language Validator is proposed to verify generated captions so that key semantic concepts can be effectively preserved. Our experiments on two public datasets (MVSD and MSR-VTT) manifest significant improvements over state-of-the-art approaches on all metrics, especially for BLEU-4 and CIDEr. Our code is available at https://github.com/baiyang4/D-LSG-Video-Caption.
[{"first":"Yang","last":"Bai","middle":[]},{"first":"Junyan","last":"Wang","middle":[]},{"first":"Yang","last":"Long","middle":[]},{"first":"Bingzhang","last":"Hu","middle":[]},{"first":"Yang","last":"Song","middle":[]},{"first":"Maurice","last":"Pagnucco","middle":[]},{"first":"Yu","last":"Guan","middle":[]}]
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Audio-visual speech recognition (AVSR) attracts a surge of research interest recently by leveraging multimodal signals to understand human speech. Mainstream approaches addressing this task have developed sophisticated architectures and techniques for multi-modality fusion and representation learning. However, the natural heterogeneity of different modalities causes distribution gap between their representations, making it challenging to fuse them. In this paper, we aim to learn the shared representations across modalities to bridge their gap. Different from existing similar methods on other multimodal tasks like sentiment analysis, we focus on the temporal contextual dependencies considering the sequence-to-sequence task setting of AVSR. In particular, we propose an adversarial network to refine frame-level modality-invariant representations (MIR-GAN), which captures the commonality across modalities to ease the subsequent multimodal fusion process. Extensive experiments on public benchmarks LRS3 and LRS2 show that our approach outperforms the state-of-the-arts.
[{"first":"Yuchen","last":"Hu","middle":[]},{"first":"Chen","last":"Chen","middle":[]},{"first":"Ruizhe","last":"Li","middle":[]},{"first":"Heqing","last":"Zou","middle":[]},{"first":"Eng Siong","last":"Chng","middle":[]}]
Generative Adversarial Text to Image Synthesis
Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal. However, in recent years generic and powerful recurrent neural network architectures have been developed to learn discriminative text feature representations. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to generate highly compelling images of specific categories, such as faces, album covers, and room interiors. In this work, we develop a novel deep architecture and GAN formulation to effectively bridge these advances in text and image model- ing, translating visual concepts from characters to pixels. We demonstrate the capability of our model to generate plausible images of birds and flowers from detailed text descriptions.
[{"first":"Scott","last":"Reed","middle":[]},{"first":"Zeynep","last":"Akata","middle":[]},{"first":"Xinchen","last":"Yan","middle":[]},{"first":"Lajanugen","last":"Logeswaran","middle":[]},{"first":"Bernt","last":"Schiele","middle":[]},{"first":"Honglak","last":"Lee","middle":[]}]
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Cross-modal retrieval has become an active study ﬁeld with the expanding scale of multimodal data. To date, most existing methods transform multimodal data into a common rep- resentation space where semantic similarities between items can be directly measured across different modalities. How- ever, these methods typically suffer from following limita-tions: 1) They usually attempt to bridge the modality gap by designing losses in the common representation space which may not be sufﬁcient to eliminate potential heterogeneity of different modalities in the common space. 2) They typically treat labels as independent individuals and ignore la- bel relationships which are important for constructing semantic links between multimodal data. In this work, we propose a novel Dual Adversarial Graph Neural Networks (DAGNN) composed of the dual generative adversarial net- works and the multi-hop graph neural networks, which learn modality-invariant and discriminative common representa- tions for cross-modal retrieval. Firstly, we construct the dual generative adversarial networks to project multimodal data into a common representation space. Secondly, we leverage the multi-hop graph neural networks, in which a layer aggre- gation mechanism is proposed to exploit multi-hop propagation information, to capture the label correlation dependency and learn inter-dependent classiﬁers. Comprehensive experiments conducted on two cross-modal retrieval benchmark datasets, NUS-WIDE and MIRFlickr, indicate the superiority of DAGNN.
[{"first":"Shengsheng","last":"Qian","middle":[]},{"first":"Dizhan","last":"Xue","middle":[]},{"first":"Huaiwen","last":"Zhang","middle":[]},{"first":"Quan","last":"Fang","middle":[]},{"first":"Changsheng","last":"Xu","middle":[]}]
Video Captioning by Adversarial LSTM
Vi deo Captioning by Adve rsarial LSTM Yan g Yan g , Member , IEEE , Ji e Zhou, Ji angbo Ai , Y i Bin, Ala n Hanjali c, Fellow, IEEE,
Video Captioning by Adversarial LSTM
Abstract-In this paper, we propose a novel approach to video captioning based on adversarial learning and long shortterm memory (LSTM). With this solution concept, we aim at compensating for the deficiencies of LSTM-based video captioning methods that generally show potential to effectively handle temporal nature of video data when generating captions but also typically suffer from exponential error accumulation. Specifically, we adopt a standard generative adversarial network (GAN) architecture, characterized by an interplay of two competing processes: a "generator" that generates textual sentences given the visual content of a video and a "discriminator" that controls the accuracy of the generated sentences. The discriminator acts as an "adversary" toward the generator, and with its controlling mechanism, it helps the generator to become more accurate. For the generator module, we take an existing video captioning concept using LSTM network. For the discriminator, we propose a novel realization specifically tuned for the video captioning problem and taking both the sentences and video features as input. This leads to our proposed LSTM-GAN system architecture, for which we show experimentally to significantly outperform the existing methods on standard public datasets.
Video Captioning by Adversarial LSTM
Index Terms-Video captioning, adversarial training, LSTM.
Video Captioning by Adversarial LSTM
V IDEO captioning is referred to as the problem of generating a textual description for a given video content. The interdisciplinary nature of this problem opens vast new possibilities for interacting with video collections and there has been increased research effort on this topic observable over the past years [1]- [5]. This interdisciplinary nature, however, also poses significant research challenges at the intersection between the fields of natural language processing and computer vision. Typically, these challenges have been pursued as extrapolations of the solutions proposed earlier for image captioning [6]- [8]. These solutions perform classification in the visual domain with the goal to generate salient regions, linking jiangboml@gmail.com; yi.bin@hotmail.com; shenhengtao@hotmail.com; yanliji@uestc.edu.cn).
Video Captioning by Adversarial LSTM
V IDEO captioning is referred to as the problem of generating a textual description for a given video content. The interdisciplinary nature of this problem opens vast new possibilities for interacting with video collections and there has been increased research effort on this topic observable over the past years [1]- [5]. This interdisciplinary nature, however, also poses significant research challenges at the intersection between the fields of natural language processing and computer vision. Typically, these challenges have been pursued as extrapolations of the solutions proposed earlier for image captioning [6]- [8]. These solutions perform classification in the visual domain with the goal to generate salient regions, linking jiangboml@gmail.com; yi.bin@hotmail.com; shenhengtao@hotmail.com; yanliji@uestc.edu.cn).
Video Captioning by Adversarial LSTM
A. Hanjalic is with the Multimedia Computing Group, Intelligent Systems Department, Delft University of Technology, 2628CD Delft, The Netherlands (e-mail: a.hanjalic@tudelft.nl).
Video Captioning by Adversarial LSTM
Color versions of one or more of the figures in this paper will be available online at http://ieeexplore.ieee.org.
Video Captioning by Adversarial LSTM
Digital Object Identifier 10.1109/TIP.2018.2855422 these regions with some predefined textual attributes and then synthesize a sentence by completing a predefined generative model using the recognized attribute [1], [3], [9]- [12].
Video Captioning by Adversarial LSTM
Digital Object Identifier 10.1109/TIP.2018.2855422 these regions with some predefined textual attributes and then synthesize a sentence by completing a predefined generative model using the recognized attribute [1], [3], [9]- [12].
Video Captioning by Adversarial LSTM
Different from static pictures [13], [14], the content of a video is significantly more rich and unlikely to be captured well by simply extrapolating the methods developed for images. This richness comes mainly through the temporal aspect of video content. And comparing to the retrieval and annotation technology [15]- [18], the captioning task relies more on narrowing the semantic gap between the visual and textual information. In order to take this aspect into account, Yao et al. [1] proposed a 3D Convolutional Neural Network (3D-CNN) structure, applying convolution not only spatially, but also temporally. However, 3D-CNN can only capture the information over a short period of time due to the limit of the convolution kernel size. Venugopalan et al. [3] implemented a Long-Short Term Memory (LSTM) network, a variant of a Recurrent Neural Network (RNNs), to model the global temporal structure in an entire video snippet. However, this method was shown to accumulate the grammatical errors exponentially and to result in decreasing association among the generated words with the increasing video length. Furthermore, the traditional caption generative models usually select words with maximum probability, which typically results in a rather monotonous set of generated sentences, like for instance around the words "playing" and "doing", both appearing rather often in the common training data sets.
Video Captioning by Adversarial LSTM
Different from static pictures [13], [14], the content of a video is significantly more rich and unlikely to be captured well by simply extrapolating the methods developed for images. This richness comes mainly through the temporal aspect of video content. And comparing to the retrieval and annotation technology [15]- [18], the captioning task relies more on narrowing the semantic gap between the visual and textual information. In order to take this aspect into account, Yao et al. [1] proposed a 3D Convolutional Neural Network (3D-CNN) structure, applying convolution not only spatially, but also temporally. However, 3D-CNN can only capture the information over a short period of time due to the limit of the convolution kernel size. Venugopalan et al. [3] implemented a Long-Short Term Memory (LSTM) network, a variant of a Recurrent Neural Network (RNNs), to model the global temporal structure in an entire video snippet. However, this method was shown to accumulate the grammatical errors exponentially and to result in decreasing association among the generated words with the increasing video length. Furthermore, the traditional caption generative models usually select words with maximum probability, which typically results in a rather monotonous set of generated sentences, like for instance around the words "playing" and "doing", both appearing rather often in the common training data sets.
Video Captioning by Adversarial LSTM
In order to eliminate this deficiency of LSTM, in this paper, we propose a novel approach that expands the LSTM concept towards an adversarial learning concept. As illustrated in Fig. 1, this expansion involves adding a "discriminator" module to the system architecture, which acts as an adversary with respect to the sentence generator. While the generator has the objective to make the generated sentences as close to its existing generative model as possible, the discriminator has the objective to ensure that generated sentences are reasonable and natural for people better understanding. This interplay of two concurring processes has recently been introduced as adversarial learning, with a Generative Adversarial Network (GAN) [19]- [21] as a basic realization. Consequently, we refer to our proposed approach as LSTM-GAN. We will demonstrate that this expansion of the LSTM concept will enable the video captioning process to improve the accuracy and diversity of generated captions and their robustness to increasing video length.
Video Captioning by Adversarial LSTM
In order to eliminate this deficiency of LSTM, in this paper, we propose a novel approach that expands the LSTM concept towards an adversarial learning concept. As illustrated in Fig. 1, this expansion involves adding a "discriminator" module to the system architecture, which acts as an adversary with respect to the sentence generator. While the generator has the objective to make the generated sentences as close to its existing generative model as possible, the discriminator has the objective to ensure that generated sentences are reasonable and natural for people better understanding. This interplay of two concurring processes has recently been introduced as adversarial learning, with a Generative Adversarial Network (GAN) [19]- [21] as a basic realization. Consequently, we refer to our proposed approach as LSTM-GAN. We will demonstrate that this expansion of the LSTM concept will enable the video captioning process to improve the accuracy and diversity of generated captions and their robustness to increasing video length.
Video Captioning by Adversarial LSTM
Applying a GAN to the context of video captioning is, however, not straightforward. A GAN is designed for real-valued, but continuous data and may have difficulty handling sequences of discrete words or tokens as mentioned An illustration of the modular structure of the proposed video captioning model characterizerd by an interplay of the generator G that generates text sentences and the discriminator D (adversary) that verifies the sentences. The optimization goal is that G deceives D, by generating sentences that are not distinguishable from reference sentences.
Video Captioning by Adversarial LSTM
in [22]. The reason lies in that the gradient of the loss from the discriminator based on the output of the generator is used to move the generator to slightly change the way the sentences are generated. However, if the output of the generator consists of discrete tokens, the slight change guidance by the discriminator may not work because there may be no token in the used dictionary to signal the desired level of change towards the generator [20].
Video Captioning by Adversarial LSTM
In order to overcome this problem, we propose a embedding layer which can transform the discrete outputs into a consecutive representation [23]. Besides that, since the outputs of our generative model are a sequence, ordinary discriminative model, consisted of several fully connected layers, has a poor ability for classifying the sequence-sentence. For solving this problem, we propose a new realization of the discriminative model. Specifically, we replace the fully connected layer, as originally proposed in [19], with a novel convolutional structure, previously proposed by Zhang et al. [25], Kim [26], and Collobert et al. [27]. Our discriminative model consists of convolutional layer, max-pooling layer and fully connected layer. The convolutional layer will produce local features and retain the local coherence around each word of the sequence-sentence. After max-pooling layer, the most important information of the sentence will be effectively extracted. Those informations are denoted by a fixed length of vector. Additionally, we also introduce a multimodal input for the discriminative model. We sent not only the sentence to the discriminative module but also the video feature generated from our first LSTM layer (Encoder) of generative module. The novel methods for incorporating the original inputs with the video feature will help to generate more relevant discriptions about the input video. This method has been confirmed to be effective by our experiment.
Video Captioning by Adversarial LSTM
In order to overcome this problem, we propose a embedding layer which can transform the discrete outputs into a consecutive representation [23]. Besides that, since the outputs of our generative model are a sequence, ordinary discriminative model, consisted of several fully connected layers, has a poor ability for classifying the sequence-sentence. For solving this problem, we propose a new realization of the discriminative model. Specifically, we replace the fully connected layer, as originally proposed in [19], with a novel convolutional structure, previously proposed by Zhang et al. [25], Kim [26], and Collobert et al. [27]. Our discriminative model consists of convolutional layer, max-pooling layer and fully connected layer. The convolutional layer will produce local features and retain the local coherence around each word of the sequence-sentence. After max-pooling layer, the most important information of the sentence will be effectively extracted. Those informations are denoted by a fixed length of vector. Additionally, we also introduce a multimodal input for the discriminative model. We sent not only the sentence to the discriminative module but also the video feature generated from our first LSTM layer (Encoder) of generative module. The novel methods for incorporating the original inputs with the video feature will help to generate more relevant discriptions about the input video. This method has been confirmed to be effective by our experiment.
Video Captioning by Adversarial LSTM
To our knowledge, we are the first to propose the method for generating video description via advesarial learning. The remainder of this paper is organized as follows. In Section II, we review the existing work related to the problem of video captioning and position our proposed solution with respect to it. Section III presents our proposed LSTM-GAN video captioning model. The model is evaluated experimentally in Section IV where we compare our approach with the relevant existing methods on four public datasets: MSVD, MSR-VTT, M-VAD and MSR-VTT. This comparison revealed that our approach significantly outperforms the existing methods on these datasets, which justifies our methodological and algorithmic design choices. Section V concludes the paper by a discussion of the obtained video captioning performance and pointers to future work on this topic.
Video Captioning by Adversarial LSTM
Generating a textual description for a given video content, also called video captioning, has been showing increasingly strong potential in computer vision. The primary challenges of this research lie in two aspects: adequately extracting the information from the video sequences and generating grammar-correct sentences easy for the human to understand. The early research for generating video descriptions mainly focused on extracting useful information e.g., object, attribute, and preposition, from given video content. Krishnamoorthy et al. [27] aim at generating more precise words to describe the objects in the video. Their method includes a content planning stage and a surface realization stage. In their work, object detection and activity recognition modular are used to extract the related words about the video and then it applies a template-based approach to generate sentences. Reference [28] is another work about using templatebased approach. Different from [27], the biggest contribution of this paper is building a hierarchical semantic model to classify different words. Besides, this model can even detect unseen verbs with the help of knowledge mining from webscale textual corpora. No doubt that the performances of those methods are limited by the accuracy of the detection of the word and the robustness of the template. In order to overcome the problem abovementioned, some novel approaches based on RNNs [28]- [35] are proposed with the development of deep learning. Different from the template-based approach in [27], Pan et al. [10] take advantage of the CNN architecture to get the representation of every frame and then applies an average pooling operation to those frame presentations. For generating video description of the inputted video, they take advantage of the RNNs architecture to produce sequence output with an end-to-end training pattern. What's more, it is noteworthy that they innovatively embed the sentence and the video content into a feature with a uniform dimension. And by measuring the feature diversity of the embedded sentences and input videos, they get the value of relevance loss between these two features. By minimizing this loss, the semantic gap will become narrowed. With the popularization of Convolutional Neural Network for an impressive performance in feature extraction, generating the description for a video has made great breakthroughs. Pan et al. [10] used both 2D CNN and 3D CNN to process video clips and average pooling operation to process over all the clips to generate a single D v -dimension vector which will be used to generate descriptions for video. But all the abovementioned methods could not work well for a short video clip containing multi-event. Yu et al. [36] proposed an attention model which use an attention mechanism to select the most important video features and has yielded impressive performance in multi-event videos.
Video Captioning by Adversarial LSTM
Generating a textual description for a given video content, also called video captioning, has been showing increasingly strong potential in computer vision. The primary challenges of this research lie in two aspects: adequately extracting the information from the video sequences and generating grammar-correct sentences easy for the human to understand. The early research for generating video descriptions mainly focused on extracting useful information e.g., object, attribute, and preposition, from given video content. Krishnamoorthy et al. [27] aim at generating more precise words to describe the objects in the video. Their method includes a content planning stage and a surface realization stage. In their work, object detection and activity recognition modular are used to extract the related words about the video and then it applies a template-based approach to generate sentences. Reference [28] is another work about using templatebased approach. Different from [27], the biggest contribution of this paper is building a hierarchical semantic model to classify different words. Besides, this model can even detect unseen verbs with the help of knowledge mining from webscale textual corpora. No doubt that the performances of those methods are limited by the accuracy of the detection of the word and the robustness of the template. In order to overcome the problem abovementioned, some novel approaches based on RNNs [28]- [35] are proposed with the development of deep learning. Different from the template-based approach in [27], Pan et al. [10] take advantage of the CNN architecture to get the representation of every frame and then applies an average pooling operation to those frame presentations. For generating video description of the inputted video, they take advantage of the RNNs architecture to produce sequence output with an end-to-end training pattern. What's more, it is noteworthy that they innovatively embed the sentence and the video content into a feature with a uniform dimension. And by measuring the feature diversity of the embedded sentences and input videos, they get the value of relevance loss between these two features. By minimizing this loss, the semantic gap will become narrowed. With the popularization of Convolutional Neural Network for an impressive performance in feature extraction, generating the description for a video has made great breakthroughs. Pan et al. [10] used both 2D CNN and 3D CNN to process video clips and average pooling operation to process over all the clips to generate a single D v -dimension vector which will be used to generate descriptions for video. But all the abovementioned methods could not work well for a short video clip containing multi-event. Yu et al. [36] proposed an attention model which use an attention mechanism to select the most important video features and has yielded impressive performance in multi-event videos.
Video Captioning by Adversarial LSTM
The combination of CNN and LSTM architecture has become the mainstream form for video captioning research which also demonstrated a excellent performance. Both feature extraction and words generation methods are important for the quality of description for the input video. Although the LSTM scheme has proved promising performance for handling the temporal nature of video data in the temporal process, the LSTM scheme critical deficiency is shown to accumulate the grammatical errors exponentially and may result in decreasing association among the generated words with the increasing video length. Based on the problem, we consider if there is a structure that can discriminate whether the generated descriptions are reasonable and relevant to the video. Inspired by the generative adversarial network firstly for generating an image, we proposed our model LSTM-GAN incorporating a joint LSTMs with adversarial learning. This model consists of a generative model and discriminative model. The generative model is used for encoding the video clips and generates sentences, while the discriminative model is trying to distinguish whether the input sentences are from reference sentence or generated sentences (as in Fig. 2).
Video Captioning by Adversarial LSTM
Generative Adversarial Networks (GAN), introduced by Goodfellow et al. [19], has achieved promising success in generating realistic synthetic real-valued data. On account of impressive performance, GAN has been widely applicated in computer vision (CV) and natural language processing (NLP). Original Generative Adversarial Networks consist of generative model and discriminative model. The generative model tries to produce data, which is to mix the spurious with the genuine. While the discriminative model learns to determine whether the data is from genuine distribution or not. Typical applications can be found in, for instance, image synthesis [22] where discrete data with normal distribution are used to generate realistic images. Besides showing the excellent performances in the image processing, GAN structures also work well in natural language processing recently. Li et al. [37] proposed a dialog generation method using Adversarial reinforce model. They adopted policy gradient training method to encourage the generator to generate utterances that are indistinguishable from human-generated dialogs. Later, Press et al. [38] presented a text generating model with RNNs for both generator and discriminator. Different from policy gradient training in [37], this work applied curriculum learning, as a result, it vastly improves the quality of generated sequences. For captioning research, Dai et al. [39] firstly presented an image captioning method, which can generate sentences much more closely to what human saying than original caption models.
Video Captioning by Adversarial LSTM
Generative Adversarial Networks (GAN), introduced by Goodfellow et al. [19], has achieved promising success in generating realistic synthetic real-valued data. On account of impressive performance, GAN has been widely applicated in computer vision (CV) and natural language processing (NLP). Original Generative Adversarial Networks consist of generative model and discriminative model. The generative model tries to produce data, which is to mix the spurious with the genuine. While the discriminative model learns to determine whether the data is from genuine distribution or not. Typical applications can be found in, for instance, image synthesis [22] where discrete data with normal distribution are used to generate realistic images. Besides showing the excellent performances in the image processing, GAN structures also work well in natural language processing recently. Li et al. [37] proposed a dialog generation method using Adversarial reinforce model. They adopted policy gradient training method to encourage the generator to generate utterances that are indistinguishable from human-generated dialogs. Later, Press et al. [38] presented a text generating model with RNNs for both generator and discriminator. Different from policy gradient training in [37], this work applied curriculum learning, as a result, it vastly improves the quality of generated sequences. For captioning research, Dai et al. [39] firstly presented an image captioning method, which can generate sentences much more closely to what human saying than original caption models.
Video Captioning by Adversarial LSTM
In summary, our work presents the first effort to incorporate the GAN with the video caption model. Our model can overcome the aforementioned problem and the performance of our model is also excellent through our experimental results.
Video Captioning by Adversarial LSTM
Fig. 2 depicts our LSTM-GAN architecture which consists of a generator G, and a discriminator D. The generator G uses an encoder-decoder architecture to generate descriptions for relevant videos under the umbrella of incorporating a CNN architecture as the discriminator D which is for evaluating whether the generated sentences reasonable or not. Specifically, we begin this section by introducing the fundamental Long Short-Term Memory Networks (LSTM) and Generative Adversarial Network (GAN) model briefly. In the remainder of the module, we elaborate on the algorithm's theory and design choices underlying the proposed framework in more detail.
Video Captioning by Adversarial LSTM
Traditional RNNs [40], [41] is designed to learn complex temporal dynamics by mapping the input sequences to a sequence of hidden states and then generating outputs via the following recurrence equations as Eq. 1:
Video Captioning by Adversarial LSTM
Traditional RNNs [40], [41] is designed to learn complex temporal dynamics by mapping the input sequences to a sequence of hidden states and then generating outputs via the following recurrence equations as Eq. 1:
Video Captioning by Adversarial LSTM
where the W h , U h denote the weight matrices and b denotes the bias, ψ is an element-wise non-linear function, such as RELU or hyperbolic tangent. x t is the input, h t ∈ R M is the hidden state with M hidden units, o t is the output at time t. The traditional RNNs have proven successful in text generation and speech recognition. But it is difficult to handle well about long-range temporal dependencies videos because of the exploding and vanishing gradient problem. The LSTM network proposed by Hochreiter and Schmidhuber [41] has demonstrated to be able to effectively prevent the gradient vanishing and explosion problems [41] during back-propagation through time (BPTT) [42]. This is because it incorporates memory units, which facilitate the network to learn long-range temporal dependencies, to forget previously hidden states and to update them given new information. More specifically, as illustrated in Fig. 3 and implemented in our framework, LSTM incorporates several control gates and a memory cell. Let x t , c t and h t represent the input, cell memory and hidden control states at each time t respectively. Given a sequence of inputs (x 1 , . . . , x T ), the LSTM will compute the hidden control sequence (h 1 , . . . , h T ) and the cell memory sequence (c 1 , . . . , c T ). Formally, this process can be described by the following set of equations:
Video Captioning by Adversarial LSTM
where the W h , U h denote the weight matrices and b denotes the bias, ψ is an element-wise non-linear function, such as RELU or hyperbolic tangent. x t is the input, h t ∈ R M is the hidden state with M hidden units, o t is the output at time t. The traditional RNNs have proven successful in text generation and speech recognition. But it is difficult to handle well about long-range temporal dependencies videos because of the exploding and vanishing gradient problem. The LSTM network proposed by Hochreiter and Schmidhuber [41] has demonstrated to be able to effectively prevent the gradient vanishing and explosion problems [41] during back-propagation through time (BPTT) [42]. This is because it incorporates memory units, which facilitate the network to learn long-range temporal dependencies, to forget previously hidden states and to update them given new information. More specifically, as illustrated in Fig. 3 and implemented in our framework, LSTM incorporates several control gates and a memory cell. Let x t , c t and h t represent the input, cell memory and hidden control states at each time t respectively. Given a sequence of inputs (x 1 , . . . , x T ), the LSTM will compute the hidden control sequence (h 1 , . . . , h T ) and the cell memory sequence (c 1 , . . . , c T ). Formally, this process can be described by the following set of equations:
Video Captioning by Adversarial LSTM
(2) Fig. 3. A diagram of a basic LSTM memory cell used in our paper, where the input gate, forget gate and output gate represents by i t , g t , o t respectively.
Video Captioning by Adversarial LSTM
Here, stands for element-wise product and W j -like matrices are the LSTM weight parameters. Additionally, σ and ϕ denote the sigmoid and hyperbolic non-linear functions respectively.
Video Captioning by Adversarial LSTM
In a nutshell, a GAN-based learning approach [19] involves simultaneous training of two network models, generator G and discriminator D. In the field of image super-resolution, image generation, image to image translation, Neuro-Linguistic Programming, GAN has made great contributions [43]- [46]. Through the interplay of the two learning processes, the G and D models facilitate each other interactively to individually reach their goals. The generator G tries to generate real data given a noize z∼P noise (z), while the discriminator D ∈ [0, 1] aims at classifing the real data x∼ p data (x) and the fake data G(z)∼P G (z) generated from G. More specifically, the generator G also learns to generate samples from the generator distribution P G by transforming a noise data z∼P noise (z) into a sample G z (z will be replaced with input video to generate corresponding descriptions in our later experiments). Similarly, the goal of D is to distinguish between samples from the true data distribution P data and the generator's distribution P G as accurate as possible. After a period of competition between the two processes, the two network models will achieve some degree of balance with the optimal discriminator being D(x) = P data (x)/(P data (x)+ P G (z)) and with the generator being able to generate data which are difficult for the discriminator D to distinguish whether from training data or synthetic data. This "game" is steered towards convergence by the optimization criteria expressed through loss functions, designed for both G and D. We define the following loss functions L G and L D :
Video Captioning by Adversarial LSTM
In a nutshell, a GAN-based learning approach [19] involves simultaneous training of two network models, generator G and discriminator D. In the field of image super-resolution, image generation, image to image translation, Neuro-Linguistic Programming, GAN has made great contributions [43]- [46]. Through the interplay of the two learning processes, the G and D models facilitate each other interactively to individually reach their goals. The generator G tries to generate real data given a noize z∼P noise (z), while the discriminator D ∈ [0, 1] aims at classifing the real data x∼ p data (x) and the fake data G(z)∼P G (z) generated from G. More specifically, the generator G also learns to generate samples from the generator distribution P G by transforming a noise data z∼P noise (z) into a sample G z (z will be replaced with input video to generate corresponding descriptions in our later experiments). Similarly, the goal of D is to distinguish between samples from the true data distribution P data and the generator's distribution P G as accurate as possible. After a period of competition between the two processes, the two network models will achieve some degree of balance with the optimal discriminator being D(x) = P data (x)/(P data (x)+ P G (z)) and with the generator being able to generate data which are difficult for the discriminator D to distinguish whether from training data or synthetic data. This "game" is steered towards convergence by the optimization criteria expressed through loss functions, designed for both G and D. We define the following loss functions L G and L D :
Video Captioning by Adversarial LSTM
log(1 − D(G(z (i) ))),
Video Captioning by Adversarial LSTM
where {z (1) , . . . , z (m) } are a mini-batch of m noise samples by random initialization and {x (1) , . . . , x (m) } are a mini-batch of m samples from true data distribution p data (x).
Video Captioning by Adversarial LSTM
where {z (1) , . . . , z (m) } are a mini-batch of m noise samples by random initialization and {x (1) , . . . , x (m) } are a mini-batch of m samples from true data distribution p data (x).
Video Captioning by Adversarial LSTM
Consider a video V including a sequence of n sample frames where V = {v 1 , v 2 , . . . , v n }, with associated caption S where S = {w 1 , w 2 , . . . , w m } consisting of m words. Let v i ∈ R D v and w j ∈ R D w denote the D v -dimensional visual presentations of the i-th frame in video V and the D w -dimensional textual features of the j-th word in sentence S, respectively. In our work, our goal is to maximize the conditional probability of an output sequence (w 1 , . . . , w m ) given an input sequence  (v 1 , . . . , v n ). The conditional probabilities over the sentences can be defined as follows: p(w 1 , . . . , w m |v 1 , . . . , v n ).
Video Captioning by Adversarial LSTM
This problem is similar to the problem of machine translation in natural language processing, where a sequence of words serves as input into a generative model that outputs a sequence of words as the translation result. What is different from aforementioned is that, in our work, we replace the textual input by our video frames and look forward to a sequence of caption as output. What is more, we not only expect to get the relevant description of the input videos but also to make the sentences natural and reasonable for people to understand.
Video Captioning by Adversarial LSTM
For the sake of overcoming the above-mentioned problem, in this section, we devise our model to generate video description under the umbrella of an adversarial system. Specifically, our overall framework consists of a generative model G and discriminative model D. The generative model G, similar to sequence-to-sequence models [47], defines the policy that generates a sequence of the relevant description given a short video. The discriminative model D is a binary classifier that takes a sequence of sentences {s, y} as input and outputs a label D(S) ∈ [0, 1] indicating whether the sentence is natural, reasonable and grammatical correct. In particular, several variants of our designed model are utilized to compare with other methods. We now elaborate on the implementation of our designed architecture.
Video Captioning by Adversarial LSTM
For the sake of overcoming the above-mentioned problem, in this section, we devise our model to generate video description under the umbrella of an adversarial system. Specifically, our overall framework consists of a generative model G and discriminative model D. The generative model G, similar to sequence-to-sequence models [47], defines the policy that generates a sequence of the relevant description given a short video. The discriminative model D is a binary classifier that takes a sequence of sentences {s, y} as input and outputs a label D(S) ∈ [0, 1] indicating whether the sentence is natural, reasonable and grammatical correct. In particular, several variants of our designed model are utilized to compare with other methods. We now elaborate on the implementation of our designed architecture.
Video Captioning by Adversarial LSTM
1) Objective Function: In order to achieve faster convergence of the objective, we firstly pre-training the generative model G and the discriminative model D, respectively. For G, similar to sequence-to-sequence models [47], our goal is to estimate the conditional probability p(S|V ) where V = (v 1 , v 2 , . . . , v t ) is an input sequence consisting of a sample of frames and S = (w 1 , w 2 , . . . , w t 1 ) is the corresponding output sequence as a descriptive texture for the input video. t and t 1 represents the length of the video and the input sentence respectively. As sequence-to-sequence models [3], we conclude the follow objective function:
Video Captioning by Adversarial LSTM
1) Objective Function: In order to achieve faster convergence of the objective, we firstly pre-training the generative model G and the discriminative model D, respectively. For G, similar to sequence-to-sequence models [47], our goal is to estimate the conditional probability p(S|V ) where V = (v 1 , v 2 , . . . , v t ) is an input sequence consisting of a sample of frames and S = (w 1 , w 2 , . . . , w t 1 ) is the corresponding output sequence as a descriptive texture for the input video. t and t 1 represents the length of the video and the input sentence respectively. As sequence-to-sequence models [3], we conclude the follow objective function:
Video Captioning by Adversarial LSTM
From Equation (5) for our model with θ and output sequence S = (w 1 , w 2 , . . . , w t 1 ), we could get the optimal θ with the follow formula:
Video Captioning by Adversarial LSTM
log p(w t |h n+t −1 , w 1 , . . . , w t −1 ; θ). (6) where h n+t −1 denotes the hidden state at time step n + t − 1 which will be introduced detaily in the next section. For D, our primary purpose is to train a classifier which can be used for sentence encoding and mapping the input sentence to an output D(S) ∈ [0, 1] representing the probability of S is from the ground-truth captions, rather than from adversarial generator. The objective function of D for pre-training can be formalized into a cross-entropy loss as follow: (7) where m denote the number of examples in a batch, Y i and D(S i )) represent the real label and predicted value of discriminator respectively. Hence, when we train our overall framework called LSTM-GAN, the whole training procedure for the LSTM-GAN is same as in Fig. 2. We aim at minimizing the log likelihood formulated as follows:
Video Captioning by Adversarial LSTM
log p(w t |h n+t −1 , w 1 , . . . , w t −1 ; θ). (6) where h n+t −1 denotes the hidden state at time step n + t − 1 which will be introduced detaily in the next section. For D, our primary purpose is to train a classifier which can be used for sentence encoding and mapping the input sentence to an output D(S) ∈ [0, 1] representing the probability of S is from the ground-truth captions, rather than from adversarial generator. The objective function of D for pre-training can be formalized into a cross-entropy loss as follow: (7) where m denote the number of examples in a batch, Y i and D(S i )) represent the real label and predicted value of discriminator respectively. Hence, when we train our overall framework called LSTM-GAN, the whole training procedure for the LSTM-GAN is same as in Fig. 2. We aim at minimizing the log likelihood formulated as follows:
Video Captioning by Adversarial LSTM
2) Generative Model: As mentioned before, We use a joint recurrent neural networks, also called encoder-decoder LSTM similar to sequence-to-sequence models [47], as the generative model. The encoder architecture is used to encode the video features into a fixed dimension vector. While the decoder architecture decodes the vector into natural sentences. To begin with, we adopt VGG16 [48] as the CNN architecture to map the sequence frames V = (v 1 , v 2 , . . . , v t ) into a feature matrix W v ∈ R D d ×D t = (w D 1 , . . . , w D t ). D d and D t denote the dimensions of a feature vector and the number of frames, respectively. Particularly, the encoder LSTM net, referred to as "encoder", maps the input embedding presentations w D 1 , . . . , w D t , namely features matrix, into a sequence of hidden states h 1 , h 2 , . . . , h t mentioned in previous section by using the update function as eq. (2) recursively.
Video Captioning by Adversarial LSTM
2) Generative Model: As mentioned before, We use a joint recurrent neural networks, also called encoder-decoder LSTM similar to sequence-to-sequence models [47], as the generative model. The encoder architecture is used to encode the video features into a fixed dimension vector. While the decoder architecture decodes the vector into natural sentences. To begin with, we adopt VGG16 [48] as the CNN architecture to map the sequence frames V = (v 1 , v 2 , . . . , v t ) into a feature matrix W v ∈ R D d ×D t = (w D 1 , . . . , w D t ). D d and D t denote the dimensions of a feature vector and the number of frames, respectively. Particularly, the encoder LSTM net, referred to as "encoder", maps the input embedding presentations w D 1 , . . . , w D t , namely features matrix, into a sequence of hidden states h 1 , h 2 , . . . , h t mentioned in previous section by using the update function as eq. (2) recursively.
Video Captioning by Adversarial LSTM
According to the above description, we make it clear that the last status h t as the presentations of the whole video, generated from "encoder", will be sent to the decoder LSTM which is referred to as "decoder". Specifically, given the h t (in our Fig. (2) marked as h t −1 for straightforward) and a corresponding sentence S also referred to textual description in Fig. 1, encoded with one-hot vectors (1-of-N encoding, where N is the size of the vocabulary), our decoder LSTM is conditioned step by step on the t th word and on the previous h t −1 , and is trained to produce the next word of the description for input video. We commit ourselves to minimize objective function as eq. (5) to generate an excellent performance generative model. The probability of those words is modeled via a softmax function applied on the output of the decoder. As we know, those words should be in an one-hot format which not only are high dimensionality but also are discrete and discontinuous. Passing those words with a one-hot format to the D (Discriminator) will make it difficult to pass the gradient update. Although score function based algorithms, such as REINFORCE [49] obtains unbiased gradient estimation for discrete variables by using Monte Carlo estimation. However, the variance of the gradient estimation could be large [50]. In order to cope with this problem, we adopt a soft-argmax function similar to the one proposed in [24]:
Video Captioning by Adversarial LSTM
According to the above description, we make it clear that the last status h t as the presentations of the whole video, generated from "encoder", will be sent to the decoder LSTM which is referred to as "decoder". Specifically, given the h t (in our Fig. (2) marked as h t −1 for straightforward) and a corresponding sentence S also referred to textual description in Fig. 1, encoded with one-hot vectors (1-of-N encoding, where N is the size of the vocabulary), our decoder LSTM is conditioned step by step on the t th word and on the previous h t −1 , and is trained to produce the next word of the description for input video. We commit ourselves to minimize objective function as eq. (5) to generate an excellent performance generative model. The probability of those words is modeled via a softmax function applied on the output of the decoder. As we know, those words should be in an one-hot format which not only are high dimensionality but also are discrete and discontinuous. Passing those words with a one-hot format to the D (Discriminator) will make it difficult to pass the gradient update. Although score function based algorithms, such as REINFORCE [49] obtains unbiased gradient estimation for discrete variables by using Monte Carlo estimation. However, the variance of the gradient estimation could be large [50]. In order to cope with this problem, we adopt a soft-argmax function similar to the one proposed in [24]:
Video Captioning by Adversarial LSTM
Here, W e ∈ R Z ×C is a word embedding matrix (to be learned) which is similar to the GloVe [51] and transforms the one-hot encoding of words to a dense lower dimensional embedding, C is the dimension of the embedded word (1024 in our experiments) and Z is the size of vocabulary in our training data. V is the set of parameters and encodes the h t −1 to a vector. w t −1 represents the generated word of LSTM at t th step. L is a big enough integer which would make the vector of softmax V h t −1 L closes to a one-hot form. Each value of it is constrained to be either approximately 0 or 1 which can help the w t −1 more close to W e [t − 1] (suppose the value at the t − 1 position is the largest of V h t −1 ) and also help the word embedding to be more smooth and speed up the loss function to convergence. ε denotes a function that maps the decoder output space to a word space.
Video Captioning by Adversarial LSTM
Here, W e ∈ R Z ×C is a word embedding matrix (to be learned) which is similar to the GloVe [51] and transforms the one-hot encoding of words to a dense lower dimensional embedding, C is the dimension of the embedded word (1024 in our experiments) and Z is the size of vocabulary in our training data. V is the set of parameters and encodes the h t −1 to a vector. w t −1 represents the generated word of LSTM at t th step. L is a big enough integer which would make the vector of softmax V h t −1 L closes to a one-hot form. Each value of it is constrained to be either approximately 0 or 1 which can help the w t −1 more close to W e [t − 1] (suppose the value at the t − 1 position is the largest of V h t −1 ) and also help the word embedding to be more smooth and speed up the loss function to convergence. ε denotes a function that maps the decoder output space to a word space.
Video Captioning by Adversarial LSTM
In the discriminator D, our primary purpose is to maximize the probability of assigning the correct label to both training sentences and generated sentences from G. As well konwn, deep discriminative models such as deep neural network (DNN) [52], convolutional neural network (CNN) [25] and recurrent convolutional neural network (RCNN) [53] have shown an impressive performance in complicated sequence classification tasks. In our paper, referring to [54] which has recently been shown of great performance in text classification using CNN, we choose the CNN as our discriminator.
Video Captioning by Adversarial LSTM
In the discriminator D, our primary purpose is to maximize the probability of assigning the correct label to both training sentences and generated sentences from G. As well konwn, deep discriminative models such as deep neural network (DNN) [52], convolutional neural network (CNN) [25] and recurrent convolutional neural network (RCNN) [53] have shown an impressive performance in complicated sequence classification tasks. In our paper, referring to [54] which has recently been shown of great performance in text classification using CNN, we choose the CNN as our discriminator.
Video Captioning by Adversarial LSTM
As illustrated in Fig. 2, our discriminator consists of a convolution layer and a max-pooling operation, which can capture the most useful local features produced by the convolutional layers [25], [26], [54], over the entire sentence for each feature map. The input sentences to our discriminator contain both the ground-truth sentences as the true label (also called textual description in Fig. 1) and generated sentences generated by our generator as the false label. For convenience, we fix the length of input sentences by adopting the length of longest sentence in a mini-batch (padded 0 when necessary). A sentence of length T is represented as a matrix X d ∈ R C×T = (x d1 , . . . , x dT ) by concatenating the word embeddings as columns, where T is the length of sentence and C is the dimension of a word. Then a kernel W c ∈ R C×l applies a convolution operation to a window size of T words to produce a feature map as one of the representations of the input sentence. The specific process like Fig. 4 could be formulated as follow: where f(·) is a nonlinear activation function (in our experiments, we use the RELU), b ∈ R T −l+1 is the bias vector and * represents the convolution operator. We then apply a max-overtime pooling operation over the generated feature maps and take its maximum value, Out = max{Out 1 , . . . , Out T −l+1 }. As proved by Collobert et al. [26], the max-pooling operation can not only help capture the most important feature by effectively filtering out less informative compositions of words, but also guarantees that the extracted features are independent of the length of the input sentence. To be more persuasive, we conduct a contrast experiment using maxpooling and mean-pooling respectively on discriminator. The accuracy of classification with max-pooling improves over the mean-pooling by 1.8% which proves that max-pooling have a better capacity to classify the sentences. The dataset for classification consists of the ground-truth of corresponding video and the false sentences generated from the generator. The above process describes how the features are extracted from CNN for the sentences. Although the present CNN architecture process a impressive performance in complicated sequence classification tasks. But in this paper, we devote to generate the sentence that not only needs to be natural and reasonable for people to understand but also can describe the input video precisely. In order to overcome this difficulty, we adopt a linear connection method which could integrate the video feature and pooled feature from different kernels into a new representation. Given a video feature F ∈ R H extracted from the last hidden layer in encoder of generator, where H is the dimension of hidden layer, we concatenate it with its corresponding textual feature Out ∈ R H 1 , where H 1 is the dimension of Out. We will get a synthetic feature vector F new ∈ R H +H 1 . We then pass F new to a fully connected softmax layer to get the probability D(X d ) ∈ [0, 1], an output close to 1 indicates a bigger probability that X is drawn from the real data distribution or not.
Video Captioning by Adversarial LSTM
As illustrated in Fig. 2, our discriminator consists of a convolution layer and a max-pooling operation, which can capture the most useful local features produced by the convolutional layers [25], [26], [54], over the entire sentence for each feature map. The input sentences to our discriminator contain both the ground-truth sentences as the true label (also called textual description in Fig. 1) and generated sentences generated by our generator as the false label. For convenience, we fix the length of input sentences by adopting the length of longest sentence in a mini-batch (padded 0 when necessary). A sentence of length T is represented as a matrix X d ∈ R C×T = (x d1 , . . . , x dT ) by concatenating the word embeddings as columns, where T is the length of sentence and C is the dimension of a word. Then a kernel W c ∈ R C×l applies a convolution operation to a window size of T words to produce a feature map as one of the representations of the input sentence. The specific process like Fig. 4 could be formulated as follow: where f(·) is a nonlinear activation function (in our experiments, we use the RELU), b ∈ R T −l+1 is the bias vector and * represents the convolution operator. We then apply a max-overtime pooling operation over the generated feature maps and take its maximum value, Out = max{Out 1 , . . . , Out T −l+1 }. As proved by Collobert et al. [26], the max-pooling operation can not only help capture the most important feature by effectively filtering out less informative compositions of words, but also guarantees that the extracted features are independent of the length of the input sentence. To be more persuasive, we conduct a contrast experiment using maxpooling and mean-pooling respectively on discriminator. The accuracy of classification with max-pooling improves over the mean-pooling by 1.8% which proves that max-pooling have a better capacity to classify the sentences. The dataset for classification consists of the ground-truth of corresponding video and the false sentences generated from the generator. The above process describes how the features are extracted from CNN for the sentences. Although the present CNN architecture process a impressive performance in complicated sequence classification tasks. But in this paper, we devote to generate the sentence that not only needs to be natural and reasonable for people to understand but also can describe the input video precisely. In order to overcome this difficulty, we adopt a linear connection method which could integrate the video feature and pooled feature from different kernels into a new representation. Given a video feature F ∈ R H extracted from the last hidden layer in encoder of generator, where H is the dimension of hidden layer, we concatenate it with its corresponding textual feature Out ∈ R H 1 , where H 1 is the dimension of Out. We will get a synthetic feature vector F new ∈ R H +H 1 . We then pass F new to a fully connected softmax layer to get the probability D(X d ) ∈ [0, 1], an output close to 1 indicates a bigger probability that X is drawn from the real data distribution or not.
Video Captioning by Adversarial LSTM
Previous literature [19], [55] has discussed the difficulty in training GAN model using the gradient-based method. In order to reduce the instability in training process, we initialize the LSTM parameters for generator and CNN parameters for discriminator by pre-training a standard encoder-decoder LSTM model and a standard CNN classification model aforementioned.
Video Captioning by Adversarial LSTM
Previous literature [19], [55] has discussed the difficulty in training GAN model using the gradient-based method. In order to reduce the instability in training process, we initialize the LSTM parameters for generator and CNN parameters for discriminator by pre-training a standard encoder-decoder LSTM model and a standard CNN classification model aforementioned.
Video Captioning by Adversarial LSTM
Attention structure is a kind of extension of LSTM. It has been widely used in many previous jobs [6], [56], [57]. Rather than compress an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed. For example, when a captioning model uses "a child is running on the ground" to describe a picture, the attention model will focus on the area where the child located in the picture when the model generate the word "child", while generating the word "ground", the model will focus on the ground in the picture.
Video Captioning by Adversarial LSTM
Attention structure is a kind of extension of LSTM. It has been widely used in many previous jobs [6], [56], [57]. Rather than compress an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed. For example, when a captioning model uses "a child is running on the ground" to describe a picture, the attention model will focus on the area where the child located in the picture when the model generate the word "child", while generating the word "ground", the model will focus on the ground in the picture.
Video Captioning by Adversarial LSTM
In our model with attention, we use attention structure in Generative Model when generating words. At each time step t, the attention model accept the video's visual information, which is a n by d matrix, where "n" is the number of the visual vectors and "d" is the dimension of visual vector. In the attention structure, each visual vector is multipeld by different weight α, which reflecting the importance of the corresponding visual vector in that time step. After that, the vectors are added together and become the input of LSTM unit to generate a new word. For example, when generating the word "boy", the weight for visual vectors with boy's information will be larger; While generating the word "soccer", the weight for visual vectors with soccer's information will be larger. The visual information mentioned above are the output of hidden layers of Encoder illustrated in Fig. 2 in our paper. The weight α at timestamp t is the result of softmax for the combination for hidden state at timestamp t-1 and the visual information. For a video with n visual contexts C = c 1 , c 2 , . . . , c n , we have:
Video Captioning by Adversarial LSTM
where h t −1 is the transposed vector of hidden state at the last time stamp, U c is the mapping matrix and c i is i th of visual vector.
Video Captioning by Adversarial LSTM
In our model with attention, we use attention mechanism for generating the description of the input video. At each time step t, the attention unit accepts video visual information vectors c t , which are the output of hidden layers of Encoder structure illustrated in Fig. 2 in our paper. After the encoding stage, we get the (c 1 , . . . , c n ) where n denotes the num of frames of input video.(problem definition use the a n, need to check) As eq.12, the vectors are multiplied by different weights in attention unit and these weights can determine which frame of input video s t should be concerned with current time step. After that, the y t and the last hidden status represent the new input for the next LSTM unit to generate the new word. The weights mentioned above are calculated dynamically and the sum of those weights is 1.
Video Captioning by Adversarial LSTM
And eq. 2 with attention can be written as follows:
Video Captioning by Adversarial LSTM
In this section we describe the experimental validation of our proposed video captioning approach in detail.
Video Captioning by Adversarial LSTM
To verify the impressive performance of our video captioning by abversarial training approach, we evaluate and compare our experimental results on four large public datasets, including MSVD [9], MSR-VTT [58], M-VAD [59] and MPII-MD [60].
Video Captioning by Adversarial LSTM
To verify the impressive performance of our video captioning by abversarial training approach, we evaluate and compare our experimental results on four large public datasets, including MSVD [9], MSR-VTT [58], M-VAD [59] and MPII-MD [60].
Video Captioning by Adversarial LSTM
1) MSVD: MSVD dataset consists of 1970 short video snippets downloaded from YouTube. Each video snippet is annotated with around 40 textual descriptions collected via crowdsourcing. This results in 80839 sentences. In our experiments, we split the data into train, validation and test sets, containing 1200, 100 and 670 videos snippets, respectively.
Video Captioning by Adversarial LSTM
2) MSR-VTT: MSR-VTT is a new large-scale benchmark video captioning dataset specially suitable for the video-to-text translation task. This dataset was created by using 257 popular queries from a commercial video search engine and by collecting 118 videos for each query. Each video is annotated with about 20 natural sentences provided by 1,327 crowdsourcing workers. In total, MSR-VTT provides 10K web video clips (41.2 hours) and 200K clip-sentence pairs, covering various semantic categories and diverse visual content.
Video Captioning by Adversarial LSTM
3 
Video Captioning by Adversarial LSTM
Similar to traditional machine translation, the generated descriptive sentences for the correlative video also can be measured by comparing a set of reference sentences. Recently, some common metrics in machine translation also are used for evaluating visual captioning, i.e., BLEU [61],ROUGEL [62] and METEOR [63]. ROUGE-L simply to compile statistics the maximum matches of generated sentences and reference sentences. BLEU-N (N = 1, 2, 3, 4) usually measures the precision of N-gram matches. METEOR is used to measure semantic matcher which is more robust for more consistent with human's judgment. In our experiment, for convenient comparison and robust results, we adopt BLEU and METEOR to evaluate our proposed approach following [5] and [66].
Video Captioning by Adversarial LSTM
Similar to traditional machine translation, the generated descriptive sentences for the correlative video also can be measured by comparing a set of reference sentences. Recently, some common metrics in machine translation also are used for evaluating visual captioning, i.e., BLEU [61],ROUGEL [62] and METEOR [63]. ROUGE-L simply to compile statistics the maximum matches of generated sentences and reference sentences. BLEU-N (N = 1, 2, 3, 4) usually measures the precision of N-gram matches. METEOR is used to measure semantic matcher which is more robust for more consistent with human's judgment. In our experiment, for convenient comparison and robust results, we adopt BLEU and METEOR to evaluate our proposed approach following [5] and [66].
Video Captioning by Adversarial LSTM
We used the evaluation script provided by Chen et al. [66] to compute scores on our datasets.
Video Captioning by Adversarial LSTM
To empirically verify the merit of our LSTM-GAN models, we compared the following state-of-the-art methods on the four mentioned datasets.
Video Captioning by Adversarial LSTM
• LSTM [11]: LSTM, incorporating CNN with RNN framework, attempts to directly translate from video pixels to natural language. The video representation is generated by performing mean pooling over the frame features across the entire video.
Video Captioning by Adversarial LSTM
• S2TV [3]: S2VT adopts a stack of two LSTMs for the encoding and decoding of the inputs respectively and word presentations are learnt jointly in a parallel manner. Besides that, S2VT incorporates both RGB and optical flow inputs.
Video Captioning by Adversarial LSTM
• S2TV [3]: S2VT adopts a stack of two LSTMs for the encoding and decoding of the inputs respectively and word presentations are learnt jointly in a parallel manner. Besides that, S2VT incorporates both RGB and optical flow inputs.
Video Captioning by Adversarial LSTM
• LSTM-E [10]: LSTM-E integrates 2D CNN and 3D CNN to extract video feature representation, and simultaneously explores the learning of LSTM and visual-semantic embedding for video captioning.
Video Captioning by Adversarial LSTM
• LSTM-E [10]: LSTM-E integrates 2D CNN and 3D CNN to extract video feature representation, and simultaneously explores the learning of LSTM and visual-semantic embedding for video captioning.
Video Captioning by Adversarial LSTM
• TA [1]: TA combines the frame representation from GoogleNet and video clip representation based on a 3D CNN trained on hand-crafted descriptors. whats more, the model add a weighted attention mechanism to dynamically attend to specific temporal regions of the video while generating sentence.
Video Captioning by Adversarial LSTM
• TA [1]: TA combines the frame representation from GoogleNet and video clip representation based on a 3D CNN trained on hand-crafted descriptors. whats more, the model add a weighted attention mechanism to dynamically attend to specific temporal regions of the video while generating sentence.
Video Captioning by Adversarial LSTM
For video representation, we extract the video features from VGG16 network. We take the output of the 4096-way fc7 layer from VGG16 pre-trained on the ImageNet ILSVRC12 dataset [10] as the input representation. Before training the overall model LSTM-GAN, we first separately pre-train the G and D to speed up the convergence of the overall model. For the pre-training process of G, we set the video features as the input of encoder and the corresponding reference sentences as the output of the decoder. For the pre-training process of D, we first collect both the reference sentences and generated sentences from the output of G as the training data. Following [24], we used a confusion training strategy. In detail, we randomly swap two or three words to construct tweaked counterpart sentences as negative samples. Referring to the generated sentences from pre-traing G which generates repeated word frequently about some specific words, we also select those sentences including aforementioned word to copy twice or three times as the incorrect sentence for D. Besides that, we employ filter windows (l of W c ) of sizes 3, 4, 5 with 300 feature maps each, hence each sentence is represented as a 900-dimensional vector. For ensuring the correlation between generated description and input video, we concatenate the video representation to the sentence vector as a multi-modal input [5]. In this way, we can make D have the ability to distinguish the sentences with grammar mistakes from correct ones.
Video Captioning by Adversarial LSTM
For video representation, we extract the video features from VGG16 network. We take the output of the 4096-way fc7 layer from VGG16 pre-trained on the ImageNet ILSVRC12 dataset [10] as the input representation. Before training the overall model LSTM-GAN, we first separately pre-train the G and D to speed up the convergence of the overall model. For the pre-training process of G, we set the video features as the input of encoder and the corresponding reference sentences as the output of the decoder. For the pre-training process of D, we first collect both the reference sentences and generated sentences from the output of G as the training data. Following [24], we used a confusion training strategy. In detail, we randomly swap two or three words to construct tweaked counterpart sentences as negative samples. Referring to the generated sentences from pre-traing G which generates repeated word frequently about some specific words, we also select those sentences including aforementioned word to copy twice or three times as the incorrect sentence for D. Besides that, we employ filter windows (l of W c ) of sizes 3, 4, 5 with 300 feature maps each, hence each sentence is represented as a 900-dimensional vector. For ensuring the correlation between generated description and input video, we concatenate the video representation to the sentence vector as a multi-modal input [5]. In this way, we can make D have the ability to distinguish the sentences with grammar mistakes from correct ones.
Video Captioning by Adversarial LSTM
For the training process, we adopt the Stochastic Gradient Descent (SGD) optimization function, for which we set the momentum attribute to 0.9. We first set the learning rate as 0.001. When the loss became unstable and started to fluctuate repeatedly, we decrease the learning rate to 0.0001 for followup training. In order to avoid over-fitting, we set the dropout ratio to 0.5 for all full-connected layers. In addition, we also added the weight decay and set the value to 0.01. The bathsize in our experiment is 128. If the memory capacity is large enough, we can also set a bigger value. All experiments were implemented in Theano, using a NVIDIA GTX1080 GPU with 8GB memory. The model was trained for about one day. Table I summarizes the obtained results and Fig. 5 shows the log-likelihood of the convergence performance when training the network. Overall, the results on two evaluation metrics consistently demonstrate that our proposed LSTM-GAN achieves better performance than all the existing techniques including nonattention models (LSTM, S2VT, LSTM-E) and attentionbased approaches (TA). Comparing to the two the metrics of METEOR and BLEU@4, we note that our LSTM-GAN method with attention achieved the best score of 30.4 and Fig. 6. Examples to demostrate the effectiveness of our model to generate much richer lexicon. We display the caption from ground-truth, LSTM-GAN(our model), and LSTM model respectively. We can observe that our model can describe the event of the video appropriately and generate more representative words like "sauce, frying, loaf". 42.9 respectively, outperforming all other methods. Table I shows the experimental result compared to other methods. By additionally incorporating attention mechanism to LSTM model, LSTM-GAN leads to a performance boost, demonstrating that adversarial training has the ability to improve the performance of our caption model. Additionally, we notice that our proposed LSTM-GAN (without attention) makes also a relative improvement over S2VT (RGB) which has a stack of two LSTMs layer one for encoding video and the other for decoding. The result effectively indicates that LSTM incorporating with adversarial training do benefit the learning of video sentence generation. But we also notice the method with the performance closest to this was S2VT with optical flow feature. Good performance of S2VT (Optical Flow) may be due to the usage of optical flow features, which are important for depicting the motion information of object in video. By incorporating with optical flow features, the model may generate more relevant descriptions to the video. After using attention mechanism to better handle the video features, our LSTM-GAN model has both sentence correction capability and feature processing capability. So that our LSTM-GAN with attention model outperforms all other models, including TA, which also incorporates with a weighted attention mechanism. Fig. 6 provides some examples come from ground-truth, LSTM-GAN (our model), and LSTM (without adversarial training) respectively. we can notice that our LSTM-GAN model can generate richer lexicon, which also is logically correct and more relevant to the video.
Video Captioning by Adversarial LSTM
2) Performance on MSR-VTT Dataset: Table II lists the statistics and comparison in MSR-VTT datasets. We compare our experimental results with the baseline SA-LSTM proposed  by Xu et al. [60] which achieve the attention mechanism on MSR-VTT datasets. From the statistics in Table II, our proposed approaches LSTM-GAN achieves 26.1 and 36.0 on METEOR and B4 respectively which outperforms all other methods based on different feature extraction approaches. But we also notice that our LSTM-GAN model without attention has the better score about METEOR but has poor performance in B4 compared with the LSTM. The cause of the problem is most likely our LSTM-GAN generates longer sentences which cause the relatively lower score about B4. Actually, our model with adversarial learning generates more words (in total about 2000 words for all test data) than the model without adversarial learning. When added the attention mechanism which will help to select most important and relevant information about the video clips, our model outperforming all other methods. The fact shows that, with the help of attention mechanism, our LSTM-GAN can generate more reasonable descriptions for videos although with a longer sequence output. Table III lists the statistics and comparison in M-VAD and MPII-MD datasets, which are both more challenging due to the high diversity of visual and textual content. For M-VAD dataset, we compare our experimental results with the baseline TA [1] and LSTM [11]. From the first part of statistics in Table III, our proposed approaches LSTM-GAN achieves 6.3 on METEOR, which improves over the TA by 2.0% and over the LSTM by 0.2%. Comparing to our baseline model (LSTM without GAN), our model improves over it from 5.9% to 6.3%, proving the effectiveness of our proposed model.
Video Captioning by Adversarial LSTM
2) Performance on MSR-VTT Dataset: Table II lists the statistics and comparison in MSR-VTT datasets. We compare our experimental results with the baseline SA-LSTM proposed  by Xu et al. [60] which achieve the attention mechanism on MSR-VTT datasets. From the statistics in Table II, our proposed approaches LSTM-GAN achieves 26.1 and 36.0 on METEOR and B4 respectively which outperforms all other methods based on different feature extraction approaches. But we also notice that our LSTM-GAN model without attention has the better score about METEOR but has poor performance in B4 compared with the LSTM. The cause of the problem is most likely our LSTM-GAN generates longer sentences which cause the relatively lower score about B4. Actually, our model with adversarial learning generates more words (in total about 2000 words for all test data) than the model without adversarial learning. When added the attention mechanism which will help to select most important and relevant information about the video clips, our model outperforming all other methods. The fact shows that, with the help of attention mechanism, our LSTM-GAN can generate more reasonable descriptions for videos although with a longer sequence output. Table III lists the statistics and comparison in M-VAD and MPII-MD datasets, which are both more challenging due to the high diversity of visual and textual content. For M-VAD dataset, we compare our experimental results with the baseline TA [1] and LSTM [11]. From the first part of statistics in Table III, our proposed approaches LSTM-GAN achieves 6.3 on METEOR, which improves over the TA by 2.0% and over the LSTM by 0.2%. Comparing to our baseline model (LSTM without GAN), our model improves over it from 5.9% to 6.3%, proving the effectiveness of our proposed model.
Video Captioning by Adversarial LSTM
For MPII-MD datasets, we achieve METEOR score of 7.2%, outperforming all the existing methods including SMT [60], LSTM [11], Visual-Labels [67] and S2VT [3]. Similar to the observations on MSVD, and MSR-VTT, Our LSTM with GAN model exhibits better performance than LSTM without for video captioning.
Video Captioning by Adversarial LSTM
For MPII-MD datasets, we achieve METEOR score of 7.2%, outperforming all the existing methods including SMT [60], LSTM [11], Visual-Labels [67] and S2VT [3]. Similar to the observations on MSVD, and MSR-VTT, Our LSTM with GAN model exhibits better performance than LSTM without for video captioning.
Video Captioning by Adversarial LSTM
In this paper, we presented a first attempt to introduce the concept of adversarial learning in solving the video capturing problem. We believe that this concept has potential to significantly improve the quality of the captions, which is due to its ability to better control the capture generation process. This control is in this case done by the discriminator module, acting as an adversary to the caption generation module. In addition to making the fundamental adversarial learning framework based on the GAN paradigm suitable for dealing with discrete generator outputs, with our novel realization of the discriminator, we further improved the control mechanism. This was achieved by making the input into the discriminator multimodal. In this way, the sentences coming out of the generator were not only validated for grammatical correctness, but also for their relevance to the video content. The potential of our LSTM-GAN framework to improve the quality and diversity of captions was also demonstrated experimentally, through an elaborate experimental study involving multiple baseline approaches, four popular datasets, and two widely used evaluation metrics. We believe that the performance of LSTM-GAN could further be improved by relying on Reinforcement Learning. Reinforcement Learning has proven effective for tasks similar to video captioning, like for instance dialog generation [20].
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
. Images generated by I2I task, Direct T2I task, Guided T2I task (Use I2I to guide T2I to better encode text feature and synthesize images) and the corresponding ground truth (GT). objects with semantic details is the heterogeneous gap between high-level concepts in text descriptions and pixel-level contents of synthetic images. Building an effective synthesizer to bridge this domain gap is difficult.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Many approaches [11], [16], [24], [25], [41], [42] based on Generative Adversarial Networks (GANs) [9] bridge the domain gap by utilizing a discriminator to distinguish the synthesized text-image pair and the ground-truth pair. However, such a discriminator alone is usually insufficient to model underlying semantic consistency between text and image [23], and consequently, results in semantic or structural errors in synthesized images (see Figure 1, the "Direct T2I" column). Recently, the attention mechanism [13], [22], [35] has been exploited to address this problem, which guides the generator to better match certain visual words with corresponding image subregions. But using word-level attention alone does not ensure global semantic consistency due to the diversity between text and image modalities [23]. Thus, MirrorGAN [23] models Text-to-Image and Image-to-Text together to enhance global cross-domain semantic consistency. However, the Image-to-Text in MirrorGAN [23] is still a cross-domain generation, which is not easier than homogeneous generation task such as I2I task. Thus, the problem of semantic inconsistency between heterogeneous information still remains. SEGAN [13] introduces a new contrastive loss and a Semantic Consistency Module (SCM) to better align the synthesized image and the ground truth in feature space. But still due to the heterogeneous semantic inconsistency, SEGAN cannot extract effective text features that can guide the synthesis of realistic and detailed images.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Our observation is that Image-to-Image (I2I) synthesis belongs to a homogeneous generation task, whose information gap is much smaller than that between heterogeneous gap. I2I can generate a synthesized image that has much stronger semantic consistency with the ground truth image. Thus, I2I may effectively guide T2I to better encode text features and synthesize images. As shown in Figure 1, T2I guided by I2I (last column) produces much better results than direct T2I synthesis.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In addition, the recent AttnGAN [35] and its subsequent improvements [10], [13], [17], [22], [31] adopt the word-level attention mechanism to enhance the local semantic matching between word features and local image features. In these algorithms, during the process of image synthesis, the weights of word features are fixed. Consequently, if the attention mechanism does not produce accurate weight estimation in one pass, then important words will be neglected, and image details will be missing. Thus, our second technical development is an attention update mechanism so that attention module can gradually focus on important words in a progressive way during the process of image synthesis.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Based on the above observations, we propose a new Knowledge-Transfer Generative Adversarial Network (KT-GAN) for T2I synthesis with two new components: (1) a Semantic Distillation Mechanism (SDM) that uses the I2I task to guide the T2I task for both text feature encoding and image generation, and (2) an Alternate Attention-Transfer Mechanism (AATM) to better identify important words in text. The main contributions of this paper are as follows: (i) We designed a Semantic Distillation Mechanism with a novel distillation loss function, to use I2I to guide T2I for better text feature extraction and image synthesis.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(ii) We designed an Alternate Attention-Transfer Mechanism to progressively refine word-level attention weights and enrich details of the synthesized image.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(iii) We validated our KT-GAN on two datasets: CUB-Bird [32] and large-scale MS-COCO [19]. Extensive experimental results and analysis demonstrate the effectiveness of KT-GAN and significantly improved performance compared against most previous most state-of-the-art methods on all four evaluation metrics.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [40], image/video captioning [39], person Reidentification [33] and visual question answering [29]. In T2I synthesis, recently, AttnGAN [35] introduces the word-level visual attention mechanism for T2I synthesis, it enhances the synthesis of fine-grained details at different image regions. The following work, obj-GAN [17] proposes a object-driven attention mechanism to further improve the detail synthesis, and produces finer images. A limitation of both AttnGAN [35] and obj-GAN [17] is that weight of words are fixed, which sometimes results in attention mechanism neglecting many important words. Recently, SEGAN [13] introduce the attention regularization term to filter out unimportant words and highlight the important words. But it is difficult to find the adaptive threshold in the regularization term.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Recently, [43] propose the attention update mechanism to transfer the pose of a given person to a target pose. The mechanism can effectively and dynamically utilize pose and appearance features to smoothly guide the pose transfer process. Inspired by [43], we propose the suitable attention update mechanism to progressively focus on important words.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Knowledge Distillation (KD) [15], [34] with neural networks was pioneered by Hinton [8], which is a transfer learning method that aims to improve the training of a student network by relying on knowledge borrowed from a powerful teacher network. This has also been addressed for model compression [2], [7], cross-domain task [1], [20] and continual learning tasks [21], [6]. Reference [2] propose two distinct teacher-student frameworks based on knowledge distillation mechanism for person detection. Reference [20] apply domaininvariant feature distillation mechanism for cross-domain sentiment classification. Lifelong GAN [21] employs knowledge distillation to address catastrophic forgetting for conditional generative continual learning tasks. However, above works belongs to homologous information distillation mechanism, because both teacher task and student task belong to the same or similar tasks. In KT-GAN, we solve the more challenging heterogeneous information distillation task. In our KT-GAN, teacher task (I2I) is quite different from student task (T2I).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We first revisit the attention mechanism in AttnGAN (Sec. III-A), then introduce our two new mechanisms: the Alternate Attention-Transfer Mechanism (AATM) (Sec. III-B) and Semantic Distillation Mechanism (SDM) (Sec. III-C). By integrating the AATM and SDM into the AttnGAN, we get our proposed KT-GAN.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
The AttnGAN [35] introduced an Attentional Generative Network (AGN) to guide the synthesis of different sub-regions in the image following their most relevant words. The input text description is transformed into the sentence feature s and word features W 0 by a pre-trained bi-directional LSTM text encoder in [35].
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
As shown in left part of Figure 2, the AGN has m blocks (B 0 , B 1 , · · · , B m−1 ) and the corresponding m generators (G 0 , G 1 , · · · , G m−1 ). The generators take the hidden states (H 0 , H 1 , · · · , H m−1 ) as input and generate images of small-to-large scales (I 0 , I 1 , · · · , I m−1 ):
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, z ∼ N(0, 1). F ca is a conditioning augmentation module [41] that converts a sentence feature s to a conditioning feature for the generator. F ca , F attn i , F i , and G i are modeled as neural networks. Here, denotes the concatenation of two maps along depth axis.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Consider the i -th block B i , the black dotted box in the middle row of Figure 2 
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
). For more details please refer to AttnGAN [35].
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We can observe that the input of word feature is always W 0 in the block B i (i = 1, 2, 3, · · · , m − 1) of AttnGAN [35]. A problem for the attention mechanism in AttnGAN is that word weights calculated in a single attentional block are not guaranteed to be correct. As a result, some visually important words could be neglected and their semantics are not reflected in the synthesized image. In order to tackle this problem, we design an Alternate Attention-Transfer Mechanism (AATM) to iteratively and progressively identify visually important words in the sentence. We construct the AATM by introducing a Word Feature Update module into the block B i (i = 1, 2, 3, · · · , m − 1), as illustrated in the red dotted box in Figure 2.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Each block B i in AATM contains two modules: Word Feature Update module (components in yellow in Fig. 2) and Image Feature Update module (components in black in Fig. 2). Firstly, the Word Feature Update module updates the weight of word feature based on the image feature and word feature from the last block. With these blocks, important words will gradually aggregate their weights and they will get highlighted. Secondly, the image features should also be updated according to such change, i.e., image features are synchronized to indicate where to draw the detail on the image according to the updated word features.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
1) Word Feature Update Module: In the i -th block B i , it takes in image features H i−1 and word features W i−1 from the B i−1 block, and outputs the updated word features W i through a three-step procedure.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(Step 1) Calculate an NCC Matrix R * ∈ R T ×N between H i−1 ∈ RD ×N and W i−1 ∈ R D×T : (i) We map word features to the same latent semantic space of the image features by
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, r i, j encodes the dot-product similarity between the i th word in the sentence and j th sub-region in the image.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Step 2) Calculate attentional weight maskR of words: (i) We reshape R * ∈ R T ×N to R ∈ R T by maxpooling. Each element in R represents the maximum similarity of a word to all image sub-regions, which is regarded as this word's weight. (ii) In order to match R with the word feature matrix W i−1 ∈ R T ×D , we reshape R to attention weight mask R ∈ R T ×D by repeating the column of R for D times.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(Step 3) Update the word feature W i by
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
where denotes element-wise product, α is the retention factor of current word feature information, β is the attenuation parameter of the word feature information in the previous stage, which reduces the interference of non-important words to some extent. By multiplying the transformed word features The AATM is embedded in the generator of the student task. The SDM contains three main components: the teacher task (Image-to-Image task, I2I), the student task (Text-to-Image task, I2I), and the Semantic Distillation Loss (SDL).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
W i−1 with the weightR, features W i of certain words are either preserved or suppressed. The residual connection ⊕ and the progressive feedback from discriminators of previous blocks together help important words to aggregate their weights quickly.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
2) Image Feature Update Module: We use the whole operations (Eq. 1) F i in AttnGAN as the Image Feature Update module in our each block B i . However, in AttnGAN the input word feature of F i is always W 0 , but in our AATM, the word features and image features need to be updated alternately, thus, in our each B i , we need
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
). With this modification, our AATM can be implemented correctly.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Besides introducing a better attention mechanism (AATM) into the T2I, we also design a new Semantic Distillation Mechanism (SDM) to guide the text encoder to provide better input features for the T2I. As illustrated in the green box in Figure 3, the SDM trains a student task (T2I) under the guidance of a trained teacher task (I2I), which performs the supervised cross-task semantic transfer. Our SDM contains three main components: the teacher task (I2I), the student task (T2I), and the Semantic Distillation Loss (SDL).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In the teacher task, we train an I2I task to obtain a good image encoder and a good image generator. We design an SDL to train a good text encoder, GTE (Guided Text Encoder), that matches with the trained image encoder in I2I. In the student task, we train a T2I task to get a good image generator. The implementation steps are as follows.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Step 1: Train networks in the teacher task (I2I).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Step 2: Train text encoder by SDL.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Step 3: Train networks in the student task (T2I).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In Step 1, I2I transforms an input image into a global image feature v ∈ R D by an image encoder. This image encoder is initialized using a pre-trained Inception-V3 [28] on ImageNet [26], and is then fine-tuned during the I2I training. We modify AttnGAN [35] from an T2I synthesizer to an I2I synthesizer. Note that choosing the structure of AttnGAN [35] as the teacher's network is appropriate, because it aligns well with the T2I task. The modification of AttnGAN mainly includes two operations. (i) Remove the attention mechanism in the generator, use global image feature v as the input of the generator. (ii) Introduce the perceptual loss (L per ) [4] into the I2I to improve the quality of synthesized image. More details of these modifications can be found in Appendix-A.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Step 2, we use use the trained image encoder in I2I task to guide training of an effective text encoder for the T2I task.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In Step 3, we use the text encoder trained by SDL in Step 2 to generate text feature as the input to T2I. Meanwhile, we use the generator and discriminator trained in Step 1 as the initial generator and discriminator. Rather than training the generator and discriminator from scratch, this inheritance greatly improves the generator's performance.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
The core of SDM is SDL. In this distillation, image encoder needs to be fixed, because it was already trained in I2I, and can provide effective feature template for text encoder in T2I to follow. In training process, we should design the SDL to (1) globally, push the sentence feature s to match the fixed global image feature v, and (2) locally, push the word feature W 0 to match the fixed image feature's sub-regions V 0 . The DAMSM loss in [35] is a widely used function to match image features and text features. We made two main modifications on DAMSM Loss. (1) First, unlike AttnGAN [35] that trains both image encoder and text encoder, we modify DAMSM loss to fix image encoder and only train text encoder. (2) Second, DAMSM can not effectively deal with imbalanced easy/difficult data samples in training process. Inspired by [18], Focal Loss can better balance the easy and hard samples. Thus, we further revise the construction of SDL following the design of Focal Loss.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
The SDL is composed of L S * and L W * . The distillation loss L S * ("S * " stands for "Sentence") is designed for matching sentence feature s and global image feature v. The distillation loss L W * between Word feature W 0 and local image feature V 0 ("W * " stands for "Words") is designed similarly.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
2) The Distillation Loss L S * : For a batch of image-sentence
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
, firstly we define the probability of matching between sentence feature s i and fixed image feature v * i (trained in I2I) as P(s i , v * i ):
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, d(x, y) is cosine distance between x and y. Then, we define the L S * as the negative log matching probability as
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, γ is tunable focusing parameter γ ≥ 0 in [18].
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
3) The Distillation Loss L W * : In L W * , we aim to make the word feature W 0 = {w j 0 ∈ R D | j = 1, 2, · · · , T } (output of the text encoder) to align with the fixed image feature's sub-
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(1) Following [14], [17], [35], we define a normalized image-text similarity matrix R + ∈ R T ×N through two operations:
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
encode the dotproduct similarity between the i -th word in the sentence and j -th sub-region in the image.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
, and then normalized to
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(2) Same as [14], [17], [35], we also define the attentiondriven word-image matching score:
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
(2.a) We define the dynamic representation of the image with respect to the i -th word using a vector
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We can define the attention-driven word-image matching score between image v * and sentence s,
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
where β 0 is a factor weighing the importance of the most relevant word-to-region pair. As described in [35], when
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Based on the attention-driven word-image matching score, for a batch of image-sentence pairs
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
, we define the probability of matching between sentence feature s i and fixed image feature v
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Finally, follow the design of Focal Loss [18], we define the Distillation Loss L W * by introducing a modulating factor (1 − P(s i , v * i )) γ with a tunable focusing parameter γ ≥ 0:
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, the value of γ in L W * is the same as the value of γ in L S * Finally, the Semantic Distillation Loss is defined as
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, the subscript S DL stands for "Semantic Distillation Loss".
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
: At the Block-B i , the Generative loss L G i and Discriminative loss L D i are defined as
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
where the unconditional loss is trained to generate images towards the true image distribution to fool the discriminator, and the conditional loss is trained to generate images to match text descriptions. The discriminator D i is trained to classify the input into the class of real or fake images by minimizing the cross-entropy loss
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
where I i is from the true image distribution p data at the i th scale, and I i is from distribution p G i of the generative images at the same scale.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
To generate realistic images, the final loss function of the generator and discriminator are defined as
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, we utilize the DAMSM loss [35] to make generated images better conditioned on text descriptions.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
A. Experiment Settings 1) Implementation Details: The resolution of the images participating in the evaluation are 256 × 256. All hyperparameter values are listed in the Appendix-B. We find the suitable values for these hyper-parameters by a series of ablation studies in the Appendix-C. In the training stage: (i) Train I2I task; (ii) Train SDM; (iii) Train T2I task. In the testing stage: Only input features of sentence and words to the generator of the Student Network.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
2) Datasets: Two widely used datasets are used. The CUB-Bird dataset [32] contains 11, 788 bird images belonging to 200 categories, and 10 visual description sentences for each image. We pre-process and split the images following the same pipeline in [25], [41]. The MS-COCO dataset [19] contains 80k training images and 40k test images, and each image has 5 text annotations.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
3) Evaluation: We compare KT-GAN and other state-ofthe-art algorithms using four measures: (1) Inception Score (IS) [27]; (2) Fréchet Inception Distance (FID) [12]; (3) We also compare the Rank-1 score in text-to-image retrieval [5]; (4) Human perceptual test.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
IS uses fine-tuned inception models provided by [41] to compute the KL-divergence between the conditional class  distribution and the marginal class distribution. A larger IS (better) means that a T2I generator can synthesize a higher diversity of images for all classes, where each image belongs more clearly to a specific class. FID computes the Fréchet distance between synthetic and realistic images based on the extracted features from a pre-trained Inception-V3 network [28]. A lower FID (better) implies a closer distance between generated image distribution and real-world image distribution. The Rank-1 score denotes the most relevant synthesized images for each text sentence in text-to-image retrieval. A bigger Rank-1 score (better) implies the synthesized image has better consistency with the given text. The trained retrieval model provided by [5] was used to calculate the Rank-1. The Human perceptual test aims to judge whether the generated images are well-conditioned on the text descriptions from human subjective perception. In order to conduct the Human perceptual test, we randomly select 1000 text descriptions in the CUB-Bird test set and 2000 text descriptions in the MS-COCO test set. Given the same text description, 30 volunteers (not including any author) are asked to rank the images generated by different methods. The average ratio ranked as the best by human users are calculated to evaluate the compared methods. A bigger score (better) in the Human perceptual test implies the synthesized image has better consistency with the given text description.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We evaluate the effectiveness of two new components, AATM and SDM, in terms of three measures. The results are documented in Table I. (I) We introduce AATM to replace the attention mechanism in AttnGAN [35], i.e. AttnGAN+AATM. As shown in Table I In all, Table I 
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We compare IS, FID and Rank-1 of different designs in the Word Feature Update Module (Eq. 2) on the CUB-Bird dataset. Due to GPU memory constraints, we did not try the KT-GAN with more than three blocks. And AATM is employed over the last two blocks. Thus, AATM can only be applied to blocks B 1 and/or B 2 , and there are three possible variants: (I) AATM (B 1 ) indicates that AATM is only implemented in block B 1 . As shown in Table II, compared with AttnGAN, the performance of AttnGAN+AATM (B 1 ) gains the moderate improvements in these three measures. (II) AttnGAN+AATM indicates that AATM is implemented in both blocks B 1 and B 2 in this paper. Compared with AttnGAN+AATM (B 1 ), the performance of AttnGAN+AATM (B 1 ) further gains the moderate improvements in these three measures. It demonstrates that progressively adding the AATM into the block (B i (i = 1, 2, . . . , m − 1)) can effectively improve the performance of generator. (III) In order to show the effectiveness of the residual connection, we drop out residual connection in Eq. 2, i.e. W i = α · [R W i−1 ]. The performance of "AttnGAN+AATM w/o Res" is degraded in these three measures. It indicates that the accumulation of word information from previous blocks also play an important role in synthesizing process. Figure 4 compares AttnGAN and AttnGAN+AATM by visualizing the iterative update on word weights and their corresponding attention maps. Weights for the top-5 words are listed. The attention maps are plotted on synthesized images: each word's relevant region is brighter. In Figure 4, the AttnGAN generally can not effectively accumulate important word information and improve attention maps. In the CUB-Bird example (left column), word weights and attention in Block-B 2 are not better than in Block-B 1 and remain incorrect (e.g., "a" "the" "short" and "beak"). Similarly, in the MS-COCO example (right column), the AttnGAN's attention always focuses on certain words and misses important visual information from some other words, and eventually leads to bad synthesis. In contrast, our AATM progressively aggregates  the weights of important words and enhances the accuracy of their attention. We can clearly see such improvements in Block-B 2 over Block-B 1 (e.g., left column: "beak" and "short", right column: "many", "sheep" and "field"). As for the update module for the Image Feature, the module from [35] is already effective and we simply followed that. So no ablation study on that is discussed here.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We use ablation study to verify the two designs in our SDM: (1) I2I can guides T2I to get better text encoder, and (2) I2I generators are good initial generators in T2I.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
For (1), we use the Guided Text Encoder (GTE) trained by SDL to train AttnGAN+AATM from scratch, i.e. AttnGAN+AATM+GTE. As shown in Table III (2), if we use the trained generator and discriminator in I2I as the initial generator and discriminator in T2I, i.e. AttnGAN+AATM * . As shown in Table III Thus, the effectiveness of the two designs can be demonstrated by these ablation studies. And it indicates that the SDM can help the T2I task bridge the domain gap.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We compare our KT-GAN with state-of-the-art GAN models for text-to-image synthesis on CUB-Bird and MS-COCO test datasets. The IS for our proposed KT-GAN and other compared methods are listed in Table IV. On the CUB-Bird dataset, our KT-GAN (4.85) leads to the highest IS scores. On the MS-COCO dataset, the KT-GAN (31.67) also performs better than most existing approaches except for the SD-GAN [10]. However, SD-GAN requires multiple text sentences to train the generator. If the given database only contains images with single sentence description (which is common in some practical tasks such as Story Visualization [38] and Text-to-Video [37]), SD-GAN can not be trained. In contrast, KT-GAN and AttnGAN [35] only need one sentence per image and they can be trained normally. Besides, the SD-GAN contains many Siamese branches, which is much more complex than KT-GAN. Thus, SD-GAN require much more powerful hardware devices for training.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In Table V, we compare the different models' performance using FID and Rank-1. The KT-GAN greatly improves the baseline AttnGAN [35] in terms of FID and Rank-1, on CUB-Bird and MS-COCO dataset respectively. And KT-GAN achieves the best score in terms of Rank-1 on the two standard datasets. However, the FID of KT-GAN is lower than that   DMGAN is better than that of our KT-GAN. In the semantic consistent aspect, our KT-GAN enhances the semantic consistent from global and local aspects. Thus, the "Rank-1" of our KT-GAN are better than that of DMGAN. Besides, because the Dynamic Memory Module in DMGAN contains too many learning parameters. So, the DMGAN's generator is more complex than our KT-GAN. In all, the results of FID demonstrates that KT-GAN performs better in capturing the feature distribution of more complex real images. The results of Rank-1 demonstrates the KT-GAN leads to better semantic consistency between synthesized images and its text description.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In Table VI, we compared our KT-GAN with AttnGAN [35] and DM-GAN [22] using Human perceptual test. After the volunteers finished the experiment, we counted the votes for each method in the two datasets. The results of subjective test shows that KT-GAN is more effective in terms of semantic consistency. These results demonstrate the superiority of KT-GAN for generating visually realistic and semantically consistent images.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Visualization. For qualitative evaluation, Figure 5 shows text-to-image synthesis examples generated by AttnGAN [35], SEGAN [13], DM-GAN [22] and KT-GAN (Ours). Observing the samples generated on the CUB-Bird dataset in the left four columns of Figure 5, images synthesized by AttnGAN [35] and SEGAN [13] are prone to semantic structure ambiguity. The quality of images synthesized by DM-GAN [22] is higher than that of AttnGAN [35] and SEGAN [13], but not as good as that of our KT-GAN. In contrast, our KT-GAN model better highlights the main object with detail, and its contrast with the background. In terms of multi-subjects image generation, e.g., the MS-COCO data (see the right four columns of Figure 5), it is more challenging to generate photo-realistic images when text descriptions are more complicated and contain multiple objects. Because KT-GAN can better bridge the domain gap between text and image, it can better capture the major objects and arrange contents  in a more meaningful way. Eventually, these lead to images with better global structure. More visualizations are given in Appendix-D.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Besides, we further evaluate the sensitivity of the proposed KT-GAN by changing just one word or phrase in the input sentence. As shown in Figure 6, the synthesized images are modified according to the changes of the input sentence, e.g., bird color ("blue" versus "yellow") and image scene ("in a large body of water" versus "on the green field"). It demonstrates that our KT-GAN has the ability to catch subtle changes of the text and retains the semantic diversities and details from text.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Although our proposed KT-GAN shows superiority in generating visually realistic and semantically consistent images, some limitations and discussion must be taken into consideration.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
First, I2I task, SDM and T2I task are not jointly optimized with complete end-to-end training due to limited computational resources. Second, we use AATM to refine the word embeddings for generator, which could be further improved. In the future sudies, we can further use some sentence parser to extract the informations of objects or details. 
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We show some failure images synthesized by our KT-GAN on the CUB-Bird test set (the first row of Figure 7) and the MS-COCO test set (the second row of Figure 7).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
On the CUB-Bird dataset (the first row of Figure 7): in the first two images, the body parts of the bird are missing; in the third and the fourth images, the bill of the bird is very strange; in the fifth image, the background is too noisy; in the sixth images, our KT-GAN creates the bird with two heads.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
On the challenging COCO datasets (the second row of Figure 7): like all existing approaches, our KT-GAN also cannot effectively extract correlated structural and semantic information to support realistic synthesis; in the first two images, KT-GAN tends to place objects corresponding to specific features at many locations throughout the image; in the last four pictures, the big problem of current methods and our KT-GAN is that objects in the images can not be correctly synthesized by the generator.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
We think the above issues are caused by: (i) The generator's generation capacity is not strong enough; so, it is necessary to design a stronger generator to better synthesize the objects or details; (ii) A single sentence contains very little semantics; Based on the limited semantic information, it is difficult for the generator to synthesize complex images, especially the complex scenes and objects on the MS-COCO dataset; Thus, it is necessary to explore more valid semantic information from more text descriptions to help the generator synthesize high-quality images.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In this paper, we propose a novel Attention-Transfer Mechanism (AATM) and a Semantic Distillation Mechanism (SDM), and build a Knowledge-Transfer Generative Adversarial Network (KT-GAN) for Text-to-Image (T2I) synthesis. The SDM uses Image-to-Image synthesis to guide the T2I synthesis to better encode text feature and synthesize photographic image. The AATM helps the generator progressively identify important words and enrich the details of synthesized image. SDM and AATM successfully bridge the heterogeneous gap and help the generator synthesize high quality images. Extensive experimental results and analysis demon-strate the effectiveness of KT-GAN and significantly improved performance compared against previous most state-of-the-art methods.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In the Figure 8, the Teacher Network includes three main module: Image Encoder module, Generator, and Discriminator. In our paper, we describe that the modification of AttnGAN mainly includes two operations. (i) Remove the attention mechanism in the generator, use global image feature v as the input of the generator. (ii) Introduce the perceptual loss (L per ) [4] into the I2I to improve the quality of synthesized image.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In addition the above two main modifications, compared with AttnGAN [35], the input feature to the synthesizer is image features instead of text features. The corresponding loss functions need to be simply modified. Thus, in the following, we describe the details of loss function in the Teacher Network.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Before describing the loss functions, we simply introduce the pipeline of synthesizer in Figure 8 in order to easily understand the following definition of the loss functions.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In Section III-C, we explain the student network. Here we elaborate the teacher's network. They are similar. we use symbols B , G , and H et al. to represent corresonding B, G, and H et al. in the student network (T2I task).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
The input image is transformed into global image feature v ∈ R D by a pre-trained Inception-V3 [28] Image Encoder on ImageNet [26]. As shown in left part of Figure 8, the teacher network has m blocks (B 0 , B 1 , · · · , B m−1 ) and the corresponding m generators (G 0 , G 1 , · · · , G m−1 ). The generators take the hidden states (H 0 , H 1 , · · · , H m−1 ) as input and generate images of small-to-large scales (Î 0 ,Î 1 , · · · ,Î m−1 ):
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, z ∼ N(0, 1). F ca is a conditioning augmentation module [41] that converts a sentence feature v to a conditioning feature for the generator. F ca , F i , and G i are modeled as neural networks. Here, denotes the concatenation of two maps along depth axis. In our paper, based on the modification (i) about AttnGAN [41], we modify the
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
. In the subsection A-B, we describe the details of the loss functions in I2I task (Teacher Network).
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Combining the above modules together, at the i -th stage of the teacher network, the Generative loss L G i and Discriminative loss L D i are defined as
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
where the unconditional loss is trained to generate images towards the true data distribution to fool the discriminator, and the conditional loss is trained to generate samples to match the real image feature embedding. The discriminator D i is trained to classify the input into the class of real or fake images by minimizing the cross-entropy loss (14), as shown at the bottom of the page, where I i is from the realistic image distribution p data at the i -th scale, andÎ i is from distribution p G i of the generative images at the same scale.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
The final objective function of the generative network and discriminative network in the Teacher Network are defined as
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Here, we train L G * and L D * using the entire training dataset.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Based on experiments on a held-out validation set, we list the hyperparameters in Table VII. Due to GPU memory constraints, we did not try the KT-GAN with more than three blocks. Thus, we set the parameter m = 3 in this paper.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In this subsection, we mainly find the suitable values for these hyper-parameters in Table VII 
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
The α, β is the important hyper-parameters in the KT-GAN. The hyper-parameters α and β belong to the AATM module. We use the AttnGAN+AATM to discuss the sensitivity of the hyper-parameters α, β. As shown in Table VIII, when the α = 0.85, β = 0.9 or α = 0.9, β = 0.85, the AttnGAN+AATM achieves the best performance in the three measures.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In the SDM, we introduce the focal loss into the cross-modal knowledge distillation stage. We hope the focal loss can better balance the hard samples and easy samples in the cross-modal knowledge matching process. Here, the γ is the important hyper-parameter in the focal loss [18]. In this subsection, we mainly find the suitable value for γ in the KT-GAN. In Table IX, when the hyper-parameter γ = 2.0, the KT-GAN gains the best performance on the CUB-Bird test set over these three measures.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
C. β 0 β 0 is a factor that determines how much to magnify the importance of the most relevant word-to-region context pair.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
So, in this subsection, we find the suitable value for the β 0 . As shown in Table X, when β 0 = 1, 2, 5, the performance of the
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
unconditional loss Fig. 9. Images of 256 × 256 resolution are generated by our KT-GAN, DM-GAN [22], SEGAN [13], and AttnGAN [35] (baseline) conditioned on text descriptions from MS-COCO test datasets. KT-GAN is stable. When β 0 = 2, 5, KT-GAN achieves the best performance on the CUB-Bird data set over these three measures.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In the training stage of KT-GAN, we also utilize the DAMSM loss [35] to make generated images better conditioned on text descriptions. Same as AttnGAN [35], we also set λ 3 = 5 in our KT-GAN. Besides, we further show the performance of KT-GAN based on different value of the hyper-parameter λ 3 in Table XI. When λ 3 = 0, 5, 10, the performance of KT-GAN is stable. HYPER-PARAMETERS η 0 , η 1 AND η 2 ON CUB-BIRD DATASET E. η 0 , η 1 , η 2 η 0 , η 1 , η 2 is hyper-parameters in the teacher network of KT-GAN. The η 0 , η 1 , η 2 balance the learning weights of the three scale perceptual losses in the teacher network. Table XII shown the main results on the CUB-Bird test set. As shown in Table XII, when η 0 = 10 −3 , η 1 = 10 −2 , η 2 = 10 −1 , KT-GAN achieves the best performance in the CUB-Bird dataset over three measures.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
In this section, we show more visual comparison results between our KT-GAN, DM-GAN [22], SEGAN [13], and  [22], SEGAN [13], and AttnGAN [35] (baseline) conditioned on text descriptions from MS-COCO test datasets. Fig. 11. Images of 256 × 256 resolution are generated by our KT-GAN, DM-GAN [22], SEGAN [13], and AttnGAN [35] (baseline) conditioned on text descriptions from CUB-Bird test datasets.
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
AttnGAN [35] (baseline) on the CUB-Bird and MS-COCO dataset in Figure 11, Figure 12, Figure 9, and Figure 10. These visual comparison results further demonstrate the generalization ability of the KT-GAN. Besides, we show more generated results on CUB-Bird dataset and MS-COCO dataset. As shown in Figure 13, we further  [22], SEGAN [13], and AttnGAN [35] (baseline) conditioned on text descriptions from CUB-Bird test datasets. show 400 images for each dataset. Since the limited size of the Appendix, you can down these figures from https://pan.baidu.com/s/11QCfAcCfWi41B2DHiWGemA, password: qqx1, and view more details. 
Discriminative Latent Semantic Graph for Video Captioning
With the tremendous growth of video materials uploaded to various online video platforms, e.g. YouTube, research in automatic video captioning has received increasing attention in recent years. Thorough video caption can lead to substantial practical impacts, e.g. content-based video retrieval and recommendation. Despite the remarkable progress of computer vision and natural language processing in video analysis and language understanding, video captioning is still a very challenging task. The task requires exploring not only complex object interactions and relationships at frame-level but also high-level story-line from video sequence. Such a task can be seen as a leap from the recognition to comprehension level.
Discriminative Latent Semantic Graph for Video Captioning
One of the main challenges of video captioning is that there is no explicit mapping between video frames and words in captions. The model needs to extract summarized visual words at a much higher semantic level. Figure 1 illustrates an intuitive example of video captioning. From a human perspective, we can interpret the overall process into several sub-tasks: 1) to detect and recognize the main objects in the video, i.e., "woman" and "stroller"; 2) to infer the action imposed to these objects, i.e., "showing" and "operates"; 3) organizing the contents into a sentence with grammatical structures, i.e., "A woman is showing how stroller operates". Early studies typically adopt encoder-decoder frameworks [24,25,28] that model video captioning as a machine translation task. These methods focus on modeling the static frame and object features and temporal changes among embedding. To overcome the drawback of embedding-based frameworks, the recent rise of graph neural networks (GNNs) has shown particular advantages in modeling relationships between objects [27,31]. However, applying GNNs to the video captioning task is not a trivial thing. Previous GNNs are mainly built on object features without jointly considering the frame-based spatio-temporal contexts in the entire video sequence.
Discriminative Latent Semantic Graph for Video Captioning
The other challenge is that the output caption needs to ensure a grammatical structure keeping content-relevant fidelity rather than producing a list of discrete concepts. To examine whether the expression of a sentence is natural or not, [27] utilizes a generative adversarial network (GAN) to control the fidelity of generated sentences. However, video captioning requires a more refined level of supervision to distinguish the real/fake sentences which is conditioned on semantic content, and to ensure grammatical correctness.
Discriminative Latent Semantic Graph for Video Captioning
The above challenges motivate us to design a new framework for video captioning with three sub-tasks, i.e. Enhanced Object Proposal, Visual Knowledge, and Sentence Validation. First, Fusion is about extracting the spatio-temporal contexts from video frames and incorporating such information in the object entities. Note that the number of frames and object proposals in videos is far more than words in captions. Therefore, the second task visual knowledge summary aims to reduce such duplicated and redundant proposals into more compact visual words. Such high-level visual words should be easier encoded by a sequential model to produce a caption. The last sentence validation task aims to examine both the fidelity and the readability of the generated caption. According to the above motivations, we design a Discriminative Latent Semantic Graph (D-LSG) framework with the following insights: 1) Graph model for feature fusion from multiple base models e.g. 2D/3D CNN and R-CNN remain unexplored. These features are often heterogeneous in data distribution, dimensions, and structure. 2D CNN represents the frame contents, while 3D CNN extracts the temporal frame changes. We consider such frame-level information as the conditions of all region-level object proposals. Therefore, the conditional graph is not in the traditional form of semi-positive indefinite affinity matrix. 2) Latent Semantic refers to the higherlevel semantic knowledge that can be extracted from the enhanced object proposals. Rather than incorporating an external auxiliary knowledge graph as [9], our key idea is to construct a dynamic graph that connects enhanced object proposals with randomly initialized nodes. In other words, the great volume of enhanced object proposals is summarized into high-level visual knowledge via the dynamic graph. 3) Discriminative module is designed as a plug-in language validator. Generated and ground truth captions can be reconstructed into visual knowledge so as to compare with that extracted from the enhanced object proposals. We adopt the Multimodel Low-rank Bi-linear (MLB) [12] pooling as metrics to provide finer-level supervision to carry out the sentence validation task. In summary, our contributions include:
Discriminative Latent Semantic Graph for Video Captioning
• To identify Enhanced Object Proposal, Visual Knowledge, and Sentence Validation sub-tasks in a unified framework for future video summarization tasks.
Discriminative Latent Semantic Graph for Video Captioning
• A Condition Graph Operation is proposed to enhance regionlevel object proposal representations with spatio-temporal information of base features of video frames. • Latent Proposal Aggregation with a dynamic graph model is proposed to compress enhanced object proposals into visual knowledge with higher semantic meanings in a latent space. • A Discriminative model is plugged as a validation network that can distinguish generated sentences from ground truth captions and encourage the generated captions to be more content-relevant and semantic-richer. • Quantitative and qualitative experiments on two datasets, MSVD [3] and MSR-VTT [26], demonstrate significant performance boost on all evaluations while achieving significant performance improvement on CIDEr.
Discriminative Latent Semantic Graph for Video Captioning
Video Captioning. As a joint field in Computer Vision (CV) and Natural Language Processing (NLP), video captioning has received increasing research interest. Early research on video captioning mainly focused on template-based language models [7,19]. With the rapid development of deep learning techniques, recent methods have developed sequence-learning based methods with an encoderdecoder structure [24,25,28] and considered video captioning as a machine translation task [24]. For example, The work of Yao et al. [28] adopted a temporal attention mechanism to summarize the visual features for each generated word dynamically. More recently, object-level information has drawn more attention [10,22,29]. For instance, the work of [10] applied two LSTM layers to construct temporal structures at both frame-level and object-level. [22] proposed a visual reasoning approach on videos over both space and time. Zhang et al. [29] proposed an object-aware aggregation with bidirectional temporal graph (OA-BTG), which captures detailed temporal dynamics for salient objects in videos. In summary, these previous methods focus on modeling global information or temporal structure of salient objects. We believe our proposed D-LSG framework initialised three new research questions for video captioning. First, manipulation of interaction between different objects via graph modeling will lead to a new research trend in video captioning. Our proposed conditional graph operation is so far the first approach that can incorporate heterogeneous features from multiple base models. Second, how to extract visual knowledge from enhanced object proposals shares the same spirit of the traditional Bag-of-Visual-Words (BoVW) paradigm. The dynamic graph can be seamlessly included in end-to-end training and is much more cost-efficient. Finally, we point out that the sentence validation requires finer supervision in the discriminative model so that the fidelity and sentence structure can be both preserved. Discriminative modeling. So far, many studies have investigated image and video captioning using discriminative modeling. The work of [20] applied a discriminator model to distinguish ground truth captions from generated captions, which relies on the Gumbel-Softmax approximation [11], whereas [5] utilizes the policy gradient and their discriminator focuses on caption naturalness and image relevance. The work of [16] designed a multi-discriminator system that encourages better multi-sentence video description. However,  Figure 2: Overview of proposed LSG framework. Base features from 2D/3D CNN and R-CNN provide object and contexts features at frame and region levels. Conditional Graph Operation is applied to appearance and motion channels to compute Enhanced Object Proposalsˆandˆ. frames ofˆandˆare selected into Visual Knowledge before LSTM captioning.
Discriminative Latent Semantic Graph for Video Captioning
discriminative modeling for caption generation suffers from stability issues and requires pre-trained generators. Thus, instead of using the Gumbel-Softmax or policy gradient based method, we propose a semantic relevance discriminative graph based on Wasserstein gradient penalty [8], which can directly feed output and graphbased latent semantic concepts from the generator and do not need to pre-train the generator. The proposed discriminative encoder can be plug into an end-to-end model to reconstruct captions into visual knowledge so that the fidelity and sentence structure can be validated and key semantic entities can be preserved. Graph Neural Networks. Efficient and practical neural network algorithms for processing graph-structured data have become one of the most important machine learning subareas. Recently, pioneer work [9,14,31] have tried to adopt graph neural networks to video captioning. For example, Pan et al. [14] proposed a spatio-temporal graph model for video captioning that exploits object interactions in space and time. Another graph based video captioning research [31] proposed an object relational graph based encoder, which captures more detailed object interaction features to enrich visual representation. However, the weights they used to summarize object features in temporal space are the same as the frame features, which may lead to degraded temporal information in summarized object features. In comparison, our conditional graph jointly considers object, contexts, and motion information at both region and frame levels. The work of [9] proposed a joint commonsense and relation reasoning method that applies auxiliary databases for pre-training knowledge graphs as prior knowledge for image and video captioning. Our dynamic graph in the Latent Proposal Aggregation module is able to extract high-level latent semantic concepts without an external dataset for training.
Discriminative Latent Semantic Graph for Video Captioning
The video captioning problem is essentially modeled as a sequence to sequence process. Formally, given a sequence of frames from video = { 1 , . . . , }, we aim to build an end-to-end model to generate the caption = { 1 , . . . , ′ } for the given video. Note ≠ ′ , which forms an open task that is even very challenging for humans. In this paper, we identify three key sub-tasks, namely Enhanced Object Proposal, Visual Knowledge, and Sentence Validation, details of which are introduced as follows.
Discriminative Latent Semantic Graph for Video Captioning
The overview of our proposed model is illustrated in Figure 2. The Latent Semantic Graph (LSG) consists of three parts: (1) Multiple Feature Extraction; (2) Conditional Graph Operation; (3) Latent Proposal Aggregation. A plug-in discriminative caption validator is illustrated in Figure 4. We introduce how our proposed models can address the sub-tasks next. Multiple Feature Extraction. Given input video frames , the model first extracts visual context representations. In this work, 2D CNNs and 3D CNNs are employed to extract appearance features = { } =1 and motion features = { } =1 respectively. Object proposals are extracted by R-CNNs to capture key entities with region features = { } =1 from each frame, where = { ∈ R } =1 and denotes the number of region features in each frame. Thus, the total number of object proposals is denoted as = × . Next, to make full use of motion information, we concatenate the appearance features and motion features, and apply LSTM models to learn better representations of motion features. Enhanced Object Proposal. In video captioning, one of the essential tasks is to detect and recognize the entities. The weak object proposals in region feature ∈ R × × are enhanced by their visual contexts of appearance and motion, respectively, which result in enhanced appearance proposalsˆ∈ R × and enhanced motion proposalsˆ∈ R × in a graph structure, where is the feature dimension used in graph operation.ˆandˆtogether form the enhanced object proposals. Visual Knowledge. The Latent Proposal Aggregation (LPA) module introduces a dynamic graph that can summarize the enhanced appearance and motion features to latent semantic proposals as dynamic visual words:
Discriminative Latent Semantic Graph for Video Captioning
∈ R × and ∈ R × . Note ≪ .
Discriminative Latent Semantic Graph for Video Captioning
Language Decoder. Visual knowledge extracted by the LPA is then used to generate corresponding captions. We adopt the language generation decoder that are commonly used in VQA and video captioning fields [1,9,22,29]. The language decoder consists of an attention LSTM network for weighting dynamic visual words and a language LSTM network for caption generation. At each time step, attention LSTM takes current word embedding and global visual vector¯= [ =1 , =1 ] ∈ R 2 as input and output current hidden state .The is then treated as the query of the attention operation to weight sum the object and motion visual words to context feature , ∈ R . The language LSTM then takes the current context features and current attention LSTM hidden states and output current predicted word probability distribution ∈ R , where is the vocabulary size.
Discriminative Latent Semantic Graph for Video Captioning
There has been significant research investigating the dependencies between objects and complex content in generating video captions. However, learning spatio-temporal dependencies remains a challenging issue. Compared to conventional spatio-temporal convolution and recursive neural networks, graph models provide a new solution to model dependencies. In this work, we propose the LSG model that can efficiently encode object-level features from videos as highly summarized visual words with higher semantic level. To progressively generate the high-level concepts representing visual features, the essential parts of the LSG model are divided into two components: conditional graph operation and latent proposal aggregation.
Discriminative Latent Semantic Graph for Video Captioning
Conditional Graph Operation. In video captioning, one of the key challenges is to model the complex object-level interactions and relationships. Another challenge is to learn informative object-level features that are in context of frame-based background information. To encode object-level information as highly summarized latent semantic objects and motion visual words conditioned on frame-based background information, we first aggregate objectlevel features into appearance and motion features respectively via graph operation. Since we have the object-level region features, frame-level motion features and appearance features after the Multiple Feature Extraction step mentioned in Section 3.1, we build a graph neural network to model object-level interactions, where each region feature out of all region features is regarded as a node. To modeling the frame-based conditioning, instead of relying only on the local region features, we take the full picture into account, which takes both frame-level motion and appearance features and object-level region features. Specifically, we pass messages of the region features to frame-level features at each frame :
Discriminative Latent Semantic Graph for Video Captioning
whereˆrepresents the ℎ representation of enhanced appearance proposal, and ∈ R × denotes learnable parameters. Specially, F is a kernel function that aims to encode relations between frame-level features and detailed region features . In this study, we define F as:
Discriminative Latent Semantic Graph for Video Captioning
where and are linear functions followed by Tanh activation function. This step aims to project features from different modalities to a common feature space and compute the similarity to represent the degree of connectivity between region features and frame-level features in the graph. Alternatively, the equation can be written as:
Discriminative Latent Semantic Graph for Video Captioning
where ∈ R × = F ( ( ) ( ) ) denotes the relation coefficient matrix between appearance features and region features. Meanwhile, region features are aggregated to motion features asî n the same process. Withˆandˆthat contain object-level information on the condition of frame-level features, we then need to summarize the enhanced proposals to obtain informative semantic concept candidates or proposals with less redundancy. Latent Proposal Aggregation. To further summarize the enhanced object proposals, we propose a latent proposal aggregation method to generate visual words dynamically based on the enhanced fea-turesˆandˆinspired by [30]. First, we augment the original enhanced proposal nodes {ˆ} =1 and {ˆ} =1 with a set of additional latent nodes, and then aggregate information from the enhanced proposals to the latent nodes in a graph structured manner. Specifically, we introduce a set of object visual words = { ∈ R } =1 , which means potential object candidates in the given video. Note that indicates the number of visual words, so that we can summarize the enhanced proposals into informative dynamic visual words. The aggregation process is defined as:
Discriminative Latent Semantic Graph for Video Captioning
where denotes the ℎ object visual word and ∈ R denotes learnable parameters for the ℎ object visual word. Following the same process, we can derive the motion visual words = { ∈ R } =1 that represent potential motion candidates in the given video. Therefore, with LSG, we extract the high-level representation and summarize information as dynamic visual words from a video that models both object-level interaction and frame-level condition. The latent semantic visual words are then fed into the language  Figure 4: An overview of our discriminative modeling process. We score an input sentence in a semantic concept perspective of view. The model reconstructs the visual knowledge based on the input sentence and output comparison scores with visual knowledge encoded from the corresponding video.
Discriminative Latent Semantic Graph for Video Captioning
decoder to generate captions as mentioned in Section 3.1. Although the output sequence is generated based on the visual words, there's still potential to obtain video description with more meaningful semantic concepts for more informative caption generation.
Discriminative Latent Semantic Graph for Video Captioning
While other discriminative models for video captioning mainly focus on fluency and visual relevance of the generated descriptions, we aim to generate meaningful captions from the perspective of semantic concepts. In our approach, we design a discriminative model as a language validation process that encourages the generated captions to contain more informative semantic concepts via reconstructing the visual words or knowledge based on the input sentences under the condition of corresponding true visual words encoded by LSG. Specifically, based on the visual knowledge and encoded from input video features, we propagate information from the generated captions to reconstruct visual knowledge and discriminate the reconstructed visual words from ground-truth and generated captions in an adversarial training manner. The process of the discriminative modeling is summarized in Algorithm 1 and described as follows.
Discriminative Latent Semantic Graph for Video Captioning
Given the output word sequence from the language decoder, the discriminative model aims to distinguish the generated captions and ground-truth with regard to the semantic concepts in the corresponding sentences. To prevent the discriminative model from easily distinguishing between real and fake samples without learning useful information (the ground truth caption is the one-hot integer datatype while the generated caption contains probability distributions) and to stabilize the training process, we employ the WGAN-GP architecture as it uses the earth-mover distance to capture the difference between real and fake samples, which is well suited to our problem.
Discriminative Latent Semantic Graph for Video Captioning
The discriminative model first extracts sentence features ∈ R ′ × from the given caption ∈ R ′ × by using several layers of 1D CNN with residual connections. Then, we adopt a graph-based structure to obtain the reconstructed object and motion visual words˜and˜from the caption:
Discriminative Latent Semantic Graph for Video Captioning
.
Discriminative Latent Semantic Graph for Video Captioning
Note that visual words from the LSG model stops gradient descent before passing to the discriminative model so that it does not affect the caption generation. Following the same process, we can derive the aggregated motion visual words˜. Then, we compare the visual words with the aggregated visual words˜by Multimodal Low-rank Bi-linear pooling (MLB) [12], which is recognized to be efficient in tasks such as VQA. To be specific, for each pair of visual words˜and :
Discriminative Latent Semantic Graph for Video Captioning
where is the sigmoid activation function, ⊙ denotes the Hadamard product, and are learnable parameters. Note that is a scalar which means the score of the input aggregated proposal compared to the original one. is score of ℎ object visual word pair and is score of ℎ motion visual word pair. Then the overall comparison score can be expressed as:
Discriminative Latent Semantic Graph for Video Captioning
where ′ is the number of visual words selected out of for reconstruction and comparison. The intuition behind is that the captions contain less semantic information than video. Instead of adding and as the discriminative model's output, we weight them adaptively based on the sentence feature since sentences have different proportions of object and motion concepts. We calculate the output of the discriminative model as following:
Discriminative Latent Semantic Graph for Video Captioning
where , ∈ R are learned parameters, and¯∈ R is mean pooled sentence feature. ( | ) is the output of the discriminative model that learns to give real captions large values and minimize the values of generated captions. For the ground-truth caption and generated caption , the loss function of the discriminative model is defined as:
Discriminative Latent Semantic Graph for Video Captioning
whereˆis sampled along straight lines between real caption and generated caption . ( ∇ˆ(ˆ) 2 − 1) 2 is the gradient penalty term that forces the gradient from the generated sample to the real sample to be as small as possible to satisfy the Lipschitz constraint [8]. Thus, for the generator, the loss function is calculated as:
Discriminative Latent Semantic Graph for Video Captioning
where is the hyperparameter that controls the weight ofL . L is the caption generation loss.
Discriminative Latent Semantic Graph for Video Captioning
Overall, the LSG model aims to summarize input video into high-level visual words to generate informative captions, and the discriminative modeling enhances the generated captions to be more semantically relevant. 
Discriminative Latent Semantic Graph for Video Captioning
In this section, we present our experimental results on two public datasets: MSVD [3] and MSR-VTT [26]. We compare our D-LSG with other state-of-the-art methods and an in-depth ablation study is provided to better understand our method.  [15], METEOR [6], CIDEr [23] and ROUGE-L [13]. BLEU-4 measures the fraction of overlapping n-grams (here n = 4) between predicted sentences and reference sentences. METEOR calculates the precision and recall between predicted sentences and references based on uni-gram, which extends exact word matching to various match levels. CIDEr evaluates the consensus between a predicted sentence and reference sentences of the corresponding image or video based on the number of overlapping units such as n-gram. ROUGE-L computes recall and precision scores of the longest common subsequences (LCS) between the generated and each reference sentence. For all metrics, a higher value represents better performance of the generated captions. Data Preprocessing and Feature Extraction. We follow the process of [22] for corpus preprocessing and feature extraction. For corpus preprocessing, captions are first converted to lower case and punctuations are removed. Then, captions with more than 26 words are truncated and captions with less than 26 words are zero-padded. Besides, words that appear less than twice and five times are deleted in MSVD and MSR-VTT, respectively. For feature extraction, 2D and 3D CNN feature extractors are InceptionResNetV2 (IRV2) [21] and I3D [2]. Features from 26 frames are uniformly sampled in each video. Faster-RCNN [18] is adopted to extract the 36 region features for each frame out of the 26 sampled frames. Implementation Details. The Adam optimizer is applied with a learning rate 8 × 10 −4 for LSG model. For the discriminative model, we applied the Adam optimizer with ascent learning rates from 2 × 10 −4 to 8 × 10 −4 . The size of hidden states for all LSTM models is 1024 and 1536 in MSVD and MSR-VTT datasets, respectively. The feature size for all graph operations is set to 1024 for both datasets. Layer normalization is applied on top of the LSTM layer and graph nodes to speed up convergence. The word embedding size is set to 300 without the use of any pre-trained embedding such as glove. Feature dimension in CNN for discriminative model word feature extraction is set to 512. The training batch size is set to 128 for both datasets. Beam search is applied during inference with size 5.
Discriminative Latent Semantic Graph for Video Captioning
We compare our proposed D-LSG model with the state-of-theart models on the MSVD and MSR-VTT datasets to evaluate our model's performance, and the results are listed in Table 1. The results illustrate that our model achieves the best performance on the MSVD and MSR-VTT datasets for all evaluation metrics, which indicates the effectiveness of our proposed model for the video captioning task. The detailed analysis of results on the MSVD and MSR-VTT datasets is shown below.
Discriminative Latent Semantic Graph for Video Captioning
Comparison with encoder-decoder models. We first compare our model with traditional encoder-decoder based models, including PickNet [4] and MARN [17]. We can observe from Table 1 that the performance gains significant improvement, which means object information plays an important role in video captioning task.
Discriminative Latent Semantic Graph for Video Captioning
Comparision with object-based models. We then compare with several recent studies that consider detailed object information, including OA-BTG [29] and RMN [22]. We can observe that our D-LSG gains better performance in all metrics on both datasets, proving the effectiveness of utilizing graph-based models to learn object-level features. OA-BTG builds a bi-directional temporal graph based on  object features. However, the interaction between different objects is less encoded in the graph structure. RMN utilizes an attention mechanism for encoding the object-level features. However, the object interactions are only considered in the motion modeling module. We observe that our D-LSG model provides more obvious improvements on CIDEr than BLUE-4, which indicates that modeling interactions between different objects help generate rich semantic captions. Comparision with GNN-based models. Finally, we compare our model with the most recent approaches that adopt GNN based methods, including ORG-TRL [31], S-T Graph [14], and C-R Reasoning [9]. Our model outperforms the mentioned GNN based models and achieves excellent performance in BLUE-4 and CIDEr metrics. The BLUE-4 metric focuses on the fluency and logic of the generated captions and the CIDEr metric mainly focuses on content-relevant words in videos. The performance proves that D-LSG successfully captures high-level semantic concepts. In contrast, while ORG-TRL employs GCN to model object interactions, ORG-TRK does not take frame-level information into account when conducting objectlevel graph convolution. Our proposed LSG has better performance, indicating that aggregating object-level information conditioned on frame-level features helps the GNN learn better object representations. C-R Reasoning utilizes extra datasets to build semantic knowledge graphs. On the contrary, we propose to use discriminative modeling to enhance the semantic extraction via reconstructing visual words. This demonstrates that D-LSG can succeed in extracting semantic information without using auxiliary databases. 
Discriminative Latent Semantic Graph for Video Captioning
We then verify the effectiveness of the proposed D-LSG method through ablation studies on the MSVD and MSR-VTT dataset as shown in Table 2: (1)   Effect of Graph. Comparing the results of CGO only and LSG, we observe a noticeable performance decrease on both datasets, which indicates the importance of summarizing frame-level features to latent concepts or visual words. Comparing the results of LAP only and LSG, the performance also decreases. This is because LAP does not employ the conditional graph operation so that the visual words obtained in this model lack detailed object-level information. Since the performance drop of the LAP only model compared with the CGO only model is more pronounced, we can conclude that summarizing only frame-level information is not representative enough for semantic concepts, which also implies CGO's effectiveness modeling detailed object information.
Discriminative Latent Semantic Graph for Video Captioning
Effect of latent proposal number. We also evaluated how the number of visual words affects the quality of the generated captions on the MSVD and MSR-VTT datasets. Figure 5 illustrates the performance on CIDEr using different numbers of visual words for both MSVD and MSR-VTT datasets. As for the MSVD dataset, a small number of latent proposals provide better performance on CIDEr. However, when the number of proposals increases, the performance drops significantly. The intuition behind this is that the videos and captions in the MSVD dataset are short, so the data do not have enough semantic information to construct a large number of visual words, which results in performance drops. On the contrary, MSR-VTT with longer videos and captions suffers more performance drop when the number of proposals is small, which means that a small number of visual knowledge is not enough to represent the semantic concept of a given video.
Discriminative Latent Semantic Graph for Video Captioning
The above examples imply that The LSG model is able to summarize video content into semantic concepts with a proper number of visual words. Effect of discriminative modeling based on Graph. For the MSVD dataset, comparing LSG and D-LSG, we observe that ME-TEOR and ROUGE-L have a slight improvement, while BLEU-4 and CIDEr show large improvements, especially for CIDEr. Though the advantage is less apparent on the MSR-VTT dataset, the increase of CIDEr is also noticeable when comparing the improvement on other evaluation metrics. Since the mechanism of CIDEr is to punish words that are less informative of the video content, it may indicate that the discriminative structure can enrich the semantic concepts of the generated sentences, which means the model is capable of helping LSG capture and summarize key semantic concepts more effectively from input video features. The increase of BLEU-4 indicates that the discriminative model is capable of organizing enriched semantic concepts while keeping the grammatical structure of the generated captions. Figure 6 shows examples of the generated captions on MSVD and MSR-VTT datasets. Compared to the ground truth captions, we can observe that the generated captions contain important objects (e.g.,"man","guitar", "stroller") and motion (e.g., "playing", "showing") information even when the information is rare such as "peeling" and "folding". Also, the results indicate the visual words are capable of representing video content, and the model can capture frame-based background information rather than only focusing on detailed object information (e.g., "stadium" is detected in the example at bottom-right).
Discriminative Latent Semantic Graph for Video Captioning
We have presented the first work to introduce graph neural networks and discriminative modeling to process spatio-temporal information for the video captioning task jointly. As for the Latent Semantic Graph, from the experiment results, we conclude that the Conditional Graph Operation effectively models detailed object-level interactions and relationships. Besides, considering frame-level conditions is conducive to object-level interactive representation learning. The Latent Proposal Aggregation component also succeeded in summarizing high-level visual knowledge from input video features. Also, the discriminative modeling enriched the generated captions' semantic information via visual knowledge reconstruction and discriminative training. On two public datasets, our D-LSG model has outperformed the current state-of-the-art approaches, which verifies the effectiveness of our method.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Human perception of the world intrinsically comprises multiple modalities, including vision, audio, text, etc. (McGurk and MacDonald, 1976;Baltrušaitis et al., 2018). Audio-visual speech recognition (AVSR) leverages both audio and visual modalities to understand human speech, improving the noise-robustness of audio-only speech recognition with noise-invariant lip movement information (Sumby and Pollack, 1954). Thanks to recent advances of deep learning techniques, AVSR research has gained a remarkable progress (Afouras et al., 2018a;Ma et al., 2021;Shi et al., 2022b).
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Currently, the mainstream AVSR approaches are centered around developing sophisticated architectures and techniques for multi-modality fusion, 1 Code is available at https://github.com/YUCHE N005/MIR-GAN.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Modality-specific including simple feature concatenation (Makino et al., 2019;Ma et al., 2021;Pan et al., 2022), recurrent neural network (Petridis et al., 2018;Xu et al., 2020) and cross-modal attention module (Afouras et al., 2018a;Lee et al., 2020). Despite the advances, these approaches are often challenged by the representation gap persisting between naturally heterogeneous modalities (Hazarika et al., 2020). Recently in some other multimodal tasks like sentiment analysis (Hazarika et al., 2020;Yao and Mihalcea, 2022) and cross-modal retrieval , there have been research works proposing to learn two distinct representations to benefit multimodal learning. The first representation is modality-invariant, where multiple modalities of a same utterance are mapped to a shared space, indicating the homogeneous semantic meaning from the speaker. In addition, they also learn modality-specific representations that are private to each modality. Given an utterance, each modality contains some unique features with respect to speaker-sensitive information (Tsiros, 2013). Combing these two representations provides a holistic view of multimodal data for downstream tasks (Yang et al., 2022). However, these methods focus on utterance-level representations that could be easily mapped to either shared or individual modality space using similarity cost functions, which does not apply to AVSR task that requires sequence-to-sequence mapping with temporal contextual dependencies (Petridis et al., 2018).
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Motivated by above observations, we propose an adversarial network to refine frame-level modalityinvariant representations (MIR-GAN) for capturing the commonality across modalities, which bridges their heterogeneous gap to ease the subsequent multimodal fusion. In particular, we first design a MIR generator to learn modality-invariant representations over the shared audio-visual modality space. Meanwhile, a modality discriminator is proposed to strengthen its modality agnosticism via adversarial learning. Moreover, to further enrich its contextual semantic information, we propose a mutual information maximization strategy to align the refined representations to both audio and visual modality sequences. Finally, both modality-invariant and -specific representations are fused for downstream speech recognition. Empirical results demonstrate the effectiveness of our approach. In summary, our main contributions are:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
• We present MIR-GAN, an AVSR approach to refine frame-level modality-invariant representations, which captures the commonality across modalities and thus bridges their heterogeneous gap to ease multimodal fusion.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
• We first learn modality-invariant representations with a MIR generator, followed by another modality discriminator to strengthen its modality agnosticism via adversarial learning. Furthermore, we propose a mutual information maximization strategy to enrich its contextual semantic information. Finally, both modality-invariant and -specific representations are fused for downstream recognition.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
• Our proposed MIR-GAN outperforms the state-of-the-arts on LRS3 and LRS2 benchmarks. Extensive experiments also show its superiority on ASR and VSR tasks. and EG-seq2seq (Xu et al., 2020) leverage recurrent neural network for audio-visual fusion. In addition, cross-modal attention has also become popular recently for multimodal interaction and fusion in AVSR tasks, such as TM-seq2seq (Afouras et al., 2018a), DCM (Lee et al., 2020) and MMST (Song et al., 2022). Despite the effectiveness, these fusion techniques are often challenged by the representation gap between naturally heterogeneous modalities. Recently, multimodal self-supervised learning has been popular for capturing unified crossmodal representations, like AV-HuBERT (Shi et al., 2022a) andu-HuBERT (Hsu andShi, 2022), which achieve the state-of-the-art but require abundant unlabeled data and computing resources. In this work, we propose a supervised learning scheme to efficiently refine modality-invariant representations for bridging the heterogeneous modality gap.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Modality-Invariant and -Specific Representations. Recent studies in many multimodal tasks suggest that the model benefits from both shared and individual modality representations, including multimodal sentiment analysis (Hazarika et al., 2020;Yang et al., 2022), person re-identification (Wei et al., 2021;Huang et al., 2022), cross-modal retrival (Zeng et al., 2022)  builds an individual network for each modality, with a shared identity loss to learn modalityinvariant representations. However, these methods map utterance-level representations to modalityinvariant or -specific spaces via similarity cost functions, while AVSR is sequence-to-sequence task that requires contextual semantic information. To this end, we propose an adversarial network with mutual information maximization to refine framelevel modality-invariant representations that subjects to temporal contextual dependencies.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Adversarial Network. The concept of adversarial network starts from GAN (Goodfellow et al., 2014), which has attracted a surge of research interests due to its strong ability of generating high-quality novel samples according to existing data. The bestknown applications include image-to-image trans- lation (Isola et al., 2017) and image synthesis (Denton et al., 2015;Radford et al., 2015). Recently, GAN is further applied to multimodal tasks such as text-to-image synthesis (Reed et al., 2016;Tan et al., 2020), video captioning (Yang et al., 2018;Bai et al., 2021) and cross-modal retrieval (Qian et al., 2021). In this work, we leverage the strong distinguishing ability of adversarial network to strengthen the modality agnosticism of the learned modality-invariant representations.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
The overall architecture of our proposed MIR-GAN is illustrated in Fig. 2. First, we have two front-end modules 2 to process the input streams, which generate two modality sequences, i.e., f v , f a ∈ R T ×D , where T is number of frames and D is embedding size. These two sequences are then fed by visual and audio encoders respectively to generate modality-specific representations, i.e., f spe v , f spe a ∈ R T ×D . Based on that, we propose a MIR generator to learn modality-invariant representations by extracting the shared information of two modalities, i.e., f inv va ∈ R T ×D . Meanwhile, we design a modality discriminator to strengthen its modality agnosticism via adversarial learning. In addition, to further 2 Details are presented in Appendix A.3. enrich its contextual semantic information, we propose a mutual information maximization (MIM) strategy to align the refined representations to both audio and visual modality sequences. Finally, both modality-invariant and -specific representations are fused for downstream speech recognition.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
As illustrated in Fig. 2 (d), we introduce a pair of visual and audio encoders to learn modality-specific representations. Following Transformer (Vaswani et al., 2017) architecture, they first employ selfattention modules to capture the contextual dependencies within each modality, followed by crossattention modules for interaction between two modalities, which can initially narrow their gap to benefit the subsequent modality-invariant representation learning. Finally, there are feed-forward networks to generate the modality-specific outputs.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
With learned modality-specific representations, we propose MIR-GAN to refine frame-level modalityinvariant representations. First, we design a MIR generator to extract the shared information of two modalities, which generates a modality-invariant representation f inv va ∈ R T ×D . Meanwhile, we design a modality discriminator to strengthen its modality agnosticism via adversarial learning. Here we take the visual modality for example (m = v). Fig. 2 (b) details the architecture of proposed MIR generator G, where we design a hybrid-modal attention (HMA) module to extract out the part of information in each modality-specific representation that is related to both modalities:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
(1) where the subscript m denotes modality. The resulted features are then added to input sequence f va to form the final modality-invariant representation:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
where the "Norm" denotes layer normalization (Ba et al., 2016), "Conv" denotes 1×1 convolution followed by PReLU activation (He et al., 2015).
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Hybrid-Modal Attention (HMA) first involves a cross-attention sub-module to extract the information in each modality-specific representation that is related to both modalities, with the query input f va comprising both visual and audio sequence information, as shown in Fig. 3:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
(3) To further make the extracted feature invariant to modalities, we design a parallel convolutional network to learn a mask for filtering out the modalityspecific information:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
where "Conv" denotes 1×1 convolutional layer, ∥ denotes feature concatenation, σ denotes Sigmoid activation, ⊗ denotes element-wise multiplication. As a result, the output representation s m from HMA involves information regarding both visual and audio modalities, making the final output f inv va (in Eq. 2) invariant to modalities.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
With the generated modality-invariant representation, we further design a modality discriminator D to strengthen its modality agnosticism via adversarial learning. As shown in Fig. 2 (c), the discriminator consists of two linear layers followed by Sigmoid activation to predict a scalar between 0 and 1 for each frame, indicating which modality it belongs to (i.e., 0 for visual and 1 for audio):
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Therefore, for frames in modality-specific representations f spe v and f spe a , we hope the discriminator can correctly classify the modality type, i.e., 0 or 1. In contrast, in order to strengthen the modality agnosticism of refined representation f inv va , we hope it can confuse the discriminator with the output around 0.5, i.e., a medium between two modalities.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
With above designs of generator and discriminator, the adversarial training objective of MIR-GAN can be mathematically formulated as:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
where f inv va = G(f spe v , f spe a , f va ), E denotes the expectation over all the temporal frames in current data batch. Details of the corresponding optimization strategy are illustrated in Alg. 1.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
The MIR-GAN successfully refines the modalityinvariant representation by focusing on the modality commonality and agnosticism, while the original semantic information may not be preserved. To this end, we further design a mutual information maximization (MIM) strategy via contrastive learning to enrich the contextual semantic information in refined modality-invariant representation.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
In particular, we formulate a contrastive loss function to maximize the mutual information between modality-invariant representation f inv va and the modality-specific representations f spe v , f spe a :
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
where ⟨ ·, · ⟩ denotes cosine similarity, τ is temperature parameter. The subscripts i and j denote frame index, where f inv va /f spe v /f spe a ∈ R T ×D . The constructed positive and negative samples are distinguished by frame index. As same frame of different representations express similar semantic meanings, we assign them as positive samples to strengthen consistency, while the mismatched frames are pulled apart from each other. As a result, the MIM strategy can enrich the semantic information in final modality-invariant representation.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
The optimization strategy of MIR-GAN is detailed in Alg. 1. After the forward-propagation process, we calculate L GAN and L MIM according to Eq. 6 and Eq. 7. Meanwhile, the downstream speech recognition loss L rec is calculated as the crossentropy between recognized text and the groundtruth transcription. The final training objective of MIR-GAN can therefore be written as:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
where λ GAN , λ MIM are weighting parameters to balance different training objectives. Inspired by GAN training strategy (Goodfellow et al., 2014), we split the back-propagation process into two steps. First, we maximize L GAN to update the discriminator, where the generator is detached from optimization. According to Eq. 6, maximizing the first term of L GAN (i.e., L D ) trains the discriminator to correctly classify the two modalities, while increasing the second term amounts to informing discriminator that f inv va is modality-specific and can either be visual or audio 3 (this is opposite to what we desire as modality-invariant). Second, we freeze discriminator and update the rest network, where minimizing L G pushes the discrimination output of f inv va to 0.5, 3 which is a medium between visual and audio modalities, i.e., modality-agnostic. In addition, L rec optimizes the downstream speech recognition model and L MIM implements the MIM strategy. The entire system is trained in an end-toend manner with well-tuned weighting parameters.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Datasets. We conduct experiments on two largescale public benchmarks, LRS3 (Afouras et al.,
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Require: Training data D that contains visual-audio pairs (xv, xa) and the text transcription y. The MIR-GAN network θ that consists of front-ends θ vf and θ af , encoders θvae, MIR generator θG, modality discriminator θD and downstream speech recognition model θrec.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Hyper-parameter weights λGAN , λMIM . 1: Randomly initialize the entire system θ. 2: if select self-supervised setting then 3:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Load the pre-trained AV-HuBERT for speech recognition model θrec and front-ends θ vf , θ af 4: end if 5: while not converged do 6:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
for (xv, xa) ∈ D do 7:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
FORWARD-PROPAGATION:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
TRAINING OBJECTIVES: 14:
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
LGAN ( (Chung et al., 2017). LRS3 dataset collects 433 hours of transcribed English videos in TED and TEDx talks from over 5000 speakers, which is the largest publicly available labeled audio-visual speech recognition dataset. LRS2 dataset contains 224 hours of video speech, with a total of 144K clips from BBC programs. Model Configurations. We first build a base model with only front-ends and downstream speech recognition module, which follows Transformer architecture with 24 encoder layers and 9 decoder layers. Based on that, we build the MIR-GAN with N E = 3 visual & audio encoder layers and N G = 3 MIR generator layers. To maintain similar model size, we only use 12 encoder layers and 9 decoder layers in the recognition model. The number of parameters in our base model and MIR-GAN are 476M and 469M respectively. We also use Conformer (Gulati et al., 2020) as our backbone. In addition, we implement a self-supervised setting by loading pre-trained AV-HuBERT 4 . Following prior work (Shi et al., 2022b), we employ data augmentation and noisy test set based on MUSAN  (Watanabe et al., 2017), "CTC" denotes CTC loss (Graves et al., 2006), "DataAug" denotes noise augmentation, "LM" denotes language model rescoring. The noisy test set is synthesized using MUSAN noise (Snyder et al., 2015).
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
noise (Snyder et al., 2015). More detailed settings are presented in Appendix A.3 -A.5.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Baselines. To evaluate our proposed MIR-GAN, we select some popular AVSR methods for comparison, which can be roughly divided into two groups. The first is supervised learning method, including TM-seq2seq/CTC ( 
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
We conduct experiments on two public datasets under supervised and self-supervised settings, depending on whether use the AV-HuBERT pre-trained model. Results show that our proposed MIR-GAN achieves the state-of-the-art under both settings. LRS3 Benchmark.   , where "TF"/"CF" denote Transformer/Conformer backbone, "Sup"/"SelfSup" denote supervised/self-supervised setting, "3"/2" denote LRS3/LRS2 test set. margin under both supervised and self-supervised settings (3.2% vs. 3.7%, 2.2% vs. 2.6%). In addition, we also observe promising gains of performance in noisy testing conditions. As a result, our proposed MIR-GAN achieves new state-of-the-art under both supervised and self-supervised settings on two public benchmarks, which demonstrates its superiority on AVSR task. Table 3 presents the ablation study of each component in MIR-GAN. There are three parts of ablation that are independent with each other, i.e., each study is conducted where other two components are kept same as the full MIR-GAN. Importance of Representations. We first investigate the importance of modality-invariant andspecific representations by discarding each of them. When removing the refined modality-invariant representations from multi-modality fusion, the downstream speech recognition performance degrades a lot under all configurations, which verifies its significance of bridging the modality gap. Similarly, we observe that the modality-specific representations also plays an important role in AVSR. Importance of Modules. In this part, we study the role of each module in the proposed MIR-GAN. The visual and audio encoders are designed to extract deep modality-specific representations, which contributes to performance gains of MIR-GAN. Then we replace the core module -MIR generator with simple feature concatenation in refining modality-invariant representations, which results in significant performance degradation. Another key module -modality discriminator also contributes 
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
a lot in MIR-GAN by strengthening the modality agnosticism of refined representations from MIR generator. In this sense, we conclude that all the modules in proposed MIR-GAN contribute positively to the multimodal representation learning.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
With the adversarial training strategy illustrated in Alg. 1, the proposed modality discriminator effectively strengthens the modality agnosticism of the refined representations from generator. To verify its effectiveness, we remove the adversarial training strategy from MIR-GAN, which results in similar performance degradation to the previous case without modality discriminator. Therefore, it demonstrates the key role of this strategy in learning modality-invariant representations, where further visualization is shown in Fig. 5. Meanwhile, we design a MIM strategy to enrich the contextual semantic information in the refined modality-invariant representations, and similar performance drops can be observed in absence of such strategy. Furthermore, we visualize the attention maps in Fig. 4 to show its effectiveness. The clear diagonals in (c) and (d) indicate the strong ability of MIM strategy to align modality-invariant and -specific representations, which enriches the contextual semantic information in the former.  Visualizations of Modality-Invariant and -Specific Representations. Fig. 5 presents the t-SNE visualization of modality-invariant andspecific representations to illustrate the principle of MIR-GAN. First, we observe from (a) base model that the two modality-specific representations are distantly separated, indicating the heterogeneous gap between different modalities (Hazarika et al., 2020). With the proposed MIR-GAN (no modality discriminator), the two modalities are pushed closer by the interaction between encoders, and the refined modality-invariant representations serve as a medium between them. However, these refined representations are still entangled with audio modality-specific representations 5 , making them less modality-invariant. Finally, the proposed discriminator effectively strengthens their modality agnosticism via adversarial learning, which are dispersed between two modalities to capture their commonality and thus bridge the heterogeneous modality gap. As a result, the subsequent multi-modality fusion process would be eased and generate better features for downstream recognition.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Comparison with Utterance-Level Approaches As illustrated in §2, prior works have investigated utterance-level modality-invariant and -specific representations with similarity cost functions, includ-5 Audio modality plays the dominant role in AVSR task.  Table 5: Performance on single-modality inputs with LRS3 benchmark. "AV", "A" and "V" indicate the input modality during both finetuning and inference stages. The missing modality is replaced by zero embeddings.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
ing MISA (Hazarika et al., 2020), MCLNet (Hao et al., 2021) and VI-REID (Feng et al., 2019). We implement them in our framework as comparison to our proposed MIR-GAN, where we employ their designed similarity cost functions on frame-level representations. As illustrated in Table 4, these utterance-level approaches can also improve AVSR results but still underperforms our proposed approach by a large margin. Performance on Single-Modality Inputs. Furthermore, Table 5 presents the performance of our MIR-GAN on single-modality inputs. First, we observe that in all models using both modalities performs better than single modality, and the audio-only case achieves much better results than visual-only case, which shows the dominance of audio modality in AVSR task. Under two single-modality cases, our proposed MIR-GAN both achieves significant improvement over the base model, and the best performance outperforms or matches previous stateof-the-arts in both supervised and self-supervised settings (2.3% vs. 2.3%, 34.2% vs. 33.6%; 1.3% vs. 1.4%, 26.6% vs. 26.9%). Therefore, even with missing modality, our MIR-GAN can still refine effective modality-invariant representations to benefit the downstream speech recognition, which further verifies the generality of our approach.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
In this paper, we propose MIR-GAN, an adversarial network to refine frame-level modality-invariant representations for AVSR, which captures the commonality across modalities to ease the multimodal fusion process. MIR-GAN first learns modalityinvariant representation with MIR generator, followed by a modality discriminator to strengthen its modality agnosticism via adversarial learning. Furthermore, we propose a mutual information maximization strategy to enrich its contextual semantic information. Finally, both modality-invariant and -specific representations are fused to provide a holistic view of multimodal data for downstream task. Experiments on public benchmarks show that our MIR-GAN achieves the state-of-the-art.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
The main novelty of our proposed MIR-GAN is refining frame-level modality-invariant representations via adversarial learning. It is promising to combine this approach with the popular selfsupervised pre-training to learn unified multimodal representations. In this work, we only load pretrained AV-HuBERT for the front-ends and speech recognition model, while the proposed modules (i.e., encoders, generator, discriminator) are still trained from scratch. In future, we may include the entire MIR-GAN into self-supervised learning scheme, together with the adversarial learning to refine better multimodal representations.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
All the data used in this paper are publicly available and are used under the following five licenses: the Creative Commons BY-NC-ND 4.0 License and Creative Commons Attribution 4.0 International License, the TED Terms of Use, the YouTube's Terms of Service, and the BBC's Terms of Use. The data is collected from TED and BBC and contain thousands of speakers from a wide range of races.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
To protect the anonymity, only the mouth area of a speaker is visualized wherever used in the paper.  Table 6: WER (%) of our MIR-GAN and prior works on LRS2 benchmark. "S2S" denotes sequence-to-sequence loss (Watanabe et al., 2017), "CTC" denotes CTC loss (Graves et al., 2006), "DataAug" denotes noise augmentation, "LM" denotes language model rescoring. The noisy test set is synthesized using MUSAN noise (Snyder et al., 2015).
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
part differs from trainval in that the duration of its video clips are at a much wider range. Since there is no official development set provided, we randomly select 1,200 samples from trainval as validation set (∼ 1 hour) for early stopping and hyper-parameter tuning. In addition, it provides a standard test set (0.9 hours) for evaluation. LRS2 7 (Chung et al., 2017) is a large-scale publicly available labeled audio-visual (A-V) datasets, which consists of 224 hours of video clips from BBC programs. The training data is divided into three parts: pretrain (195 hours), train (28 hours) and val (0.6 hours), which are all transcribed at sentence level. An official test set (0.5 hours) is provided for evaluation use.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
The data preprocessing for above two datasets follows the LRS3 preprocessing steps in prior work (Shi et al., 2022a). For the audio stream, we extract the 26-dimensional log filter-bank feature at a stride of 10 ms from input raw waveform. For the video clips, we detect the 68 facial keypoints using dlib toolkit (King, 2009) and align the image frame to a reference face frame via affine transformation. Then, we convert the image frame to gray-scale and crop a 96×96 region-of-interest (ROI) centered on the detected mouth. During train-ing, we randomly crop a 88×88 region from the whole ROI and flip it horizontally with a probability of 0.5. At inference time, the 88×88 ROI is center cropped without horizontal flipping. To synchronize these two modalities, we stack each 4 neighboring acoustic frames to match the image frames that are sampled at 25Hz.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Front-ends. We introduce the modified ResNet-18 from prior work (Shi et al., 2022a) as visual frontend, where the first convolutional layer is replaced by a 3D convolutional layer with kernel size of 5×7×7. The visual feature is flattened into an 1D vector by spatial average pooling in the end. For audio front-end, we use one linear projection layer followed by layer normalization (Ba et al., 2016). MIR-GAN. We build the MIR-GAN framework based on Transformer, where the embedding dimension/feed-forward dimension/attention heads in each Transformer layer are set to 1024/4096/16 respectively. In addition, we also employ Conformer as backbone, where the depth-wise convolution kernel size is set to 31. We use a dropout of p = 0.1 after the self-attention block within each Transformer layer, and each Transformer layer is dropped (Fan et al., 2019) at a rate of 0.1.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Following prior work (Shi et al., 2022b), we use many noise categories for data augmentation. We select the noise categories of "babble", "music" and "natural" from MUSAN noise dataset (Snyder et al., 2015), and extract some "speech" noise samples from LRS3 dataset. All categories are divided into training, validation and test partitions. During training process, we randomly select one noise category and sample a noise clip from its training partition. Then, we randomly mix the sampled noise with input clean audio, at signal-to-noise ratio (SNR) of 0dB with a probability of 0.25.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
At inference time, we evaluate our model on clean and noisy test sets respectively. Specifically, the system performance on each noise type is evaluated separately, where the testing noise clips are added at five different SNR levels: {−10, −5, 0, 5, 10}dB. At last, the testing results on different noise types and SNR levels will be averaged to obtain the final noisy WER result.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Training. We follow the sequence-to-sequence finetuning configurations of AV-HuBERT (Shi et al., 2022b) to train our systems. We use Transformer decoder to decode the encoded features into unigram-based subword units (Kudo, 2018), where the vocabulary size is set to 1000. The temperature τ in Eq. 7 is set to 0.1, and the weighting parameters λ GAN /λ MIM in Eq. 8 are set to 0.01/0.005 respectively. The entire system is trained for 60K steps using Adam optimizer (Kingma and Ba, 2014), where the learning rate is warmed up to a peak of 0.001 for the first 20K updates and then linearly decayed. The finetuning process takes ∼ 1.4 days on 4 NVIDIA-V100-32GB GPUs.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Inference. No language model is used during inference. We employ beam search for decoding, where the beam width and length penalty are set to 50 and 1 respectively. All the hyper-parameters in our systems are tuned on validation set. Since our experimental results are quite stable, a single run is performed for each reported result.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
In this section, we describe the baselines for comparison.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
• TM-seq2seq (Afouras et al., 2018a): TM-seq2seq proposes a Transformer-based AVSR system to model the A-V features separately and then attentively fuse them for decoding, and uses sequence-to-sequence loss (Watanabe et al., 2017) as training criterion. • EG-seq2seq (Xu et al., 2020): EG-seq2seq builds a joint audio enhancement and multimodal speech recognition system based on the element-wise attention gated recurrent unit (Zhang et al., 2019), where the A-V features are concatenated before decoding.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
• LF-MMI TDNN (Yu et al., 2020): LF-MMI TDNN proposes a joint audio-visual speech separation and recognition system based on time-delay neural network (TDNN), where the A-V features are concatenated before fed into the recognition network. MoCo+wav2vec employs self-supervised pre-trained audio and visual front-ends, i.e., wav2vec 2.0 (Baevski et al., 2020) and MoCo v2 (Chen et al., 2020), to generate better audio-visual features for fusion and decoding.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
• AV-HuBERT (Shi et al., 2022a,b): AV-HuBERT employs self-supervised learning to capture deep A-V contextual information, where the A-V features are masked and concatenated before fed into Transformer encoder to calculate masked-prediction loss for pretraining, and cross-entropy based sequence-tosequence loss is used for finetuning.
Generative Adversarial Text to Image Synthesis
In this work we are interested in translating text in the form of single-sentence human-written descriptions directly into image pixels. For example, "this small bird has a short, pointy orange beak and white belly" or "the petals of this flower are pink and the anther are yellow". The problem of generating images from visual descriptions gained interest in the research community, but it is far from being solved.
Generative Adversarial Text to Image Synthesis
Traditionally this type of detailed visual information about an object has been captured in attribute representationsdistinguishing characteristics the object category encoded into a vector (Farhadi et al., 2009;Kumar et al., 2009;Parikh & Grauman, 2011;Lampert et al., 2014), in particular to enable zero-shot visual recognition (Fu et al., 2014;Akata et al., 2015), and recently for conditional image generation (Yan et al., 2015).
Generative Adversarial Text to Image Synthesis
While the discriminative power and strong generalization  properties of attribute representations are attractive, attributes are also cumbersome to obtain as they may require domain-specific knowledge. In comparison, natural language offers a general and flexible interface for describing objects in any space of visual categories. Ideally, we could have the generality of text descriptions with the discriminative power of attributes.
Generative Adversarial Text to Image Synthesis
Recently, deep convolutional and recurrent networks for text have yielded highly discriminative and generalizable (in the zero-shot learning sense) text representations learned automatically from words and characters (Reed et al., 2016). These approaches exceed the previous stateof-the-art using attributes for zero-shot visual recognition on the Caltech-UCSD birds database (Wah et al., 2011), and also are capable of zero-shot caption-based retrieval. Motivated by these works, we aim to learn a mapping directly from words and characters to image pixels.
Generative Adversarial Text to Image Synthesis
To solve this challenging problem requires solving two subproblems: first, learn a text feature representation that captures the important visual details; and second, use these fea-arXiv:1605.05396v2 [cs.NE] 
Generative Adversarial Text to Image Synthesis
Generative Adversarial Text to Image Synthesis tures to synthesize a compelling image that a human might mistake for real. Fortunately, deep learning has enabled enormous progress in both subproblems -natural language representation and image synthesis -in the previous several years, and we build on this for our current task.
Generative Adversarial Text to Image Synthesis
However, one difficult remaining issue not solved by deep learning alone is that the distribution of images conditioned on a text description is highly multimodal, in the sense that there are very many plausible configurations of pixels that correctly illustrate the description. The reverse direction (image to text) also suffers from this problem but learning is made practical by the fact that the word or character sequence can be decomposed sequentially according to the chain rule; i.e. one trains the model to predict the next token conditioned on the image and all previous tokens, which is a more well-defined prediction problem.
Generative Adversarial Text to Image Synthesis
This conditional multi-modality is thus a very natural application for generative adversarial networks (Goodfellow et al., 2014), in which the generator network is optimized to fool the adversarially-trained discriminator into predicting that synthetic images are real. By conditioning both generator and discriminator on side information (also studied by Mirza & Osindero (2014) and Denton et al. (2015)), we can naturally model this phenomenon since the discriminator network acts as a "smart" adaptive loss function.
Generative Adversarial Text to Image Synthesis
Our main contribution in this work is to develop a simple and effective GAN architecture and training strategy that enables compelling text to image synthesis of bird and flower images from human-written descriptions. We mainly use the Caltech-UCSD Birds dataset and the Oxford-102 Flowers dataset along with five text descriptions per image we collected as our evaluation setting. Our model is trained on a subset of training categories, and we demonstrate its performance both on the training set categories and on the testing set, i.e. "zero-shot" text to image synthesis. In addition to birds and flowers, we apply our model to more general images and text descriptions in the MS COCO dataset (Lin et al., 2014).
Generative Adversarial Text to Image Synthesis
Key challenges in multimodal learning include learning a shared representation across modalities, and to predict missing data (e.g. by retrieval or synthesis) in one modality conditioned on another. Ngiam et al. (2011) trained a stacked multimodal autoencoder on audio and video signals and were able to learn a shared modality-invariant representation. Srivastava & Salakhutdinov (2012) developed a deep Boltzmann machine and jointly modeled images and text tags. Sohn et al. (2014) proposed a multimodal conditional prediction framework (hallucinating one modality given the other) and provided theoretical justification.
Generative Adversarial Text to Image Synthesis
Many researchers have recently exploited the capability of deep convolutional decoder networks to generate realistic images. Dosovitskiy et al. (2015) trained a deconvolutional network (several layers of convolution and upsampling) to generate 3D chair renderings conditioned on a set of graphics codes indicating shape, position and lighting. Yang et al. (2015) added an encoder network as well as actions to this approach. They trained a recurrent convolutional encoderdecoder that rotated 3D chair models and human faces conditioned on action sequences of rotations. Reed et al. (2015) encode transformations from analogy pairs, and use a convolutional decoder to predict visual analogies on shapes, video game characters and 3D cars.
Generative Adversarial Text to Image Synthesis
Generative adversarial networks (Goodfellow et al., 2014) have also benefited from convolutional decoder networks, for the generator network module. Denton et al. (2015) used a Laplacian pyramid of adversarial generator and discriminators to synthesize images at multiple resolutions. This work generated compelling high-resolution images and could also condition on class labels for controllable generation. Radford et al. (2016) used a standard convolutional decoder, but developed a highly effective and stable architecture incorporating batch normalization to achieve striking image synthesis results.
Generative Adversarial Text to Image Synthesis
The main distinction of our work from the conditional GANs described above is that our model conditions on text descriptions instead of class labels. To our knowledge it is the first end-to-end differentiable architecture from the character level to pixel level. Furthermore, we introduce a manifold interpolation regularizer for the GAN generator that significantly improves the quality of generated samples, including on held out zero shot categories on CUB.
Generative Adversarial Text to Image Synthesis
The bulk of previous work on multimodal learning from images and text uses retrieval as the target task, i.e. fetch relevant images given a text query or vice versa. However, in the past year, there has been a breakthrough in using recurrent neural network decoders to generate text descriptions conditioned on images (Vinyals et al., 2015;Mao et al., 2015;Karpathy & Li, 2015;Donahue et al., 2015). These typically condition a Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) on the top-layer features of a deep convolutional network to generate captions using the MS COCO (Lin et al., 2014) and other captioned image datasets.  incorporated a recurrent visual attention mechanism for improved results.
Generative Adversarial Text to Image Synthesis
Other tasks besides conditional generation have been considered in recent work. Ren et al. (2015) generate answers to questions about the visual content of images. This approach was extended to incorporate an explicit knowledge base . Zhu et al. (2015) applied sequence models to both text (in the form of books) and movies to perform a joint alignment.
Generative Adversarial Text to Image Synthesis
In contemporary work Mansimov et al. (2016) generated images from text captions, using a variational recurrent autoencoder with attention to paint the image in multiple steps, similar to DRAW (Gregor et al., 2015). Impressively, the model can perform reasonable synthesis of completely novel (unlikely for a human to write) text such as "a stop sign is flying in blue skies", suggesting that it does not simply memorize. While the results are encouraging, the problem is highly challenging and the generated images are not yet realistic, i.e., mistakeable for real. Our model can in many cases generate visually-plausible 64×64 images conditioned on text, and is also distinct in that our entire model is a GAN, rather only using GAN for post-processing.
Generative Adversarial Text to Image Synthesis
Building on ideas from these many previous works, we develop a simple and effective approach for text-based image synthesis using a character-level text encoder and class-conditional GAN. We propose a novel architecture and learning strategy that leads to compelling visual results. We focus on the case of fine-grained image datasets, for which we use the recently collected descriptions for Caltech-UCSD Birds and Oxford Flowers with 5 humangenerated captions per image (Reed et al., 2016). We train and test on class-disjoint sets, so that test performance can give a strong indication of generalization ability which we also demonstrate on MS COCO images with multiple objects and various backgrounds.
Generative Adversarial Text to Image Synthesis
In this section we briefly describe several previous works that our method is built upon.
Generative Adversarial Text to Image Synthesis
Generative adversarial networks (GANs) consist of a generator G and a discriminator D that compete in a twoplayer minimax game: The discriminator tries to distinguish real training data from synthetic images, and the generator tries to fool the discriminator. Concretely, D and G play the following game on V(D,G):
Generative Adversarial Text to Image Synthesis
Goodfellow et al. (2014) prove that this minimax game has a global optimium precisely when p g = p data , and that under mild conditions (e.g. G and D have enough capacity) p g converges to p data . In practice, in the start of training samples from D are extremely poor and rejected by D with high confidence. It has been found to work better in practice for the generator to maximize log(D(G(z))) instead of minimizing log(1 − D(G(z))).
Generative Adversarial Text to Image Synthesis
To obtain a visually-discriminative vector representation of text descriptions, we follow the approach of Reed et al.
Generative Adversarial Text to Image Synthesis
(2016) by using deep convolutional and recurrent text encoders that learn a correspondence function with images. The text classifier induced by the learned correspondence function f t is trained by optimizing the following structured loss:
Generative Adversarial Text to Image Synthesis
where {(v n , t n , y n ) : n = 1, ..., N } is the training data set, ∆ is the 0-1 loss, v n are the images, t n are the corresponding text descriptions, and y n are the class labels. Classifiers f v and f t are parametrized as follows:
Generative Adversarial Text to Image Synthesis
where φ is the image encoder (e.g. a deep convolutional neural network), ϕ is the text encoder (e.g. a characterlevel CNN or LSTM), T (y) is the set of text descriptions of class y and likewise V(y) for images. The intuition here is that a text encoding should have a higher compatibility score with images of the correspondong class compared to any other class and vice-versa.
Generative Adversarial Text to Image Synthesis
To train the model a surrogate objective related to Equation 2 is minimized (see Akata et al. (2015) for details). The resulting gradients are backpropagated through ϕ to learn a discriminative text encoder. Reed et al. (2016) found that different text encoders worked better for CUB versus Flowers, but for full generality and robustness to typos and large vocabulary, in this work we always used a hybrid characterlevel convolutional-recurrent network.
Generative Adversarial Text to Image Synthesis
Our approach is to train a deep convolutional generative adversarial network (DC-GAN) conditioned on text features encoded by a hybrid character-level convolutionalrecurrent neural network. Both the generator network G and the discriminator network D perform feed-forward inference conditioned on the text feature.
Generative Adversarial Text to Image Synthesis
We use the following notation. The generator network is denoted G : R Z × R T → R D , the discriminator as D :
Generative Adversarial Text to Image Synthesis
where T is the dimension of the text description embedding, D is the dimension of the image, and Z is the dimension of the noise input to G. We illustrate our network architecture in Figure 2.
Generative Adversarial Text to Image Synthesis
In the generator G, first we sample from the noise prior z ∈ R Z ∼ N (0, 1) and we encode the text query t using text encoder ϕ. The description embedding ϕ(t) is first compressed using a fully-connected layer to a small dimension (in practice we used 128) followed by leaky-ReLU and 
Generative Adversarial Text to Image Synthesis
Our text-conditional convolutional GAN architecture. Text encoding ϕ(t) is used by both generator and discriminator. It is projected to a lower-dimensions and depth concatenated with image feature maps for further stages of convolutional processing.
Generative Adversarial Text to Image Synthesis
then concatenated to the noise vector z. Following this, inference proceeds as in a normal deconvolutional network: we feed-forward it through the generator G; a synthetic imagex is generated viax ← G(z, ϕ(t)). Image generation corresponds to feed-forward inference in the generator G conditioned on query text and a noise sample.
Generative Adversarial Text to Image Synthesis
In the discriminator D, we perform several layers of stride-2 convolution with spatial batch normalization (Ioffe & Szegedy, 2015) followed by leaky ReLU. We again reduce the dimensionality of the description embedding ϕ(t) in a (separate) fully-connected layer followed by rectification.
Generative Adversarial Text to Image Synthesis
When the spatial dimension of the discriminator is 4 × 4, we replicate the description embedding spatially and perform a depth concatenation. We then perform a 1 × 1 convolution followed by rectification and a 4 × 4 convolution to compute the final score from D. Batch normalization is performed on all convolutional layers.
Generative Adversarial Text to Image Synthesis
The most straightforward way to train a conditional GAN is to view (text, image) pairs as joint observations and train the discriminator to judge pairs as real or fake. This type of conditioning is naive in the sense that the discriminator has no explicit notion of whether real training images match the text embedding context.
Generative Adversarial Text to Image Synthesis
However, as discussed also by (Gauthier, 2015), the dynamics of learning may be different from the nonconditional case. In the beginning of training, the discriminator ignores the conditioning information and easily rejects samples from G because they do not look plausible. Once G has learned to generate plausible images, it must also learn to align them with the conditioning information, and likewise D must learn to evaluate whether samples from G meet this conditioning constraint.
Generative Adversarial Text to Image Synthesis
In naive GAN, the discriminator observes two kinds of inputs: real images with matching text, and synthetic images with arbitrary text. Therefore, it must implicitly separate two sources of error: unrealistic images (for any text), and
Generative Adversarial Text to Image Synthesis
Algorithm 1 GAN-CLS training algorithm with step size α, using minibatch SGD for simplicity. 1: Input: minibatch images x, matching text t, mismatchingt, number of training batch steps S 2: for n = 1 to S do G ← G − α∂L G /∂G {Update generator} 14: end for realistic images of the wrong class that mismatch the conditioning information. Based on the intuition that this may complicate learning dynamics, we modified the GAN training algorithm to separate these error sources. In addition to the real / fake inputs to the discriminator during training, we add a third type of input consisting of real images with mismatched text, which the discriminator must learn to score as fake. By learning to optimize image / text matching in addition to the image realism, the discriminator can provide an additional signal to the generator.
Generative Adversarial Text to Image Synthesis
Algorithm 1 summarizes the training procedure. After encoding the text, image and noise (lines 3-5) we generate the fake image (x, line 6). s r indicates the score of associating a real image and its corresponding sentence (line 7), s w measures the score of associating a real image with an arbitrary sentence (line 8), and s f is the score of associating a fake image with its corresponding text (line 9). Note that we use ∂L D /∂D to indicate the gradient of D's objective with respect to its parameters, and likewise for G. Lines 11 and 13 are meant to indicate taking a gradient step to update network parameters.
Generative Adversarial Text to Image Synthesis
Generative Adversarial Text to Image Synthesis
Generative Adversarial Text to Image Synthesis
Deep networks have been shown to learn representations in which interpolations between embedding pairs tend to be near the data manifold (Bengio et al., 2013;Reed et al., 2014). Motivated by this property, we can generate a large amount of additional text embeddings by simply interpolating between embeddings of training set captions. Critically, these interpolated text embeddings need not correspond to any actual human-written text, so there is no additional labeling cost. This can be viewed as adding an additional term to the generator objective to minimize:
Generative Adversarial Text to Image Synthesis
where z is drawn from the noise distribution and β interpolates between text embeddings t 1 and t 2 . In practice we found that fixing β = 0.5 works well.
Generative Adversarial Text to Image Synthesis
Because the interpolated embeddings are synthetic, the discriminator D does not have "real" corresponding image and text pairs to train on. However, D learns to predict whether image and text pairs match or not. Thus, if D does a good job at this, then by satisfying D on interpolated text embeddings G can learn to fill in gaps on the data manifold in between training points. Note that t 1 and t 2 may come from different images and even different categories. 1
Generative Adversarial Text to Image Synthesis
If the text encoding ϕ(t) captures the image content (e.g. flower shape and colors), then in order to generate a realistic image the noise sample z should capture style factors such as background color and pose. With a trained GAN, one may wish to transfer the style of a query image onto the content of a particular text description. To achieve this, one can train a convolutional network to invert G to regress from samplesx ← G(z, ϕ(t)) back onto z. We used a simple squared loss to train the style encoder:
Generative Adversarial Text to Image Synthesis
where S is the style encoder network. With a trained generator and style encoder, style transfer from a query image x onto text t proceeds as follows:
Generative Adversarial Text to Image Synthesis
wherex is the result image and s is the predicted style.
Generative Adversarial Text to Image Synthesis
In this section we first present results on the CUB dataset of bird images and the Oxford-102 dataset of flower images. CUB has 11,788 images of birds belonging to one of 200 different categories. The Oxford-102 contains 8,189 images of flowers from 102 different categories.
Generative Adversarial Text to Image Synthesis
As in Akata et al. (2015) and Reed et al. (2016), we split these into class-disjoint training and test sets. CUB has 150 train+val classes and 50 test classes, while Oxford-102 has 82 train+val and 20 test classes. For both datasets, we used 5 captions per image. During mini-batch selection for training we randomly pick an image view (e.g. crop, flip) of the image and one of the captions.
Generative Adversarial Text to Image Synthesis
For text features, we first pre-train a deep convolutionalrecurrent text encoder on structured joint embedding of text captions with 1,024-dimensional GoogLeNet image embedings  as described in subsection 3.2. For both Oxford-102 and CUB we used a hybrid of character-level ConvNet with a recurrent neural network (char-CNN-RNN) as described in (Reed et al., 2016). Note, however that pre-training the text encoder is not a requirement of our method and we include some end-to-end results in the supplement. The reason for pre-training the text encoder was to increase the speed of training the other components for faster experimentation. We also provide some qualitative results obtained with MS COCO images of the validation set to show the generalizability of our approach.
Generative Adversarial Text to Image Synthesis
We used the same GAN architecture for all datasets. The training image size was set to 64 × 64 × 3. The text encoder produced 1, 024-dimensional embeddings that were projected to 128 dimensions in both the generator and discriminator before depth concatenation into convolutional feature maps.
Generative Adversarial Text to Image Synthesis
As indicated in Algorithm 1, we take alternating steps of updating the generator and the discriminator network. We used the same base learning rate of 0.0002, and used the ADAM solver (Ba & Kingma, 2015) with momentum 0.5. The generator noise was sampled from a 100-dimensional unit normal distribution. We used a minibatch size of 64 and trained for 600 epochs. Our implementation was built on top of dcgan.torch 2 .
Generative Adversarial Text to Image Synthesis
We compare the GAN baseline, our GAN-CLS with imagetext matching discriminator (subsection 4.2), GAN-INT learned with text manifold interpolation (subsection 4.3) and GAN-INT-CLS which combines both.
Generative Adversarial Text to Image Synthesis
Results on CUB can be seen in Figure 3. GAN and GAN-CLS get some color information right, but the images do not look real. However, GAN-INT and GAN-INT-CLS show plausible images that usually match all or at least part of the caption. We include additional analysis on the robustness of each GAN variant on the CUB dataset in the supplement. a tiny bird, with a tiny beak, tarsus and feet, a blue crown, blue coverts, and black cheek patch this small bird has a yellow breast, brown crown, and black superciliary an all black bird with a distinct thick, rounded bill. this bird is different shades of brown all over with white and black spots on its head and back
Generative Adversarial Text to Image Synthesis
the gray bird has a light grey head and grey webbed feet GT Figure 3. Zero-shot (i.e. conditioned on text from unseen test set categories) generated bird images using GAN, GAN-CLS, GAN-INT and GAN-INT-CLS. We found that interpolation regularizer was needed to reliably achieve visually-plausible results.
Generative Adversarial Text to Image Synthesis
this flower is white and pink in color, with petals that have veins.
Generative Adversarial Text to Image Synthesis
these flowers have petals that start off white in color and end in a dark purple towards the tips.
Generative Adversarial Text to Image Synthesis
bright droopy yellow petals with burgundy streaks, and a yellow stigma.
Generative Adversarial Text to Image Synthesis
a flower with long pink petals and raised orange stamen.
Generative Adversarial Text to Image Synthesis
the flower shown has a blue petals with a white pistil in the center Figure 4. Zero-shot generated flower images using GAN, GAN-CLS, GAN-INT and GAN-INT-CLS. All variants generated plausible images. Although some shapes of test categories were not seen during training (e.g. columns 3 and 4), the color information is preserved.
Generative Adversarial Text to Image Synthesis
Results on the Oxford-102 Flowers dataset can be seen in Figure 4. In this case, all four methods can generate plausible flower images that match the description. The basic GAN tends to have the most variety in flower morphology (i.e. one can see very different petal types if this part is left unspecified by the caption), while other methods tend to generate more class-consistent images. We speculate that it is easier to generate flowers, perhaps because birds have stronger structural regularities across species that make it easier for D to spot a fake bird than to spot a fake flower.
Generative Adversarial Text to Image Synthesis
Many additional results with GAN-INT and GAN-INT-CLS as well as GAN-E2E (our end-to-end GAN-INT-CLS without pre-training the text encoder ϕ(t)) for both CUB and Oxford-102 can be found in the supplement.
Generative Adversarial Text to Image Synthesis
In this section we investigate the extent to which our model can separate style and content. By content, we mean the visual attributes of the bird itself, such as shape, size and color of each body part. By style, we mean all of the other factors of variation in the image such as background color and the pose orientation of the bird.
Generative Adversarial Text to Image Synthesis
The text embedding mainly covers content information and typically nothing about style, e.g. captions do not mention the background or the bird pose. Therefore, in order to generate realistic images then GAN must learn to use noise sample z to account for style variations.
Generative Adversarial Text to Image Synthesis
To quantify the degree of disentangling on CUB we set up two prediction tasks with noise z as the input: pose verifi- cation and background color verification. For each task, we first constructed similar and dissimilar pairs of images and then computed the predicted style vectors by feeding the image into a style encoder (trained to invert the input and output of generator). If GAN has disentangled style using z from image content, the similarity between images of the same style (e.g. similar pose) should be higher than that of different styles (e.g. different pose).
Generative Adversarial Text to Image Synthesis
To recover z, we inverted the each generator network as described in subsection 4.4. To construct pairs for verification, we grouped images into 100 clusters using K-means where images from the same cluster share the same style.
Generative Adversarial Text to Image Synthesis
For background color, we clustered images by the average color (RGB channels) of the background; for bird pose, we clustered images by 6 keypoint coordinates (beak, belly, breast, crown, forehead, and tail).
Generative Adversarial Text to Image Synthesis
For evaluation, we compute the actual predicted style variables by feeding pairs of images style encoders for GAN, GAN-CLS, GAN-INT and GAN-INT-CLS. We verify the score using cosine similarity and report the AU-ROC (averaging over 5 folds). As a baseline, we also compute cosine similarity between text features from our text encoder.
Generative Adversarial Text to Image Synthesis
We present results on Figure 5. As expected, captions alone are not informative for style prediction. Moreover, consistent with the qualitative results, we found that models incorporating interpolation regularizer (GAN-INT, GAN-INT-CLS) perform the best for this task.
Generative Adversarial Text to Image Synthesis
We demonstrate that GAN-INT-CLS with trained style encoder (subsection 4.4) can perform style transfer from an unseen query image onto a text description. Figure 6 shows that images generated using the inferred styles can accurately capture the pose information. In several cases the style transfer preserves detailed background information such as a tree branch upon which the bird is perched.
Generative Adversarial Text to Image Synthesis
Disentangling the style by GAN-INT-CLS is interesting because it suggests a simple way of generalization. This way
Generative Adversarial Text to Image Synthesis
The bird has a yellow breast with grey features and a small beak.
Generative Adversarial Text to Image Synthesis
This is a large white bird with black wings and a red head.
Generative Adversarial Text to Image Synthesis
A small bird with a black head and wings and features grey wings.
Generative Adversarial Text to Image Synthesis
This bird has a white breast, brown and white coloring on its head and wings, and a thin pointy beak.
Generative Adversarial Text to Image Synthesis
A small bird with white base and black stripes throughout its belly, head, and feathers.
Generative Adversarial Text to Image Synthesis
A small sized bird that has a cream belly and a short pointed bill.
Generative Adversarial Text to Image Synthesis
This bird is completely red.
Generative Adversarial Text to Image Synthesis
This is a yellow bird. The wings are bright blue.
Generative Adversarial Text to Image Synthesis
Images (style) Figure 6. Transfering style from the top row (real) images to the content from the query text, with G acting as a deterministic decoder. The bottom three rows are captions made up by us.
Generative Adversarial Text to Image Synthesis
we can combine previously seen content (e.g. text) and previously seen styles, but in novel pairings so as to generate plausible images very different from any seen image during training. Another way to generalize is to use attributes that were previously seen (e.g. blue wings, yellow belly) as in the generated parakeet-like bird in the bottom row of Figure 6. This way of generalization takes advantage of text representations capturing multiple visual aspects. Figure 8 demonstrates the learned text manifold by interpolation (Left). Although there is no ground-truth text for the intervening points, the generated images appear plausible. Since we keep the noise distribution the same, the only changing factor within each row is the text embedding that we use. Note that interpolations can accurately reflect color information, such as a bird changing from blue to red while the pose and background are invariant.
Generative Adversarial Text to Image Synthesis
As well as interpolating between two text encodings, we show results on Figure 8 (Right) with noise interpolation.
Generative Adversarial Text to Image Synthesis
Here, we sample two random noise vectors. By keeping the text encoding fixed, we interpolate between these two noise vectors and generate bird images with a smooth transition between two styles by keeping the content fixed.
Generative Adversarial Text to Image Synthesis
We trained a GAN-CLS on MS-COCO to show the generalization capability of our approach on a general set of images that contain multiple objects and variable backgrounds. We use the same text encoder architecture, same GAN architecture and same hyperparameters (learning rate, minibatch size and number of epochs) as in CUB 'This is a yellow bird. The wings are bright blue'
Generative Adversarial Text to Image Synthesis
'this bird is all blue, the top part of the bill is blue, but the bottom half is white' Figure 8. Left: Generated bird images by interpolating between two sentences (within a row the noise is fixed). Right: Interpolating between two randomly-sampled noise vectors.
Generative Adversarial Text to Image Synthesis
and Oxford-102. The only difference in training the text encoder is that COCO does not have a single object category per class. However, we can still learn an instance level (rather than category level) image and text matching function, as in (Kiros et al., 2014).
Generative Adversarial Text to Image Synthesis
Samples and ground truth captions and their corresponding images are shown on Figure 7. A common property of all the results is the sharpness of the samples, similar to other GAN-based image synthesis models. We also observe diversity in the samples by simply drawing multiple noise vectors and using the same fixed text encoding.
Generative Adversarial Text to Image Synthesis
From a distance the results are encouraging, but upon close inspection it is clear that the generated scenes are not usually coherent; for example the human-like blobs in the baseball scenes lack clearly articulated parts. In future work, it may be interesting to incorporate hierarchical structure into the image synthesis model in order to better handle complex multi-object scenes.
Generative Adversarial Text to Image Synthesis
A qualitative comparison with AlignDRAW (Mansimov et al., 2016) can be found in the supplement. GAN-CLS generates sharper and higher-resolution samples that roughly correspond to the query, but AlignDRAW samples more noticably reflect single-word changes in the selected queries from that work. Incorporating temporal structure into the GAN-CLS generator network could potentially improve its ability to capture these text variations.
Generative Adversarial Text to Image Synthesis
In this work we developed a simple and effective model for generating images based on detailed visual descriptions.
Generative Adversarial Text to Image Synthesis
We demonstrated that the model can synthesize many plausible visual interpretations of a given text caption. Our manifold interpolation regularizer substantially improved the text to image synthesis on CUB. We showed disentangling of style and content, and bird pose and background transfer from query images onto text descriptions. Finally we demonstrated the generalizability of our approach to generating images with multiple objects and variable backgrounds with our results on MS-COCO dataset. In future work, we aim to further scale up the model to higher resolution images and add more types of text.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Cross-modal retrieval has raised widespread interests with the objective to perform retrieval across different modalities (Wang et al. 2017;Zhen et al. 2019;Liu et al. 2020). Traditionally, modeling semantically similarity links mainly focuses on single-modality scenarios, while the cross-modal retrieval requires that semantically similar items in one modality (e.g., text) can be retrieved with a query item from another modality (e.g., text).
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Since instances of different modality typically have inconsistent feature distributions and representations, the modality gap caused by the heterogeneous nature of data needs to be bridged. To bridge the gap among different multimodal data, the general solution is to map them into a common Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Figure 1: The left picture illustrates the co-occurrence of "Snow" and "Sky" in the NUS-WIDE dataset. The right picture demonstrates the co-occurrence of "Animals" and "Plant" in MIRFlickr dataset.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
representation space. Traditional methods (Hotelling 1936;Yao, Mei, and Ngo 2015) typically take linear projections as basic models, which often maximize the cross-modal pairwise item correlation or classification accuracy to learn the common representation. Recently, deep neural networks (DNN) are emerging tools to automatically learn feature representations, which have been increasingly utilized in the cross-modal retrieval task (Ngiam et al. 2011;Peng and Qi 2019;Andrew et al. 2013). The DNN-based cross-modal retrieval has become an active study field to exploit nonlinear relationships and achieved great performance improvements. Most existing methods, such as DSCMR (Zhen et al. 2019) and ACMR (Wang et al. 2017), mainly consider how to transform the original representation to the common representation and design handcraft loss functions based on the common representation. However, there may exist modalityspecific features in the common representation in the transform process, which may not be fully eliminated by the existing methods and lead to the performance decline. Therefore, we have to face the Challenge 1: How to bridge the modality gap so that modality-specific information can be eliminated well and semantic information can be fully preserved in the common representation space?
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Another drawback of existing supervised methods is that labels are treated as independent individuals and the label correlations in multimodal data are ignored. For example, cross-modal datasets such as MIRFlickr (Huiskes, Thomee, and Lew 2008) and NUS-WIDE (Chua et al. 2009), often contain multiple labels for each sample, where the semantic dependencies among multiple labels are important for crossmodal retrieval and representation learning. As illustrated in Figure 1, some labels co-occur in images with a great chance and we can observe that "Sky" usually appears together with "Snow", and "Plant" usually adjoins "Animals". Nevertheless, some pairs of labels hardly occurs in the physical world. Therefore, we need to address the Challenge 2: How to explore and capture the label correlation, which contains rich semantic information, to guide the model learning discriminative features?
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
To overcome the above challenges, we propose a novel deep cross-modal representation learning approach termed Dual Adversarial Graph Neural Networks (DAGNN) for multi-label cross-modal retrieval. For Challenge 1, we propose the dual generative adversarial networks (Dual GAN) to generate common representations and further reconstruct the original representations of the other modality. The generator models learn to fit the joint data distribution of different modalities, while the discriminator models learn to discriminate between original data and reconstructed data. By optimizing the adversarial loss and the modality invariance loss, the modality-specific features in the common space can be sufficiently eliminated and the semantic information can be fully preserved. For Challenge 2, we propose the multi-hop graph neural networks (Multi-hop GNN) to generate interdependent classifiers by exploring the label correlation dependency, which takes a set of labels and their corresponding prior label representations, e.g., word-embedding vectors, as input. The correlation matrix of GNN is calculated according to the statistical features of labels. However, resulting from the limitation of the calculation method and different relative levels between nodes, some pairs of relative nodes may not be connected directly but be connected by some paths in the graph. Therefore, we design a novel layer aggregation mechanism, in which a linear transform is applied on the concatenation of all previous GNN layers, to hierarchically exploit multi-hop propagation information. The learned label classifiers are then employed to classify the common representation, which is generated by the Dual GAN, to perform the end-to-end training process by the classification loss and guide the model learning the underlying semantic structure. In brief, the contributions of our work are listed as follows:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
• We propose a novel Dual Adversarial Graph Neural Networks (DAGNN) by designing Dual GAN to bridge the modality gap and constructing Multi-hop GNN to hierarchically explore and capture the label correlation dependency, which can learn modality-invariant and discriminative representation for multi-label cross-modal retrieval.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
• We propose a novel cross-modal representation learning method by adopting the dual generative adversarial networks to generate common representations, further cross-reconstruct original representations and discriminate modality-specific features between original representations and reconstructed representations, which can explore and eliminate modality-specific information while the semantic information can be fully preserved.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
• We leverage the multi-hop graph neural networks to learn inter-dependent classifiers by exploiting information in the label graph, which can utilize the label correlation to guide the model learning discriminative representations. Furthermore, a layer aggregation mechanism is proposed to flexibly leverage different neighborhood ranges for nodes, which can explicitly guide GNN utilizing multihop propagation information and enable better structureaware representation.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
• Extensive experiments demonstrate the superiority of DAGNN compared with ten cross-modal retrieval baselines on two public datasets NUS-WIDE and MIRFlickr.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
We firstly introduce some notations that are used in this paper. Assuming that the training set 
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where c denotes the category number. If the ith sample belongs to the jth category, y ij = 1; otherwise, y ij = 0.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
The key to the cross-modal retrieval is learning modalityspecific transformation functions:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
for the text modality, where θ v and θ t are trainable parameters of the two transformation functions, and d represents the number of dimensions in the common representation space. It enables the direct comparison of samples which are in the common representation space but from different modalities, and the similarity of the samples belonging to the same category would be higher than that from different categories. Therefore, the relevant samples of one modality can be retrieved by taking a query from another modality.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
As demonstrated in Figure 2, the overall architecture of our model consists of two main components, i .e., the dual generative adversarial networks and the multi-hop graph neural networks.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Dual generative adversarial networks (Dual GAN) include two sub-networks trained in an end-to-end style: ImgGAN for feature learning of image modality, TxtGAN for feature learning of text modality. We adopt VGGNet (Simonyan and Zisserman 2015) and Multi-Layer Perception (MLP) (Rumelhart, Hinton, and McClelland 1986) as the backbone of the ImgGAN and the TxtGAN, respectively. The inputs of ImgGAN are the raw images and the inputs of TxtGAN are the bag-of-words (BoW) features provided by datasets.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
ImgGAN. We adopt the convolutional layers in 19layer VGGNet, which is pretrained on the ImageNet (Deng et al. 2009), as convolutional layers of ImgGAN. 4096dimensional features are generated from fc7 layer as the original representations of raw images, denoted as h v i ∈ R dv (1 ≤ i ≤ n). Then, several fully-connected layers map h v i to the common representation, denoted as
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where d is the dimensionality of the common representation space and θ v = {θ cnn , θ G v 1 }. Then, the common representation will pass the generator of the second GAN (denoted as G v 2 ) to reconstruct the semantic representation of text modality, denoted as r t i :
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
The MLP in TxtGAN, which is pretrained via performing general classification task, consists of two fullyconnected layers f c t1 and f c t2 (d t → 4096 → 300) and generates features h t i ∈ R dt (1 ≤ i ≤ n). Then, several fully-connected layers are followed to obtain the common representation, denoted as t i :
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where d is the dimensionality of the common representation space and θ t = {θ mlp , θ G t 1 }. Then, the common representation passes the generator of the second GAN (denoted as G t 2 ) and reconstruct the image representation, denoted as r v i : After the cross-reconstruction, discriminators of Img-GAN and TxtGAN take the original representations and the reconstructed representations as input and predict the modalities of the input:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
of which the first dimension represents the predicted probability of the input representation belonging to image modality and the other one corresponds to text modality.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Category labels in our benchmark datasets, such as plant, tree, f lower, contain valuable semantic correlations and these labels should not be only involved in the classification loss calculation stage. Motivated by (Chen et al. 2019), we propose Multi-hop Graph Neural Networks (Multi-hop GNN) to learn inter-dependent classifiers by capturing and exploring the label correlations. Firstly, we build a graph G = (V, E) where the category label set is taken as the graph vertice set V while the number of categories c is the number of vertices, and E is the edge set. Each vertex is associated with a d (0) -dimensional feature vector. Feature matrix Q = {q i } c i=1 , Q ∈ R c×d (0) represent the features of vertices in the graph, where q i corresponds to the feature of the i-th vertex, and d (0) represents the dimensionality of the label-level word embedding. We also introduce a correlation matrix A ∈ R c×c , where A ij is the weight of the edge between the ith vertex and the jth vertex. We employ Multihop GNN to map the feature matrix Q ∈ R c×d (0) of these vertices into the corresponding inter-dependent classifiers, i .e., Z = {z i } c i=1 , Z ∈ R c×d . Several classical GNNs, such as GAT (Veličković et al. 2017), GCN (Kipf and Welling 2017), are tested as the backbone of the Multi-hop GNN.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Layer Aggregation Mechanism. We denote the output of the lth GNN layer as H l . As shown in Figure 3, we apply a concatenation [H 1 , H 2 , ..., H k ] to combine all layers of a k-layer GNN. Then a layer aggregation mechanism is leveraged to compute aggregated features:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where W ag ∈ R d×(d (0) +d (1) +...+d (k) ) , b ag ∈ R d are parameters of the aggregation layer. Some relative nodes in Figure 2 may be connected by some paths in the graph. Therefore, the layer aggregation mechanism can hierarchically exploit multi-hop propagation information and enable better structure-aware representation. We denote all parameters of the Multi-hop GNN as θ GN N . Then, by applying the learned classifiers to cross-modal representations {v i } n i=1 and {t i } n i=1 , we can obtain the predicted score as:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
whereŷ v ij is the predicted score of the ith image sample belonging to the jth category andŷ t ij is the predicted score of the ith text sample belonging to the jth category.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Our proposed correlation matrix A is based on statistical information as label co-occurrence patterns. Firstly, we compute conditional probability, i .e., P (L i |L j ), which represents the occurrence probability of label L i when label L j appears. We count the co-occurrence frequency M ∈ N c×c of label pairs and the occurrence frequency N ∈ N c of all labels in the training set. Then we calculate the conditional probability matrix:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
However, this matrix may be faced with two problems while performing as the correlation matrix. Firstly, some rare co-occurrence patterns between labels may be noise, which results from the existence of long-tail distributions. Secondly, the co-occurrence frequencies between labels may not be completely consistent in the training and testing scenarios, which can affect the generalization ability. Therefore, we propose a binary process with the threshold τ :
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where B is the binary correlation matrix. The feature of one node is the weighted sum of features of itself and its adjacent nodes after every layer in GNN. Thus, the binary correlation matrix can result in over-smoothing, which means features of different nodes become indistinguishable. In order to solve this problem, the following re-weighted trick is applied:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where A is the re-weighted correlation matrix, and p controls the weight assigned to a vertex itself and its neighbors. Via setting a suitable p, propagation information from different nodes can be integrated appropriately. Remarkably, when p → 0, the neighboring information will be completely ignored. It is worth noting that the value of p(> 0) does not influence the propagation process of GAT.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
We can learn the instance representation via minimizing the classification error and the dissimilarity of samples from same semantic categories, while maximizing the dissimilarity of samples from different categories. Furthermore, we can bridge the modality gap while the generators try to generate modality-invariant representations and the discriminators try to discriminate between modalities. We adopt the following objective function to measure the classification loss:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where · 2 denotes the l2 norm,
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Furthermore, we also measure the modality invariance loss that consists of inter-modal and intra-modal invariance losses in the common representation space:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
is the cosine similarity function. The first term of Equation 10 can be written as − 1 n 2 1≤i,j≤n log( e S vt ij Γ ij 1+e Γ ij ). When S vt ij = 0, minimizing this loss equals to minimizing the similarity Γ ij , otherwise, it is equivalent to maximizing Γ ij .
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
In order to sufficiently bridge the modality gap, we compute the adversarial loss of the Dual GAN:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where y vv i = y vt i = [1, 0](1 ≤ i ≤ n) are one-hot modal labels for image instances and y tv i = y tt i = [0, 1](1 ≤ i ≤ n) are one-hot modal labels for text instances.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Combining Equation 9, 10 and 11, we obtain the final objective function of DAGNN which can be optimized via Adam algorithm (Kingma and Ba 2015). Since the optimization goals of L adv and L cla , L mdi are opposite, the optimization process runs as a minimax game of the two concurrent sub-process:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
where the hyper-parameters α, β are trade-off factors of three components. Following (Ganin and Lempitsky 2015), we perform the minimax optimization by applying Gradient Reversal Layer (GRL), which is transparent during the forward-propagation, but multiplies the gradients by −1 during the back-propagation. Adding GRL before the first layer of the modality discriminators D v and D t , the minimax optimization can be performed simultaneously.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
We conduct sufficient experiments on two public multi-label cross-modal datasets NUS-WIDE and MIRFlickr to verify the superiority of DAGNN. By default, we adopt GAT-based Multi-hop GNN in DAGNN.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
We construct a subset of this dataset, which contains 190,421 image-text pairs covering the 21 most commonly used concepts, through pruning samples that are either unlabeled or that have no tag information. Every image-text pair is represented by a 224x224 RGB array and an index vector of 1,000 of the most common tags. Following (Zhen et al. 2019;Su, Zhong, and Zhang 2019), we randomly pick up 2,000 image-text pairs as the testing set and the rest as the training set.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
MIRFlickr (Huiskes, Thomee, and Lew 2008) is composed of 25,000 image-text pairs labeled by 24 categories. Every pair is represented by a 224x224 RGB array and a tag vector with the dimension of 500. Then 2,000 image-text pairs are randomly selected as the testing set and the rest are used for training.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Five traditional methods CFA (Li et al. 2003), CCA (Hotelling 1936), PLS-C2A (Tenenhaus 1998), JRL (Zhai, Peng, and Xiao 2014) and ml-CCA (Ranjan, Rasiwasia, and Jawahar 2015), and five deep learning-based methods Multimodal DBN (Srivastava and Salakhutdinov 2012), Corr-AE (Feng, Wang, and Li 2014), DCCA (Andrew et al. 2013), ACMR (Wang et al. 2017) and DSCMR (Zhen et al. 2019) are selected as the baseline methods to compare with DAGNN. For evaluating cross-modal retrieval performance on all approaches, we utilize the mean Average Precision (mAP) score (Wang et al. 2017;Zhen et al. 2019).
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
We utilize mini-batch Adam optimizer (Kingma and Ba 2015) to optimize the whole model. The batch size m is set as 1024 for NUS-WIDE and 100 for MIRFlickr. The initial learning rates of the optimizer are 0.00005 on both datasets.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
The selection of hyper-parameters. First, we choose hyper-parameters τ and p by employing the grid search and finally set τ = 0.3 for the GAT-based Multi-hop GNN and τ = 0.4, p = 0.5 for the GCN-based Multi-hop GNN on both NUS-WIDE and MIRFlickr datasets. Second, we validate the hyper-parameters α and β in Equation 12 and finally set α = 0.2, β = 0.2 for both datasets.  The convolutional layers of ImgGAN have the same configuration with 19-layer VGGNet and the text MLP of TxtGAN is pretrained by performing the general classification task as mentioned in the Methodology Section. Then one fully-connected layer (G v 1 and G t 1 ) activated by Rectified Linear Unit (ReLU) (Nair and Hinton 2010) is followed in both ImgGAN and TxtGAN. The output dimensionality of G v 1 and G t 1 is both 1,024. In ImgGAN, the generator G t 2 consists of two fully-connected layers with the number of hidden units 512 and 300, which are activated by LeakyReLU (Maas, Hannun, and Ng 2013) with the negative slope of 0.2. In TxtGAN, the generator G v 2 consists of two fully-connected layers with the number of hidden units 2,048 and 4,096, which are activated by ReLU. The multi-hop graph neural networks consist of five GAT layers on NUS-WIDE and four GAT layers on MIRFlickr together with one aggregation layer, in which the output dimensionality of each GAT layer and aggregation layer is 1,024. For labeling representation, we adopt 300-dimensional Glove (Pennington, Socher, and Manning 2014) word embeddings as inputs. Table 1 shows the mAP results of DAGNN and baseline methods on NUS-WIDE and MIRFlickr in two cross-modal retrieval tasks. Image2T ext refers the query as the image modality and the database as the text modality, while T ext2Image refers the situation that the query as the text modality and the database as the image modality. From the results, we have following observations: (1) Some traditional methods with deep features can also achieve high mAP scores. Specifically, linear methods CCA, PLS-C2A, and ml-CCA obtained promising results which can compete with some supervised deep learning methods (Corr-AE and DCCA). This demonstrates that our f cnn and f mlp have transformed input instances into approximately linear spaces, greatly reducing the difficulties in cross-modal retrieval.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
(2) Deep learning methods that utilize label information (ACMR and DSCMR) outperform other traditional methods, which shows that nonlinear transformation models have competitive advantages than traditional linear transformation models. (3) The proposed DAGNN model beats all baselines on the two datasets consistently. Compared to DSCMR, the state-of-the-art deep cross-modal retrieval method, our method achieves 4.7% and 2.2% improvements in mAP scores in Image2T ext and T ext2Image tasks on NUS-WIDE respectively and achieves 5.2% and 1.8% improvements on MIRFlickr. It demonstrates that DAGNN can better capture the underlying semantic correlation of labels and bridge the modality gap.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
The objective function of the proposed DAGNN consists of three terms, aiming at optimizing the classification loss, the modality invariance loss, and the adversarial loss. To verify the effect of objective function terms and the Multi-hop GNN on the cross-modal retrieval performance, several variants are designed as following:
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
DAGNN-1 abandons L cla and retains L mdi and L adv . DAGNN-2 abandons L mdi and retains L cla and L adv . DAGNN-3 abandons L adv and retains L cla and L mdi . DAGNN-4 abandons the cross-reconstruction and adopts the discriminator to discriminate between common representations of image and text modalities.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
DAGNN-5 abandons the multi-hop graph neural networks and adopts a linear classifier instead. We conduct the cross-modal retrieval task with these variants on two datasets. The optimization procedure of all variants is similar to the proposed DAGNN. The overall results are shown in Table 2. According to the results, we have following observations: (1) DAGNN outperforms DAGNN-1, DAGNN-2, and DAGNN-3, which indicates that each of the three terms in objective function contributes to the final results.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
(2) DAGNN outperforms DAGNN-1 with a large gap while the margin between DAGNN and DAGNN-2 or DAGNN-3 is small, which proves L cla is the major component of the objective function. (3) DAGNN-2 outperforms DAGNN-3, which indicates that the adversarial loss is more sufficient than the modality invariance loss for bridging the modality gap. (4) DAGNN outperforms DAGNN-4, which demonstrates the effectiveness of the Dual GAN. Compared with ACMR, the adversarial reconstruction technique can sufficiently eliminate modality-specific features and the exploitation of pre-trained true data can enhance the stability of the adversarial learning. (5) DAGNN performs far superior to DAGNN-5, which indicates the effectiveness of the Multi-hop GNN. Compared with utilizing the linear classifier, our GNN-based method can learn inter-dependent classifiers which effectively exploit and preserve the semantic structure of labels.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
Different GNN Models By default, we utilize Multihop GAT to learn inter-dependent classifiers. To verify the effectiveness of Multi-hop GAT, we try different GNN models and the number of GNN layers, which are shown in Figure 4. When using GNN models without the layer aggregation mechanism (GAT, GCN), the cross-modal retrieval mAP achieves the best performances with the layer number of one or two and declines with more layers. Compared with two traditional GNN models, Multi-hop GNN can hierarchically exploit multi-hop propagation information and get better performance as the model becomes deeper.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
To evaluate if the meaningful semantic structure can be preserved by our method, we visualize the learned classifiers on MIRFlickr dataset in different learning manners. Firstly, we adopt the t-SNE (van der Maaten and Hinton 2008) to visualize the classifiers learned by our proposed DAGNN. Secondly, we abandon the Multi-hop GNN and directly parametrize classifiers, the learned classifiers with the same training settings are also visualized by t-SNE.  ure 5 shows the results of the two methods. We can observe that classifiers in Figure 5(a) tend to be close to semantically similar classifiers, which demonstrates our method can maintain more semantic structures than another one. For example, compared with Figure 5(a), the classifier baby is far from its similar classifiers people, male and f emale, and the classifier tree stays away from its relative classifiers plant and f lower in Figure 5(b). This visualization performance moreover demonstrates effectiveness of DAGNN in modeling label dependencies.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval
In this work, we propose a novel Dual Adversarial Graph Neural Networks (DAGNN) to learn common representations for cross-modal retrieval. To better bridge the modality gap and preserve the underlying semantic structure, we introduce an end-to-end framework composed of the dual generative adversarial networks and the multi-hop graph neural networks, which can preserve the semantical relationships among multimodal instances and capture the latent semantic structure of labels. Additionally, Multi-hop GNN is proposed to flexibly leverage different neighbor ranges for nodes, which can learn better structure-aware representation via hierarchically exploiting multi-hop propagation information. Comprehensive experimental results on two public benchmark datasets indicate that DAGNN outperform stateof-the-art methods in cross-modal retrieval.